User:Taniguchi
Cycle Compatibility and New Tool
From Cycle 11 (Pipeline 2024), hsd_imaging calls tsdimaging, not the former sdimaging.
About This Guide
Most recently updated for CASA Version 6.6.1 using Python 3.8
This guide describes some examples for creating and perfecting the Total Power (TP) imaging products using the ALMA Cycle 11 Pipeline, for pipeline & manually calibrated data.
If your data were manually imaged by ALMA, you should instead consult the scriptForImaging.py delivered with your data.
The Section Restore Pipeline Calibration and Prepare for Re-imaging describes the first steps to do. After that, the individual sections are self-contained (and they typically assume the "Restore" has been performed). It illustrates how to completely re-run the pipeline from beginning to end in order to reproduce the pipeline run done at your ARC.
Additional documentation on the Cycle 11 pipeline can be found in the Pipeline User's Guide which can also be found at the ALMA Science Portal. The User's guide describes how to obtain the ALMA Pipeline, how to use it to calibrate and image ALMA interferometric (IF) and single-dish (SD) data, and a description of the Pipeline WebLog.
Note that the scripts described in this guide have only been tested in Linux (RedHat 8) and Python 3.8. Before CASA 5.6.x, the pipeline scripts are not written in python 3 and may not work properly.
Getting and Starting CASA
If you do not already have CASA installed on your machine, you will have to download and install it.
Download and installation instructions are available here:
http://casa.nrao.edu/casa_obtaining.shtml
In this guide, we process ALMA Cycle 8 data with CASA 6.6.1.17 and pipeline-2024.1.0.8.
To use pipeline tasks, you must start CASA with
casa --pipeline
Restore Calibration and Prepare for Re-imaging
STEP 1: Follow instructions in your QA2 report for restoring pipeline calibrated data using the *scriptForPI.py. In general, scriptForPI.py is only compatible with CASA versions similar to the one used for its creation. See the Table at https://almascience.org/processing/science-pipeline for details. For running *scriptForPI.py, you move to the "script" folder containing script files and run the script using the following command (modify <uid_name> accordingly):
execfile('member.<uid_name>.scriptForPI.py')
If you want to obtain the data sets that are the same as the products without any changes, just run scriptForPI.py. scriptForPI.py stops at the calibration stage without atmospheric correction and baseline subtraction, if the script folder contains member.<uid_name>.casa_piperestorescript.py. Follow the procedures in ALMA Pipeline User's Guide to obtain the final datasets with atmospheric correction and baseline subtraction.
If the script folder does not contain member.<uid_name>.casa_piperestorescript.py, but contains member.<uid_name>.casa_pipescript.py, scriptForPI.py uses the latter python script. In that case, all of the stages included in casa_pipescript.py are running including atmospheric model correction and baseline subtraction. Running scriptForPI.py without casa_piperestorescript.py enables us to obtain all of the intermit files (<uid_name>.ms, <uid_name>.ms.atmcor.atmtypeX, and <uid_name>.ms.atmcor.atmtype1_bl, please see ALMA Pipeline User's Guide for suffix).
Once completed, the following files and directories will be present. More information on the structure in the ALMA Archival Data Primer
- calibrated/
- This directory contains a file(s) called <uid_name>.ms.split.cal (one for each execution in the MOUS), products/, working/, and rawdata/ subdirectories. In the working/ subdirectory, <uid_name>.ms are restored. For further data processing, see ALMA Pipeline User's Guide.
- If you run the script without casa_piperestorescript.py, only the working subdirectory is created and all data is stored in this subdirectory.
- calibration/
- This directory contains auxproducts.tgz, auxcaltables.tgz, caltables.tgz, auxproducts.tgz, and flagversions.tgz. The text files (.txt) include information of the applied commands.
- product/
- This directory contains the original image products (fits format).
- qa/
- This directory contains the original weblog and the QA2 and QA0 reports. The QAs reports contain summaries of the scheduling block (SB), and calibration and imaging results.
- If you want to obtain the same results with the QA2 report, you should check the CASA version in the reports and run the scriptForPI.py script with the same version.
- raw/
- This directory contains the raw asdm.sdm(s).
- script/
- This directory contains the file scriptForPI.py (named member.<uid_name>.scriptForPI.py) which internally runs member.<uid_name>.hsd_calimage.casa_piperestorescript.py and other necessary tasks to restore the data.
- The folder also contains member.<uid_name>.hsd_calimage.casa_pipescript.py, a full CASA pipeline script that reproduces all pipeline products.
Here, we run the scriptForPI.py without casa_piperestorescript.py to obtain the final images and evaluate the pipeline procedures.
You can find the results of each pipeline task in WebLog.
STEP 2: Copy member.<uid_name>.casa_pipesescript.py in "calibrated/working/" directory. Change to a directory that contains the calibrated data suitable for running pipeline imaging tasks (i.e. *.ms) called "calibrated/working" after the pipeline restore.
Start CASA 6.6.1 with
casa --pipeline
Change applied atmospheric model (hsd_atmcor stage)
You can find which atmospheric model is applied from the ms name: <uid>.ms.atmcor.atmtypeX, where X is the applied model (X = 1-4).
The relevant tasks for imaging pipeline reprocessing described in this CASA guide are hifa_importdata, hif_mstransform, hifa_flagtargets, hifa_imageprecheck, hif_checkproducts, hif_checkproductsize, hif_uvcontsub, hif_makeimlist, hif_makeimages. Each type means as follows; atmType=1 (tropical), 2 (mid latitude summer), 3 (mid latitude winter), and 4 (subarctic summer). The pipeline applies the most optimal one for each EB. You can check the other models in the ATM Heurestics Plots.
If you want to change the applied atmospheric model, you can apply :
# Be sure to edit mymss!
For further reference, the description of pipeline tasks for interferometric and single dish data reduction can be found in the Pipeline Reference Manual
Change baseline subtraction (hsd_baseline stage)
Next, chose the example below that best fits your use case. Due to the need to preserve the indentation of the python commands, the examples will work best if you copy the entire block of python commands (orange-shaded regions) for a particular example into its own python script, check that the indentation is preserved, edit the USER SET INPUTS section, and then execute the file.
Re-determine and Apply Pipeline Continuum Subtraction using Pipeline Tasks
The following script splits off the calibrated science target data for all spws and fields for each execution, applies any flagging commands found in the <uid_name>_flagtargetstemplate.txt file(s) (one for each execution), uses the existing cont.dat file to fit and subtract the continuum emission, leaving the result in the CORRECTED column. Before running this script, you can manually modify both the <uid_name>_flagtargetstemplate.txt file(s) and cont.dat file to add flag commands or change the cont.dat frequency ranges. Once you're happy with the script, you can run it in a CASA session (that was started with the --pipeline option) using execfile(script_name).
## Edit the USER SET INPUTS section below and then execute
