User:Taniguchi

Cycle Compatibility and New Tool of Single-Dish Pipeline

From Cycle 11 (Pipeline 2024), hsd_imaging calls tsdimaging, not the former sdimaging. The pipeline stage hsd_imaging grids/images total power and spectral data according to a specified gridding kernel.

About This Guide

Most recently updated for CASA Version 6.6.1 using Python 3.8

This guide describes some examples for creating and perfecting the Total Power (TP) imaging products using the ALMA Cycle 11 Pipeline, for pipeline & manually calibrated data.

If your data were manually imaged by ALMA, you should instead consult the scriptForImaging.py delivered with your data.

The Section Restore Pipeline Calibration and Prepare for Re-imaging describes the first steps. After that,

Additional documentation on the Cycle 11 pipeline can be found in the Pipeline User's Guide which can also be found at the ALMA Science Portal. After that, the individual sections show how you can change the applied atmospheric model and the baseline in the pipeline task. The User's guide describes how to obtain the ALMA Pipeline, how to use it to calibrate and image ALMA interferometric (IF) and single-dish (SD) data, and a description of the Pipeline WebLog.

Note that the scripts described in this guide have only been tested in Linux (RedHat 8) and Python 3.8. Before Cycle 7 data whose scripts are written for before CASA 5.6.x versions (Python 2.X), their pipeline scripts may not run properly after with CASA 6.6.X version (Python 3.X).

Getting and Starting CASA

If you have not installed CASA on your machine, you will have to download and install it. Download and installation instructions are available here: http://casa.nrao.edu/casa_obtaining.shtml

In this guide, we process ALMA Cycle 8 data with CASA 6.6.1.17 and pipeline-2024.1.0.8. To use pipeline tasks, you must start CASA with

casa --pipeline

Restore Calibration and Prepare for Re-imaging

STEP 1: Follow instructions in your QA2 report for restoring pipeline calibrated data using the *scriptForPI.py. In general, scriptForPI.py is only compatible with CASA versions similar to the one used for its creation. See the Table at https://almascience.org/processing/science-pipeline for details. For running *scriptForPI.py, you move to the "script" folder containing script files and run the script using the following command (modify <uid_name> accordingly):

execfile('member.<uid_name>.scriptForPI.py')

If you want to obtain the data sets that are the same as the products without any changes, just run scriptForPI.py.

scriptForPI.py stops at the calibration stage without atmospheric correction and baseline subtraction, if the script folder contains member.<uid_name>.casa_piperestorescript.py.

Follow the procedures in ALMA Pipeline User's Guide to obtain the final datasets with atmospheric correction and baseline subtraction.

If the script folder does not contain member.<uid_name>.casa_piperestorescript.py, but contains member.<uid_name>.casa_pipescript.py, scriptForPI.py uses the latter script.

In that case, all of the stages including atmospheric model correction and baseline subtraction are processed.

Running scriptForPI.py with casa_pipescript.py enables us to obtain all of the intermediate files (<uid_name>.ms, <uid_name>.ms.atmcor.atmtypeX, and <uid_name>.ms.atmcor.atmtype1_bl, please see ALMA Pipeline User's Guide for suffix).

Once completed, the following files and directories will be present. More information on the structure in the ALMA Archival Data Primer

• calibrated/
  • This directory contains a file(s) called <uid_name>.ms.split.cal (one for each execution in the MOUS), products/, working/, and rawdata/ subdirectories. In the working/ subdirectory, <uid_name>.ms are restored. For further data processing, see ALMA Pipeline User's Guide.
  • If you run the script without casa_piperestorescript.py, only the working subdirectory is created and all data is stored in this subdirectory.
• calibration/
  • This directory contains auxproducts.tgz, auxcaltables.tgz, caltables.tgz, auxproducts.tgz, and flagversions.tgz. The text files (.txt) include information of the applied commands.
• product/
  • This directory contains the original image products (fits format).
• qa/
  • This directory contains the original weblog and the QA2 and QA0 reports. The QAs reports contain summaries of the scheduling block (SB), and calibration and imaging results.
  • If you want to obtain the same results with the QA2 report, you should check the CASA version in the reports and run the scriptForPI.py script with the same version.
• raw/
  • This directory contains the raw asdm.sdm(s).
• script/
  • This directory contains the file scriptForPI.py (named member.<uid_name>.scriptForPI.py) which internally runs member.<uid_name>.hsd_calimage.casa_piperestorescript.py and other necessary tasks to restore the data.
  • The folder also contains member.<uid_name>.hsd_calimage.casa_pipescript.py, a full CASA pipeline script that reproduces all pipeline products.

Here, we run the scriptForPI.py without casa_piperestorescript.py to obtain the final images and evaluate the pipeline procedures. To do that, run the following command in the script folder before you run member.<uid_name>.scriptForPI.

#In the script folder
mv *casa_piperestorescript.py ../

You can find the results of each pipeline task in the WebLog. The WebLog is created under the subdirectory "calibrated/working/pipeline-*/html/". The following shows an example of the contents of casa_pipescript.py after running scriptForPI.py.

context = h_init()
context.set_state('ProjectSummary', 'proposal_code', '2021.1.00172.L')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectStructure', 'ous_entity_id', 'uid://A001/X1525/X290')
context.set_state('ProjectStructure', 'ous_part_id', 'X476396803')
context.set_state('ProjectStructure', 'ous_title', 'Undefined')
context.set_state('ProjectStructure', 'ps_entity_id', 'uid://A001/X1525/X294')
context.set_state('ProjectStructure', 'ousstatus_entity_id', 'uid://A001/X15a0/X18e')
context.set_state('ProjectStructure', 'ppr_file', '/opt/pipelinedriver/2023JUN/mnt/dataproc/2021.1.00172.L_2023_09_06T09_21_41.745/SOUS_uid___A001_X1590_X30a8/GOUS_uid___A001_X1590_X30a9/MOUS_uid___A001_X15a0_X18e/working/PPR_uid___A001_X15a0_X18f.xml')
context.set_state('ProjectStructure', 'recipe_name', 'hsd_calimage')
try:
    hsd_importdata(vis=['uid___A002_Xfc10af_X33f1', 'uid___A002_Xfc10af_X40b7', 'uid___A002_X10bf6e3_X59f', 'uid___A002_X10bf6e3_Xe1a', 'uid___A002_X10c0a33_X5a9c', 'uid___A002_X10c2033_X5b9', 'uid___A002_X10c2033_X6c55'], session=['session_3', 'session_3', 'session_4', 'session_4', 'session_5', 'session_6', 'session_7'])
    hsd_flagdata(pipelinemode="automatic")
    h_tsyscal(pipelinemode="automatic")
    hsd_tsysflag(pipelinemode="automatic")
    hsd_skycal(pipelinemode="automatic")
    hsd_k2jycal(dbservice=False)
    hsd_applycal(pipelinemode="automatic")
    hsd_atmcor(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_imaging(pipelinemode="automatic")
finally:
    h_save()

STEP 2: Copy files in the script folder to the 'calibrated/working' directory. Change to a directory that contains the calibrated data suitable for running pipeline imaging tasks (i.e. *.ms) called "calibrated/working" after the pipeline restore.

# In the script directory
cp member.uid* ../calibrated/working/

# Move to "calibrated/working/" subdirectory
cd ../calibrated/working/

After running scriptForPI.py, information is added in casa_pipescript.py etc. These files are necessary for the following tasks. In the following task, edit casa_pipescript.py in the "calibrated/working" subdirectory.

Change applied atmospheric model (hsd_atmcor stage)

You can find which atmospheric model is applied from the measurement set name: <uid>.ms.atmcor.atmtypeX, where X (1-4) indicates the applied model; atmType=1 (tropical), 2 (mid-latitude summer), 3 (mid-latitude winter), and 4 (subarctic summer). Details about the atmospheric models are summarized in Sawada et al. (2021). The pipeline applies the most optimal one for each EB. You can check the other models in the ATM Heurestics Plots in the WebLog (see the attached figure).

hsd_atmcor stage results in Weblog.

If you want to change the applied atmospheric model, you can select them by giving either a single integer (apply to all EBs) or a list of integers (models per EB) to atmtype parameter. Furthermore, you can constrain parameters such as antenna and spectral window. See more details in the Pipeline Reference Manual.

You can edit the pipeline hsd_atmcor command in casa_pipescript.py as below:

hsd_atmcor(spw = '25', atmtype=2)

In most cases, the automatically selected models work well. The strong ozone lines sometimes affect this stage.

Change baseline subtraction (hsd_baseline stage)

At the hsd_baseline stage in the WebLog, you can find the applied baseline as red lines as well as before and after baseline subtraction spectra (blue lines). The light-blue ranges indicate the ranges where the pipeline judges that lines are detected. The pipeline conducts baseline fitting using a cubic spline function if there is no specific description. Broad lines are sometimes subtracted as baselines in a cubic spline function. The attached figure shows an example of this case (spw 21).

Example of fail in baseline subtraction. The broad line is subtracted.

For spw 21, the best baseline is likely the 1st-order polynomial baseline. If you want to change the baseline in the pipeline task, edit the casa_pipescript.py as follows:

hsd_baseline(linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1)

The stage hsd_baseline runs twice in casa_pipescript.py and you need to put this command twice. The linewindow parameter is set at channel range(s) where the line(s) is detected. This corresponds to the light-blue range(s). The indicated example means that there is a line from 500 to 1900 channels in spw 21. The applied baseline function is the 1st-order polynomial. Regarding how to set the parameters, see more details in the Pipeline Reference Manual.

After changing the parameters you want to change in casa_pipescript.py, run member.<uid_name>.casa_pipescript.py in CASA.

#In the "calibrated/working" subdirectory
execfile('member.<uid_name>.casa_pipescript.py')

You will obtain the new calibrated measurement sets with baseline subtraction (_bl). The attached figure shows the result of the manual baseline fitting using hsd_baseline(linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1).

Applied manual baseline fit. The 1st-order polynomial line is applied.

NOTE: The current single-dish pipeline applies the same fit function to all spectral windows. Thus, if you change fitfunc = 'poly', this is applied to all of the spectral windows even if the cubic spline function works well in the original results. The results are overwritten and please carefully use this method.

If you want to restore the original results except for a few spectral windows, please keep the results in another directory, and run member.<uid_name>.casa_pipescript.py in the "calibrate/working/" directory.

The individual fitting function will be developed in future pipeline versions.

Currently, CASA task sdbaseline is recommended as mentioned in ALMA Pipeline User's Guide.