View source for Single-Dish (Total Power) Data Processing with Pipeline

[[Category:ALMA]]
== Cycle Compatibility and New Tool of Single-Dish Pipeline ==
From Cycle 11 (Pipeline 2024), hsd_imaging calls '''tsdimaging''', not the former sdimaging. The pipeline stage hsd_imaging grids/images total power and spectral data according to a specified gridding kernel.

== About This Guide ==
{{checked_6.6.1}}

'''This guide describes some examples for creating and perfecting the Total Power (TP) imaging products using the ALMA Cycle 11 Pipeline, for pipeline & manually calibrated data.''' 

If your data were manually imaged by ALMA, you should instead consult the scriptForImaging.py delivered with your data. 

The Section [[#Restore Pipeline Calibration and Prepare for Re-imaging (all Options)|Restore Pipeline Calibration and Prepare for Re-imaging]] describes the first steps.

Additional documentation on the Cycle 11 pipeline can be found in the [https://almascience.org/processing/alma_pipeline_user_s_guide_for_release_2024-1.pdf Pipeline User's Guide] which can also be found at [https://almascience.nrao.edu/processing/science-pipeline the ALMA Science Portal].
After that, the individual sections show how you can change the applied atmospheric model and the baseline in the pipeline task. 
The User's guide describes how to obtain the ALMA Pipeline, how to use it to calibrate and image ALMA interferometric (IF) and single-dish (SD) data, and the contents of the Pipeline WebLog.

Note that the scripts described in this guide have only been tested in Linux (RedHat 8) and Python 3.8. 
For pre-Cycle 7 data, for which scripts are written for CASA versions before 5.6.x (Python 2.X), their pipeline scripts may not run properly with CASA versions after 6.6.X (Python 3.X).

== Getting and Starting CASA ==

If you have not installed CASA on your machine, you will have to download and install it.
Download and installation instructions are available here:
http://casa.nrao.edu/casa_obtaining.shtml

In this guide, we process ALMA Cycle 8 data with CASA 6.6.1.17 and pipeline-2024.1.0.8.
To use pipeline tasks, you must start CASA with 

<pre style="background-color: #fffacd;">
casa --pipeline
</pre>

== Restore Calibration and Prepare for Re-imaging ==

'''STEP 1:''' Follow instructions in your QA2 report for restoring pipeline calibrated data using the '''<code>*scriptForPI.py</code>'''. In general, '''<code>scriptForPI.py</code>''' is only compatible with CASA versions similar to the one used for its creation. See the Table at https://almascience.org/processing/science-pipeline for details.
For running <code>*scriptForPI.py</code>, you move to the "script" folder containing script files and run the script using the following command (modify <uid_name> accordingly): 

<source lang="python">
execfile('member.<uid_name>.scriptForPI.py')
</source>

If you want to obtain the data sets that are the same as the products without any changes, just run '''<code>scriptForPI.py</code>'''.

'''<code>scriptForPI.py</code>''' stops at the calibration stage without atmospheric correction and baseline subtraction, if the script folder contains '''<code>member.<uid_name>.casa_piperestorescript.py</code>'''.

Follow the procedures in [https://almascience.nrao.edu/processing/science-pipeline ALMA Pipeline User's Guide] to obtain the final datasets with atmospheric correction and baseline subtraction.

If the script folder '''does not contain <code>member.<uid_name>.casa_piperestorescript.py</code>''', but '''contains <code>member.<uid_name>.casa_pipescript.py</code>''', '''<code>scriptForPI.py</code>''' uses the latter script.

In that case, all of the stages including atmospheric model correction and baseline subtraction are processed.

Running '''<code>scriptForPI.py</code>''' with '''<code>casa_pipescript.py</code>''' enables us to obtain all of the intermediate files (<uid_name>.ms, <uid_name>.ms.atmcor.atmtypeX, and <uid_name>.ms.atmcor.atmtype1_bl, please see [https://almascience.nrao.edu/processing/science-pipeline ALMA Pipeline User's Guide] for suffix).

Once completed, the following files and directories will be present. More information on the structure is in the [https://almascience.nrao.edu/documents-and-tools/cycle11/archive-primer ALMA Archival Data Primer]
* calibrated/
** This directory contains a file(s) called <uid_name>.ms.split.cal (one for each execution in the MOUS), products/, working/, and rawdata/ subdirectories. In the working/ subdirectory, <uid_name>.ms are restored. For further data processing, see [https://almascience.nrao.edu/processing/science-pipeline ALMA Pipeline User's Guide].
** If you run the script without <code>casa_piperestorescript.py</code>, only the working subdirectory is created and all data is stored in this subdirectory.
* calibration/
** This directory contains auxproducts.tgz, auxcaltables.tgz, caltables.tgz, auxproducts.tgz, and flagversions.tgz. The text files (.txt) include information of the applied commands. 
* product/
** This directory contains the original image products (fits format).   
* qa/
** This directory contains the original weblog and the QA2 and QA0 reports. The QAs reports contain summaries of the scheduling block (SB), and calibration and imaging results. 
** If you want to obtain the same results with the QA2 report, you should check the CASA version in the reports and run the '''<code>scriptForPI.py</code>''' script with the same version.
* raw/ 
** This directory contains the raw asdm.sdm(s). 
* script/ 
** This directory contains the file scriptForPI.py (named '''<code>member.<uid_name>.scriptForPI.py</code>''') which internally runs '''<code>member.<uid_name>.hsd_calimage.casa_piperestorescript.py</code>''' and other necessary tasks to restore the data.
** The folder also contains '''<code>member.<uid_name>.hsd_calimage.casa_pipescript.py</code>''', a full CASA pipeline script that reproduces all pipeline products.


Here, we run the '''<code>scriptForPI.py</code>''' without '''<code>casa_piperestorescript.py</code>''' to obtain the final images and evaluate the pipeline procedures.
To do that, run the following command in the script folder before you run '''<code>member.<uid_name>.scriptForPI</code>'''.

<pre style="background-color: #fffacd;">
#In the script folder
mv *casa_piperestorescript.py ../
</pre>

You can find the results of each pipeline task in the WebLog.
The WebLog is created under the subdirectory "calibrated/working/pipeline-*/html/".
The following shows an example of the contents of '''<code>casa_pipescript.py</code>''' after running '''<code>scriptForPI.py</code>'''.

<source lang="python">
context = h_init()
context.set_state('ProjectSummary', 'proposal_code', '2021.1.00172.L')
context.set_state('ProjectSummary', 'proposal_title', 'unknown')
context.set_state('ProjectSummary', 'piname', 'unknown')
context.set_state('ProjectStructure', 'ous_entity_id', 'uid://A001/X1525/X290')
context.set_state('ProjectStructure', 'ous_part_id', 'X476396803')
context.set_state('ProjectStructure', 'ous_title', 'Undefined')
context.set_state('ProjectStructure', 'ps_entity_id', 'uid://A001/X1525/X294')
context.set_state('ProjectStructure', 'ousstatus_entity_id', 'uid://A001/X15a0/X18e')
context.set_state('ProjectStructure', 'ppr_file', '/opt/pipelinedriver/2023JUN/mnt/dataproc/2021.1.00172.L_2023_09_06T09_21_41.745/SOUS_uid___A001_X1590_X30a8/GOUS_uid___A001_X1590_X30a9/MOUS_uid___A001_X15a0_X18e/working/PPR_uid___A001_X15a0_X18f.xml')
context.set_state('ProjectStructure', 'recipe_name', 'hsd_calimage')
try:
    hsd_importdata(vis=['uid___A002_Xfc10af_X33f1', 'uid___A002_Xfc10af_X40b7', 'uid___A002_X10bf6e3_X59f', 'uid___A002_X10bf6e3_Xe1a', 'uid___A002_X10c0a33_X5a9c', 'uid___A002_X10c2033_X5b9', 'uid___A002_X10c2033_X6c55'], session=['session_3', 'session_3', 'session_4', 'session_4', 'session_5', 'session_6', 'session_7'])
    hsd_flagdata(pipelinemode="automatic")
    h_tsyscal(pipelinemode="automatic")
    hsd_tsysflag(pipelinemode="automatic")
    hsd_skycal(pipelinemode="automatic")
    hsd_k2jycal(dbservice=False)
    hsd_applycal(pipelinemode="automatic")
    hsd_atmcor(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_imaging(pipelinemode="automatic")
finally:
    h_save()
</source>


'''STEP 2:''' Copy files in the script folder to the 'calibrated/working' directory. Change to a directory that contains the calibrated data suitable for running pipeline imaging tasks (i.e. *.ms) called "calibrated/working" after the pipeline restore.
<source lang="python">
# In the script directory
cp member.uid* ../calibrated/working/

# Move to "calibrated/working/" subdirectory
cd ../calibrated/working/
</source>

After running '''<code>scriptForPI.py</code>''', information is added in '''<code>casa_pipescript.py</code>''' etc.
These files are necessary for the following tasks.
In the following task, edit '''<code>casa_pipescript.py</code>''' in the "calibrated/working" subdirectory.

== Change applied atmospheric model (hsd_atmcor stage) ==

You can find which atmospheric model is applied from the measurement set name: <uid>.ms.atmcor.atmtypeX, where X (1-4) indicates the applied model; atmType=1 (tropical), 2 (mid-latitude summer), 3 (mid-latitude winter), and 4 (subarctic summer). 
Details about the atmospheric models are summarized in [https://iopscience.iop.org/article/10.1088/1538-3873/abe0ab/pdf Sawada et al. (2021)].
The pipeline applies the most optimal one for each EB. You can check the other models in the ATM Heurestics Plots in the WebLog (see the attached figure). 

[[File:Fig atmcor.png|thumb|<caption>hsd_atmcor stage results in Weblog.]]

If you want to change the applied atmospheric model, you can select them by giving either a single integer (apply to all EBs) or a list of integers (models per EB) to atmtype parameter.
Furthermore, you can constrain parameters such as antenna and spectral window. See more details in the [https://almascience.nao.ac.jp/processing/reference-manual-2024.pdf Pipeline Reference Manual].   

You can edit the pipeline hsd_atmcor command in '''<code>casa_pipescript.py</code>''' as below:
<source lang="python">
hsd_atmcor(spw = '25', atmtype=2)
</source>

In most cases, the automatically selected models work well. The strong ozone lines sometimes affect this stage.

== Change baseline subtraction (hsd_baseline stage)==

At the hsd_baseline stage in the WebLog, you can find the applied baseline as red lines, as well as before and after baseline subtraction spectra (blue lines).
The light-blue ranges indicate the ranges where the pipeline judges that lines are detected.
The pipeline conducts baseline fitting using a cubic spline function if there is no specific description. 
Broad lines are sometimes subtracted as baselines in a cubic spline function.
The attached figure shows an example of this case (spw 21).

[[File:Spectral_plot_before_subtraction_uid_A002_Xfc10af_X33f1.ms.atmcor_Sgr_A_star_ant0_spw21_pol0.png|thumb|<caption> Example of fail in baseline subtraction. The broad line is subtracted.]]

For spw 21, the best baseline is likely a 1st-order polynomial baseline.
If you want to change the baseline in the pipeline task, edit the '''<code>casa_pipescript.py</code>''' as follows:

<source lang="python">
hsd_baseline(linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1)
</source>
The stage hsd_baseline runs twice in casa_pipescript.py and you need to put this command twice. 
The linewindow parameter is set at channel range(s) where the line(s) is detected.
This corresponds to the light-blue range(s). 
The indicated example means that there is a line from 500 to 1900 channels in spw 21.
The applied baseline function is the 1st-order polynomial.
Regarding how to set the parameters, see more details in the [https://almascience.nao.ac.jp/processing/reference-manual-2024.pdf Pipeline Reference Manual]. 

After changing the parameters you want to change in <code>casa_pipescript.py</code>, run '''<code>member.<uid_name>.casa_pipescript.py</code>''' in CASA. 
<source lang="python">
#In the "calibrated/working" subdirectory
execfile('member.<uid_name>.casa_pipescript.py')
</source>

You will obtain the new calibrated measurement sets with baseline subtraction (_bl).
The attached figure shows the result of the manual baseline fitting using hsd_baseline(linewindow={21:[500,1900]}, fitfunc='poly', fitorder = 1).  

[[File:Maual_BL_guide2.png|thumb|<caption> Applied manual baseline fit. The 1st-order polynomial line is applied.]]

'''NOTE: The current single-dish pipeline applies the same fit function to all spectral windows. Thus, if you change fitfunc = 'poly', this is applied to all of the spectral windows even if the cubic spline function works well in the original results. The results are overwritten so please use this method carefully.'''

If you want to restore the original results except for a few spectral windows, please keep the results in another directory, and run '''<code>member.<uid_name>.casa_pipescript.py</code>''' in the "calibrate/working/" directory.

The individual fitting function will be developed in future pipeline versions. 

Currently, CASA task [https://casadocs.readthedocs.io/en/stable/api/tt/casatasks.single.sdbaseline.html sdbaseline] is recommended as mentioned in [https://almascience.nrao.edu/processing/science-pipeline ALMA Pipeline User's Guide].