ALMA Cycle 4 Imaging Pipeline Reprocessing

About This Guide

This guide describes some examples for perfecting the interferometric imaging products from the ALMA Cycle 4 Pipeline. If your data were manually imaged by ALMA, you should instead consult the scriptForImaging.py delivered with your data.

Additional documentation on the Cycle 4 pipeline can be found at the ALMA Science Portal

After Section #Restore Pipeline Calibration and Prepare for Re-imaging, each reprocessing example is self-contained for ease of use.

Note that the scripts described in this guide have only been tested in Linux.

How to Decide Whether to Reprocess Pipeline Images

In order to decide whether reprocessing will be beneficial for your project you should examine both the results of the pipeline imaging via the delivered weblog and note any imaging specific comments in your README file. The details of the various imaging pipeline stages, as well as examples of weblog output can be found in the Users Guide, especially useful in this regard are Sections 7 and 8.

For Cycle 4, the primary goal of the imaging pipeline is to produce images that are of sufficient quality that QA2 can be successfully carried out, and give users a good idea of what the data may contain. In some cases these images are fine for doing science while in others, significant benefits may be obtained by re-imaging with particular science goals in mind.

Typical reasons for re-imaging include:

Imaging improvements to be gained from interactively generating emission specific clean mask and cleaning more deeply. The Cycle 4 pipeline currently uses a generic clean mask corresponding to everything within the 0.3 PB response level, and as a result only cleans down to a conservative $4\sgima$ x (dynamic range modifier) clean level. Cases with moderate to strong emission (or absorption) can significantly benefit from deeper clean with interactive clean masking. For S/N > about 100 the images can also often be improved by self-calibration coupled with deeper clean with manual clean masks.
Non-optimal continuum ranges. The pipeline uses heuristics that attempt to correctly identify continuum channels over a very broad range of science target line properties. Particularly for strong line forests (hot-cores) and occasionally for TDM continuum projects the pipeline ranges can be non-optimal -- too much in the first case and too little in the second.
Other science goal driven reprocessing needs may include
- Desire to bin channels in the imaging stage to increase the S/N of cubes
- Desire to use a different Briggs Robust image weighting than the default of robust=0.5 (smaller robust = more uniform weighting, smaller beam, poorer S/N; larger robust = more natural weighting, larger beam, better S/N)
- Desire to uv-taper images to focus on extended emission (only available manually presently)

The examples below demonstrate some of the more common ways that users may wish to perfect their imaging products to meet their science goals.

Getting and Starting CASA

If you do not already have CASA installed on your machine, you will have to download and install it.

Download and installation instructions are available here:

http://casa.nrao.edu/casa_obtaining.shtml

CASA 4.7.0 or later is required to reprocess ALMA Cycle 4 data using the scripts in this guide.

NOTE: To use pipeline tasks, you must start CASA with

casa --pipeline

Restore Pipeline Calibration and Prepare for Re-imaging (all Options)

STEP 1: Follow instructions in your README for restoring pipeline calibrated data using the scriptForPI.py. NOTE: the SPACESAVING parameter cannot be larger than 1

Once completed, the following directories will be present: calibrated/ calibration/ log/ product/ qa/ raw/ README script/

STEP 2: Change to directory that contains the calibrated data (i.e. *.ms) called "calibrated/working" after the pipeline restore and start CASA 4.7.0 or later.

casa --pipeline

STEP 3: Run the following command in CASA to copy the pipeline file that contains the frequency ranges used to create the continuum images and the continuum subtraction to the directory you will be working in.

os.system('cp ../../calibration/uid*cont.dat ./cont.dat')

Restore Pipeline Continuum Subtraction and Manually Make Image Products

The following script splits off the calibrated science target data for all spws and fields for each execution, applies any flagging commands found in the <uid_name>_flagtargetstemplate.txt file(s) (one for each execution), uses the existing cont.dat file to fit and subtract the continuum emission, leaving the result in the CORRECTED column. Before running this script, you can manually modify both the <uid_name>_flagtargetstemplate.txt file(s) and cont.dat files to add flag commands or change the cont.dat frequency ranges.

## Edit the USER SET INPUTS section below and then execute
## this script (note it must be in the 'calibrated/working' directory.

import glob as glob
__rethrow_casa_exceptions = True
pipelinemode='automatic'
context = h_init()

###########################################################
## USER SET INPUTS

## Select a title for the weblog
context.project_summary.proposal_code='Restore Continuum Subtraction'

############################################################

## Make a list of all uv-datasets appended with *.split.cal
MyVis=glob.glob('*.ms')

try:
    ## Load the *.ms files into the pipeline
    hifa_importdata(vis=MyVis, pipelinemode=pipelinemode)

    ## Split off the science target data into its own ms (called
    ## *target.ms) and apply science target specific flags
    hif_mstransform(pipelinemode=pipelinemode)
    hifa_flagtargets(pipelinemode=pipelinemode)

    ## Fit and subtract the continuum using the cont.dat for all spws all fields
    hif_uvcontfit(pipelinemode=pipelinemode)
    hif_uvcontsub(pipelinemode=pipelinemode)

finally:
    h_save()

Make Images Manually

At this point you will have created a *target.ms for each execution of your SB. Each of these measurement sets contains the original calibrated continuum + line data in the DATA column and the calibrated continuum subtracted data in the the CORRECTED column. The new CASA task for imaging tclean (which is used by the ALMA Pipeline) allows the user to select which column to use for imaging. tclean also allows a list for the vis parameter so that it is not necessary to concat the data before imaging.

**NOTE:** If you think you might want to self-calibrate your data using either the continuum or line emission it is ESSENTIAL that you first split off the column that you want to operate on before imaging. Otherwise, the CORRECTED column containing the continuum subtracted data will be overwritten when applycal is run during the self-calibration process.

To manually clean your data at this stage, there are two options:

Use modified versions of the relevant tclean commands from the "logs/<MOUS_name>.casa_commands.log". These are the exact commands originally run by the imaging pipeline to produce your imaging products.
- They will contain within them the frequency ranges (from the cont.dat) used for making the various images.
- There will be two tclean commands per image product, the first with an image name containing iter0 only makes a dirty image, while the second with iter1 makes a cleaned image.
- For example to make the aggregate continuum image but with interactive clean masking, simply copy the corresponding iter1 command (it will contain all of the spw numbers in its name), but set interactive=True, calcpsf=True, and calcres=True. If you are using the *.target.ms file(s) you can keep datacolumn='DATA'. If you have split off the DATA column and have applied self-calibration, you will want to image the datacolumn='CORRECTED'.
Use examples on the casaguide page TCLEAN_and_ALMA to formulate your own special purpose commands.

Restore Pipeline Continuum Subtraction and Make Pipeline Aggregate Continuum Image With All Channels

## Edit the USER SET INPUTS section below and then execute
## this script (note it must be in the 'calibrated/working' directory.

import glob as glob
__rethrow_casa_exceptions = True
pipelinemode='automatic'
context = h_init()

###########################################################
## USER SET INPUTS

## Select a title for the weblog
context.project_summary.proposal_code='NEW AGGREGATE CONT'

############################################################

## Move cont.dat to another name if it exists
os.system('mv cont.dat original.cont.dat')

## Make a list of all uv-datasets appended with *.split.cal
MyVis=glob.glob('*.ms')

try:
    ## Load the *.ms files into the pipeline
    hifa_importdata(vis=MyVis, pipelinemode=pipelinemode)

    ## Split off the science target data into its own ms (called
    ## *target.ms) and apply science target specific flags
    hif_mstransform(pipelinemode=pipelinemode)
    hifa_flagtargets(pipelinemode=pipelinemode)

    ## Skip the continuum subtraction steps and make an aggregate 
    ## continuum image with all unflagged channels (file named 
    ## cont.dat should NOT be present in directory).
    hif_makeimlist(specmode='cont',pipelinemode=pipelinemode)
    hif_makeimages(pipelinemode=pipelinemode)

    ## Export new images to fits format if desired.
    hif_exportdata(pipelinemode=pipelinemode)

finally:
    h_save()

Revise the cont.dat Before Pipeline Continuum Subtraction and Remake Pipeline Images

## Edit the cont.dat file(s) for the spw(s) you want 
## to change the continuum subtraction for. In this example 
## spw 17 was changed.

## Edit the USER SET INPUTS section below and then execute
## this script (note it must be in the 'calibrated/working' directory.

import glob as glob
__rethrow_casa_exceptions = True
pipelinemode='automatic'
context = h_init()

###########################################################
## USER SET INPUTS

## Select a title for the weblog
context.project_summary.proposal_code = 'NEW CONTSUB' 

## Select spw(s) that have new cont.dat parameters
## If all spws have changed use MySpw=''
MySpw='17'

############################################################

## Make a list of all uv-datasets appended with *.split.cal
MyVis=glob.glob('*.ms')

try:
    ## Load the *.ms files into the pipeline
    hifa_importdata(vis=MyVis, pipelinemode=pipelinemode)

    ## Split off the science target data into its own ms (called
    ## *target.ms) and apply science target specific flags
    hif_mstransform(pipelinemode=pipelinemode)
    hifa_flagtargets(pipelinemode=pipelinemode)

    ## Fit and subtract the continuum using revised cont.dat for all spws
    hif_uvcontfit(pipelinemode=pipelinemode)
    hif_uvcontsub(pipelinemode=pipelinemode)

    ## Make new per spw continuum for revised spw(s) and new aggregate cont
    hif_makeimlist(specmode='mfs',spw=MySpw)
    hif_makeimages(pipelinemode=pipelinemode)
    hif_makeimlist(specmode='cont',pipelinemode=pipelinemode) 
    hif_makeimages(pipelinemode=pipelinemode)    

    ## Make new continuum subtracted cube for revised spw(s)
    hif_makeimlist(specmode='cube',spw=MySpw,pipelinemode=pipelinemode) 
    hif_makeimages(pipelinemode=pipelinemode)

    ## Export new images to fits format if desired.
    hif_exportdata(pipelinemode=pipelinemode)

finally:
    h_save()

Restore Pipeline Continuum Subtraction for Subset of SPWs and Fields and Use Channel Binning for Pipeline Imaging of Cubes

## Edit the USER SET INPUTS section below and then execute
## this script (note it must be in the 'calibrated/working' directory.

import glob as glob
__rethrow_casa_exceptions = True
pipelinemode='automatic'
context = h_init()

###########################################################
## USER SET INPUTS

## Select a title for the weblog
context.project_summary.proposal_code = 'SUBSET CUBE IMAGING' 

## Select spw(s) to image and channel binning for each spcified
## MySpw. All spws listed in MySpw must have a corresponding MyNbins
## entry, even if it is 1 for no binning.
MySpw='17,23'
MyNbins='17:8,23:2'

## Select subset of sources to image by field name.
## To select all fields, set MyFields=''
MyFields='CoolSource1,CoolSource2'

## Select Briggs Robust factor for data weighting (affects angular 
## resolution of images)
MyRobust=1.5

############################################################

## Make a list of all uv-datasets appended with *.split.cal
MyVis=glob.glob('*.ms')

try:
    ## Load the *.ms files into the pipeline
    hifa_importdata(vis=MyVis, pipelinemode=pipelinemode)

    ## Split off the science target data into its own ms (called
    ## *target.ms) and apply science target specific flags
    ## In this example we split off all science targets and science 
    ## spws, however these steps could also contain the spw and field
    ## selections
    hif_mstransform(pipelinemode=pipelinemode)
    hifa_flagtargets(pipelinemode=pipelinemode)

    ## Fit and subtract the continuum using existing cont.dat
    ## for selected spws and fields only.
    hif_uvcontfit(spw=MySpw,field=MyFields,pipelinemode=pipelinemode)
    hif_uvcontsub(spw=MySpw,field=MyFields,pipelinemode=pipelinemode)   

    ## Make new continuum subtracted cube for selected spw(s) and fields
    hif_makeimlist(specmode='cube',spw=MySpw,nbins=MyNbins,field=MyFields,
                   pipelinemode=pipelinemode) 
    hif_makeimages(robust=MyRobust,pipelinemode=pipelinemode)

    ## Export new images to fits format if desired.
    hif_exportdata(pipelinemode=pipelinemode)

finally:
    h_save()

ALMA Cycle 4 Imaging Pipeline Reprocessing

Contents

About This Guide

How to Decide Whether to Reprocess Pipeline Images

Getting and Starting CASA

Restore Pipeline Calibration and Prepare for Re-imaging (all Options)

Restore Pipeline Continuum Subtraction and Manually Make Image Products

Make Images Manually

Restore Pipeline Continuum Subtraction and Make Pipeline Aggregate Continuum Image With All Channels

Revise the cont.dat Before Pipeline Continuum Subtraction and Remake Pipeline Images

Restore Pipeline Continuum Subtraction for Subset of SPWs and Fields and Use Channel Binning for Pipeline Imaging of Cubes

Navigation menu

Page actions

Page actions

Personal tools

Search

Tools