View source for Guide To Redo Calibration For ALMA Cycle 0

== How To Re-do ALMA Calibration for Cycle 0 Data Packages ==

This page describes how to redo ALMA calibration for Cycle 0 data.   If you do not need to redo calibration, proceed to the section on [[Guide_To_Processing_ALMA_Data_for_Cycle_0#DataConcatenation|Data Concatenation]] in the [[Guide_To_Processing_ALMA_Data_for_Cycle_0|Cycle 0 Data Processing Guide]].

The [[Guide_To_Processing_ALMA_DataGlossary#DataPackage|data package]] includes a calibration script in the '''scripts''' directory for each [[Guide_To_Processing_ALMA_DataGlossary#MeasurementSet|measurement set]].  This script performs all the initial calibrations required prior to combination and [[Guide_To_Processing_ALMA_DataGlossary#Imaging|imaging]] of all measurement sets corresponding to a [[Guide_To_Processing_ALMA_DataGlossary#SchedulingBlock|scheduling block]]. This script is the exact version used to generate the corresponding [[Guide_To_Processing_ALMA_DataGlossary#CalibratedData|calibrated measurement set]] found in the '''calibrated''' directory. You can use the packaged calibration script to re-calibrate step-by-step the corresponding raw measurement set (found in the '''raw''' directory) and tweak the script as desired for your own purposes. However, for Cycle 0 measurement sets, there are a couple of considerations which need to be taken into account before using the packaged calibration script.

<!-- # The directory structure provided in the Cycle 0 data package [[http://casaguides.nrao.edu/index.php?title=Guide_To_Processing_ALMA_DataGlossary#DataPackage]] does not quite match the structure expected by the calibration scripts.-->
# While the scripts and the raw data are found in different directories, the calibration script must be run in the same directory as the [[Guide_To_Processing_ALMA_DataGlossary#MeasurementSet|measurement set]] on which it is applied. We recommend copying both the measurement set and the calibration script in a new directory where the [[Guide_To_Processing_ALMA_DataGlossary#Calibration|calibration]] will be performed, so as to leave original versions intact.
# The packaged calibration script was generated so as to be used in CASA 3.4, but users are now recommended to use CASA 4.2 for all data processing.  There are several differences between CASA 3.4 scripts and CASA 4.2 scripts. To change the packaged calibration script into a CASA 4.2 script, it must be manually modified using a text editor (see the detailed instructions [[Updating_a_script_to_work_with_CASA_4.2|here]]). 
<!-- # The CASA 3.4 script should be used, therefore, only as a guide to what needs to be done, and in what order. -->
# The "raw" data supplied with the data package in the '''raw''' directory have had several a priori calibrations applied.  These include the calibration steps labeled 0-6 in the packaged calibration scripts.  In CASA 3.4, the user should hence begin re-calibration of the "raw" data with the step labeled "7" of the calibration script (or "9", if using a CASA 4.2 script). To run one (or several) specific steps in a script, the user needs to define the variable '''mysteps''' with a numerical list of desired steps to execute. Below, we show examples corresponding to running the script from the first applicable step all the way through the end of the script:
<source lang="python">
# In CASA 3.4
 mysteps=[7,8,9,10,11,12,13,14,15] 
 execfile('scriptForCalibration_3.4.py')
</source>
<source lang="python">
# In CASA  4.2
 mysteps=[9,10,11,12,13,14,15,16,17,18]
 execfile('scriptForCalibration_4.2.py')
</source>


=== General considerations on running calibration scripts===

# In addition to CASA tasks, calibration scripts call specific plotting and analysis utilities which are gathered in the analysisUtils package. This package is very easy to download and install. See [[Analysis_Utilities|this page]] for details. 
# Within the calibration script, you can identify the different steps by the identifier '''mystep = N'''. It is possible to run a single step or a sequence of subsequent steps, by defining the mysteps list accordingly.  For example:
<source lang="python">
 mysteps = [9]
 execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py') # will run only step 9
 mysteps = [10,11,12]
 execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py') # will run step 10,11,12
</source>
# Steps need to be executed in the order given by the script. For example, step 16 can be executed only after steps 9,10,11,12,13,14 and 15 have been executed.
# It is also possible to go back and repeat a step that has already been performed, but one must be careful about flags. Flags are applied to the data when the command {{flagdata}} is called, but also when calibration tables are applied. This means that if you are going back to perform step 9 after having run steps 9, 10, and 11, the data on which you are applying step 9 will have applied flags that were created in steps 9, 10 and 11. To avoid this, you need to make sure that if you are running step N, you reload the flags as they were at the end of step N-1. This is possible by using the {{flagmanager}} task, which allows one to save the state of flags at a given point in the script, and to reload that state of flags later. Several calls to {{flagmanager}} are placed throughout the calibration script, and you can add your own. For example, in step 14 of the reduction script (CASA 4.2), you will find a call to {{flagmanager}} with '''mode=save''':

  flagmanager(vis = 'uid___A002_X554543_X207.ms.split',
    mode = 'save',
    versionname = 'BeforeGainCalibration')

This command takes a snapshot of the flags just before [[Guide_To_Processing_ALMA_DataGlossary#GainCalibration|gain calibration]] . After running steps 15 and 16 (gain calibration), some additional flags have probably been applied to the data. If you want to re-run step 15 and 16, you will need to reload the state of flags at the end of step 14 using {{flagmanager}} with '''mode=restore''':

<source lang="python">
mysteps=[15,16]  
execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py')  # first executions of steps 15 and 16
# You decide to modify the script in steps 15 and 16 and run these steps again
flagmanager(vis = 'uid___A002_X554543_X207.ms.split',  # reload flags as recorded at the end of step 14
    mode = 'restore',
    versionname = 'BeforeGainCalibration')
mysteps=[15,16]  
execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py')  # execute (modified) steps 15 and 16 
</source>

=== Example: re-calibration of the TW Hydra data===

In our TW HYdra example, there are 3 [[Guide_To_Processing_ALMA_DataGlossary#MeasurementSet|measurement sets]] in the '''raw''' directory that are ready for [[Guide_To_Processing_ALMA_DataGlossary#Calibration|calibration]], and 3 corresponding calibration scripts in the scripts directory, all written for CASA 3.4. You can find those scripts already updated to CASA 4.2 here:

* [[File:Uid_A002_X554543_X207.ms.scriptForCalibration_4.2.py]] 
* [[File:Uid_A002_X554543_X3d0.ms.scriptForCalibration_4.2.py]] 
* [[File:Uid_A002_X554543_X667.ms.scriptForCalibration_4.2.py]] 

These scripts follow the steps described [[Updating_a_script_to_work_with_CASA_4.2|here]].  These can be used to calibrate and combine these data sets under CASA 4.2. First you will need to place the datasets and the scripts in the same directory (either the raw directory or another created directory if you don't want to overwrite the original raw datasets). The scripts be run in CASA 4.2 as follows. 

<source lang="python">
# In CASA 4.2
 thesteps = [9,10,11,12,13,14,15,16,17,18]
 execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py')
 execfile('uid___A002_X554543_X3d0.ms.scriptForCalibration_4.2.py')
 execfile('uid___A002_X554543_X667.ms.scriptForCalibration_4.2.py')
</source>