Updating a script to work with CASA 4.2: Difference between revisions

From CASA Guides
Jump to navigationJump to search
Amoullet (talk | contribs)
No edit summary
Ekeller (talk | contribs)
No edit summary
 
(21 intermediate revisions by 2 users not shown)
Line 1: Line 1:


In the ALMA archive for Cycle 0 data, the packaged calibration script found in the scripts directory is written for execution in CASA 3.4. While it possible to use the packaged calibration script as it is in CASA 3.4, it is now recommended to use CASA 4.2 for data processing. To change the packaged calibration script into a CASA 4.2 script, it must be manually modified using a text editor, following the instructions below:
In the ALMA archive for Cycle 0 data, the packaged calibration script found in the ''scripts'' directory is written for execution in CASA 3.4. While it possible to use the packaged calibration script as it is in CASA 3.4, it is now recommended to use CASA 4.2 for [[Guide_To_Processing_ALMA_DataGlossary#Calibration|calibration]] and [[Guide_To_Processing_ALMA_DataGlossary#Imaging|imaging]]. To change the packaged calibration script into a CASA 4.2 script, it must be manually modified using a text editor, following the instructions below. In the script examples below 'measurementset' stands for your uid name (for example, uid___A002_X554543_X207). If, after applying those edits and launching the execution, the script either crashes or does not calibrate the data as expected, please contact the CASA HelpDesk.


* Change the list of steps at the beginning of the script to
* Change the list of steps at the beginning of the script to
<source lang="python">
<source lang="python">
step_title = {0: 'listobs',
step_title = {0: 'Import of the ASDM',
               1: 'A priori flagging',
              1: 'Fix of SYSCAL table times',
               2: 'Generation and time averaging of the WVR cal table',
              2: 'listobs',
               3: 'Generation of the Tsys cal table',
               3: 'A priori flagging',
               4: 'Generation of the antenna position cal table',
               4: 'Generation and time averaging of the WVR cal table',
               5: 'Application of the WVR, Tsys and antpos cal tables',
               5: 'Generation of the Tsys cal table',
               6: 'Split out science SPWs and time average',
               6: 'Generation of the antenna position cal table',
               7: 'Listobs, clear pointing table, and save original flags',
               7: 'Application of the WVR, Tsys and antpos cal tables',
               8: 'Initial flagging',
               8: 'Split out science SPWs and time average',
               9: 'Putting a model for the flux calibrator(s)',
               9: 'Listobs, clear pointing table, and save original flags',
               10: 'Save flags before bandpass cal',
               10: 'Initial flagging',
               11: 'Bandpass calibration',
               11: 'Putting a model for the flux calibrator(s)',
               12: 'Save flags before gain cal',
               12: 'Save flags before bandpass cal',
               13: 'Gain calibration',
               13: 'Bandpass calibration',
               14: 'Save flags before applycal cal',
               14: 'Save flags before gain cal',
               15: 'Application of the bandpass and gain cal tables'}
               15: 'Gain calibration',
               16: 'Save flags before applycal',
               17: 'Application of the bandpass and gain cal tables',
              18: 'Split out corrected column'}
</source>
</source>


* add "import os" and "if 'applyonly' not in globals(): applyonly = False" on line 27. The code under the steps list should now read:
 
* add "import analysisUtils as au", "import analysisUtils as aU", "import os" and "if 'applyonly' not in globals(): applyonly = False" on line 27. The code under the steps list should now read:


<source lang="python">
<source lang="python">
import analysisUtils as au
import analysisUtils as aU
import os
import os
if 'applyonly' not in globals(): applyonly = False  #(AM) added
if 'applyonly' not in globals(): applyonly = False  #(AM) added
Line 34: Line 40:
</source>
</source>


* comment out any call to aU and es tasks (aU.plotWVRSolutions, aU.plotbandpass, es.checkCalTable,...)
 
* Add "if applyonly != True:" in front of any aU and es routines calls. For example:
 
<source lang="python">
if applyonly != True: es = aU.stuffForScienceDataReduction()
if applyonly != True:  es.checkCalTable('measurementset.ms.split.ap_pre_bandpass', msName='measurementse.ms.split', interactive=False)
</source>
 
 
* Replace the CASA 3.4 version search to CASA 4.2
<source lang="python">
if re.search('^4.2', casadef.casa_version) == None: #(AM)
sys.exit('ERROR: PLEASE USE THE SAME VERSION OF CASA THAT YOU USED FOR GENERATING THE SCRIPT: 4.2')
</source>
 


* In step 7:  
* In step 7:  
** the label of the step must be changed to 9 ('mystep = 9')
** the label of the step must be changed to 9 ('mystep = 9') (the steps 0-8 will be skipped since they do not need to be applied on the "raw" data supplied with the data package. Hence one does not need to worry about introducing 'missing' steps)
** add 'if not os.path.exists('measurementset.ms.split.flagversions/Original.flags'):' above the flagmanager call, and ident the flagmanager call. It should now read :
** add 'if not os.path.exists('replaceuidlabelhere.ms.split.flagversions/Original.flags'):' above the flagmanager call, and ident the flagmanager call. It should now read (for the example of uid uid___A002_X554543_X207):
<source lang="python">
<source lang="python">
   if not os.path.exists('measurementset.ms.split.flagversions/Original.flags'):
   if not os.path.exists('uid___A002_X554543_X207.ms.split.flagversions/Original.flags'):
     flagmanager(vis = 'measurementset.ms.split',
     flagmanager(vis = 'uid___A002_X554543_X207.ms.split',
       mode = 'save',
       mode = 'save',
       versionname = 'Original')
       versionname = 'Original')
</source>
</source>


* In step 8:  
* In step 8:  
** the label of the step must be changed to 10 ('mystep = 10')
** the label of the step must be changed to 10 ('mystep = 10')
** any call to tflagdata must be changed to flagdata
** any call to tflagdata must be changed to flagdata


* In step 9:
* In step 9:
** the label of the step must be changed to 11 ('mystep = 11')
** the label of the step must be changed to 11 ('mystep = 11')
** if applicable (when a Solar System Object is used as a flux calibrator), in setjy change the parameter standard = 'Butler-JPL-Horizons 2010' to standard = 'Butler-JPL-Horizons 2012'
** if applicable (when a Solar System Object is used as a flux calibrator), in setjy change the parameter standard = 'Butler-JPL-Horizons 2010' to standard = 'Butler-JPL-Horizons 2012'
** if applicable (when a non solar system object is used as flux calibrator), change the clearcal command to delmod (inputs are unchanged)


* In step 10:
* In step 10:
** the label of the step must be changed to 12 ('mystep = 12')
** the label of the step must be changed to 12 ('mystep = 12')


* In step 11:
* In step 11:
** the label of the step must be changed to 13 ('mystep = 13')
** the label of the step must be changed to 13 ('mystep = 13')
** in the gaincal calls, change any calmode='ap' parameter to calmode='a'
** in the gaincal calls, change any calmode='ap' parameter to calmode='p'
 


* In step 12:
* In step 12:
** the label of the step must be changed to 14 ('mystep = 14')
** the label of the step must be changed to 14 ('mystep = 14')


* In step 13:
* In step 13:
** the label of the step must be changed to 15 ('mystep = 15')
** the label of the step must be changed to 15 ('mystep = 15')
** change the clearcal command to delmod (inputs are unchanged)
** if present, change clearcal command to delmod (inputs are unchanged)
** in the gaincal calls, change any calmode='ap' parameter to calmode='a'
** in the gaincal calls, change any calmode='ap' parameter to calmode='a'
** if a  phaseCalName loop is present, change:
<source lang="python">
    if fc[i].find('Flux density for '+phaseCalName) != -1 and re.search('in SpW=[0-9]+(?: \(ref SpW=[0-9]+\))? is: [0-9]+\.[0-9]+', fc[i]) != None:
        line = (re.search('in SpW=[0-9]+(?: \(ref SpW=[0-9]+\))? is: [0-9]+\.[0-9]+', fc[i])).group(0)
</source>
to
<source lang="python">
      if fc[i].find('Flux density for '+phaseCalName) != -1 and re.search('in SpW=[0-9]+(?: \(.*?\))? is: [0-9]+\.[0-9]+', fc[i], re.DOTALL|re.IGNORECASE) != None:
        line = (re.search('in SpW=[0-9]+(?: \(.*?\))? is: [0-9]+\.[0-9]+', fc[i], re.DOTALL|re.IGNORECASE)).group(0)
</source>


* In step 14:
* In step 14:
** the label of the step must be changed to 16 ('mystep = 16')
** the label of the step must be changed to 16 ('mystep = 16')


* In step 15:
* In step 15:
** the label of the step must be changed to 16 ('mystep = 16')
** the label of the step must be changed to 16 ('mystep = 17')


* In step 16:
** the label of the step must be changed to 16 ('mystep = 16')


* Add a step 18 (optional):
Add this piece of code, where measurementset stands for the uid label (as in, 'uid___A002_X554543_X207'):
<source lang="python">
mystep = 18
if(mystep in thesteps):
  casalog.post('Step '+str(mystep)+' '+step_title[mystep],'INFO')
  print 'Step ', mystep, step_title[mystep]


 
  os.system('rm -rf measurementset.ms.split.cal')  
In our TW HYdra example, there are 3 measurement sets in the '''raw''' data directory that are ready for calibration, and 3 corresponding calibration scripts in the '''scripts''' directory, all written for CASA 3.4. You can find [here] those datasets updated to CASA 4.2, which can be used to calibrate and combine these data sets in CASA 4.2.  They can be run in CASA 4.2 as follows.
  split(vis = 'measurementset.ms.split',
<!--  (Start from the '''raw''' directory.)-->
    outputvis = 'measurementset.ms.split.cal',
 
    datacolumn = 'corrected',
<source lang="python">
    keepflags=True)
# In CASA 4.2
thesteps = [9,10,11,12,13,14,15,16,17,18]
execfile('uid___A002_X554543_X207.ms.scriptForCalibration_4.2.py')
execfile('uid___A002_X554543_X3d0.ms.scriptForCalibration_4.2.py')
execfile('uid___A002_X554543_X667.ms.scriptForCalibration_4.2.py')
execfile('scriptForFluxCalibration_4.2.py')
</source>
</source>

Latest revision as of 20:46, 29 March 2017

In the ALMA archive for Cycle 0 data, the packaged calibration script found in the scripts directory is written for execution in CASA 3.4. While it possible to use the packaged calibration script as it is in CASA 3.4, it is now recommended to use CASA 4.2 for calibration and imaging. To change the packaged calibration script into a CASA 4.2 script, it must be manually modified using a text editor, following the instructions below. In the script examples below 'measurementset' stands for your uid name (for example, uid___A002_X554543_X207). If, after applying those edits and launching the execution, the script either crashes or does not calibrate the data as expected, please contact the CASA HelpDesk.

  • Change the list of steps at the beginning of the script to
step_title = {0: 'Import of the ASDM',
              1: 'Fix of SYSCAL table times',
              2: 'listobs',
              3: 'A priori flagging',
              4: 'Generation and time averaging of the WVR cal table',
              5: 'Generation of the Tsys cal table',
              6: 'Generation of the antenna position cal table',
              7: 'Application of the WVR, Tsys and antpos cal tables',
              8: 'Split out science SPWs and time average',
              9: 'Listobs, clear pointing table, and save original flags',
              10: 'Initial flagging',
              11: 'Putting a model for the flux calibrator(s)',
              12: 'Save flags before bandpass cal',
              13: 'Bandpass calibration',
              14: 'Save flags before gain cal',
              15: 'Gain calibration',
              16: 'Save flags before applycal',
              17: 'Application of the bandpass and gain cal tables',
              18: 'Split out corrected column'}


  • add "import analysisUtils as au", "import analysisUtils as aU", "import os" and "if 'applyonly' not in globals(): applyonly = False" on line 27. The code under the steps list should now read:
import analysisUtils as au
import analysisUtils as aU
import os
if 'applyonly' not in globals(): applyonly = False  #(AM) added
try:
  print 'List of steps to be executed ...', mysteps
  thesteps = mysteps
except:
  print 'global variable mysteps not set.'


  • Add "if applyonly != True:" in front of any aU and es routines calls. For example:
if applyonly != True: es = aU.stuffForScienceDataReduction() 
if applyonly != True:  es.checkCalTable('measurementset.ms.split.ap_pre_bandpass', msName='measurementse.ms.split', interactive=False)


  • Replace the CASA 3.4 version search to CASA 4.2
if re.search('^4.2', casadef.casa_version) == None: #(AM)
 sys.exit('ERROR: PLEASE USE THE SAME VERSION OF CASA THAT YOU USED FOR GENERATING THE SCRIPT: 4.2')


  • In step 7:
    • the label of the step must be changed to 9 ('mystep = 9') (the steps 0-8 will be skipped since they do not need to be applied on the "raw" data supplied with the data package. Hence one does not need to worry about introducing 'missing' steps)
    • add 'if not os.path.exists('replaceuidlabelhere.ms.split.flagversions/Original.flags'):' above the flagmanager call, and ident the flagmanager call. It should now read (for the example of uid uid___A002_X554543_X207):
  if not os.path.exists('uid___A002_X554543_X207.ms.split.flagversions/Original.flags'):
    flagmanager(vis = 'uid___A002_X554543_X207.ms.split',
      mode = 'save',
      versionname = 'Original')


  • In step 8:
    • the label of the step must be changed to 10 ('mystep = 10')
    • any call to tflagdata must be changed to flagdata


  • In step 9:
    • the label of the step must be changed to 11 ('mystep = 11')
    • if applicable (when a Solar System Object is used as a flux calibrator), in setjy change the parameter standard = 'Butler-JPL-Horizons 2010' to standard = 'Butler-JPL-Horizons 2012'
    • if applicable (when a non solar system object is used as flux calibrator), change the clearcal command to delmod (inputs are unchanged)


  • In step 10:
    • the label of the step must be changed to 12 ('mystep = 12')


  • In step 11:
    • the label of the step must be changed to 13 ('mystep = 13')
    • in the gaincal calls, change any calmode='ap' parameter to calmode='p'


  • In step 12:
    • the label of the step must be changed to 14 ('mystep = 14')


  • In step 13:
    • the label of the step must be changed to 15 ('mystep = 15')
    • if present, change clearcal command to delmod (inputs are unchanged)
    • in the gaincal calls, change any calmode='ap' parameter to calmode='a'
    • if a phaseCalName loop is present, change:
     if fc[i].find('Flux density for '+phaseCalName) != -1 and re.search('in SpW=[0-9]+(?: \(ref SpW=[0-9]+\))? is: [0-9]+\.[0-9]+', fc[i]) != None:
        line = (re.search('in SpW=[0-9]+(?: \(ref SpW=[0-9]+\))? is: [0-9]+\.[0-9]+', fc[i])).group(0)

to

      if fc[i].find('Flux density for '+phaseCalName) != -1 and re.search('in SpW=[0-9]+(?: \(.*?\))? is: [0-9]+\.[0-9]+', fc[i], re.DOTALL|re.IGNORECASE) != None:
        line = (re.search('in SpW=[0-9]+(?: \(.*?\))? is: [0-9]+\.[0-9]+', fc[i], re.DOTALL|re.IGNORECASE)).group(0)
  • In step 14:
    • the label of the step must be changed to 16 ('mystep = 16')


  • In step 15:
    • the label of the step must be changed to 16 ('mystep = 17')


  • Add a step 18 (optional):

Add this piece of code, where measurementset stands for the uid label (as in, 'uid___A002_X554543_X207'):

mystep = 18
if(mystep in thesteps):
  casalog.post('Step '+str(mystep)+' '+step_title[mystep],'INFO')
  print 'Step ', mystep, step_title[mystep]

  os.system('rm -rf measurementset.ms.split.cal') 
  split(vis = 'measurementset.ms.split',
    outputvis = 'measurementset.ms.split.cal',
    datacolumn = 'corrected',
    keepflags=True)