Antennae Band7 - Calibration for CASA 3.3
Overview
This portion of the AntennaeBand7 CASA Guide will cover the calibration of the raw visibility data. To skip to the imaging portion of the guide, see: Antennae Band7 - Imaging.
If you haven't already downloaded the raw data, you may do that now by clicking on the region closest to your location and downloading the file named 'Antennae_Band7_UnCalibratedMSandTablesForReduction.tgz':
Once the download has finished, unpack the file:
# In a terminal outside CASA
tar -xvzf Antennae_Band7_UnCalibratedMSandTablesForReduction.tgz
cd Antennae_Band7_UnCalibratedMSandTablesForReduction
# Start CASA
casapy
The data have already been converted to CASA Measurement Set (MS) format using the CASA task importasdm. Accompanying the data are some basic calibration tables you will need for the following reduction, as well as the *.ms.flagversions files that are automatically generated by importasdm.
Initial Inspection
We will eventually concatenate the nine datasets used here into one large dataset. However, we will keep them separate for now, as some of the steps to follow require individual datasets (specifically, the application of the Tsys and WVR tables). We therefore start by defining an array "basename" that includes the names of the six files in chronological order. This will simplify the following steps by allowing us to loop through the files using a simple for-loop in python. Remember that if you log out of CASA, you will have to re-issue this command. We will remind you of this in the relevant sections by repeating the command at the start.
# In CASA
basename=['uid___A002_X1d54a1_X5','uid___A002_X1d54a1_X174','uid___A002_X1d54a1_X2e3',
'uid___A002_X1d5a20_X5','uid___A002_X1d5a20_X174','uid___A002_X1d5a20_X330']
The usual first step is then to get some basic information about the data. We do this using the task listobs, which will output a detailed summary of each dataset supplied.
# In CASA
for name in basename:
listobs(vis=name+'.ms')
Note that after cutting and pasting a for-loop you often have to press return several times to execute. The output will be sent to the CASA logger. You will have to scroll up to see the individual output for each of the datasets. Here is an example of the most relevant output for the first file in the list.
2011-07-13 03:09:07 INFO PlotMS::summary+ MeasurementSet Name: /Users/despada/Desktop/Imaging/Antennae/Datasets/band7/uid___A002_X207fe4_X3a.ms MS Version 2 2011-07-13 03:09:07 INFO PlotMS::summary+ ================================================================================ 2011-07-13 03:09:07 INFO PlotMS::summary+ Observer: Unknown Project: T.B.D. 2011-07-13 03:09:07 INFO PlotMS::summary+ Observation: ALMA(12 antennas) 2011-07-13 03:09:07 INFO PlotMS::summary Data records: 211116 Total integration time = 4997.38 seconds 2011-07-13 03:09:07 INFO PlotMS::summary+ Observed from 03-Jun-2011/21:08:39.6 to 03-Jun-2011/22:31:56.9 (UTC) 2011-07-13 03:09:07 INFO PlotMS::summary Fields: 26 2011-07-13 03:09:07 INFO PlotMS::summary+ ID Code Name RA Decl Epoch SrcId 2011-07-13 03:09:07 INFO PlotMS::summary+ 0 none 3c279 12:56:11.1666 -05.47.21.5247 J2000 0 2011-07-13 03:09:07 INFO PlotMS::summary+ 1 none Titan 12:42:08.4083 -01.42.40.8166 J2000 1 2011-07-13 03:09:07 INFO PlotMS::summary+ 2 none NGC4038 - A* 12:01:53.1701 -18.52.37.9200 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 3 none NGC4038 - A* 12:01:51.9030 -18.51.49.9437 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 4 none NGC4038 - A* 12:01:52.4309 -18.51.49.9437 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 5 none NGC4038 - A* 12:01:52.9587 -18.51.49.9437 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 6 none NGC4038 - A* 12:01:53.4866 -18.51.49.9436 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 7 none NGC4038 - A* 12:01:54.0144 -18.51.49.9436 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 8 none NGC4038 - A* 12:01:52.1669 -18.51.56.4319 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 9 none NGC4038 - A* 12:01:52.6948 -18.51.56.4318 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 10 none NGC4038 - A* 12:01:53.2226 -18.51.56.4318 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 11 none NGC4038 - A* 12:01:53.7505 -18.51.56.4318 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 12 none NGC4038 - A* 12:01:51.9030 -18.52.02.9201 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 13 none NGC4038 - A* 12:01:52.4309 -18.52.02.9200 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 14 none NGC4038 - A* 12:01:52.9587 -18.52.02.9200 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 15 none NGC4038 - A* 12:01:53.4866 -18.52.02.9200 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 16 none NGC4038 - A* 12:01:54.0144 -18.52.02.9199 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 17 none NGC4038 - A* 12:01:52.1669 -18.52.09.4082 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 18 none NGC4038 - A* 12:01:52.6948 -18.52.09.4082 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 19 none NGC4038 - A* 12:01:53.2226 -18.52.09.4082 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 20 none NGC4038 - A* 12:01:53.7505 -18.52.09.4081 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 21 none NGC4038 - A* 12:01:51.9030 -18.52.15.8964 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 22 none NGC4038 - A* 12:01:52.4309 -18.52.15.8964 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 23 none NGC4038 - A* 12:01:52.9587 -18.52.15.8963 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 24 none NGC4038 - A* 12:01:53.4866 -18.52.15.8963 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ 25 none NGC4038 - A* 12:01:54.0144 -18.52.15.8963 J2000 2 2011-07-13 03:09:07 INFO PlotMS::summary+ (nVis = Total number of time/baseline visibilities per field) 2011-07-13 03:09:08 INFO PlotMS::summary Spectral Windows: (9 unique spectral windows and 2 unique polarization setups) 2011-07-13 03:09:08 INFO PlotMS::summary+ SpwID #Chans Frame Ch1(MHz) ChanWid(kHz)TotBW(kHz) Ref(MHz) Corrs 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 4 TOPO 184550 1500000 7500000 183300 I 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 3840 TOPO 344845.586 488.28125 1875000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 1 TOPO 343908.086 1875000 1875000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 3 3840 TOPO 354971.074 488.28125 1875000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 4 1 TOPO 343908.086 1875000 1875000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 5 128 TOPO 344900.518 15625 2000000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 6 1 TOPO 343892.705 1796875 1796875 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 7 128 TOPO 354916.143 15625 2000000 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary+ 8 1 TOPO 343892.705 1796875 1796875 344908.33 XX YY 2011-07-13 03:09:08 INFO PlotMS::summary Sources: 63 2011-07-13 03:09:08 INFO PlotMS::summary+ ID Name SpwId RestFreq(MHz) SysVel(km/s) 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 0 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 9 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 10 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 11 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 12 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 13 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 14 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 15 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 16 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 17 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 18 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 19 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 20 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 1 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 2 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 3 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 3c279 4 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 0 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 9 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 10 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 11 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 12 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 13 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 14 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 15 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 16 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 17 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 18 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 19 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 20 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 1 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 2 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 3 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 Titan 4 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 Titan 5 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 Titan 6 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 Titan 7 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 0 Titan 8 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 3c279 5 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 3c279 6 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 3c279 7 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 1 3c279 8 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 0 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 9 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 10 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 11 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 12 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 13 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 14 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 15 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 16 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 17 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 18 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 19 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 20 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 1 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 2 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 3 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 4 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 5 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 6 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 7 - - 2011-07-13 03:09:08 INFO PlotMS::summary+ 2 NGC4038 - A* 8 - - 2011-07-13 03:09:08 INFO PlotMS::summary Antennas: 12 'name'='station' 2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 0-3: 'DV02'='A015', 'DV04'='J505', 'DV05'='A067', 'DV06'='T704', 2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 4-7: 'DV07'='A004', 'DV08'='A072', 'DV09'='A008', 'DV10'='A009', 2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 8-11: 'DV11'='A016', 'DV12'='A011', 'PM01'='T702', 'PM03'='J504' 2011-07-13 03:09:42 INFO PlotMS::summary ================================================================================
This output shows that three fields were observed: 3c279, Titan, and the different pointing of the Antennae mosaic. Field 0 (3c279) will serve as the gain calibrator and bandpass calibrator; field 1 (Titan) will serve as the flux calibrator; and field 2 (Antennae) is, of course, the science target.
Note that there are more than two SpwIDs even though the observations were set up to have two spectral windows. The spectral line data themselves are found in spectral windows 1,3, which have 3840 channels each. The first one (spw 1) is centered on the CO(3-2) emission line in the Antennae. There is one additional spectral window (spw 3) in the Upper Side Band (USB). The additional spectral window is used to measure the continuum emission in the galaxy, and may contain other emission lines as well. The spectral resolution and total bandwidth of these two spectral windows are 488.28 MHz and 1.875 GHz (3840 channels), respectively, corresponding to 0.426 km/s and 1634 km/s. There are two spectral windows (spw 5 and 7) fully covering spw 1 and 3 but in TDM mode (2 GHz bandwidth, 128 channels), which are used for tsys calibration.
Spectral windows 2,4,6,8 contain channel averages of the data in spectral windows 1,3,5,7, respectively. These are not useful for the offline data reduction. Spectral window 0 contains the WVR data. You may notice that there are additional SpwIDs listed in the "Sources" section which are not listed in the "Spectral Windows" section. These spectral windows are reserved for the WVRs of each antenna (seven in our case). At the moment, all WVRs point to spw 0, which contains nominal frequencies. The additional spectral windows (spw 9-15) are therefore not used and can be ignored.
Another important thing to note is that the position of Titan is listed as 00:00:00.0000 +00.00.00.0000. This is due to the fact that for ephemeris objects, the positions are currently not stored in the asdm. This will be handled correctly in the near future, but at present, we have to fix this offline. We will correct the coordinates below by running the procedure fixplanets, which takes the position from the pointing table.
The final column of the listobs output in the logger (not shown above) gives the scan intent. This information is used later to flag the pointing scans and the hot and ambient load calibration scans, using scan intent as a selection option. Also these intents will be used in the future for pipeline processing.
A total of 12 antennas were used for the dataset listed above. Note that numbering in python always begins with "0", so the antennas have IDs 0-11. To see what the antenna configuration looked like at the time of this observation, we will use the task plotants.
# In CASA
plotants(vis=basename[0]+'.ms', figfile=basename[0]+'_plotants.png')
This will plot the antenna configuration on your screen as well as save it under the specified filename for future reference. This will be important later on when we need to choose a reference antenna, since the reference antenna should be close to the center of the array (as well as stable and present for the entire observation).
You can check using plotants for the other datasets that the configuration stays relatively constant during the course of the observations.
As for Delay correction, no large delays are found in phase in any of the datasets, so bandpass calibration should remove the small phase delays (i.e., less than one wrap over the bandpass).
Data edition
Flagging
The first editing we will do is some a priori flagging with flagdata and flagautocorr. We will start by flagging the shadowed data and the autocorrelation data. ALMA data contains both the cross correlation and autocorrelation data, but here we are only interested in the cross-correlation data. Additionally, for compact configurations of the array, one antenna can shadow another, blocking its view. These data also need to be flagged. Remember that you first need to redefine the "basename" array if you logged out of CASA prior to starting this section.
# In CASA
basename_all=["uid___A002_X1ff7b0_Xb","uid___A002_X1ff7b0_X1c8","uid___A002_X207fe4_X3a","uid___A002_X207fe4_X1f7","uid___A002_X207fe4_X3b9","uid___A002_X207fe4_X4d7","uid___A002_X215db8_X18","uid___A002_X215db8_X1d5","uid___A002_X215db8_X392","uid___A002_X2181fb_X49"]
Now we will loop over the datasets, running the two flagging commands:
# In CASA
for name in basename_all:
flagdata(vis=name+'.ms', flagbackup = F, mode = 'shadow')
flagautocorr(vis=name+'.ms')
There are a number of scans in the data that were used by the online system for pointing calibration. These scans are no longer needed, and we can flag them easily with flagdata by selecting on 'intent':
# In CASA
for name in basename_all:
flagdata(vis=name+'.ms', mode='manualflag', flagbackup = F, intent='*POINTING*')
Similarly, we can flag the scans corresponding to atmospheric calibration:
# In CASA
for name in basename_all:
flagdata(vis=name+'.ms', mode='manualflag', flagbackup = F, intent='*ATMOSPHERE*')
We will then store the current flagging state for each dataset using the flagmanager:
# In CASA
for name in basename:
flagmanager(vis = name+'.ms', mode = 'save', versionname = 'Apriori')
Fixing frequency labelling problem
Frequency information for spectral window 3 and 7 (Upper Side Band) need to be fixed.
# In CASA
for i in basename_all:
ms1 = i+'.ms'
bbFreq = 355.90833
spwToCorrect = [3,7]
tb.open(ms1+'/SPECTRAL_WINDOW', nomodify=False)
for i in range(tb.nrows()):
if i in spwToCorrect:
numChan1 = tb.getcell(columnname='NUM_CHAN', rownr=i)
chanFreq1 = tb.getcell(columnname='CHAN_FREQ', rownr=i)
centerFreq1 = (max(chanFreq1) + min(chanFreq1)) / 2.0
print i, numChan1, centerFreq1, bbFreq, max(chanFreq1)-min(chanFreq1)
chanFreq1 = chanFreq1 + (bbFreq*1e9 - centerFreq1)
centerFreq1 = (max(chanFreq1) + min(chanFreq1)) / 2.0
print i, numChan1, centerFreq1, bbFreq, max(chanFreq1)-min(chanFreq1)
for j in range(numChan1):
chanFreq1[j] = centerFreq1 + (centerFreq1 - chanFreq1[j])
tb.putcell(columnname='CHAN_FREQ', rownr=i, thevalue=chanFreq1)
tb.close()
Creating Tsys
Next we do the Tsys calibration. Tsys measurements correct for the atmospheric opacity (to first-order) and allow the calibration sources to be measured at elevations that differ from the science target. The Tsys tables for these datasets were provided with the downloadable data, and have been obtained as follows:
numms=len(basename_all)
for msnumber in range (0,numms):
asdm=basename_all[msnumber]
#Gencal. This makes a gain table from the TDM mode (2GHz, 128 channels per baseband, XX and YY polarizations).
# Gencal needs to be executed on each ms individually. If it is done on
# the concatenated data set, only values for the first observation are generated.
#Don't worry about 'No scr col generation' message.
#band 5
os.system('rm -rf '+ asdm+'.tsys.cal')
gencal(vis=asdm+".ms", caltype='tsys', caltable=asdm+'.tsys.cal')
#Plot the tsys calibration tables:Time and Freq. domain
plotcal(caltable=asdm+'.tsys.cal', xaxis="time", iteration='antenna',yaxis="amp",overplot=False, plotrange=[0, 0, 0, 0],plotsymbol=".",figfile=asdm+'.tsys-time.cal.png',timerange='<400000')
plotcal(caltable=asdm+'.tsys.cal', xaxis="freq", yaxis="amp", subplot=111, overplot=False, plotrange=[0, 0, 0, 0],plotsymbol=".",figfile=asdm+'.tsys-spec-tdm.cal.png',timerange='<400000')
#at the end of the band there are high values.
#without the timerange< it does not work...
sleep(5)
#Instance for tsys interpolation band5
interpolatetsys=aU.InterpolateTsys(asdm+'.tsys.cal')
#Ignoring spectral window [0, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] because it is WVR related
#Correcting time stamps
interpolatetsys.correctBadTimes()
# this sets the tsys when atmcal was actually observed.
interpolatetsys.assignFieldAndScanToSolution() # This associates the field and scan numbers to the table properly. This will allow you to use the field id to identify which tsys you want to map to which field source in the dataset.
#Plot the tsys calibration tables with corrected times: Time domain. spw 5 and 7 are TDM mode.
plotcal(caltable=asdm+'.tsys.cal', xaxis="time", iteration='antenna',yaxis="amp",overplot=False, plotrange=[0, 0, 0, 0],plotsymbol=".",figfile=asdm+'.tsys-time-correctbadtimes.cal.png',timerange='<400000')
#without the <number it does not work.
sleep(5)
#TDM to FDM interpolation
interpolatetsys.getTdmFdmSpw()
interpolatetsys.interpolateTsys()
# many errors!!! but just means that some channels are missing.
plotcal(caltable=asdm+'.tsys.cal.fdm', xaxis="freq",yaxis="amp",overplot=False,plotrange=[0, 0, 0, 0],plotsymbol=".",figfile=asdm+'.tsys-spec-fdm.cal.png',timerange='<400000')
We inspect them with the task plotcal:
Aside from the large amplitudes in the edge channels (which we will handle below), the plots look acceptable. Note that in the lowest spectral window ID (spw 1), Tsys rises toward higher frequencies. This is due to a spectral line from O2 at XX GHz.
Applying calibration tables for WVR and Tsys
We will now apply the tsys and the WVR calibration tables to the data with the task applycal. We do this in two steps. First we will cycle over the three datasets from the first day of observations, because we have to correct the delay error for DV07 for those data. For the last three datasets (i.e. those taken during the second day), we do not need to correct the delays, so we just apply the WVR tables. In the call to applycal, we will specify interpolation="nearest". This is important for the WVR corrections, and it doesn't make a difference for the delay corrections because they have no time dependence.
First, remember to redefine the "basename_all" array if you've logged out of CASA at any point up until now.
# In CASA
basename_all=["uid___A002_X1ff7b0_Xb","uid___A002_X1ff7b0_X1c8","uid___A002_X207fe4_X3a","uid___A002_X207fe4_X1f7","uid___A002_X207fe4_X3b9","uid___A002_X207fe4_X4d7","uid___A002_X215db8_X18","uid___A002_X215db8_X1d5","uid___A002_X215db8_X392","uid___A002_X2181fb_X49"]
Now loop through the datasets as described above, applying the tsys and wvr calibration tables to all the datasets and sources:
# In CASA
for asdm in basename_all:
print asdm
#plotcal(caltable=directory+asdm+'.tsys.cal.fdm', xaxis="freq",yaxis="amp",overplot=False,plotrange=[0, 0, 0, 0],plotsymbol=".",timerange='<400000')
for fieldname in ['Titan','3c279','NGC4038*','Antenna*']:
applycal(vis=directory+asdm+".ms", spw='1,3', flagbackup=F, field=fieldname, gainfield=fieldname,interp=['nearest','nearest'], gaintable=[directory+asdm+".tsys.cal.fdm",directory+asdm+'.wvr.cal'])
Now you can use plotms to show some of the before-and-after effects of calibration. Just re-run the previous plotms command, repeated below, or if the window is still open, check 'force reload'. This will display the uncorrected phases across the band. To display the corrected phases, you will need to select the CORRECTED data column in the axes tab and replot.
# In CASA
plotms(vis=basename[0]+'.ms', xaxis='channel', yaxis='phase',
spw='1', antenna='', correlation='XX', avgtime='1e8',
coloraxis='baseline', avgscan=T, selectdata=T, field='1037*')
Now we split out the CORRECTED data column of the datasets with the task split. We give the resulting datasets the extension "_K_WVR" to indicate that the delay tables and WVR tables have been applied in the DATA column, and we specify spectral windows 0-8 to get rid of the "WVR placeholder" spectral windows. Since split will not overwrite existing files, we start by removing any previous versions of the split ms's before running the split command.
# In CASA
for asdm in basename_all:
os.system('rm -rf '+asdm+'.split.ms')
split(vis=directory+asdm+'.ms', outputvis=asdm+'.split.ms', datacolumn='corrected', spw='1,3')
We will apply the Tsys tables with applycal. We do this for each field separately so that the appropriate calibration data are applied to the right fields. The "field" parameter specifies the field to which we will apply the calibration, and the "gainfield" parameter specifies the field from which we wish to take the calibration solutions from the gaintable.
Without giving the full recipe here, we suggest at this point that you use plotms to plot channel-averaged amplitudes as a function of time, comparing the DATA and CORRECTED columns after applying the Tsys correction. This way you can check that calibration has done what was expected, which is put the data onto the Kelvin temperature scale.
We then once again split out the CORRECTED data column, this time only retaining spectral windows 1,3. This will get rid of the extraneous spectral windows, including the channel averaged spectral windows and spw 0, which is the one that contained the WVR data.
# In CASA
for name in basename:
os.system('rm -rf '+name+'_line.ms*')
split(vis=name+'_K_WVR.ms', outputvis=name+'_line.ms',
datacolumn='corrected', spw='1,3,5,7')
The WVR and Tsys tables are now applied in the DATA column of the new measurement sets. These datasets have the extension "_line" to indicate that they only contain the spectral line data and no longer the "channel average" spectral windows. These measurement sets therefore have four spectral windows, with IDs 0-3.
Now that we have applied the Tsys calibration and WVR corrections, in plotms we can have a look to the original and corrected data. We perform an additional data inspection for other possible problems, such as bad data, phase jumps, etc.
Additional Data Inspection
Now that the data are concatenated into one dataset, we will do some additional inspection with plotms. First we will plot amplitude versus channel, averaging over time and baselines in order to speed up the plotting process.
# In CASA
plotms(vis='ngc3256_line.ms', xaxis='channel', yaxis='amp',
averagedata=T, avgbaseline=T, avgtime='1e8', avgscan=T)
From this plot we see that the edge channels have abnormally high amplitudes. We will use flagdata to remove the edge channels from both sides of the bandpass:
# In CASA
flagdata(vis='ngc3256_line.ms', flagbackup=T, spw=['*:0~16','*:125~127'])
Next, we will look at amplitude versus time, averaging over all channels and colorizing by field. Since the observations take place over two days, you will need to zoom in to examine the data in more detail. In particular, note the difference in Titan's amplitude between the two days and the change in amplitude during the second day. The plot on the right shows a zoom-in on the first day of observations, only showing spw 1 in this case. Scans on Titan are colored red, NGC3256 is orange, and the calibrator 1037-295 is colored black. If you select other spws, you can see some outlying points, which will be flagged later on.
# In CASA
plotms(vis='ngc3256_line.ms', xaxis='time', yaxis='amp',
averagedata=T, avgchannel='128', coloraxis='field',
iteraxis='spw')
Checking flux calibrator and fixing its position
Titan is our primary flux calibrator. However, for the second day of observations, Titan had moved too close to Saturn, and Saturn's rings moved into the primary beam. Another way to see this is to plot amplitude versus uv-distance and colorize by scan:
# In CASA
plotms(vis='ngc3256_line.ms', xaxis='uvdist', yaxis='amp', field='1',
averagedata=T, avgchannel='128', avgtime='1e8', coloraxis='scan')
If you use the "Mark a region" and "Locate" tools in plotms, you will see the signature of a bright, resolved source during the latter three (last day's) scans. We will therefore need to flag the Titan scans for the second day:
# In CASA
flagdata(vis = 'ngc3256_line.ms', flagbackup = T,
timerange='>2011/04/16/12:00:00', field='Titan')
We also find that during the first day, the Titan observations in spw 2 and 3 are also affected by Saturn. These spectral windows are at lower frequencies and therefore correspond to slightly larger primary beams. This results in Saturn just being picked up in spw 2 and 3, but not in spw 0 and 1. We do not flag these data here, but have to take this effect into account when we do the flux calibration later.
Next, we will fix the position of Titan in the combined dataset. Recall that the position of the Titan field is currently set to 00:00:00.0000 +00.00.00.0000. The following procedure will replace this with the actual mean position observed by the telescopes and, at the same time, it will recalculate the uvw coordinates.
First initialize the CASA script with:
# In CASA
execfile(casadef.python_library_directory+'/recipes/fixplanets.py')
Now fix the position of Titan:
# In CASA
fixplanets('ngc3256_line.ms', 'Titan', True)
The third parameter in fixplanets, set to True, indicates that the uvw-coordinates for Titan are recalculated. Note that for Cycle 0 data, the coordinates of ephemeris objects will be treated correctly in the data.
Now check to see that the coordinates for Titan have been corrected:
2011-07-13 03:08:32 INFO PlotMS::summary+ 0 none 3c279 12:56:11.1666 -05.47.21.5247 J2000 0 2011-07-13 03:08:32 INFO PlotMS::summary+ 1 none Titan 12:42:09.7695 -01.42.26.8843 J2000 1 2011-07-13 03:08:32 INFO PlotMS::summary+ 2 none NGC4038 - A* 12:01:53.1701 -18.52.37.9200 J2000 2
Further flagging on each individual dataset
Continue to inspect the data with plotms, plotting different axes and colorizing by the different parameters. Don't forget to average the data if possible to speed the plotting process. Have a look to the attached file to see the different problems that have been identified. The times to insert in flagdata can be obtained using plotms Tools Hover/Display. Instead of using the following flagdata commands, you can also flag by hand in plotms. To do this, select your bad data by clicking on the 'Mark Regions" button, then on 'Flag".
We indicated how to flag the bad data in different instances of the flagdata command.
asdm="uid___A002_X207fe4_X3a"
2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 0-3: 'DV02'='A015', 'DV04'='J505', 'DV05'='A067', 'DV06'='T704',
2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 4-7: 'DV07'='A004', 'DV08'='A072', 'DV09'='A008', 'DV10'='A009',
2011-07-13 03:09:08 INFO PlotMS::summary+ ID= 8-11: 'DV11'='A016', 'DV12'='A011', 'PM01'='T702', 'PM03'='J504'
#Quality: ~, phase scatter is ~twice higher than other datasets, spectrum phase is not flat, delay problem?
#wvr: ok
#Antennas: 12
#tsys: DV04 is slightly above the others (AIV-4694). Several antennas have ripples in the spectrum. Probably no need to flag.
# - Phase fluctuation was large in the beginning, bandpass
# - DV04 Trx/Tsys was higher than the other antennas
# - PM01 Pol Y total power jumps (AIV-4260)
# - CDMP reported "RES DELAY FAILURE" (CSV-859)
#Source is at ~20 deg higher elevation than calibrators. Antennae ~ 60 while calibrator is at ~40 deg.
#Continuum phase: large scatter, ~ 100 deg. Specially first scan of phase calibrator. Bandpass scans are a bit noisy. First scan on phase calibrator is noisy. Flagged manually in plotms.
#Continuum phase: Small phase jumps ~ 20 deg *&PM01 and *&PM03. Don't think that it is just the atmosphere, as even short baselines show these small jumps.
#Continuum phase: DV09 phase jump. Flag: 21:24:09~21:35:35
flagdata(flagbackup = F,vis = directory+asdm+'.ms',mode='manualflag',antenna='DV09',timerange='21:24:09~21:35:35')
#Spectrum phase. *&PM03,spw=3,corr='YY' need to be removed. Weird behaviour also found in amplitude.
flagdata(flagbackup = F,vis = directory+asdm+'.ms',mode='manualflag',antenna='PM03',correlation='YY',spw='3')
#Spectrum amp. DV08,DV06? few subscans need to be removed in the first scan (phase calibration). Confirmed in Continuum amp. Flagged manually in plotms.
#Spectrum phase. Phases not flat in most baselines. delay problem. It can be calibrated.
#Spectrum phase: DV12 spw=1 corr=YY some negative peaks
flagdata(vis = directory+asdm+'.ms',antenna='DV12',spw='1:638~641;1663~1664;2430~2433',correlation='YY')
#Antennae: Line is detected.
flagmanager(vis =directory+ asdm+'.ms',mode = 'save',versionname = 'FlagFinal')
Bandpass and gain calibration
First we split the different datasets only with spectral windows 1 and 3 and applied tsys and wvr corrections.
for asdm in basename_all:
os.system('rm -rf '+asdm+'.split.ms')
split(vis=directory+asdm+'.ms', outputvis=asdm+'.split.ms', datacolumn='corrected', spw='1,3')
Next we will inspect the bandpass calibrator by plotting the phase as a function of time and frequency. For the first plot we use avgscan=T and avgtime='1E6' to average in time over all scans, and we specify coloraxis='baseline' to colorize by baseline. For the second, we use spw='0:30~3810' and avgchannel='3840' to average over the central 3780 channels of the first spectral window. For both plots we will iterate on antenna, so you will need to use the green arrows at the bottom of the plotms GUI to view the plots for different antennas.
# In CASA
plotms(vis= asdm+'.split.ms', xaxis='freq', yaxis='phase', selectdata=True,
field='3c279', avgtime='1E6', avgscan=T, coloraxis='baseline', iteraxis='antenna')
# In CASA
plotms(vis= asdm+'.split.ms', xaxis='time', yaxis='phase', selectdata=True,
field='3c279', spw='0:40~3800,1:40~3800', avgchannel='3840', avgscan=T,
coloraxis='baseline', iteraxis='antenna')
The top plot on the right shows the time-averaged phase as a function of frequency for the calibrator 3c279, with all baselines shown at once (i.e., without iteraxis='antenna') for presentation purposes. We see that the phase variations across the spectral windows are modest, typically ~30 degrees. The second plot shows how the phase varies as function of time. For clarity, we only show baselines with one antenna and only for the first day of observations. There are clearly phase variations on short time scales that we wish to correct for before calculating the bandpass solutions.
Hence, we run gaincal on the bandpass calibrator to determine phase-only gain solutions. We will use solint='int' for the solution interval, which means that one gain solution will be determined for every integration time. This short integration time is possible because the bandpass calibrator is a very bright point source, so we have very high signal-to-noise and a perfect model. This will correct for any phase variations in the bandpass calibrator as a function of time, a step which will prevent decorrelation of the vector-averaged bandpass solutions. We will then apply these solutions on-the-fly when we run bandpass.
We will use the average of channels 40~3800 and two spw to increase our signal-to-noise in the determination of the antenna-based phase solutions. Averaging over a subset of channels near the center of the bandpass is acceptable when the phase variation as a function of channel is small. For our reference antenna, we choose DV06. We call the output calibration table asdm+'.g1.cal'. We then check the time variations of the phase solutions with plotcal. We will plot the XX and YY polarization products separately and make different subplots for each of the spectral windows. This is done by setting the "iteration" parameter to "spw" and specifying subplot=221. By setting the parameter "figfile" to a non-blank value, it will also generate png files of the plots.
Now that we have a first measurement of the phase variations as a function of time, we can determine the bandpass solutions with bandpass. We will apply the phase calibration table on-the-fly with the parameter "gaintable". Now that the phases are corrected, the data can be time-averaged over longer intervals to maximise SNR in each individual channel. We determine bandpass solutions for both days separately because we find that for some antennas, the response across the bandwidth has changed slightly in between the two sessions. We first calculate the bandpass solution for the first day. Do not worry about the message "Insufficient unflagged antennas", which relates to the flagged edge channels. We then plot the bandpass solutions with plotcal again, and we see that the solutions seem reasonable.
for asdm in basename_all:
os.system('rm -rf '+asdm+'.g1.cal,'+asdm+'.b1.cal')
gaincal(vis = asdm+'.split.ms',spw = '0:40~3800,1:40~3800',caltable = asdm+'.g1.cal',field = '3c279',selectdata=T,solint = 'int',refant = 'DV06',calmode='p')
plotcal(caltable = asdm+'.g1.cal',iteration = 'spw,antenna',subplot = 221,plotrange=[0,0,-180,180],xaxis = 'time',yaxis = 'phase')
bandpass(vis = asdm+'.split.ms',gaintable = [asdm+'.g1.cal'],caltable = asdm+'.b1.cal',field = '3c279',bandtype='B',solint = 'inf',combine = 'scan', solnorm=T,refant = 'DV06',minblperant=3,minsnr=2,fillgaps=1)
plotcal(caltable = asdm+'.b1.cal',xaxis = 'freq',yaxis = 'amp',plotrange = [0,0,0.7,1.3],iteration = 'spw',subplot = 211,poln='X',figfile=asdm+'-b1-X-amp_vs_freq.png',showgui=F)
where
- caltable = asdm+'.b1.cal': Output bandpass calibration table
- gaintable = 'asdm+'.g1.cal': Gain calibration table to apply
- minblperant=3: Minimum number of baselines required per antenna for each solve
- minsnr=2: Minimum SNR for solutions
- solint='inf': This setting, combined with the default combine='scan', sets the solution interval to the entire observation
- combine='scan': The solutions cross scans
- bandtype='B': The default type of bandpass solution, which does a channel by channel solution for each specified spw
- fillgaps=1: Interpolate channel gaps 1 channel wide
- solnorm=T: Normalize the bandpass amplitudes and phases of the corrections to unity
Flux Calibration
The first step is to set the flux density for Titan using the task setjy. We will use the Butler-JPL-Horizons 2010 model. We will do a new gain calibration, this time applying the bandpass calibration solutions on-the-fly. We solve for amplitude and phase simultaneously and determine average solutions per scan. Finally, we will bootstrap the flux density of the secondary calibrator from that of Titan using the task fluxscale.
for asdm in basename_all:
os.system('rm -rf '+asdm+'.g2.cal,'+asdm+'.f1.cal')
# Put in Model for Titan
default('setjy')
setjy(vis = asdm+'.split.ms',field = 'Titan',standard = 'Butler-JPL-Horizons 2010')
#2011-05-30 06:48:35 INFO setjy Titan spwid= 1 [I=2.817, Q=0, U=0, V=0] Jy, (JPL-Butler Solar System Object)
# Do gaincal in Titan and 3c279
gaincal(vis = asdm+'.split.ms',gaintable = asdm+'.b1.cal',caltable = asdm+'.g2.cal',refant = 'DV06',solint = 'int',field = 'Titan, 3c279')
#plotcal(caltable = asdm+'.g2.cal',xaxis = 'time',yaxis = 'amp', iteration = 'antenna',subplot = 421, plotrange = [0,0,0,0],field = 'Titan, 3c279')
#plotcal(caltable = asdm+'.g2.cal',xaxis = 'time',yaxis = 'phase', iteration = 'antenna',subplot = 421, plotrange=[0,0,-180,180],field = 'Titan, 3c279')
# Get flux densities using fluxscale
fluxscale(vis = asdm+'.split.ms',caltable = asdm+'.g2.cal',fluxtable = asdm+'.f1.cal',reference = 'Titan',transfer = '3c279')
The flux density of Titan is 2.846 Jy at 340 GHz (spw 0) and raises to 3.036 Jy at 350 GHz (spw 1):
#2011-07-13 07:31:04 INFO setjy Titan spwid= 0 [I=2.846, Q=0, U=0, V=0] Jy, (JPL-Butler Solar System Object) #2011-07-13 07:31:13 INFO setjy Titan spwid= 1 [I=3.036, Q=0, U=0, V=0] Jy, (JPL-Butler Solar System Object)
The new flux table asdm+'.f1.cal' replaces the previous asdm+'.g2.cal table in future application of the calibration to the data, i.e. the new flux table contains both asdm+'.g2.cal2 and the newly acquired flux scaling. Unlike the gain calibration steps, this is not an incremental table.
- caltable = 'asdm+'.g2.cal: the output gain calibration table
- minsnr=1.0: To reject solutions with a signal-to-noise less than 1.0
- calmode = 'ap': To solve for amplitude and phase
- solint='inf': Together with the default for the "combine" parameter, this setting will solve for one solution per scan
- solnorm=F: We do not want to normalize the solutions to unity since we wish to relate the measured amplitudes of the secondary calibrator (3c279) to the flux calibrator (Titan)
- gaintable = asdm+'.b1.cal': We apply the bandpass calibration on-the-fly
Now we will examine the amplitude and phase solutions as a function of time, iterating on spectral window and plotting the XX and YY correlations separately for clarity. Zoom in on the GUI to examine the plots in more detail. You can also use the "Mark Region" and "Locate" buttons on the toolbar to identify points. Use the field parameter to select which calibrator you want to plot the solutions for.
We find that the flux density of 1037-295 is ~10 Jy in the higher frequency spectral windows and ~10.2 Jy in the lower frequency spectral windows. This fluxes agree very well with the 0.8mm measurements from the SMA calibrator [1]: 17 Feb 2011 SMA 8.93 ± 0.82.
Applying the calibrations
Now we will use applycal to apply the bandpass and gaincal tables that we generated in the previous sections. First, we will apply the solutions from the secondary calibrator to the science target and the secondary calibrator itself.
for asdm in basename_all:
applycal(vis = asdm+'.split.ms', gaintable = [asdm+'.b1.cal',asdm+'.f1.cal'])
Checking the calibration
We plot the corrected amplitudes and phases of 3c279 as a function of time and frequency, to check that the phases are close to zero and the amplitudes are constant. The result of the amplitude versus time plot is shown to the right.
# In CASA
plotms(vis = asdm+'.split.ms', xaxis='time', yaxis='amp',
ydatacolumn='corrected', selectdata=True, field='3c279',
averagedata=True, avgchannel='128', avgtime='',
avgscan=False, avgbaseline=F, coloraxis='spw')
plotms(vis = asdm+'.split.ms', xaxis='time', yaxis='pha',
ydatacolumn='corrected', selectdata=True, field='3c279',
averagedata=True, avgchannel='128', avgtime='',
avgscan=False, avgbaseline=F, coloraxis='spw')
Let's now use plotms to examine the corrected amplitude and phase of Antennae as a function of frequency, just by changing the field to field='NGC*','Antennae*'
Split source data and smoothing
Different names were given to Antennae galaxies in different datasets. We smooth the velocity to 20 km/s (width = 47).
basename_all_NGC=["uid___A002_X1ff7b0_Xb","uid___A002_X207fe4_X3a","uid___A002_X207fe4_X3b9","uid___A002_X2181fb_X49"]
for asdm in basename_all_NGC:
os.system('rm -rf Antennae-'+asdm+'.cal.ms')
split(vis = asdm+'.split.ms',field = ['NGC*'],outputvis = 'Antennae-'+asdm+'.cal.ms',spw='0,1',width=47)
listobs(asdm+'.split.ms')
basename_all_Ant=["uid___A002_X1ff7b0_X1c8","uid___A002_X207fe4_X1f7","uid___A002_X207fe4_X4d7","uid___A002_X215db8_X18","uid___A002_X215db8_X1d5","uid___A002_X215db8_X392"]
for asdm in basename_all_Ant:
os.system('rm -rf Antennae-'+asdm+'.cal.ms')
split(vis = asdm+'.split.ms',field = ['Antennae*'],outputvis = 'Antennae-'+asdm+'.cal.ms',spw='0,1',width=47)
listobs(asdm+'.split.ms')
Fixing frequency information
As you will see in the split command, there is frequency information problem. We fix that using fixncw.py (check if this fix is already in the new CASA release)
execfile('fixncw.py')
fixncw('Antennae-'+asdm+'.cal.ms') # fix for negative spectral channels.
listobs('Antennae-'+asdm+'.cal.ms')
Concatenating datasets
Once each individual dataset is calibrated, we finally concatenate the data sets corresponding to one either north and south mosaics into one big measurement set. We define an array "comvis" that contains the names of the measurement sets we wish to concatenate, and then we run the task concat.
# In CASA
basename_all_South=["uid___A002_X1ff7b0_X1c8","uid___A002_X207fe4_X1f7","uid___A002_X207fe4_X4d7","uid___A002_X215db8_X18","uid___A002_X215db8_X1d5","uid___A002_X215db8_X392"]
comvis_South=[]
basename_all_North=["uid___A002_X1ff7b0_Xb","uid___A002_X207fe4_X3a","uid___A002_X207fe4_X3b9","uid___A002_X2181fb_X49"]
comvis_North=[]
for asdm in basename_all_South:
comvis_South.append('Antennae-'+asdm+'.cal.ms')
for asdm in basename_all_North:
comvis_North.append('Antennae-'+asdm+'.cal.ms')
os.system('rm -rf Antennae_South.cal.ms,Antennae_North.cal.ms')
concat(vis=comvis_South,concatvis='Antennae_South.cal.ms')
concat(vis=comvis_South,concatvis='Antennae_North.cal.ms')
