Difference between revisions of "DataWeightsAndCombination"

From CASA Guides
(How Do I Know the Situation For My Data?)
 
(195 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<div style="background-color:#E0FFFF;border:4px solid #FF9966;">
+
The purpose of this guide is to describe the CASA visibility weights, and how they relate to data combination. If you are trying to combine ALMA data from Cycles 0, 1, and/or 2, you should be aware of the issues described here. Only data calibrated in >=CASA 4.2.2 pipeline, or manual reduction in >=CASA 4.3 are free from any issues. All of the data to be combined must be in a consistent weight state.
<div style="font-size:250%; color:red; text-align:center;">
 
This page is currently under construction.
 
</div>
 
</div>
 
  
 
== Principles of Data Weighting ==
 
== Principles of Data Weighting ==
  
For optimal imaging performance it is critical that in a relative sense each visibility in the data have the correct weight -- that is, data with better sensitivity have more weight than data with less sensitivity. Formally, the
+
==== Visibility Weights ====
visibility weights should be equal to 1/&sigma;<sub>ij</sub><sup>2</sup> where &sigma;<sub>ij</sub> is the variance or rms noise of a given visibility.
+
 
 +
For optimal imaging performance it is critical that in a ''relative'' sense each visibility in the data have the correct weight after calibration -- that is, data with better sensitivity have more weight than data with less sensitivity. Formally, the post-calibration visibility weights should be equal to 1/&sigma;<sub>ij</sub><sup>2</sup> where &sigma;<sub>ij</sub> is the rms noise of a given visibility.
  
 
<math>  
 
<math>  
\sigma_{ij}=\frac{2k}{\eta_{q}\eta_{c}A_{eff}}  
+
\sigma_{ij} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}}  
 
</math>
 
</math>
 
<math>  
 
<math>  
\sqrt{\frac{T_{sys,i} T_{sys,j}}{\Delta\nu_{ch} t_{ij}}},
+
\sqrt{\frac{T_{sys,i} T_{sys,j}}{2\Delta\nu_{ch} t_{ij}}}
 +
</math>
 +
<math>
 +
\times 10^{26},
 
</math>
 
</math>
  
Line 21: Line 21:
 
''k'' is Boltzmann's constant.  
 
''k'' is Boltzmann's constant.  
  
''A<sub>eff</sub>'' is the effective antenna area which is equal to the aperture efficiency x the geometric area of the antenna. The aperture efficiency depends on the rms antenna surface accuracy and the geometric area on the antenna dish radius<sup>2</sup>.  
+
''A<sub>eff</sub>'' is the effective antenna area which is equal to the aperture efficiency (''&eta;<sub>a</sub>'') x the geometric area of the antenna (&pi; r<sup>2</sup>). The aperture efficiency depends on the rms antenna surface accuracy and is about 0.75 for ALMA dishes.  
  
''&eta;<sub>q</sub>'' and ''&eta;<sub>c</sub>'' are the quantization and correlator efficiencies, respectively. These have values near 1 and will be ignored for the purposes of this casaguide, but see the ALMA Technical Handbook for more information.
+
''&eta;<sub>q</sub>'' and ''&eta;<sub>c</sub>'' are the quantization and correlator efficiencies, respectively. These have values of about 0.88 and 0.96, respectively. See the ALMA Technical Handbook for more information.
  
''T<sub>sys,i</sub>'' is the system temperature for antenna i,  and T<sub>sys,j</sub>'' is the system temperature for antenna j
+
''T<sub>sys,i</sub>'' is the system temperature for antenna i,  and ''T<sub>sys,j</sub>'' is the system temperature for antenna j
  
''&Delta;&nu;<sub>ch</sub>'' is the channel width.
+
''&Delta;&nu;<sub>ch</sub>'' is the effective channel frequency width.
  
 
''t<sub>ij</sub>'' is the integration time per visibility.  
 
''t<sub>ij</sub>'' is the integration time per visibility.  
  
Note, when this equation is extended to a whole dataset rather than a single visibility, T<sub>sys,i</sub>T<sub>sys,j</sub> is replaced by the average ''T<sub>sys</sub><sup>2</sup>'' and the factors ''sqrt[N(N-1)]'' and ''n<sub>p</sub>'' are included in the denominator, where N is the number of antennas and ''n<sub>p</sub>'' is the number of polarizations (1 for single pol and 2 for dual pol data).
+
Note, this equation can be extended from a single visibility to the expected imaging rms noise for one channel in an entire dataset by replacing ''T<sub>sys,i</sub>T<sub>sys,j</sub>'' by the average ''T<sub>sys</sub><sup>2</sup>'', ''t<sub>ij</sub>'' by the total time on source, and the <math>\sqrt{2}</math> in the denominator by <math>\sqrt{N(N-1)}</math> , where ''N'' is the number of antennas. Additionally, ''<math>\sqrt{n_p}</math>'' is included in the denominator, where ''n<sub>p</sub>'' is the number of polarizations (1 for single pol and 2 for dual or full pol data):
 +
 
 +
<math>
 +
\sigma_{image} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}}
 +
</math>
 +
<math>
 +
\sqrt{\frac{T_{sys}^{2}}{N(N-1)\Delta\nu_{ch} t_{source}n_{p}}}
 +
</math>
 +
<math>
 +
\times 10^{26},
 +
</math>
 +
 
 +
'''In order to correctly combine and image data that have different ''T<sub>sys</sub>'', ''&Delta;&nu;<sub>ch</sub>'', ''t<sub>ij</sub>'', or antenna size it is essential to use visibility weights proportional to 1/&sigma;<sub>ij</sub><sup>2</sup>. The remainder of this guide deals with ensuring that the individual visibility weights are correct in your ALMA data.'''
  
'''In order to correctly combine and image data that have different T<sub>sys, &Delta;&nu;<sub>ch</sub>, t<sub>ij</sub>, or antenna size it is essential to use visibility weights proportional to 1/&sigma;<sub>ij</sub><sup>2</sup>. The remainder of this guide deals with ensuring that the individual visibility weights are correct in your ALMA data.'''
+
==== Relative Overall Weights between Configurations ====
  
Additionally, when combining data from different antenna configurations, one will get optimal overall sensitivity to all spatial scales by matching the surface brightness sensitivity at each uv-distance. This can only be achieved by having time-on-source per configuration in the right proportion. This topic is covered in ALMA Memo http://library.nrao.edu/public/memos/alma/memo598.pdf and informs the relative amount that ALMA observes 7m-array data versus 12m-array data, and compact versus extended 12m-array configurations. However, since telescope time is expensive, one typically does not actually observe in the optimal proportion, in that case one will not fully realize the expected "impact" of adding the less sensitive config data. In that case, one may chose to "upweight" the less sensitive config explicitly by changing its data weights, above and beyond 1/&sigma;<sub>ij</sub><sup>2</sup>. However, it should be noted that there are no free lunches, and such a change will come at the expense of overall image sensitivity though it may very well be the optimal choice for your science case. Finding the optimal up-weighting is usually a matter of experimentation, but can easily be explored using the visweightscale parameter in concat. As a general rule of thumb extra up-weighting by more than a factor of 2-3 is not recommended. Before messing with this however, its always best to start with data that simply has the correct 1/&sigma;<sub>ij</sub><sup>2</sup> weights.
+
Additionally, when combining data from different antenna configurations, one will get optimal overall sensitivity to all spatial scales by matching the surface brightness sensitivity at each uv-distance. This can only be achieved by having time-on-source per configuration in the right proportion. This topic is covered in [http://library.nrao.edu/public/memos/alma/memo598.pdf ALMA Memo 598]. This memo informs the relative amount of time that ALMA observes a project with the 7m-array versus the 12m-array, and compact versus extended 12m-array configurations. However, since telescope time is expensive, one typically does not actually observe in the optimal proportion, in that case one will not fully realize the expected "impact" of adding the less sensitive configuration data. In that case, one may chose to "up-weight" the less sensitive configuration explicitly by changing its data weights, above and beyond 1/&sigma;<sub>ij</sub><sup>2</sup>. However, it should be noted that nothing comes for free, and such a change will come at the expense of overall image sensitivity though it may very well be the optimal choice for your science case -- for example, if you are particularly interested in large scale structures you might apply extra up-weighting to the dataset(s) with the shortest baselines. Finding the optimal up-weighting is a matter of experimentation, but can easily be explored using the visweightscale parameter in {{concat}}. As a general rule of thumb extra up-weighting by more than a factor of 4 or so is not recommended. Before experimenting with this however, its always best to start with data that simply has the correct 1/&sigma;<sub>ij</sub><sup>2</sup> weights. Additionally, some degree of relative re-weighting can also be accomplished simply by using the Briggs Robust parameter, without explicitly changing the visibility weights, and generally it is also a good idea to try this first before more aggressive up-weighting is attempted.
  
 
== Weights in CASA ==
 
== Weights in CASA ==
  
A memo describing weights in CASA, in particular the significant changes that were made with CASA 4.2.2, can be found at http://casa.nrao.edu/Memos/CASA-data-weights.pdf
+
A memo describing weights in CASA, in particular the significant changes that were made with CASA 4.2.2, can be found at [http://casa.nrao.edu/Memos/CASA-data-weights.pdf CASA-data-weights.pdf]
  
To summarize the situation specifically as applied to ALMA data reduction in the ALMA archive and delivered to PIs:
+
To summarize the situation specifically for ALMA data processed by the ALMA Project:
  
:'''1) CASA 4.2.1 and earlier''': Weights were only scaled by 1/[(Tsys(i) * Tsys(j)] using calwt=True at the applycal stage for Tsys table. Assuming that (i) there aren't any unflagged antennas with significantly low gain due to for example pointing errors, or hardware problems, and (ii) all the antennas have similar surface errors -- usually good assumptions for ALMA data, data calibrated this way are to good approximation internally consistent and can produce good imaging results, but should not be combined with other data that have different &Delta;&nu;<sub>ch</sub>, t<sub>ij</sub>, or antenna size without further modification. An example of what you can do in this situation is given in XX.
+
:'''CASE 1) CASA 4.2.1 and earlier''': Weights were calculated on a per spw bases and scaled by ''N<sub>chan</sub>/[(Tsys(i) * Tsys(j)]'' using calwt=True at the applycal stage for Tsys table. Assuming that (i) there aren't any unflagged antennas with significantly low gain due to for example pointing errors, or hardware problems, and (ii) all the antennas have similar efficiencies -- usually good assumptions for ALMA data, then to good approximation the data weights are close to internally consistent and can produce good imaging results, but should not be combined with other data that have different &Delta;&nu;<sub>ch</sub>, t<sub>ij</sub>, or antenna size without further modification.  
:'''2) CASA 4.2.2 and later''': Upon import data weights are scaled by 2&Delta;&nu;<sub>ch</sub>&Delta;t<sub>ij</sub> and also scaled by 1/[(Tsys(i) * Tsys(j)] using calwt=True at the applycal stage for Tsys table. Additionally:
+
:'''CASE 2) CASA 4.2.2 and later''': Upon import data weights are scaled by 2&Delta;&nu;<sub>ch</sub>&Delta;t<sub>ij</sub> and then also scaled by ''1/[(Tsys(i) * Tsys(j)]'' using calwt=True at the applycal stage for Tsys table (thus making them per channel weights). Additionally:
::'''(A) For data calibrated by the 4.2.2 CASA Pipeline''' the weights are further modified by [gain(i)<sup>2</sup> * gain(j)<sup>2</sup>] when the amplitude gain table is applied using calwt=True. Since the amplitude gains are directly proportional to the effective Antenna area, scaling the weights by the amplitude gains will take into account antenna size differences, and also down-weight antennas with comparatively low gain. Thus, these weights are completely correct.
+
::'''(A) For data calibrated by the 4.2.2 CASA Pipeline''' the weights are further modified by [gain(i)<sup>2</sup> * gain(j)<sup>2</sup>] when the amplitude gain table is applied using calwt=True. Since the amplitude gains are directly proportional to the effective Antenna area, scaling the weights by the amplitude gains will take into account antenna size differences, and also down-weight antennas with comparatively low gain. Thus, these weights are correct (but see [https://casaguides.nrao.edu/index.php/DataWeightsAndCombination#Absolute_Accuracy_of_the_Data_Weights Absolute Accuracy of the Data Weights]).
 
::'''(B) For data manually calibrated in CASA 4.2.2''', unfortunately calwt=False was still used to apply the antenna gain table, thus, these data have weights that are not correct in a relative sense when compared to other data with different antenna size by the factor [gain(i)<sup>2</sup> * gain(j)<sup>2</sup>].
 
::'''(B) For data manually calibrated in CASA 4.2.2''', unfortunately calwt=False was still used to apply the antenna gain table, thus, these data have weights that are not correct in a relative sense when compared to other data with different antenna size by the factor [gain(i)<sup>2</sup> * gain(j)<sup>2</sup>].
:'''3) CASA 4.3 and later:''' Data calibrated in either the pipeline or manually will have completely correct weights. An example of this situation is demonstrated in https://casaguides.nrao.edu/index.php/M100_Band3_Combine_4.3
+
:'''CASE 3) CASA 4.3 and later:''' Data calibrated in either the pipeline or manually will have correct weights (but see [https://casaguides.nrao.edu/index.php/DataWeightsAndCombination#Absolute_Accuracy_of_the_Data_Weights Absolute Accuracy of the Data Weights]). An example of this situation is demonstrated in https://casaguides.nrao.edu/index.php/M100_Band3_Combine_4.3
  
== How Do I Know the Situation For My Data?==
+
== How Do I Know the Situation For My ALMA Data?==
  
* Data taken in Cycle 0 and Cycle 1 were reduced in 4.2.1 or earlier versions and correspond to Situation 1 above. ACA 7m-array data were first offered in Cycle 1. If you want to combine 12m-array and 7m-array data from Cycle 1 is very likely you need to correct weights before imaging.
+
First, remember that good imaging performance requires that the '''relative weights''' be correct for all data included in the imaging. For all of the situations below, the data are sufficiently internally consistent that they can be imaged fine on their own or with other data that share the same situation and have the same data properties. Problems arise when data are combined that (a) do not share the same weight situation and/or differences in antenna size, visibility integration time, or channel bandwidth are not accounted for.  
  
* The situation for Cycle 2 is more confused (this includes actual Cycle 2 projects and carry-over projects or parts of projects from Cycle 1).
+
'''All datasets to be combined should be independently assessed to determine their weight situation and thus whether corrective action is required'''
  
:Key dates:
+
* '''Cycle 0''' and '''Cycle 1''' data were reduced in 4.2.1 or earlier versions and correspond to Case 1 above. ACA 7m-array data were first offered in Cycle 1. If you want to combine 12m-array and 7m-array data taken during Cycle 1, it is very likely you need to correct the visibility weights before imaging.
::Start Cycle 2: June 1, 2014
+
 
::CASA 4.2.2 release date: Sept. 4, 2014
+
* '''Cycle 2''' data has a more confused range of possibilities (this includes actual Cycle 2 projects and carry-over projects or parts of projects from Cycle 1). Recall from Case 2 above, that the key factor that determines your situation for Cycle 2 is whether the data were pipeline or manually calibrated.
::Pipeline release date: Oct. 20, 2014
+
** Key dates:
: How to distinguish between pipeline and manual data reduction.
+
*** Start Cycle 2: June 1, 2014
:: In your data delivery, the README file should say but there were some oversights on this front.
+
*** CASA 4.2.2 release date: Sept. 4, 2014
:: A sure way to tell: look in the directory called '''script''' for a particular member_ouss. If you see a file with PPR*.xml, the data was calibrated by the pipeline. Otherwise it was done manually.
+
*** Pipeline release date: Oct. 20, 2014
 +
** The earlier in Cycle 2 your data was taken, the more likely it was manually calibrated, also if you have any very narrow spws, high frequency, etc it is more likely data were manually calibrated.
 +
** If the data were manually calibrated early in Cycle 2 it is likely in Case 1 (certainly the case for data earlier than Sept. 2014). After Oct. 1, 2014 it is likely to be in Case 2B.
 +
** How can I distinguish between pipeline and manual data reduction for Cycle 2 data?
 +
*** In your data delivery, the README file should say but there were some oversights on this front. A sure way to tell: look in the directory called '''script''' within your data delivery for a particular member_ouss. If you see a file with '''PPR*.xml''', the data was calibrated by the pipeline. Otherwise it was done manually.
  
 
== What Are the Options for Adjusting the Weights for Older Reductions?==
 
== What Are the Options for Adjusting the Weights for Older Reductions?==
  
If the data weights are not correct in the data you want to combine there are three options to correct the situation. These different methods carry different levels of pain/complexity depending on your situation. For example, for data manually calibrated in 4.2.2, Option 1 is pretty easy, but harder for older data/ versions of CASA. The situation can also be extra confusing if your data fall into multiple categories above. For example, it is not uncommon that in Cycle 2 the 12m-array data could be pipeline calibrated, but the 7m-array data done manually.  
+
If the data weights are not correct in ALL the datasets you want to combine there are three options to correct the situation. These different methods carry different levels of pain/complexity depending on your situation. The situation can also be extra confusing if your data fall into multiple categories above. For example, it is not uncommon that in Cycle 2 the 12m-array data could be pipeline calibrated, but the 7m-array data done manually. Pay close attention to recommendations in <span style="color:#B22222">Red</span> and the Caveats for each option.
  
 
====Option 1: Re-calibrate your data in CASA 4.2.2 or later====  
 
====Option 1: Re-calibrate your data in CASA 4.2.2 or later====  
  
:<span style="color:#B22222"> &lArr; This option is easiest to implement for Case 2B (Data observed in Cycle 2 manually calibrated in 4.2.2 with calwt=False for amplitude/flux gain table. </span>
+
:<span style="color:#B22222"> &rArr; Option 1 is easiest to implement for Case 2B (Data observed in Cycle 2 manually calibrated in 4.2.2 or 4.3.1 with calwt=False for amplitude/flux gain table). </span>
 +
 
 +
* All data imported in 4.2.2 (or later) will automatically adjust the weights by 2&Delta;&nu;<sub>ch</sub>&Delta;t<sub>ij</sub>. The Tsys application should also already be correct in your scripts. To correct for antenna size and weight by the gains you must change calwt=True for the amplitude table {{applycal}}.
 +
 
 +
* Where do I make the calwt change?
 +
** As described in your README file, you can obtain a fully calibrated measurement set by executing the '''scriptForPI.py'''. For manually calibrated data, within the '''script''' directory you will also find scripts with the format uid*scriptForCalibration.py, one for each execution for that Scheduling Block -- the sciptForPI.py actually executes these scripts, thus this is where any change must be made.
 +
** Toward the bottom of the uid*scriptForCalibration.py scripts (typically Step 17 or 18) you will find a step called '''# Application of the bandpass and gain cal tables'''. Change the calwt=False in all of the {{applycal}} calls to calwt=True.
 +
 
 +
:'''Caveat 1:''' You must have the raw ALMA ASDM to run the '''scriptForPI.py''' (see the README file for more information).
 +
 
 +
:'''Caveat 2:''' Most (all except early Cycle 0) ALMA manual calibration scripts have within them the CASA version used to create the script. For example, if your manual calibration script says the following, using any other version than 4.2.2 will give an error (even though using for example 4.3.1 would be fine in this case):
  
Importing your data in 4.2.2 (or later) will automatically adjust the weights by 2&Delta;&nu;<sub>ch</sub>&Delta;t<sub>ij</sub>. The Tsys application should already be correct in your scripts. Be sure however, to change calwt=True for the amplitude table applycal.  
+
if re.search('^4.2.2', casadef.casa_version) == None:
 +
sys.exit('ERROR: PLEASE USE THE SAME VERSION OF CASA THAT YOU USED FOR GENERATING THE SCRIPT: 4.2.2')
  
::'''Caveat 1:''' You must have the raw ALMA ASDM to correctly start over, unless you are in the CASA 4.2.2 manual calibration case. In that case you can simply change calwt=True for the amplitude table applycal and re-run.  
+
::You must change the version number in the script to match the version you want to use or the script will not run.  
  
::'''Caveat 2:''' Most (all except early Cycle 0) ALMA manual calibration scripts have within them the CASA version used to create the script. For example, if your script says the following, using any other version than 4.3.1 will give an error:
+
:'''Caveat 3:''' Scripts from earlier than 4.2.1 are likely to have commands that make them incompatible to run directly in later versions of CASA. It may be difficult for a non-expert to update the uid*scriptForCalibration.py script(s) to current syntax.
  
if re.search('^4.3.1', casadef.casa_version) == None:
+
====Option 2: Make an approximate overall relative correction to your calibrated science target data:====
sys.exit('ERROR: PLEASE USE THE SAME VERSION OF CASA THAT YOU USED FOR GENERATING THE SCRIPT: 4.3.1')
 
  
::You must change the version number to match the version you want to use or the script will not run.
+
:<span style="color:#B22222"> &rArr; Option 2 is easy to implement for Cases 1 and 2B, but is not as accurate as Options 1 or 3. However, it is typically adequate for most purposes, and is MUCH better than doing nothing. </span>
  
::'''Caveat 3:''' Scripts from earlier than 4.2.1 are likely to have commands that make them incompatible to run directly in later versions of CASA. It may be difficult for a non-expert to update the script to current syntax.
+
* All ALMA data reduced in CASA should already have 1/Tsys<sup>2</sup> weighting which accounts for many of the factors that make data from some baselines more sensitive than others within an individual dataset. If we assume that antennas with abnormally low gains (typically due to poor pointing, hardware failures) have already been flagged and all antennas meet the surface accuracy specifications, then the amplitude gains will be fairly constant from antenna to antenna, with the value dominated by the antenna size. Other parameters that can often be different between datasets include the &Delta;&nu;<sub>ch</sub>, and t<sub>ij</sub>.
 +
* Thus, a reasonably good approximation for the relative weight scaling for data_a compared to data_b is:
 +
** Case 1: (antenna_size_b/antenna_size_a)<sup>4</sup> x (&Delta;&nu;<sub>ch</sub>_b/&Delta;&nu;<sub>ch</sub>_a) x (t<sub>ij</sub>_b/t<sub>ij</sub>_a)
 +
** Case 2B: (antenna_size_b/antenna_size_a)<sup>4</sup>
 +
* such an overall scale factor can be applied during the {{concat}} step using the visweightscale parameter. Note that because its a ratio, it doesn't matter if you use the antenna radius or diameter as long as you are consistent. An example is given below.
  
====Option 2: Run the task statwt on your calibrated science target data====
+
:'''Caveat 1:''' This option requires that the datasets to be combined have only one (unflagged) antenna size per dataset. If this is not the case, then Options 1 or 3 must be used.
  
This task attempts to assess the sensitivity per visibility and adjust the weights accordingly. It is very commonly used for JVLA data (including their pipeline).  
+
:'''Caveat 2:''' This option also requires that both datasets to be combined are in the same weight situation (i.e. mixed Cases cannot be handled using simple overall relative scaling).
  
::'''Caveat 1:''' One must limit the calculation to line-free channels. For complex line projects this can be painful, however, typically the line-free channels are already known from the continuum subtraction, and can be reused here. However, it is best run statwt before continuum subtraction.
+
====Option 3: Run the task {{statwt}} on your calibrated science target data====
  
====Option 3: Make an approximate overall correction:====
+
:<span style="color:#B22222"> &rArr; Option 3 is pretty easy to implement for both Cases 1 and 2B, and will account for (likely small) Gain variations from antenna to antenna within the individual datasets, whereas Option 2 does not. However, see Caveat 1, for very complex line emission it may be difficult to get a good result. </span>
  
Using the task listobs, it is easy to check whether the datasets you wish to combine have different channel widths or visibility integration times (&Delta;&nu;<sub>ch</sub> or t<sub>ij</sub>).
+
* This task attempts to assess the sensitivity per visibility and adjust the weights accordingly. It is very commonly used for JVLA data (including their pipeline).  
  
== Example of Correcting Weights from CASA 4.2.1 Calibrated Data Using Option 3==
+
:'''Caveat 1:''' One must limit the calculation to line-free channels using the fitspw parameter. For complex line projects this can be painful, however, typically the line-free channels are already known from the continuum subtraction, and can be reused here. However, it is best to run statwt before continuum subtraction, and it should always be run on the fully channelized data before any spectral averaging is applied.
 +
:'''Caveat 2:''' {{statwt}} does not currently account for channel-dependent flagging. It is critical to explicitly avoid the edge channels for any TDM spws in the data (128 channels per spw) data using fitspw, (example: for a dataset with only TDM spws, fitspw='*,7~120').
 +
 
 +
== Example of Correcting Weights Using Option 2 (Relative Scaling)==
  
 
<figure id="12m_WT.png">
 
<figure id="12m_WT.png">
[[File:12m_WT.png|200px|thumb|right|<caption> 12m weights.</caption>]]
+
[[File:12m_WT_4.2.1.png|200px|thumb|right|<caption> M100 12m-array weights from 4.2.1 reduction.</caption>]]
 
</figure>
 
</figure>
  
 
<figure id="7m_WT_old.png">
 
<figure id="7m_WT_old.png">
[[File:7m_WT_old.png|200px|thumb|right|<caption> 7m weights.</caption>]]
+
[[File:7m_WT_4.2.1.png|200px|thumb|right|<caption> M100 7m-array weights from 4.2.1 reduction.</caption>]]
 
</figure>
 
</figure>
  
Below we show what could have been done to correct the M100 SV data if it was reduced in CASA 4.2.1 (rather than 4.3 as demonstrated in https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3).
+
Below we show what could have been done to approximately correct the weights in the M100 SV 7m-array and 12m-array data if it was reduced in CASA 4.2.1 (or earlier), i.e. Case 1 (For a complete treatment of the data processed in CASA 4.3 see https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3).
  
In CASA 4.2.1 and earlier, the data weights are 1 upon import, later in the standard calibration procedure, applycal scales the weights by 1/[(Tsys(i) * Tsys(j)] if calwt=True for the Tsys table applycal. As an example, we plot the  
+
In CASA 4.2.1 and earlier, the data weights are 1 upon import, later in the standard calibration procedure, applycal scales the weights by ''N<sub>chan</sub>/[(Tsys(i) * Tsys(j)]'' because calwt=True for the Tsys table applycal. As an example, we plot the weights of 7m and 12m data imported in CASA 4.2.1. No averaging can be turned on  
weights of 7m and 12m data imported in CASA 4.2.1. No averaging can be turned on  
 
 
when plotting the weights.  
 
when plotting the weights.  
  
Line 120: Line 152:
 
</source>
 
</source>
  
As you can see from these plots, the weights are quite similar at this  
+
The Tsys for the 12m data for the spw containing CO(1-0) ranges from about 100 to 150 K, and the number
 +
of channels per spw is 3840, yielding an expected range for the 4.2.1 weights of 0.38 to 0.17 in good
 +
agreement with Fig. 1.
 +
 
 +
As you can see from Figures 1 and 2, the weights are quite similar at this  
 
stage because the data were taken under similar weather conditions and  
 
stage because the data were taken under similar weather conditions and  
hence Tsys, and this is the only thing the weights have been scaled by so far.  
+
hence Tsys, and this is the only thing the weights have been scaled by so far besides the
 +
number of channels per spw which is also the same.  
  
 
Recall that the rms noise in a single channel for a single visibility is:
 
Recall that the rms noise in a single channel for a single visibility is:
  
 
<math>  
 
<math>  
\sigma_{ij}=\frac{2k}{A_{eff}}  
+
\sigma_{ij} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}}  
 
</math>
 
</math>
 
<math>  
 
<math>  
\sqrt{\frac{T_{sys,i} T_{sys,j}}{\Delta\nu_{ch} t_{ij}}}
+
\sqrt{\frac{T_{sys,i} T_{sys,j}}{2\Delta\nu_{ch} t_{ij}}}
 +
</math>
 +
<math>
 +
\times 10^{26},
 
</math>
 
</math>
  
<figure id="Intcombo_0.193_WT.png">
+
Beyond the obvious difference in the antenna dish sizes (''A<sub>eff</sub>''), looking at the listobs output for these two datasets the same channel width was used but that the integration time per visibility is 10.1 sec for the 7m-array and 6.05 sec for the 12m-array. Since dish area is in the denominator of  
[[File:Intcombo_0.193_WT.png|200px|thumb|right|<caption> 7m and 12m weights after scaling by relative sensitivity.</caption>]]
 
</figure>
 
 
 
Beyond the obvious difference in the antenna dish sizes, looking at the listobs output for these two datasets, we see that the same channel width was used but that the integration time per visibility is 10.1 sec for the 7m-array and 6.05 sec for the 12m-array. Since dish area is in the numerator of  
 
 
the radiometer equation and integration time per visibility is in the  
 
the radiometer equation and integration time per visibility is in the  
denominator, and assuming WT propto 1/sigma<sup>2</sup>,  
+
denominator, and assuming WT &prop; 1/&sigma;<sub>i</sub><sub>j</sub><sup>2</sup>,  
 
the '''7m weight should be scaled by: (7./12.)<sup>4</sup> x (10.1/6.05) = 0.193'''  
 
the '''7m weight should be scaled by: (7./12.)<sup>4</sup> x (10.1/6.05) = 0.193'''  
 
to account for the difference in telescope size and integration time
 
to account for the difference in telescope size and integration time
 
per visibility.
 
per visibility.
  
An easy way to perform this overall scaling is using the visweightscale parameter in concat.
+
12m data:
 +
<pre style="background-color: #fffacd;">
 +
 
 +
ObservationID = 0        ArrayID = 0
 +
  Date        Timerange (UTC)          Scan  FldId FieldName            nRows    SpwIds  Average Interval(s)    ScanIntent
 +
  10-Aug-2011/19:38:05.8 - 19:50:22.8    11      1 M100                      1500  [0,1,2,3]  [6.05, 6.05, 6.05, 6.05] [CALIBRATE_WVR#ON_SOURCE,OBSERVE_TARGET#ON_SOURCE]
 +
 
 +
Spectral Windows:  (4 unique spectral windows and 1 unique polarization setups)
 +
  SpwID  Name  #Chans  Frame  Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz) BBC Num  Corrs
 +
  0              3840  TOPO  113726.419      488.281  1875000.0 114663.6750        1  XX  YY
 +
  1              3840  TOPO  111851.419      488.281  1875000.0 112788.6750        2  XX  YY
 +
  2              3840  TOPO  103663.431      -488.281  1875000.0 102726.1750        3  XX  YY
 +
  3              3840  TOPO  101850.931      -488.281  1875000.0 100913.6750        4  XX  YY
 +
</pre>
 +
 
 +
 
 +
7m data:
 +
<pre style="background-color: #fffacd;">
 +
 
 +
ObservationID = 0        ArrayID = 0
 +
  Date        Timerange (UTC)          Scan  FldId FieldName            nRows    SpwIds  Average Interval(s)    ScanIntent
 +
  17-Mar-2013/04:44:04.3 - 04:50:43.7    11      1 M100                      261  [0,1,2,3]  [10.1, 10.1, 10.1, 10.1] [OBSERVE_TARGET#ON_SOURCE]
 +
 
 +
Spectral Windows:  (6 unique spectral windows and 1 unique polarization setups)
 +
  SpwID  Name                          #Chans  Frame  Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz) BBC Num  Corrs
 +
  0      ALMA_RB_03#BB_1#SW-01#FULL_RES  4080  TOPO  101945.850      -488.281  1992187.5 100950.0000        1  XX  YY
 +
  1      ALMA_RB_03#BB_2#SW-01#FULL_RES  4080  TOPO  103761.000      -488.281  1992187.5 102765.1500        2  XX  YY
 +
  2      ALMA_RB_03#BB_3#SW-01#FULL_RES  4080  TOPO  111811.300      488.281  1992187.5 112807.1500        3  XX  YY
 +
  3      ALMA_RB_03#BB_4#SW-01#FULL_RES  4080  TOPO  113686.300      488.281  1992187.5 114682.1500        4  XX  YY
 +
  4      ALMA_RB_03#BB_1#SW-01#FULL_RES  4080  TOPO  111798.250      488.281  1992187.5 112794.1000        1  XX  YY
 +
  5      ALMA_RB_03#BB_2#SW-01#FULL_RES  4080  TOPO  113673.250      488.281  1992187.5 114669.1000        2  XX  YY
 +
</pre>
 +
 
 +
<figure id="Intcombo_0.193_WT.png">
 +
[[File:Intcombo_0.193_WT.png|200px|thumb|right|<caption> M100 7m-array (red) and 12m-array (black) weights after scaling by relative sensitivity. </caption>]]
 +
</figure>
 +
 
 +
''NOTE: If these data had been manually calibrated in CASA 4.2.2 with calwt=False for the gain table applycal (i.e. Case 2B), then the overall scale factor to apply would be (7./12.)<sup>4</sup> = 0.116, because the difference in visibility integration time would already be accounted for.''
 +
 
 +
An easy way to perform this overall scaling is using the visweightscale parameter in {{concat}}.
  
 
<source lang="python">
 
<source lang="python">
Line 164: Line 239:
 
</source>
 
</source>
  
These combined data with correct relative weights are now ready for imaging.
+
These combined data with correct relative weights are now ready for imaging. See https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3 for more on the imaging in CASA 4.3.
 +
 
 +
Note that the per spw 12m-array 4.2.1 derived weights are quite similar to those shown in [https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3 M100_Band3_Combine_4.3] from CASA 4.3 (Figure 3) using the correct per channel weights. This is purely accidental... for this example 1/&sigma;<sub>ij</sub><sup>2</sup> just happens to be approximately equal to  ''N<sub>chan</sub>/[(Tsys(i) * Tsys(j)]'' for an antenna with diamter=12m, &Delta;&nu;<sub>ch</sub>=488.281 kHz, t<sub>ij</sub>=6.048s, and 3840 channels.
 +
 
 +
== Example of Correcting Weights Using Option 3 (statwt)==
 +
 
 +
<figure id="12m_calwtF_WT.png">
 +
[[File:12m_calwtF_WT.png|200px|thumb|right|<caption> M100 12m-array WT vs uvdist after calwt=False for amplitude gains. </caption>]]
 +
</figure>
 +
 
 +
<figure id="7m_calwtF_WT.png">
 +
[[File:7m_calwtF_WT.png|200px|thumb|right|<caption> M100 7m-array WT vs uvdist after calwt=False for amplitude gains.</caption>]]
 +
</figure>
 +
 
 +
<figure id="12m_amp_ch_Spw0.png">
 +
[[File:12m_amp_ch_Spw0.png|200px|thumb|right|<caption> M100 12m-array amplitude vs channel. </caption>]]
 +
</figure>
 +
 
 +
<figure id="7m_amp_ch.png">
 +
[[File:7m_amp_ch.png|200px|thumb|right|<caption> M100 7m-array amplitude vs channel. </caption>]]
 +
</figure>
 +
 
 +
This is an example of running {{statwt}} on the M100 SV data. The starting point was data calibrated as if it had been manually calibrated in CASA 4.2.2, i.e. CASE 2b calwt=False for the gains. Note for this starting point, the 7m data actually have ''higher'' weight than the 12m data. This is because the average Tsys were similar but the visibility integration time of the 7m data is 10.1 versus 6.048s for the 12m data (see {{listobs}} outputs in previous section). Imaging these data would work very poorly because as already described, the 7m should actually have weights &approx; 0.193 times lower than the 12m data.
 +
 
 +
<source lang="python">
 +
os.system('rm -rf 12m_calwtF_WT.png')
 +
plotms(vis='M100_Band3_12m_CalibratedData.cf.ms.fresh', spw='0',yaxis='weight',xaxis='uvdist',
 +
      plotfile='12m_calwtF_WT.png')
 +
 
 +
 
 +
os.system('rm -rf 7m_calwtF_WT.png')
 +
plotms(vis='M100_Band3_7m_CalibratedData.cf.ms.fresh', spw='3,5',yaxis='weight',xaxis='uvdist',
 +
      plotfile='7m_calwtF_WT.png')
 +
</source>
 +
 
 +
As mentioned above in the Option 3 {{statwt}} caveats, it is essential to avoid strong line channels when calculating
 +
the statistics. What is meant by strong? At minimum, if you can see it in a uvplot it should not be included. The easiest way to check is to make a plot of the uv-data in channel space.
 +
 
 +
<source lang="python">
 +
# In CASA
 +
os.system('rm -rf 12m_amp_ch_Spw0.png')
 +
plotms(vis='M100_Band3_12m_CalibratedData.cf.ms',
 +
      yaxis='amp',xaxis='channel',spw='0',avgtime='1e8',
 +
      avgscan=True,plotfile='12m_amp_ch_Spw0.png')
 +
 
 +
 
 +
os.system('rm -rf 7m_amp_ch_Spw3_5.png')
 +
plotms(vis='M100_Band3_7m_CalibratedData.cf.ms',
 +
      yaxis='amp',xaxis='channel',spw='5',avgtime='1e8',
 +
      avgscan=True,plotfile='7m_amp_ch_Spw5.png')
 +
</source>
 +
 
 +
In statwt we need to avoid the line emission, while sampling the full spectral range. This is especially important in this example because the CO(1-0) transition is close to an oxygen line in the atmosphere -- this is what causes the noise to increase at higher channel numbers. Since we are deriving a spectrally averaged weight it would be unrealistic to only include the lower, less noisy channels.
 +
 
 +
<source lang="python">
 +
# In CASA
 +
statwt(vis='M100_Band3_12m_CalibratedData.cf.ms', spw='0',
 +
      fitspw='0:0~1800;2500~3839', datacolumn='data')
 +
 
 +
# In CASA
 +
statwt(vis='M100_Band3_7m_CalibratedData.cf.ms', spw='3,5',
 +
      fitspw='3:0~1900;2500~3839,5:0~1900;2500~3839', datacolumn='data')
 +
</source>
 +
 
 +
Now we can concat these datasets and make a plot to verify that the weights have the values expect.
 +
 
 +
<figure id="M100_statwt.png">
 +
[[File:M100_statwt.png|200px|thumb|right|<caption> M100 7m-array (purple) and 12m-array (black) weights after running statwt. </caption>]]
 +
</figure>
 +
 
 +
<source lang="python">
 +
concat(vis=['M100_Band3_12m_CalibratedData.cf.ms','M100_Band3_7m_CalibratedData.cf.ms'],
 +
      concatvis='M100_Band3_statwt.ms')
 +
 
 +
os.system('rm -rf M100_statwt.png')
 +
plotms(vis='M100_Band3_statwt.ms', spw='0,7,9',yaxis='weight',xaxis='uvdist',
 +
      coloraxis='spw',plotfile='M100_statwt.png')
 +
</source>
 +
These weights are in good agreement with those derived using the correct weights throughout CASA, i.e.
 +
Figure 3 in [https://casaguides.nrao.edu/index.php/M100_Band3_Combine_4.3 M100_Band3_Combine_4.3]. However, see the next section for caveats.
 +
 
 +
== Absolute Accuracy of the Data Weights ==
 +
 
 +
As mentioned several times in this guide, good imaging performance is only subject to correct relative weights, which is achieved as of CASA >4.2.2 provided calwt=True for the amplitude gain table. Even for Case 1 (4.2.1 and lower), the relative weights within a given dataset are reasonably accurate in most cases due to the 1/Tsys<sup>2</sup> scaling. However, there are some instances, for example, uv-model fitting where one would like to use the visibility weights as an absolute measure of 1/&sigma;<sup>2</sup>. Here we discuss the absolute accuracy of the visibility weights for options 1 and 3:
 +
 
 +
:'''Option 1:''' All PI data taken so far with ALMA are Hanning smoothed online, but it is up to the user at proposal creation time to decide if additional channel averaging is done in the correlator (typically to reduce the data rate). Data that are Hanning smoothed but not channel averaged by the ALMA online system have a '''theoretical effective channel width 2.667 times that reported by for example {{listobs}}''' from the point of view of calculating the sensitivity of a single channel (it is a factor of 2.0 for the FWHM, i.e. channel resolution). The CASA initialization of 2&Delta;&nu;<sub>ch</sub>&Delta;t<sub>ij</sub> does not currently take this into account -- CASA does not have a way of knowing presently what the online system did, only how the data appears on disk, though this will change with future versions of the online software. Thus, presently CASA weights for this case are in principle a factor of 2.667 times smaller than they should be from a sensitivity point of view. For online channel averaging of two or more channels, the difference between the channel width and the effective channel width diminishes rapidly (2x channel averaging has a factor 1.6, 4x a factor 1.2, 8x a factor of 1.1). Additionally, the current weights do not include the correlator and quantization efficiencies ''&eta;<sub>q</sub>'' and ''&eta;<sub>c</sub>''. These have values of about 0.88 and 0.96, respectively.
 +
 
 +
:<span style="color:#B22222"> &rArr; Starting in Cycle 3, the effective channel width will be recorded in the raw data and used by CASA to initialize the weights for CASA versions >=4.2.2.</span>
 +
 
 +
:'''Option 3:''' For CASA <=4.4 {{statwt}} does not take channel-specific flagging into account, only flags that apply to all channels. Thus, the user must avoid flagged channels with the fitspw parameter. Additionally, strong line emission must be avoided. Provided these conditions are met, our tests suggest that the weights produced by {{statwt}} are too low by a factor of 2 compared to the expected values. Both of these {{statwt}} issues will be fixed as of CASA 4.5.
 +
 
 +
As a consequence, for online Hanning smoothed data with no online channel averaging, the average value of the weights resulting from Options 1 and 3 will appear to approximately match (see M100 examples), but for erroneous reasons.
 +
 
 +
Our investigation into these matters continues so if the absolute weights are important to you, check back here or send a helpdesk ticket with your inquiry.

Latest revision as of 09:05, 31 March 2017

The purpose of this guide is to describe the CASA visibility weights, and how they relate to data combination. If you are trying to combine ALMA data from Cycles 0, 1, and/or 2, you should be aware of the issues described here. Only data calibrated in >=CASA 4.2.2 pipeline, or manual reduction in >=CASA 4.3 are free from any issues. All of the data to be combined must be in a consistent weight state.

Principles of Data Weighting

Visibility Weights

For optimal imaging performance it is critical that in a relative sense each visibility in the data have the correct weight after calibration -- that is, data with better sensitivity have more weight than data with less sensitivity. Formally, the post-calibration visibility weights should be equal to 1/σij2 where σij is the rms noise of a given visibility.

[math] \sigma_{ij} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}} [/math] [math] \sqrt{\frac{T_{sys,i} T_{sys,j}}{2\Delta\nu_{ch} t_{ij}}} [/math] [math] \times 10^{26}, [/math]

where:

k is Boltzmann's constant.

Aeff is the effective antenna area which is equal to the aperture efficiency (ηa) x the geometric area of the antenna (π r2). The aperture efficiency depends on the rms antenna surface accuracy and is about 0.75 for ALMA dishes.

ηq and ηc are the quantization and correlator efficiencies, respectively. These have values of about 0.88 and 0.96, respectively. See the ALMA Technical Handbook for more information.

Tsys,i is the system temperature for antenna i, and Tsys,j is the system temperature for antenna j

Δνch is the effective channel frequency width.

tij is the integration time per visibility.

Note, this equation can be extended from a single visibility to the expected imaging rms noise for one channel in an entire dataset by replacing Tsys,iTsys,j by the average Tsys2, tij by the total time on source, and the [math]\sqrt{2}[/math] in the denominator by [math]\sqrt{N(N-1)}[/math] , where N is the number of antennas. Additionally, [math]\sqrt{n_p}[/math] is included in the denominator, where np is the number of polarizations (1 for single pol and 2 for dual or full pol data):

[math] \sigma_{image} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}} [/math] [math] \sqrt{\frac{T_{sys}^{2}}{N(N-1)\Delta\nu_{ch} t_{source}n_{p}}} [/math] [math] \times 10^{26}, [/math]

In order to correctly combine and image data that have different Tsys, Δνch, tij, or antenna size it is essential to use visibility weights proportional to 1/σij2. The remainder of this guide deals with ensuring that the individual visibility weights are correct in your ALMA data.

Relative Overall Weights between Configurations

Additionally, when combining data from different antenna configurations, one will get optimal overall sensitivity to all spatial scales by matching the surface brightness sensitivity at each uv-distance. This can only be achieved by having time-on-source per configuration in the right proportion. This topic is covered in ALMA Memo 598. This memo informs the relative amount of time that ALMA observes a project with the 7m-array versus the 12m-array, and compact versus extended 12m-array configurations. However, since telescope time is expensive, one typically does not actually observe in the optimal proportion, in that case one will not fully realize the expected "impact" of adding the less sensitive configuration data. In that case, one may chose to "up-weight" the less sensitive configuration explicitly by changing its data weights, above and beyond 1/σij2. However, it should be noted that nothing comes for free, and such a change will come at the expense of overall image sensitivity though it may very well be the optimal choice for your science case -- for example, if you are particularly interested in large scale structures you might apply extra up-weighting to the dataset(s) with the shortest baselines. Finding the optimal up-weighting is a matter of experimentation, but can easily be explored using the visweightscale parameter in concat. As a general rule of thumb extra up-weighting by more than a factor of 4 or so is not recommended. Before experimenting with this however, its always best to start with data that simply has the correct 1/σij2 weights. Additionally, some degree of relative re-weighting can also be accomplished simply by using the Briggs Robust parameter, without explicitly changing the visibility weights, and generally it is also a good idea to try this first before more aggressive up-weighting is attempted.

Weights in CASA

A memo describing weights in CASA, in particular the significant changes that were made with CASA 4.2.2, can be found at CASA-data-weights.pdf

To summarize the situation specifically for ALMA data processed by the ALMA Project:

CASE 1) CASA 4.2.1 and earlier: Weights were calculated on a per spw bases and scaled by Nchan/[(Tsys(i) * Tsys(j)] using calwt=True at the applycal stage for Tsys table. Assuming that (i) there aren't any unflagged antennas with significantly low gain due to for example pointing errors, or hardware problems, and (ii) all the antennas have similar efficiencies -- usually good assumptions for ALMA data, then to good approximation the data weights are close to internally consistent and can produce good imaging results, but should not be combined with other data that have different Δνch, tij, or antenna size without further modification.
CASE 2) CASA 4.2.2 and later: Upon import data weights are scaled by 2ΔνchΔtij and then also scaled by 1/[(Tsys(i) * Tsys(j)] using calwt=True at the applycal stage for Tsys table (thus making them per channel weights). Additionally:
(A) For data calibrated by the 4.2.2 CASA Pipeline the weights are further modified by [gain(i)2 * gain(j)2] when the amplitude gain table is applied using calwt=True. Since the amplitude gains are directly proportional to the effective Antenna area, scaling the weights by the amplitude gains will take into account antenna size differences, and also down-weight antennas with comparatively low gain. Thus, these weights are correct (but see Absolute Accuracy of the Data Weights).
(B) For data manually calibrated in CASA 4.2.2, unfortunately calwt=False was still used to apply the antenna gain table, thus, these data have weights that are not correct in a relative sense when compared to other data with different antenna size by the factor [gain(i)2 * gain(j)2].
CASE 3) CASA 4.3 and later: Data calibrated in either the pipeline or manually will have correct weights (but see Absolute Accuracy of the Data Weights). An example of this situation is demonstrated in https://casaguides.nrao.edu/index.php/M100_Band3_Combine_4.3

How Do I Know the Situation For My ALMA Data?

First, remember that good imaging performance requires that the relative weights be correct for all data included in the imaging. For all of the situations below, the data are sufficiently internally consistent that they can be imaged fine on their own or with other data that share the same situation and have the same data properties. Problems arise when data are combined that (a) do not share the same weight situation and/or differences in antenna size, visibility integration time, or channel bandwidth are not accounted for.

All datasets to be combined should be independently assessed to determine their weight situation and thus whether corrective action is required

  • Cycle 0 and Cycle 1 data were reduced in 4.2.1 or earlier versions and correspond to Case 1 above. ACA 7m-array data were first offered in Cycle 1. If you want to combine 12m-array and 7m-array data taken during Cycle 1, it is very likely you need to correct the visibility weights before imaging.
  • Cycle 2 data has a more confused range of possibilities (this includes actual Cycle 2 projects and carry-over projects or parts of projects from Cycle 1). Recall from Case 2 above, that the key factor that determines your situation for Cycle 2 is whether the data were pipeline or manually calibrated.
    • Key dates:
      • Start Cycle 2: June 1, 2014
      • CASA 4.2.2 release date: Sept. 4, 2014
      • Pipeline release date: Oct. 20, 2014
    • The earlier in Cycle 2 your data was taken, the more likely it was manually calibrated, also if you have any very narrow spws, high frequency, etc it is more likely data were manually calibrated.
    • If the data were manually calibrated early in Cycle 2 it is likely in Case 1 (certainly the case for data earlier than Sept. 2014). After Oct. 1, 2014 it is likely to be in Case 2B.
    • How can I distinguish between pipeline and manual data reduction for Cycle 2 data?
      • In your data delivery, the README file should say but there were some oversights on this front. A sure way to tell: look in the directory called script within your data delivery for a particular member_ouss. If you see a file with PPR*.xml, the data was calibrated by the pipeline. Otherwise it was done manually.

What Are the Options for Adjusting the Weights for Older Reductions?

If the data weights are not correct in ALL the datasets you want to combine there are three options to correct the situation. These different methods carry different levels of pain/complexity depending on your situation. The situation can also be extra confusing if your data fall into multiple categories above. For example, it is not uncommon that in Cycle 2 the 12m-array data could be pipeline calibrated, but the 7m-array data done manually. Pay close attention to recommendations in Red and the Caveats for each option.

Option 1: Re-calibrate your data in CASA 4.2.2 or later

⇒ Option 1 is easiest to implement for Case 2B (Data observed in Cycle 2 manually calibrated in 4.2.2 or 4.3.1 with calwt=False for amplitude/flux gain table).
  • All data imported in 4.2.2 (or later) will automatically adjust the weights by 2ΔνchΔtij. The Tsys application should also already be correct in your scripts. To correct for antenna size and weight by the gains you must change calwt=True for the amplitude table applycal.
  • Where do I make the calwt change?
    • As described in your README file, you can obtain a fully calibrated measurement set by executing the scriptForPI.py. For manually calibrated data, within the script directory you will also find scripts with the format uid*scriptForCalibration.py, one for each execution for that Scheduling Block -- the sciptForPI.py actually executes these scripts, thus this is where any change must be made.
    • Toward the bottom of the uid*scriptForCalibration.py scripts (typically Step 17 or 18) you will find a step called # Application of the bandpass and gain cal tables. Change the calwt=False in all of the applycal calls to calwt=True.
Caveat 1: You must have the raw ALMA ASDM to run the scriptForPI.py (see the README file for more information).
Caveat 2: Most (all except early Cycle 0) ALMA manual calibration scripts have within them the CASA version used to create the script. For example, if your manual calibration script says the following, using any other version than 4.2.2 will give an error (even though using for example 4.3.1 would be fine in this case):
if re.search('^4.2.2', casadef.casa_version) == None:
sys.exit('ERROR: PLEASE USE THE SAME VERSION OF CASA THAT YOU USED FOR GENERATING THE SCRIPT: 4.2.2')
You must change the version number in the script to match the version you want to use or the script will not run.
Caveat 3: Scripts from earlier than 4.2.1 are likely to have commands that make them incompatible to run directly in later versions of CASA. It may be difficult for a non-expert to update the uid*scriptForCalibration.py script(s) to current syntax.

Option 2: Make an approximate overall relative correction to your calibrated science target data:

⇒ Option 2 is easy to implement for Cases 1 and 2B, but is not as accurate as Options 1 or 3. However, it is typically adequate for most purposes, and is MUCH better than doing nothing.
  • All ALMA data reduced in CASA should already have 1/Tsys2 weighting which accounts for many of the factors that make data from some baselines more sensitive than others within an individual dataset. If we assume that antennas with abnormally low gains (typically due to poor pointing, hardware failures) have already been flagged and all antennas meet the surface accuracy specifications, then the amplitude gains will be fairly constant from antenna to antenna, with the value dominated by the antenna size. Other parameters that can often be different between datasets include the Δνch, and tij.
  • Thus, a reasonably good approximation for the relative weight scaling for data_a compared to data_b is:
    • Case 1: (antenna_size_b/antenna_size_a)4 x (Δνch_b/Δνch_a) x (tij_b/tij_a)
    • Case 2B: (antenna_size_b/antenna_size_a)4
  • such an overall scale factor can be applied during the concat step using the visweightscale parameter. Note that because its a ratio, it doesn't matter if you use the antenna radius or diameter as long as you are consistent. An example is given below.
Caveat 1: This option requires that the datasets to be combined have only one (unflagged) antenna size per dataset. If this is not the case, then Options 1 or 3 must be used.
Caveat 2: This option also requires that both datasets to be combined are in the same weight situation (i.e. mixed Cases cannot be handled using simple overall relative scaling).

Option 3: Run the task statwt on your calibrated science target data

⇒ Option 3 is pretty easy to implement for both Cases 1 and 2B, and will account for (likely small) Gain variations from antenna to antenna within the individual datasets, whereas Option 2 does not. However, see Caveat 1, for very complex line emission it may be difficult to get a good result.
  • This task attempts to assess the sensitivity per visibility and adjust the weights accordingly. It is very commonly used for JVLA data (including their pipeline).
Caveat 1: One must limit the calculation to line-free channels using the fitspw parameter. For complex line projects this can be painful, however, typically the line-free channels are already known from the continuum subtraction, and can be reused here. However, it is best to run statwt before continuum subtraction, and it should always be run on the fully channelized data before any spectral averaging is applied.
Caveat 2: statwt does not currently account for channel-dependent flagging. It is critical to explicitly avoid the edge channels for any TDM spws in the data (128 channels per spw) data using fitspw, (example: for a dataset with only TDM spws, fitspw='*,7~120').

Example of Correcting Weights Using Option 2 (Relative Scaling)

Figure 1: M100 12m-array weights from 4.2.1 reduction.
Figure 2: M100 7m-array weights from 4.2.1 reduction.

Below we show what could have been done to approximately correct the weights in the M100 SV 7m-array and 12m-array data if it was reduced in CASA 4.2.1 (or earlier), i.e. Case 1 (For a complete treatment of the data processed in CASA 4.3 see https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3).

In CASA 4.2.1 and earlier, the data weights are 1 upon import, later in the standard calibration procedure, applycal scales the weights by Nchan/[(Tsys(i) * Tsys(j)] because calwt=True for the Tsys table applycal. As an example, we plot the weights of 7m and 12m data imported in CASA 4.2.1. No averaging can be turned on when plotting the weights.

# In CASA
os.system('rm -rf 7m_WT.png 12m_WT.png')
plotms(vis='m100_12m_CO.ms',yaxis='wt',xaxis='uvdist',spw='0~2:200',
       coloraxis='spw',plotfile='12m_WT.png')
#
plotms(vis='m100_7m_CO.ms',yaxis='wt',xaxis='uvdist',spw='0~2:200',
       coloraxis='spw',plotfile='7m_WT.png')

The Tsys for the 12m data for the spw containing CO(1-0) ranges from about 100 to 150 K, and the number of channels per spw is 3840, yielding an expected range for the 4.2.1 weights of 0.38 to 0.17 in good agreement with Fig. 1.

As you can see from Figures 1 and 2, the weights are quite similar at this stage because the data were taken under similar weather conditions and hence Tsys, and this is the only thing the weights have been scaled by so far besides the number of channels per spw which is also the same.

Recall that the rms noise in a single channel for a single visibility is:

[math] \sigma_{ij} (Jy) =\frac{2k}{\eta_{q}\eta_{c}A_{eff}} [/math] [math] \sqrt{\frac{T_{sys,i} T_{sys,j}}{2\Delta\nu_{ch} t_{ij}}} [/math] [math] \times 10^{26}, [/math]

Beyond the obvious difference in the antenna dish sizes (Aeff), looking at the listobs output for these two datasets the same channel width was used but that the integration time per visibility is 10.1 sec for the 7m-array and 6.05 sec for the 12m-array. Since dish area is in the denominator of the radiometer equation and integration time per visibility is in the denominator, and assuming WT ∝ 1/σij2, the 7m weight should be scaled by: (7./12.)4 x (10.1/6.05) = 0.193 to account for the difference in telescope size and integration time per visibility.

12m data:


ObservationID = 0         ArrayID = 0
  Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
  10-Aug-2011/19:38:05.8 - 19:50:22.8    11      1 M100                      1500  [0,1,2,3]  [6.05, 6.05, 6.05, 6.05] [CALIBRATE_WVR#ON_SOURCE,OBSERVE_TARGET#ON_SOURCE]

Spectral Windows:  (4 unique spectral windows and 1 unique polarization setups)
  SpwID  Name   #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz) BBC Num  Corrs
  0              3840   TOPO  113726.419       488.281   1875000.0 114663.6750        1  XX  YY
  1              3840   TOPO  111851.419       488.281   1875000.0 112788.6750        2  XX  YY
  2              3840   TOPO  103663.431      -488.281   1875000.0 102726.1750        3  XX  YY
  3              3840   TOPO  101850.931      -488.281   1875000.0 100913.6750        4  XX  YY


7m data:


ObservationID = 0         ArrayID = 0
  Date        Timerange (UTC)          Scan  FldId FieldName             nRows     SpwIds   Average Interval(s)    ScanIntent
  17-Mar-2013/04:44:04.3 - 04:50:43.7    11      1 M100                       261  [0,1,2,3]  [10.1, 10.1, 10.1, 10.1] [OBSERVE_TARGET#ON_SOURCE]

Spectral Windows:  (6 unique spectral windows and 1 unique polarization setups)
  SpwID  Name                           #Chans   Frame   Ch0(MHz)  ChanWid(kHz)  TotBW(kHz) CtrFreq(MHz) BBC Num  Corrs
  0      ALMA_RB_03#BB_1#SW-01#FULL_RES   4080   TOPO  101945.850      -488.281   1992187.5 100950.0000        1  XX  YY
  1      ALMA_RB_03#BB_2#SW-01#FULL_RES   4080   TOPO  103761.000      -488.281   1992187.5 102765.1500        2  XX  YY
  2      ALMA_RB_03#BB_3#SW-01#FULL_RES   4080   TOPO  111811.300       488.281   1992187.5 112807.1500        3  XX  YY
  3      ALMA_RB_03#BB_4#SW-01#FULL_RES   4080   TOPO  113686.300       488.281   1992187.5 114682.1500        4  XX  YY
  4      ALMA_RB_03#BB_1#SW-01#FULL_RES   4080   TOPO  111798.250       488.281   1992187.5 112794.1000        1  XX  YY
  5      ALMA_RB_03#BB_2#SW-01#FULL_RES   4080   TOPO  113673.250       488.281   1992187.5 114669.1000        2  XX  YY
Figure 3: M100 7m-array (red) and 12m-array (black) weights after scaling by relative sensitivity.

NOTE: If these data had been manually calibrated in CASA 4.2.2 with calwt=False for the gain table applycal (i.e. Case 2B), then the overall scale factor to apply would be (7./12.)4 = 0.116, because the difference in visibility integration time would already be accounted for.

An easy way to perform this overall scaling is using the visweightscale parameter in concat.

# In CASA
# Concat and scale weights
os.system('rm -rf M100_Intcombo_0.193.ms')
concat(vis=['m100_12m_CO.ms','m100_7m_CO.ms'],
       concatvis='M100_Intcombo_0.193.ms',
       visweightscale=[1,0.193])

Now plot the concatenated weights to verify they are as expected.

# In CASA
os.system('rm -rf Intcombo_0.193_WT.png')
plotms(vis='M100_Intcombo_0.193.ms',yaxis='wt',xaxis='uvdist',spw='0~2:200',
       coloraxis='spw',plotfile='Intcombo_0.193_WT.png')

These combined data with correct relative weights are now ready for imaging. See https://casaguides.nrao.edu/index.php?title=M100_Band3_Combine_4.3 for more on the imaging in CASA 4.3.

Note that the per spw 12m-array 4.2.1 derived weights are quite similar to those shown in M100_Band3_Combine_4.3 from CASA 4.3 (Figure 3) using the correct per channel weights. This is purely accidental... for this example 1/σij2 just happens to be approximately equal to Nchan/[(Tsys(i) * Tsys(j)] for an antenna with diamter=12m, Δνch=488.281 kHz, tij=6.048s, and 3840 channels.

Example of Correcting Weights Using Option 3 (statwt)

Figure 4: M100 12m-array WT vs uvdist after calwt=False for amplitude gains.
Figure 5: M100 7m-array WT vs uvdist after calwt=False for amplitude gains.
Figure 6: M100 12m-array amplitude vs channel.
Figure 7: M100 7m-array amplitude vs channel.

This is an example of running statwt on the M100 SV data. The starting point was data calibrated as if it had been manually calibrated in CASA 4.2.2, i.e. CASE 2b calwt=False for the gains. Note for this starting point, the 7m data actually have higher weight than the 12m data. This is because the average Tsys were similar but the visibility integration time of the 7m data is 10.1 versus 6.048s for the 12m data (see listobs outputs in previous section). Imaging these data would work very poorly because as already described, the 7m should actually have weights &approx; 0.193 times lower than the 12m data.

os.system('rm -rf 12m_calwtF_WT.png')
plotms(vis='M100_Band3_12m_CalibratedData.cf.ms.fresh', spw='0',yaxis='weight',xaxis='uvdist',
       plotfile='12m_calwtF_WT.png')


os.system('rm -rf 7m_calwtF_WT.png')
plotms(vis='M100_Band3_7m_CalibratedData.cf.ms.fresh', spw='3,5',yaxis='weight',xaxis='uvdist',
       plotfile='7m_calwtF_WT.png')

As mentioned above in the Option 3 statwt caveats, it is essential to avoid strong line channels when calculating the statistics. What is meant by strong? At minimum, if you can see it in a uvplot it should not be included. The easiest way to check is to make a plot of the uv-data in channel space.

# In CASA
os.system('rm -rf 12m_amp_ch_Spw0.png')
plotms(vis='M100_Band3_12m_CalibratedData.cf.ms',
       yaxis='amp',xaxis='channel',spw='0',avgtime='1e8',
       avgscan=True,plotfile='12m_amp_ch_Spw0.png')


os.system('rm -rf 7m_amp_ch_Spw3_5.png')
plotms(vis='M100_Band3_7m_CalibratedData.cf.ms',
       yaxis='amp',xaxis='channel',spw='5',avgtime='1e8',
       avgscan=True,plotfile='7m_amp_ch_Spw5.png')

In statwt we need to avoid the line emission, while sampling the full spectral range. This is especially important in this example because the CO(1-0) transition is close to an oxygen line in the atmosphere -- this is what causes the noise to increase at higher channel numbers. Since we are deriving a spectrally averaged weight it would be unrealistic to only include the lower, less noisy channels.

# In CASA
statwt(vis='M100_Band3_12m_CalibratedData.cf.ms', spw='0', 
       fitspw='0:0~1800;2500~3839', datacolumn='data')

# In CASA
statwt(vis='M100_Band3_7m_CalibratedData.cf.ms', spw='3,5', 
       fitspw='3:0~1900;2500~3839,5:0~1900;2500~3839', datacolumn='data')

Now we can concat these datasets and make a plot to verify that the weights have the values expect.

Figure 8: M100 7m-array (purple) and 12m-array (black) weights after running statwt.
concat(vis=['M100_Band3_12m_CalibratedData.cf.ms','M100_Band3_7m_CalibratedData.cf.ms'], 
       concatvis='M100_Band3_statwt.ms')

os.system('rm -rf M100_statwt.png')
plotms(vis='M100_Band3_statwt.ms', spw='0,7,9',yaxis='weight',xaxis='uvdist',
       coloraxis='spw',plotfile='M100_statwt.png')

These weights are in good agreement with those derived using the correct weights throughout CASA, i.e. Figure 3 in M100_Band3_Combine_4.3. However, see the next section for caveats.

Absolute Accuracy of the Data Weights

As mentioned several times in this guide, good imaging performance is only subject to correct relative weights, which is achieved as of CASA >4.2.2 provided calwt=True for the amplitude gain table. Even for Case 1 (4.2.1 and lower), the relative weights within a given dataset are reasonably accurate in most cases due to the 1/Tsys2 scaling. However, there are some instances, for example, uv-model fitting where one would like to use the visibility weights as an absolute measure of 1/σ2. Here we discuss the absolute accuracy of the visibility weights for options 1 and 3:

Option 1: All PI data taken so far with ALMA are Hanning smoothed online, but it is up to the user at proposal creation time to decide if additional channel averaging is done in the correlator (typically to reduce the data rate). Data that are Hanning smoothed but not channel averaged by the ALMA online system have a theoretical effective channel width 2.667 times that reported by for example listobs from the point of view of calculating the sensitivity of a single channel (it is a factor of 2.0 for the FWHM, i.e. channel resolution). The CASA initialization of 2ΔνchΔtij does not currently take this into account -- CASA does not have a way of knowing presently what the online system did, only how the data appears on disk, though this will change with future versions of the online software. Thus, presently CASA weights for this case are in principle a factor of 2.667 times smaller than they should be from a sensitivity point of view. For online channel averaging of two or more channels, the difference between the channel width and the effective channel width diminishes rapidly (2x channel averaging has a factor 1.6, 4x a factor 1.2, 8x a factor of 1.1). Additionally, the current weights do not include the correlator and quantization efficiencies ηq and ηc. These have values of about 0.88 and 0.96, respectively.
⇒ Starting in Cycle 3, the effective channel width will be recorded in the raw data and used by CASA to initialize the weights for CASA versions >=4.2.2.
Option 3: For CASA <=4.4 statwt does not take channel-specific flagging into account, only flags that apply to all channels. Thus, the user must avoid flagged channels with the fitspw parameter. Additionally, strong line emission must be avoided. Provided these conditions are met, our tests suggest that the weights produced by statwt are too low by a factor of 2 compared to the expected values. Both of these statwt issues will be fixed as of CASA 4.5.

As a consequence, for online Hanning smoothed data with no online channel averaging, the average value of the weights resulting from Options 1 and 3 will appear to approximately match (see M100 examples), but for erroneous reasons.

Our investigation into these matters continues so if the absolute weights are important to you, check back here or send a helpdesk ticket with your inquiry.