VLA CASA Pipeline-CASA4.5.3
This guide is designed for CASA 4.3.0
Introduction
The VLA calibration pipeline is described on the VLA pipeline webpage. The goal of the pipeline is to calibrate VLA data to be ready for subsequent imaging steps. At NRAO the pipeline is executed on every schedule block that the VLA observed.
We currently have two versions of the pipeline: A calibration pipeline integrated in CASA, and a scripted pipeline. This CASAguide deals with the pipeline version that is integrated in CASA. It is developed along with the ALMA pipeline and aims to use similar procedures and outputs as the ALMA pipeline. Details on the scripted pipeline can be found on the VLA pipeline webpage.
The VLA pipeline has been developed to require minimal manual intervention and to be applicable to all VLA frequency bands. All calibration steps will be applied to all data; this implies that simplicity and robustness has priority over speed. Typical pipeline runs can take from a few hours, to several days. Typically, however, a pipeline run concludes within the time span of a day.
The pipeline was introduced in May 2012 and likely works well for all data taken thereafter. It may not work well for data taken earlier and we recommend running the calibration for such data manually (VLA CASAguides are available on the VLA tutorial page).
Pipeline Requirements
The VLA pipeline has been developed for Stokes I, continuum data with typically 128MHz bandwidth per spectral window. Nevertheless it usually also executes well on other setups, although it will not be tailored for special needs. In the future, we will provide a reprocessing interface to adapt the pipeline to different setups, including spectral line, and polarization.
The pipeline will require the correct setup of the scan intent at the time of the obsbervattion. We therefore require every observer to ensure that the scan intents are correctly set in the OPT. See the OPT manual for details. In particular, the pipeline requires the intents CALIBRATE_FLUX for the flux calibration scans (this has to be one of the standard VLA calibrators 3C48, 3C138, 3C147, or 3C286), CALIBRATE_AMPLI and CALIBRATE_PHASE for the complex gain/phase calibration, and CALIBRATE_BANDPASS for the bandpass calibration (will default to the CALIBRATE_FLUX calibrator when not present). The pipeline also currently requires adequate signal-to noise for each spectral window and calibrator source.
Running the Pipeline
NRAO-initiated automatic pipeline runs
Every schedule block (SB) executed at the VLA will be batch pipeline processed. NRAO will use a pipeline version that is packaged with CASA and also available for outside users. See The CASA download page for the current (and older) pipeline versions. At NRAO we will always execute the pipeline for Stokes I continuum processing independent of the setup. Modification may this be required for non-continuum data (see below).
Once an SB is processed, the PI of the project will receive an email that pipeline calibrated data is available. Use the NRAO helpdesk to request the calibrated data. For ~10 days, the calibrated MS will be available for download. After that period, the flagging and calibration tables will be provided that can be applied to the raw measurement set. See also the VLA pipeline webpage for more details.
You may request via the helpdesk that one of the NRAO scientists to perform a quality assessment of the pipeline execution.
Starting the Pipeline Manually
The VLA pipeline webpage has details on how to run the pipeline. A CASA version that includes the pipeline is required, and started like
# In a Terminal
casa --pipeline
AT NRAO one can start the default pipeline version via
# In a Terminal
casa-pipe
Within CASA, on should load the pipeline recipes via
# In CASA
import pipeline.recipes.hifv as hifv
followed by an execution on the SDM, e.g. 16A-000.ASDM like:
# In CASA
hifv.hifv(['16A-000.ASDM'])
Although the MeasurementSet can also be used as input, we recommend the SDM-BDF that can be downloaded from the NRAO archive.
Pipeline Outputs
The pipeline output will be a calibrated MS and all calibration tables.
- The MS will have the CORRECTED column attached which can be used for subsequent imaging.
- The main directory will also contain the oussid* images as described in the tasks hifv_makeimlist and hifv_makeimages (see below).
- The main directory also contains all the calibration tables.
- The weblog is available under pipelineXXXX/html/index.html where the XXX stands for the execution timestamp.
- The same directory also contains the casapyXXX.log which is the entire CASA logger output.
- In this directory one can find the casa_pipescript.py, the actually executed pipeline commands (see below).
- In the same directory the casa_commands.log lists the actual CASA commands that were issued (see below).
- The listobs output is available under 'pipelineXXXX/html/sessionSession_default/<MSname>.listobs.txt'.
casa_pipescript.py
The 'pipelineXXXX/html' directory also contains a file called 'casa_pipescript.py' which will look like:
__rethrow_casa_exceptions = True
h_init()
try:
hifv_importdata(ocorr_mode='co', vis=['13A-398.sb17165245.eb19476558.56374.213876608796'], createmms='automatic', asis='Receiver CalAtmosphere', overwrite=True)
hifv_hanning(pipelinemode="automatic")
hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', flagbackup=False, scan=True, baseband=True, clip=True, autocorr=True, hm_tbuff='1.5int', template=False, online=True, tbuff=0.0, fracspw=0.05, shadow=True, quack=True, edgespw=True)
hifv_vlasetjy(fluxdensity=-1, scalebychan=True, reffreq='1GHz', spix=0)
hifv_priorcals(pipelinemode="automatic")
hifv_testBPdcals(pipelinemode="automatic")
hifv_flagbaddef(pipelinemode="automatic")
hifv_checkflag(pipelinemode="automatic")
hifv_semiFinalBPdcals(pipelinemode="automatic")
hifv_checkflag(checkflagmode='semi')
hifv_semiFinalBPdcals(pipelinemode="automatic")
hifv_solint(pipelinemode="automatic")
hifv_fluxboot(pipelinemode="automatic")
hifv_finalcals(pipelinemode="automatic")
hifv_applycals(pipelinemode="automatic")
hifv_targetflag(pipelinemode="automatic")
hifv_statwt(pipelinemode="automatic")
hifv_plotsummary(pipelinemode="automatic")
hif_makeimlist(nchan=-1, calmaxpix=300, intent='PHASE,BANDPASS')
hif_makeimages(masklimit=4, noise='1.0Jy', subcontms=False, target_list={}, parallel='automatic', maxncleans=1, weighting='briggs', tlimit=2.0, robust=-999.0, npixels=0)
finally:
h_save()
this is the actual pipeline script that is being executed. It contains all the 'hifv' steps. The pipeline run can be modified via this script, e.g. by commenting out individual steps, or providing different parameters (see the inline help for each task). It can then be (re-)executed via:
# In CASA
execfile('casa_pipescript.py')
We will use this method, e.g. to modify the script after being adjusted for spectral line processing (see below).
casa_commands.log
A second useful file in 'pipelineXXXX/html' is the 'casa_commands.log' file which lists all the CASA commands that the pipeline heuristics produced. Note that 'casa_commands.log' is not executable itself, but it will go through all steps with all parameters that the CASA commands are being set to.
The Pipeline Weblog
The pipeline run can be inspected in the weblog under 'pipelineXXX/html/index.html' (wher XXX is the timestamp of the execution.
We provide the weblog that we discuss in this tutorial through the following link:
https://casa.nrao.edu/Data/EVLA/Pipeline/VLApipe-guide-weblog.tar.gz (209 MB)
Extract the weblog via:
# In a Terminal
tar xzvf VLApipe-guide-weblog.tar.gz
and point your browser to html/index.html.
The main page of the weblog (XXmain) contains essential information such as the project archive code, the PI of the observations, and the the start and end time of the observations. The CASA and pipeline versions that were used are also listed on the main page (Fig. XXmain).
In addition, the filename of the MeasurementSet, receiver bands, number of antennas, on source time, min/max baseline lengths, the phase monitor rms and the filesize are listed in a table.
On the top of the page, the tabs By topic and by task will show the results of the pipeline run. An overview of the observations (Fig. XXoverview) will be obtained by clicking on the MS name.
This page provides additional information about the observations. That includes Spatial Setup (field names, target and calibraotor names), Antenna Setup (min/max baseline lengths), Spectral Setup (band designations; science bands exclude, e.g. pointing scans), and Sky Setup (min/max elevation) The page also provides graphical overviews of the scan intents as a unction of time, field names as being observed in its time sequence, and information on the weather conditions during the observations.
An important link is the 'LISTOBS OUTPUT' button. The link leads to a full listobs listing of the MS (Fig. XXlistobs}}, including the scan characteristics, with execution times, scan ids, field ids and names, associated spectral windows, integration lengths, and scan intents. Further down the spectral windows are provided with their ids, channel numbers, widths, start and central frequencies. Sources and antenna locations are part of the listobs output, too.
The By Topic page provides basic pipeline run summaries such as warnings, scores (not implemented as of CASA 4.5.3) and flagging summaries (Fig. XXtopic).
Overview of the Pipeline Steps
The pipeline is being executed in 20 individual pipeline tasks which are listed under the By Task tab (Fig. XXtasks). Each task has an associated score for success. But note that the scores are not implemented as of the CASA 4.5.3 VLA pipeline (C3R4B). Warnings and errors in tasks are also displayed by exclamation mark and cross icons next to the task names. In our example, the pipeline threw warnings in steps 1 and 20, and an error in step 4.
To obtain more details on each task execution, each tasks can be clicked on and the task results, including statistics, plots, and results can be examined. Common to each task page are details on the score (Pipeline QA; not implemented in the CASA VLA pipeline as packaged in 4.5.3), the task Input Parameters, Task Execution Statistics, and the associated CASA logs, which provide details on the actual commands that were issued as well as the associated logger outputs.
The Individual Pipeline Steps
1. hifv_importdata: Register VLA measurement sets with the pipeline
In the first step, the data is imported from the SDM (Science Data Model) archival format to the CASA MeasurementSet (MS). Basic information on the MS is being provided, such as SchedBlock ID, the number of scans and fields and the size of the MS. The MS is also being checked for the scan intents and a baseline summary of the initial flags is obtained.
In our example (Fig. XX1), a warning is issued that the data does not contain a CALIBRATE_BANDPASS scan intent. In this case, the pipeline will revert back to the flux density calibrator for bandpass calibration.
CHECK for: any errors in the import stage. That includes missing scans as in our example. Warnings will also be issued if the data were previously being processed, e.g. if the pipeline is run on an MS.
2. hifv_hanning: VLA Hanning Smoothing
This task runs Hanning-smoothes the MS. This is step will reduce the Gibbs phenomenon (ringing) when extremely bright and narrow spectral features are present, usually caused by strong rfi.
CHECK for: nothing except for complete failure. FOR SPECTRAL LINE DATA: you may decide to not run this step as spectral lines will be smoothed to a degraded spectral resolution (see also below). '
3. hifv_flagdata: VLA Deterministic flagging
This step will apply online flags. That includes antennas not on source (ANOS), scans with intents that are of no use for the pipeline such as pointing and focus scans, autorrelations, the first and last three edge channels of each spw, clipping absolute zero values, quacking (ie removing the initial integration per scan), and flagging of entire basebands if needed. The flags are reported as a fraction of the total data for the full dataset as well as broken up into the individual calibrator scans as well as the target data. A plot is provided that displays the online antenna flags as a function of time.
In our example (Fig. XXX3), the target source is flagged from a start of 3.12% of the data, adding 6.05% due to antenna not on source, other online flags 0.82%, edge channels 6.4%, clipping 0.09% of absolute zero values, and 1.40% baseband clipping. This amounts to a total of 8.71% of flagging for the scientific target. Other sources are also listed and the entire MS was flagged on a 8.84% level.
CHECK for: the percentage of the flags. Were a very large portion (or even all) of the visibilities of a calibrators flagged ? If so, try to find out why. Also have a quick look at the graph of the online flags to understand if the system behaved normally or if there was an unusually high failure of some kind.
4. hifv_setjy: Set calibrator model visibilities
Step number 4 comprises the setting of the calibrator spectral and spatial model to the standard VLA flux density calibrators. The task page lists the flux densities of the calibrator model for each spw. It also shows the models for each spw that are calculated and used to the flux density calibrator.
In our case, hifv_setjy throws an error (QA Too many flux calibrator measurements for 13A-398.sb17165245.eb19476558.56374.213876608796.ms 66/64; Fig. XXX4), which is due to the inclusion of pointing scans in this steps that are later on flagged. This is nothing to worry at this point as the X-band pointing scans are not being used for any scientific application. Pointing corrections are being calculated and applied in the online system and are not being used thereafter.
CHECK for: any unexpected fluxes or model shapes.
5. hifv_priorcals: Priorcals (gaincurves, opacities, antenna positions corrections and rq gains)
Next, the prior calibration tables are being derived. They include gain-elevation dependencies, atmospheric opacity corrections, antenna offset corrections, and requantizer gains. They are independent of the calibrator scans themselves.
In addition to the opacities themselves (calculated per spw; Fig. XX5), a plot is attached that provides more information on the weather conditions during the observation. The antenna positions are usually updated days after the move and the corrections for four antennas in our case are in the millimeter range.
CHECK for: extreme or unrealistic opacities. Also check that the antenna offsets are within reason. There should only be updates for a few antennas.
6. hifv_testBPdcals: Initial test calibrations
Now it is time to determine the delays, and the bandpass solution (gain and phase).
The plot on the main page (Fig. XX6) shows the flux density calibrator with the bandpass solution applied. The subpages show the delay, gain amplitude, gain phase, bandpass amplitude, and bandpass phase solutions for each antenna. Note that the phases will be close to zero for the reference antenna. When delays are more than +/-10ns it will be worth examining the data more closely. Some additional flagging may be needed. The gain apmlitude and phase solutions are derived per integration and they are used to correct for decorrelation for the spectral bandpass solutions. The latter are then determined over a full solution interval, usually for all bandpass scans together. Bandpasses should be smooth although they can vary substantially for wide frequency bands. The BP phases should capture the residuals after the delays are determined.
Example delays are shown in Fig. XX6delay: The delay for ea16 varies but is within a narrow range of only a few ns. These are good solutions. The delays for ea21 are fine except for the 33-35GHz frequency range where they scatter. The respective spws should be flagged if the following pipeline steps will not take do so. For ea22 the delays in the 35-37GHz range are excessive with a value of about -70ns. Maybe the pipeline will be able to calibrate these values correctly but one may need to flag the respective spws if not.
In Fig. XX6gainamp, we show some examples of the gain amplitude plots. Ea03 shows perfect solutions, whereas ea04 has elevated values until 8:06. Those should be flagged (but the pipeline may be able to detect and flag them in one of the following steps). Some of the baselines in ea18 also show low values, but they are constant in time. At this stage one can assume that they reflect the correct calibration values. It might still be worth to make a note and check if calibration dwonstream was applied correctly. The situation is different for ea25 which shows an extreme decrease of flux as a function of time. This antenna should be inspected carefully, there could be problem which will make it unusable (although the bandpass solutions seem to be ok).
Since the gain amp/phase steps are only performed to reduce decorrelation, the phase plots are the most important. We show a few solutions in Fig. XX6gainph. The reference antenna, ea09, has by definition all zero phases. The phase variations as a may change as a function of time for higher frequencies and longer baselines. Therefore both, ea03 and ea21 have good solutions. There are no jumps in the phases - remember that -180 and +180 are identical phase values and jumps between those values are only an issue of the plots, not the actual phase behavior.
Now let's have a look at the bandpasses themselves (Fig. XX6BPGain). Ea17 shows a very good bandpass solutions. Since the spws are small compared to the entire frequency range, the edges of each spw dominate the variations. This is even more extreme for the 37-39GHz range of ea18. Although this could be just the behavior of the antenna, it will be worth to keep an eye on this portion of the data as it may require flagging. Some flagging was already done for the 33-35GHz range of ea21. This corresponds to the noisy delays. Ea24 shows a few high values. They usually are fine as they are also the edges of the spws. In particular if an spw edge coincides with a baseband edge, such spikes may occur. Kepp an eye on those although they are likely not a problem in the calibration. Finally, we show the bandpass of ea25. Although the Gain Amplitude showed decreasing values as a function of time (Fig. XX6gainamp), the bandpass itself does not look suspicious and can likely be used.
The BP phases show residual, channel by channel delays. Again the reference antenna ea09 only shows zero phases by definition. Ea11 is an example of perfect phases across a bandpass. Note that again the variations are dominated by residual phases at the edges of the spw. Some variations are larger than others, but they are all in a similar range. We already swa large scatter in the bandpass amplitude of ea18 37-39GHz and the pattern is repeated in the phases. So this portion may need extra care. Finally we show ea24 once more that also reflects higher values in the amplitudes in the phases, too. At this level, the solutions should be usable.
CHECK for: strong rfi and if it was eliminated later (especially a comparison with task 14). Also check for jumps in phase and or amplitudes. If there are jumps for all but the reference antenna, maybe a different choice for the reference antenna should be considered. Also watch out for extreme delays of tens of ns, and very noisy data. There may also be some unexpected spectral and temporal behaviors of the data.
7. hifv_flagbaddef: Flag bad deformatters The data inside the telescope is undergoing a formatting stage to convert it to an optical signal on the fiber links. On the correlator end it will be deformatted back to an electronic signal. Occasionally, the timing on the deformatter can be misaligned which results in a signal similar to a square sine, or a 'bouncing' signal across the a baseband for one polarization. This step tries to identify such deformatter errors and flag the respective baseband for the affected antenna and polarization. Similar deviations are being identified phases of the signals, but in those cases it is sufficient to flag individual spws and not the entire basebands.
Here, no deformatter issues were detected in the data for the amplitudes, but the phases of a few spws are being flagged. It is always advisable to visually inspect the data, as sometimes deformatter problems are not being identified by this pipeline step. E.g. wide 'bounces', or values that don't approach zero may be missed by the deformatter problem detection algorithm. An example from a different dataset is provided in Fig. XX6example. The 'V' shape close to 5.3GHz with values close to zero are typical for a deformatter issue. If the pipeline does not detect and flags the affected baseband automatically, one should flag the affected antenna, poalrization for all spws of teh affected baseband for the entire dataset.
CHECK for: 'bounces', ie very strong variations of amplitude with low values close to zero. The pattern can repeat a few times across a baseband but should be contained to a single baseband, antenna and polarization. All spws in a faulty baseband, however, are affected. Also check the phases that this step may have flagged.
8. hifv_checkflag: Flag possible RFI on BP calibrator using rflag
Rflag as part of flagdata is a threshold-based automatic flagging algorithm in CASA. In this step, rflag is being run on the bandpass calibrator to remove relatively bright rfi and to obtain an improved bandpass calibration later on.
CHECK for: nothing in particular on this page, but some cumbersome rfi may have been eliminated in preparation of the following steps.
9. hifv_semiFinalBPdcals: Semi-final delay and bandpass calibrations
Now, the step 6 is being repeated which results in better bandpass and delay calibration.
CHECK for: similar issues as in step 6.
10. hifv_checkflag: Flag possible RFI on BP calibrator using rflag
Once more rflag is being executed. After the bright rfi has been removed in step 8 and a new bandpass solution has been applied in step 9, a new threshold will also account for weaker rfi to be removed in this step 10.
CHECK for: removal of rfi in the following steps.
11. hifv_semiFinalBPdcals: Semi-final delay and bandpass calibrations
Again, having removed more rfi, new delay and bandpass solutions are being obtained here.
CHECK for: similar to step 6.
12. hifv_solint: Determine solint and Test gain calibrations
For the final calibration, the pipeline determines the shortest and longest applicable solution interval (solint). Typically they refer to the length of an integration and a scan.
In our case (Fig. XX12) the integration time is 3 seconds which also corresponds the shortest solution interval. The longest solution interval is likely based the phase calibrator scans which typically last for ~85 minutes, minus the drive time and 'quack' flagging, the longest solution results in ~75s.
CHECK for: consistency with the data. The shortest solint should be close to the integration time and the longest to a calibration scan.
13. hifv_fluxboot: Gain table for flux density bootstrapping
Now, the fluxes are bootstrapped from the flux calibrator to the complex gain (gain and phase) calibrator. To do so, spectral indices are computed for the secondary calibrator and the absolute fluxes are determined for each channel. They are then set via setjy and reported for each spw.
For our example, we derive fluxes between 0.61 and 0.68 Jy, as depending on frequency. The spectral behavior is reported as a declining spectral index of around -0.5 (Fig. XX13).
CHECK for: values that are close to the known fluxes of the calibrator. Check the VLA calibrator manual for consistency. Since most calibrator sources are time variable AGN, some differences to the VLA catalog are expected. In particular at higher frequencies they could be up to 50%.
14. hifv_finalcals: Final Calibration Tables
The final calibration tables are now being obtained. Those are the most important ones as they are the ones that are being applied to the data in the following step. The tables are (one for each antenna): Final delay plots, BP initial gain phase, BP Amp solution, BP Phase solution, Phase (short) gain solution, Final amp time cal, Final amp freq cal, and Final phase gain cal. We have already inspected and discussed similar plots for the bandpass earlier. Let's now have a look at some of the complex gain/phase calibrator plots in Fig. XX13gains and XX13ph.
The gains vary quite a bit for this observation. Typically, the gains stay within 10% around a normalized value of 1. Here, a few spws show substantial deviations. Examples for our observations are (Fig. XX13gains): Ea02 has a drop around 5:50 and should be checked. Maybe the entire time between the adjacent calibrator scans, that are more in line withe the rest, may be flagged for this antenna. Ea04 has an inverse behavior later, around 8:00. It appears that only a subset, e.g. a baseband deviates from the rest. Ea07 is more smooth, with some variations between the individual spws but overall a smooth temporal behavior. Likely this solution can be used with no further flagging. Note that the last scan is the flux calibrator. This datum is expected to have slightly different values than those for the complex gain calibrator. Next, we look at the almost perfect ea09 gains. This is also the reference antenna. The gains in ea18 are smooth with a large dip in the first half. Maybe this in fact does calibrate out some characteristics of the observations and could be left for the moment. Around 6:40, a pointing update was performed which seems to have rectified a possibly mis-pointed ea18. Ea23 needs a sinlge spw at a single time flagged. And pointing errors are also obvious for ea25 which also shows a substantial spread across spws. We have previously seen the large decline in flux in the bandpass observations for ea25, with a different slope for each spw. This seems to be reflected also here. It is advisable to check the calibrated gains for this source and flag data is the spw amplitude variations were not calibrated properly.
Now let's have a look at the gains as a function of frequency (Fig XX13ampf). For ea02 we see that one line is below the rest. This is likley one specific time interval and indeed we have seen this in Fig. XX13amp. Ea04 has a very noisy time interval, which is also in agreement with what we have seen in the previous temporal gain plot. Ea08 shows a perfect calibration plot and ea20 repeats the extra noise in the 34-35GHz range that may need to be flagged. Ea25 now repeats the bandpass pattern that we have seen earlier and that explains the spread in Fig. XX13amp.
Now let's have a look at the phases. Ea02 clearly shows very erratic gain variations for one baseband. This is likely not recoverable. Ea04, in contrast exhibits very smooth phase variations until close to the end of the observations. This has already been observed in the gains (Fig. XX13gains) and should be looked at and likely be flagged. Ea09 is the reference antenna and per definition zero in phase. Ea13 shows smooth variations and an example for a perfect calibration table. A spread between basebands can be seen for ea15. The behavior is nevertheless smooth and the data should be calibrated nicely with this table. Ea17, however, has in addition to different behaviors for the basebands also relatively large and erratic jumps between the calibration scans. This clearly needs to be looked into further and may need flagging, although the antenna did not show any issues in previous plots.Finally, ea20 has a relatively smooth behavior until the pointing scan was executed although the variations are relatively large. After the pointing scan, however, phases vary by about +/-50degree between individual, consecutive calibrator scans which is large enough to be not reliable and to be flagged.
CHECK for: issues similar to those described in step 6. Note that carefully checking calibrator tables in this step 14 is of particular importance as they are the final tables to be applied to the target source. Since this step also contains the phase cal solutions, they should be inspected in their temporal variations to be smooth and consistent for each calibrator.
15. hifv_applycals: Apply calibrations from context The calibration itself now concludes with the applycation of the derived calibration tables on the entire dataset. That includes all calibrators as well as the target sources. Note that there's no weighting of the caltables for the VLA (calwt=F) since the swithed power calibration is not being used at this time.
In Fig. XX15, we show the results of this step. The first table shows what tables are being applied, what fields, spws, and antennas are being calibrated. The second table provides information on the flagging statistics. Failed calibration solutions result in flagged calibrator table entries and eventually the data will also be flagged as no calibration can be derived for such data. The following plots show the data of different calibrator source and spw in different plotting versions of phase and amplitude against frequency and uv-distance. To start with, the amp and phase as a function of frequency are being plotted for the complex gain/phase calibrator for each baseband. Next, the amplitudes as a function of uv-distance are plotted for the flux calibrator for each spw. They are followed by amp/time plots for all sources. Finally the amp and phases against time and amp against frequency of the target sources are being plotted for each baseband.
In Fig. XX15 we show a few examples. a) An spw that drops in the calibrated spectrum indicates that it is miscalibrated. b) Although the phases are well calibrated, residual delays are still visible. The zig-zag pattern is due to a small mismatch in the delay measurement timing of the delays ( aka. 'delay clunking'). This behavior is intrinsic to the VLA at this point. Typically the effect is averaged out over time. c) Amplitude vz uv-distance for the flux calibrator should show a flat behavior for each spw (uv radial dependencies are taken account for by the setjy model). The offset may indicate one antenna or one time that is offset from the others. This should be inspected further. d) Amplitude vz frequency for a calibrator also shows a baseline or time that should be investigated and may need additional flagging. For the target source (e) the data is usually noisy and systematic issues are difficult to identify.
CHECK for: a smooth amp vz frequency plot. Jumps may indicate mis-calibrated bandpass fluxes for a spw. Also the shape of the individual spws should be largely removed. Check for deviations from a flat amp vz uv-distance plot after the model was applied as it could indicate badly calibrated times or antennas.
16. hifv_targetflag: Targetflag After the calibration was applied, the automated flagging routing rflag is run one more time on all sources to remove rfi and other outliers from the data.
CHECK for: some rfi removal in the target. Although the flagging is performed for all fields, the calibration is done at this stage. But flagging on the target source may yield improved images. Also check that no spectral line features are being flagged. FOR SPECTRAL LINE DATA: do not run this step as spectral lines may be removed (see also below).
17. hifv_statwt: Reweight visibilities
Since the VLA pipeline is currently not using the switched power calibration, there can be some sensitivity variations of the data over time, due to changes in opacity, elevation, temperature (gradients) of the antennas, etc. So it is usually advisable to weigh the data according to the inverse of its noise. This step is done via the CASA task statwt and will increase the signal-to noise ratio. Note that features such as rfi spikes and spectral lines will be part of the rms calculations and usually results in downweighting data that includes such features.
CHECK for: improved signal-to-noise in the images. As in the previous step, spectra lines may be downweighted. FOR SPECTRAL LINE DATA: do not run this step as spectral lines may be weighted down (see also below).
18. hifv_plotsummary: VLA Plot Summary
This task produces diagnostic plots of the final data. This includes a calibrator phase for all calibrators as a function of time, and all sources, including calibrators and target as amplitude against uv-distance.
Fig. XX18 shows that the calibration around 6:00 and 6:30 is still somewhat noisy and maybe additional flagging of the calibrators may be required. Field 12, looks quite as expected and one may need to check why some values in field 0 are very low and others in field 11 are quite high. Those could correspond to individual antennas, spws, or polarizations. Also some individual Again, some editing may be required and the pipeline restarted.
CHECK for: outliers, jumps, offsets, and excessive noise.
19. hif_makeimlist: Compile a list of cleaned images to be calculated
Finally diagnostic images are being made for each spw of the phase calibrator. The images and basic parameters such as resolution (cell size) and image sizes are listed in this step and are available in the directory that the pipeline was run (usually where the SDM is located). At the moment images are being produced for each spw using the multi-frequency synthesis algorithm, ie. the continuum mode that will not have any spectral dependencies anymore but uses the improved uv-coverage due to the frequency bandwidth of each spw.
In Fig. XX19 the images are listed with 0.31" cell/pixel size ans 300 pixels on each side. Names and phase centers are given for each spw.
CHECK for: appropriate cell size for the images.
20. hif_makeimages: Calculate clean products
The images from the previous step are shown in the final pipeline task.
Imaging parameters are provided for each image. That contains beam characteristics, as well as image statistics and how they compare to the theoretical noise, the noise just based on bandwidth and integration time for typical VLA array parameters. As mentioned earlier, the score is not fully implemented in this version of the pipeline and should be ignored.
CHECK for: degraded images, strong ripples, calibrators that do not resemble a psf. Those may indicate rfi or mis-calibrated sources. If the rms is far from the theoretical noise, this could indicate that deeper cleaning is required. But that may not be important for these images.
Re-Execution of the Pipeline after Flagging
Above we showed many cases where additional flagging is required. After the additional flagging step, the pipeline can be re-executed for improved solutions.
By default, the pipeline will always revert all flags back to their original state as saved in the MeasurementSet.flagversions file. It will thus ignore all modification made.
To continue after manual flagging, one should use the flagged MeasurementSet and place it in a new directory. Do NOT copy over the related MeasurementSet.flagversions file. Then run the pipeline with the flagged MS as the input to hifv.hifv('MeaserementSet']). In that case the pipeline will not be able to revoer original flags and will proceed with the manual flags that the user has applied.
Re-Applying Pipeline Results
Pipeline calibration tables can be re-applied to raw data following the description given on the VLA pipeline webage.
Spectral Line Data
The pipeline is not optimized for calibrating spectral line data. There are some basic procedures, however, to switch off steps that may be detrimental for spectral line setups. The calibrators also require enough signal-to-noise to reliably derive bandpass, gains, phases, etc for the likely more narrow spectral line subbands. The pipeline will also flag edge channels for each spw. If the spectral line happens to be placed on spw edges, additional modifications to the script may be necessary.
As mentioned above the following pipeline steps are not advisable for spectra line data:
Step 2: hifv_hanning: Hanning smoothing dampens the Gibbs ringing from strong spectral features, usually strong, narrow rfi. Hanning smoothing, however, reduced the spectral resolution which may not be desired for spectral line. In addition, the typically narrow spectral line spws contain less rfi and Hanning may not be needed. This step is therefore typically turned off for spectral line calibration.
Step 16. hifv_targetflag: Previous flagging was only applied on the calibrator scans. But Step 16 attempts to auto-flag all field, including target fields. The rflag mode in flagdata is designed to remove outliers from a mean level. (Strong) spectral lines can fulfill this criterion and be flagged. This step should therefore be turned off, too. We recommend manual flagging or a flagdata rflag run that excludes spectra ranges with lines. This step, however, needs to be performed manually after the spectral line pipeline execution.
Step 17. hifv_statwt: A similar argument applies for the statwt step, where the visibilities are weighted by the inverse of their rms. Spectral ines will increase the rms and will therefore be downweighted. As this is not desired, the step will be turned off and should be run manually afterwards, tuned to exclude the spectral line frequency range from the weight calculations.
Given the above, we recommend to modify the casa_pipescript.py as in the example below. Surely, the SDM name (here: '3A-398.sb17165245.eb19476558.56374.213876608796' and maybe other parameters will have to be adapted for the run:
__rethrow_casa_exceptions = True
h_init()
try:
hifv_importdata(ocorr_mode='co', vis=['13A-398.sb17165245.eb19476558.56374.213876608796'], createmms='automatic', asis='Receiver CalAtmosphere', overwrite=True)
# hifv_hanning(pipelinemode="automatic")
hifv_flagdata(intents='*POINTING*,*FOCUS*,*ATMOSPHERE*,*SIDEBAND_RATIO*, *UNKNOWN*, *SYSTEM_CONFIGURATION*, *UNSPECIFIED#UNSPECIFIED*', flagbackup=False, scan=True, baseband=True, clip=True, autocorr=True, hm_tbuff='1.5int', template=False, online=True, tbuff=0.0, fracspw=0.05, shadow=True, quack=True, edgespw=True)
hifv_vlasetjy(fluxdensity=-1, scalebychan=True, reffreq='1GHz', spix=0)
hifv_priorcals(pipelinemode="automatic")
hifv_testBPdcals(pipelinemode="automatic")
hifv_flagbaddef(pipelinemode="automatic")
hifv_checkflag(pipelinemode="automatic")
hifv_semiFinalBPdcals(pipelinemode="automatic")
hifv_checkflag(checkflagmode='semi')
hifv_semiFinalBPdcals(pipelinemode="automatic")
hifv_solint(pipelinemode="automatic")
hifv_fluxboot(pipelinemode="automatic")
hifv_finalcals(pipelinemode="automatic")
hifv_applycals(pipelinemode="automatic")
# hifv_targetflag(pipelinemode="automatic")
# hifv_statwt(pipelinemode="automatic")
hifv_plotsummary(pipelinemode="automatic")
hif_makeimlist(nchan=-1, calmaxpix=300, intent='PHASE,BANDPASS')
hif_makeimages(masklimit=4, noise='1.0Jy', subcontms=False, target_list={}, parallel='automatic', maxncleans=1, weighting='briggs', tlimit=2.0, robust=-999.0, npixels=0)
finally:
h_save()
We commented out the steps 2, 16 and 17. Then run the pipeline as:
# In CASA
execfile('casa_pipescript.py')
followed by target flagging and statwt excluding the spectral line from the respective calculations.
Mixed Correlator Setups
If data were obtained in mixed correlator modes. One can run the pipeline first with the spectral line modifications. Then use split in CASA to extract the narrow bands and run the pipeline in regular mode on the remaining continuum MS.
Polarization Calibration
At this stage, the VLA pipeline does not derive and apply polarization calibration. The user can feel free to derive and add polarization calibration steps after the pipeline was run, using the pipeline calibration tables as required.
Weak Calibrators
The VLA pipeline requires a minimum of signal-to-noise for each spw and target scan. If they are not met, the pipeline will fail. We are currently implementing additional heuristics to deal with weak calibration sources. This code will be available in the upcoming version of the VLA pipeline.
Correcting Scan Intents
Scan intents should be set up correctly in the OPT before submitting the schedule block for observation.
When incorrect scan intents are identified after observations, the SDM can be modified and updated with new scan intents. The SDM metadata is structured in the form of an XML and can be manually edited. Great care, however, should be taken not to corrupt the structure of the SDM/xml.
To do so, cd into the SDM and edit the file 'Scan.xml'. To start with, we recommend to make a backup copy of the Scan.xml file in case the edits corrupt the metadata.
Each <row> section has a <scanNumber> identifier that can be compared to the listobs output. The scan intent is tagged by <scanIntent></scanIntent>, e.g.
for complex gain/phase:
<scanIntent>1 2 CALIBRATE_PHASE CALIBRATE_AMPLI</scanIntent>
for bandpass:
<scanIntent>1 1 CALIBRATE_BANDPASS</scanIntent>
for flux:
<scanIntent>1 1 CALIBRATE_FLUX</scanIntent>
Check with listobs on the imported MS (after executing importasdm or importevla) if the scan intents are now displayed correctly.
Revert back to the original Scan.xml if the above was not successful.
Scripted Pipeline
In addition to the pipeline that is delivered with CASA< one can also use the VLA scripted pipeline. It allows more detailed modifications than the CASA pipeline and can be altered to almost any circumstance. We refer to the VLA scripted Pipeline webpage for details.
Last checked on CASA Version 4.3.0.
