View source for Pipeline: Frequent VLA problems

= General Description =
The [http://go.nrao.edu/vla-pipe VLA pipeline] delivers calibrated data and some initial images of VLA observation runs. The quality of the calibration and imaging products is usually assessed through the weblog that is created in each pipeline run (see also the [https://casaguides.nrao.edu/index.php?title=VLA-S-CASA_Pipeline VLA Pipeline guide]). During the observations, the VLA may have encountered technical problems that are reflected in various ways in the weblog, where graphs show the behavior of the calibration tables as a function of time, frequency, polarization, etc., and analytical numbers describe the amount of flagging, derived fluxes, image statistics, etc. 

Here we would like to briefly describe common VLA observing problems, how they are identified in the pipeline calibration weblog, and how they can be addressed.  

 

== Radio Frequency Interference ==

By far the biggest problem is radio frequency interference (RFI). RFI is produced by internal and external sources, can be terrestrial or from satellites that operate at or spill into the observed frequency. For the VLA, please find more information on the [https://science.nrao.edu/facilities/vla/docs/manuals/obsguide/rfi Radio Frequency Interference] webpage. Although weak RFI may only slightly raise the noise of an image with little influence on the calibration tables, stronger RFI will produce artifacts that may render the data (target and calibrators) unusable, if not adequately flagged. An example for strong RFI is shown below. Flagging procedures are outlined in the [https://casaguides.nrao.edu/index.php/VLA_CASA_Flagging VLA topical CASA guide on flagging].

{|
|[[Image:CASA5.4.0-FLAG-Plotms-amp-time.png|400px|thumb|left|Example for RFI in an observation.]]
|}

It is important that all data are free of 

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Weak, intermittent RFI will increase the noise and be down-weighted in the imaging in the hifv_statwt task. Strong RFI needs to be flagged and only clean data should be calibrated and imaged. Flagging can be manual or automatic. 
</pre>

== Pointing == 
 
At higher frequencies the VLA requires regular pointing calibrations. Each pointing run will reposition the antennas to be centered on a strong source with known position. If the pointing solution fails, the amplitude of the source will drop, or drift away from the center of the antenna with the highest gain. A typical graph looks like the one shown below. The pointing solution for the first half of the run failed, which results in the source drifting away from the center of the primary beam. After a pointing update in the middle of the run, the antenna is positioned properly again (the very last data points are actually a different source, hence the drop at the edge).   

{|
|[[Image:VLApipe-gains-ea25-CASA6.2.1.png|400px|thumb|left|Example of a gain table with a failed pointing solution in the first half of the observation]]
|[[Image:IntentvsTime.png|400px|thumb|left|Plot showing the scan intents vs time for the same observation.]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: If the pointing is only off by a small amount, the gain calibration will take care of it. If it is off by a large amount, the data for this period and antenna needs to be flagged. 
</pre>

== Phase Jumps ==

Various hardware failures can cause the phase for a given antenna to be unstable in time, often with sudden, large changes in phase over time. Depending on where the problem is, this may affect just a portion of the data or up to all data on a given antenna. In the first example below, only one baseband's data is affected (large changes at each data point) while the other baseband remains near zero and is not affected. This plot, from the pipeline's Final phase gain cal section found in the 'hifv_finalcals' stage, shows the final phase solutions found for each calibrator (using the long solution interval).

{|
|[[Image:phasejumps1.png|400px|thumb|left|Example for phase jumps in an observation.]]
|}

Typical phase variations for low frequency data are a few degrees. For high frequencies tens of degrees can occur, and the cycle time between the phase calibrator and the target needs to be reduced to adequately track and interpolate the phase variations as a function of time. If the phases vary more than 360 degrees between two phase calibrator scans, then the data are completely decorrelated and cannot be calibrated anymore. 

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: If there are phase jumps, usually the data for the affected time range needs to be flagged for the antenna(s). 
</pre>

== DTS/Deformatter Problems ==


The digital transmission system (''DTS'') of each VLA antenna includes a formatting stage to convert the electronic to an optical signal before it is injected on the optical fiber link. On the correlator end the signal will be deformatted back to an electronic signal. Occasionally, the timing on the deformatter can be misaligned which results in very strong amplitude or phase slopes as a function of frequency. When this occurs, then the data are corrupt and the entire affected baseband per polarization of an antenna need to be flagged. Frequently the error shows up similar to an ''abs(sin)'' or a 'bouncing' signal across a baseband for one polarization, or, in other terms, various numbers of 'V' shapes in the data, usually in the middle of a baseband. 
	
{| 
|[[Image:VLApipe-step6-BPgain-ea18-CASA6.2.1.png|400px|thumb|left|DTS issue in one baseband]]
|[[Image:VLApipe-baddefromatterexampleBP.png|400px|thumb|left|An example of a bad deformatter from a different dataset.]]
|}
{|
|[[Image:badDTS1.png|400px|thumb|left|Another instance of a DTS issue]]
|[[Image:badDTS2.png|400px|thumb|left|DTS example that is more continuous since the gain and amplitudes are not normalized]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: The data for this baseband, antenna and polarization need to be flagged.
</pre>

== Correlator Zeros ==
 
Under some circumstances, the WIDAR correlator writes exact zeros. The pipeline will usually flag them automatically. If not, they can be removed with CASA's [https://casadocs.readthedocs.io/en/v6.2.1/api/tt/casatasks.flagging.flagdata.html flagdata] task, using the option ''mode='clip' '' with ''clipzeros=True'' or flag the zeros by hand. 

{|
|[[Image:FlaggedDataSummary.png|400px|thumb|left|If using the pipeline one can find the above table in the hifv_flagdata task. Here the percentage of data flagged due to correlator zeros is represented by the "Clipping" column.]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: The pipeline will usually catch them. If not, use CASA's flagdata task.
</pre>

== Baseband and Subband Edges ==
 

If spw roll-off frequency edges are very steep, they can degrade gain and phase solutions. Frequently this is not a big problem, but if the gain for the edge channels is close to zero, a division by the bandpass for these channels can get extremely noisy. This is particularly true for baseband edges. The ''edgespw'', ''fracspw'', and ''baseband'' parameters in ''hifv_flagdata'' can be adjusted to flag different percentages of the edges (see also [http://go.nrao.edu/vla-pipe VLA pipeline] pages). The edges can also be flagged with the CASA task [https://casadocs.readthedocs.io/en/v6.2.1/api/tt/casatasks.flagging.flagdata.html flagdata], or by hand. 

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Adjust the relevant parameters in hifv_flagdata and re-run the pipeline.  
</pre>

== Compression ==
 
Strong RFI can bring the the digital and analog receiver system into a non-linear regime (also known as compression). This is especially a problem in L and S bands. Simple RFI flagging alone will not be sufficient to remove compression. The  affected antennas/spw/pols will likely need to be flagged.  

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: For strong compression, flag the affected data. Weak compression may increase the Tsys.  
</pre>

== Resolved Calibrators ==
 
Some calibrator sources are not perfect point sources. For the VLA standard flux calibrator sources, models are provided within CASA. For resolved phase calibrators, the uv-range can be restricted during the solve for the calibration tables. Often, significant structure is noticeable directly in the visibility data. Figures 4a and 4b show, respectively, the visibility data for an unresolved and a resolved calibrator.

{|
|[[Image:3C84_AmpvsUVwave.png|400px|thumb|left|Fig. 4a: Plot showing the point-like nature of 3C84 during a K-band, B-config observation. Point-like sources will appear as horizontal lines in such plots.]]
|[[Image:3C48_AmpvsUVwave.png|400px|thumb|left|Fig. 4b: Plot showing resolved structure in 3C48 during a K-band, B-config observation. However, due to being one of the VLA's standard flux calibrators, this structure will be accounted for when setting the model for this source.]]
|} 

A solution is to use the flux.csv table. It is usually generated for ALMA in a pipeline run, but can be created before a VLA run. The uv-ranges listed there will be used in the processing. The format is like:

ms,field,spw,I,Q,U,V,spix,uvmin,uvmax,comment

and an example entry would be:

MY.ms,0,2,1,0.0,0.0,0.0,0.0,21000.0,11000.0,"# 3C48"

for a uvrange of 21000-110000 lambda. 

Above, ms is the MS name, field and spw are the IDs (not names, the ID will only be known once the data is in MS format and after executing listobs), I, Q, U, V are the Stokes flux densities in Jy (note that entries for the VLA will be ignored here, so a nominal I=1Jy will be ok), uvmin and uvmax are the uv ranges in units of lambda. Only one spw (the first) is used per field, other entries will be ignored. If uvmax is provided as 0 lambda, then this creates an inequality and uvmax is unbounded. 


If you have multi-band data, you may have to split the data per band first, then run each band through their own pipeline to make use of flux.csv.

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Restrict the uv-range for the calculations of the calibration tables. The flux.csv table can be used. 
</pre>

== Wrong Intents ==

If the intents of the data are set incorrectly for the observations, the pipeline will use the wrong calibrators for the calibration. Usually this can be fixed by overwriting the intents. The [http://go.nrao.edu/vla-pipe VLA pipeline webpage] provides instructions and a script to do this. For more complicated setups, like multiple calibrators or bands with separate calibrator scans, data may be split into smaller MSs that contain only the relevant calibrators for each target, or data reduction by hand may be needed.

== Non-ideal reference antenna == 

Sometimes, if the reference antenna has some issue, like RFI or extreme flagging, it is advisable to switch to a different reference antenna. The example below shows 
that one spw has extreme phase jumps for all antennas when ea02 was chosen as a reference antenna. This indicates that the phase jumps are likely not present on all antennas, but that phase instabilities on ea02 itself are reflected on all other antennas. Indeed, when ea09 was chosen as a reference antenna, as shown in the second plot), then the instability is shown only in ea02 and all other antennas are well-behaved. 
Delays are also a quantity that are relative to a chosen reference antenna. If the delays for all antennas show similarly high delays, then it is likely that the reference antenna has the high delays and not all other antennas. Chosing a different reference antenna would quickly reveal if this is the case. 

Use the 'refantignore' keyword to disallow the use of this antenna as a reference (in  the example one should ignore ea02 as a possible reference antenna). The [https://science.nrao.edu/facilities/vla/data-processing/pipeline Pipeline Page] provides details on the usage of this keyword. 

{|
|[[Image:Bad_refant.png|400px|thumb|left|Plots of phase solutions vs time showing that all antennas have inherited ea02's phase issue when it is used as the reference antenna.]]
|[[Image:Good_refant.png|400px|thumb|left|Plots of phase solutions vs time showing ea02 has a phase issue. Here ea09 is used as the reference antenna.]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Use 'refantignore' to remove a problematic antenna from the list of possible reference antennas. 
</pre>

== Extreme Solution Intervals ==

If the hifv_solint stage shows extreme values for the short and/or long solution intervals, then the data should be inspected and flagged. The pipeline should then be re-run to determine new solution intervals. 

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Flagging bad data. 
</pre>

== Weather ==
 
At the VLA, the weather has to meet certain conditions to run a scheduling block. The conditions vary with frequency and are more stringent for higher frequency observations (settable by the. It can happen, however, that the weather deteriorates after a scheduling block has started. High water vapor content and moving atmospheric cells can increase the system temperature and introduce extreme phase jumps. Wind (gusts) will also change the phase stability and cause more frequent pointing errors. Flagging times of bad weather conditions may help. The CASA task [https://casadocs.readthedocs.io/en/v6.2.1/api/tt/casatasks.manipulation.statwt.html statwt] will down-weight some noise variations. Also selfcal ([https://casaguides.nrao.edu/index.php/VLA_Self-calibration_Tutorial Topical Guide: VLA Self-calibration Tutorial]) will correct for phase variations. In extreme cases, however, flagging is the only method. 

For some SBs the weather data are missing from the header. This is usually not a big problem. The data can be filled, however, on request. Please contact the [https://help.nrao.edu NRAO helpdesk]. 

{|
|[[Image:WestArmPhaseJump.png|400px|thumb|left|The phase solutions of the three outer most antennas on the West Arm during a B-config observation. A phase jump can be seen between 00:20:00 and 00:30:00. Often in the extended configurations one may notice the outer antennas on a particular arm of the array show such phase jumps as the weather can be significantly different between the outer and inner antennas as the array increases in size. Note that even if the jump is due to wrapping, there was a strong phase gradient at the beginning of the observations fiollowed by a much calmer period later.]]
|}

{|
|[[Image:WeatherStationOutage.png|400px|thumb|left|A plot generated by the task plotweather which shows missing data. Such missing data is often due to power outages and glitches in the VLA's local weather monitoring station.]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Statwt, selfcal, or flagging. 
</pre>

== Decorrelation ==

Decorrelation is an effect where the individual spatial frequencies of the visibilities are misaligned. If the misalignment is random, the data is decorrelated, ie not all wave amplitudes are aligned, leading to destructive interference and thus a reduced amplitude. The biggest source of decorrelation is the atmosphere where a screen of a number of atmospheric cells with different refractive indices moves across the array, which causes errors in the delay and thus phase. The effect of decorrelation increases with observing time and is stronger for longer baselines. One correction for decorrelation is to increase the time between phase calibrator observations. At some time, however, decorrelation is constant (see the Advanced Calibration presentations at the [http://www.cvent.com/events/virtual-17th-synthesis-imaging-workshop/agenda-0d59eb6cd1474978bce811194b2ff961.aspx NRAO synthesis school]). 
 
The pipeline will correct for some degree of decorrelation for all calibrators. In extreme cases, however, data need to be flagged. If decorrelation is strong, it can be assumed that the target also shows significant decorrelation. Self-calibration is advised if the source flux is sufficient. A CASA guide for self-calibration is provided in the [https://casaguides.nrao.edu/index.php/VLA_Self-calibration_Tutorial Topical Guide: VLA Self-calibration Tutorial].

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Self-calibration. In extreme cases: flagging.
</pre>

== Shadowing ==

The pipeline flags shadowed antennas by default. If not all of shadowing is captured, or if the shadowing criteria shall be loosened (e.g. allow a small amount of shadowing), then this can be controlled by the CASA task [https://casadocs.readthedocs.io/en/v6.2.1/api/tt/casatasks.flagging.flagdata.html flagdata] 'mode='shadow'. After manually flagging the data, he 'hifv_flagdata' task call should then be modified ('shadow=False') to not do additional shadowing flagging. 

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: in CASA: flagdata mode='shadow'
</pre>

== RFI plots look worse after Flagging==
In some instances, the post-RFI flagging plots look aesthetically worse than the pre-RFI flagging plots. This is due to a poorly performing antenna (higher noise than others) that is getting heavily flagged in the RFI flagging. The post-RFI flagging plot then has less data to average together resulting in a worse looking plot. This is not a problem and the outcomes from imaging with and without the flagging of these poorly performing antennas are not scientifically different. In the case of these noisy data not getting flagged (as in the previous pipeline version), they are strongly downweighted by statwt so they do not contribute much to the final images anyway.

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Nothing to fix, but inspect closely that this is only a plotting effect.
</pre>


== Other System Issues == 

Sometimes, the calibration amplitudes vs frequency plots show some features resembling of resonances. If they are strong, maybe significantly more than 10%, the data may need to be flagged, and the calibration pipeline restarted. 

{|
|[[Image:switch1.png|400px|thumb|left|System issue that is manifested as a change in frequency in the amplitude-frequency calibration tables.]]
|[[Image:switch2.png.png|400px|thumb|left|A second example of the system issue.]]
|}

<pre style="white-space: pre-wrap; background-color: #E0FFFF;">
FIX: Stronger features should be flagged. 
</pre>