VLA CASA Pipeline-CASA4.5.3

Introduction

• With the start of Jansky VLA Full Operations (January 2013), we started a new operational model: – Deliver flagged and calibrated visibility data – You will self-calibrate and image visibility data to meet science goals, using resources at home institution or NRAO computing resources • Automated pipeline should run correctly on all “standard” Stokes I science SBs; “standard” means: – 128 MHz spws, but may work on other set-ups as well • Some constraints on strength of calibrators needed – Contains correctly labeled and complete scan intents • Current versions available: – “scripted” pipeline is a collection of python scripts that use CASA tasks wherever possible, but also uses toolkit calls; readable and easy to modify – CASA integrated pipeline is compatible with ALMA pipeline infrastructure, improved diagnostics in weblog, used as real-time pipeline since Sep 2015

=

• Real-time pipeline: – Minimal human intervention • Pipeline is run automatically on every science SB as it completes (not just “continuum”) – Pipeline output undergoes quality assurance checks by NRAO staff upon request; reports generated are archived as pipeline products • At your home institution: – Instructions for installation and operation of the VLA CASA Calibration Pipeline are available at https://science.nrao.edu/facilities/vla/data-processing/pipeline • Uses CASA 4.3.1, similar to current real-time pipeline • CASA 4.5.2 currently being validated (you are helping with this!) • Scripted pipelines for CASA versions through 4.5.0 also available – Provides more flexibility in how to use the pipeline, options suitable for spectral line datasets, mixed correlator set-ups, multi-band observations, etc. – Working to incorporate these into the CASA integrated pipeline

Pipeline Requirements

“Standard” Stokes I science SB means: – 128 MHz spws, but may work on other set-ups as well • Can work for narrower BWs, depends on the strength of the calibrators • Heuristics currently make some assumptions about the strength of the calibrators, in particular, the delay calibrator – Contains correctly labeled and complete scan intents • And also that the observation has been set up correctly! • Will the pipeline work for you? – The pipeline successfully completes on ~95% of all science SBs observed on the VLA; whether the output can be used for science depends on the science goal, and whether the observation was correctly set up • Pipeline includes Hanning smoothing, RFI flagging, and weight calculations that may not be appropriate for spectral line projects (but can modify scripted pipeline) • No polarization calibration (yet) but can use pipeline output as starting data for pol. cal. • Will probably work well for data taken since May 2012, may work for earlier EVLA data, likely that extra flagging may be needed in these cases

=

Calibrator strength: – Conservative limit on strength of BP and complex gain calibrators can be derived from requirement for initial gain calibration to work at high end of Q-band – Heuristic for delay calibration currently requires the SNR=3 limit on initial gain calibration per integration

=

• Correct observation set-up – Independent of whether you want to run the pipeline! – Remember: simple observing set-ups are always easier to calibrate – Do not skimp on calibration to spend more time on your target – you may end up not being able to calibrate the target data at all • Spending 3 minutes pointing could buy you more sensitivity than doubling the time on your target • Scan intents – The pipeline relies entirely on correct scan intents to be defined in each SB – In order for the pipeline to run successfully on an SB it must contain, at minimum, scans with the following intents: • A flux density calibrator scan that observes one of the primary calibrators (3C48, 3C138, 3C147, or 3C286) – this will also be used as the delay and bandpass calibrator if no bandpass or delay calibrator is defined • Complex gain calibrator scans

Overview of the Pipeline Procedures

The pipeline is being executed in 20 individual pipeline tasks which are listed under the By Task tab (Fig. XXtasks). Each task has an associated score for success. But note that the scores are not implemented as of the CASA 4.5.3 VLA pipeline (C3R4B). Warnings and errors in tasks are also displayed by exclamation mark and cross icons next to the task names. In our example, the pipeline threw warnings in steps 1 and 20, and an error in step 4.

Fig. XXtasks: The by Task pipeline execution steps.

To obtain more details on each task execution, each tasks can be clicked on and the task results, including statistics, plots, and results can be examined. Common to each task page are details on the score (Pipeline QA; not implemented in the CASA VLA pipeline as packaged in 4.5.3), the task Input Parameters, Task Execution Statistics, and the associated CASA logs, which provide details on the actual commands that were issued as well as the associated logger outputs.

The pipeline steps are as follows:

1. hifv_importdata: Register VLA measurement sets with the pipeline

In the first step, the data is imported from the SDM (Science Data Model) archival format to the CASA MeasurementSet (MS). Basic information on the MS is being provided, such as SchedBlock ID, the number of scans and fields and the size of the MS. The MS is also being checked for the scan intents and a baseline summary of the initial flags is obtained.

Fig. XX1: The hifv_importdata task page.

In our example (Fig. XX1), a warning is issued that the data does not contain a CALIBRATE_BANDPASS scan intent. In this case, the pipeline will revert back to the flux density calibrator for bandpass calibration.

CHECK for: any errors in the import stage. That includes missing scans as in our example. Warnings will also be issued if the data were previously being processed, e.g. if the pipeline is run on an MS.

2. hifv_hanning: VLA Hanning Smoothing

This task runs Hanning-smoothes the MS. This is step will reduce the Gibbs phenomenon (ringing) when extremely bright and narrow spectral features are present, usually caused by strong rfi.

CHECK for: nothing except for complete failure.

3. hifv_flagdata: VLA Deterministic flagging

This step will apply online flags. That includes antennas not on source (ANOS), scans with intents that are of no use for the pipeline such as pointing and focus scans, autorrelations, the first and last three edge channels of each spw, clipping absolute zero values, quacking (ie removing the initial integration per scan), and flagging of entire basebands if needed. The flags are reported as a fraction of the total data for the full dataset as well as broken up into the individual calibrator scans as well as the target data. A plot is provided that displays the online antenna flags as a function of time.

Fig. XX3: The hifv_flagdata task page.

In our example (Fig. XXX3), the target source is flagged from a start of 3.12% of the data, adding 6.05% due to antenna not on source, other online flags 0.82%, edge channels 6.4%, clipping 0.09% of absolute zero values, and 1.40% baseband clipping. This amounts to a total of 8.71% of flagging for the scientific target. Other sources are also listed and the entire MS was flagged on a 8.84% level.

CHECK for: the percentage of the flags. Were a very large portion (or even all) of the visibilities of a calibrators flagged ? If so, try to find out why. Also have a quick look at the graph of the online flags to understand if the system behaved normally or if there was an unusually high failure of some kind.

4. hifv_setjy: Set calibrator model visibilities

Step number 4 comprises the setting of the calibrator spectral and spatial model to the standard VLA flux density calibrators. The task page lists the flux densities of the calibrator model for each spw. It also shows the models for each spw that are calculated and used to the flux density calibrator.

Fig. XX4: The hifv_setjy task page.

In our case, hifv_setjy throws an error (QA Too many flux calibrator measurements for 13A-398.sb17165245.eb19476558.56374.213876608796.ms 66/64; Fig. XXX4), which is due to the inclusion of pointing scans in this steps that are later on flagged. This is nothing to worry at this point as the X-band pointing scans are not being used for any scientific application. Pointing corrections are being calculated and applied in the online system and are not being used thereafter.

CHECK for: any unexpected fluxes or model shapes.

5. hifv_priorcals: Priorcals (gaincurves, opacities, antenna positions corrections and rq gains)

Next, the prior calibration tables are being derived. They include gain-elevation dependencies, atmospheric opacity corrections, antenna offset corrections, and requantizer gains. They are independent of the calibrator scans themselves.

Fig. XX5a: The hifv_priorcals task page.

Fig. XX5b: The hifv_priorcals task page.

In addition to the opacities themselves (calculated per spw; Fig. XX5), a plot is attached that provides more information on the weather conditions during the observation. The antenna positions are usually updated days after the move and the corrections for four antennas in our case are in the millimeter range.

CHECK for: extreme or unrealistic opacities. Also check that the antenna offsets are within reason. There should only be updates for a few antennas.

6. hifv_testBPdcals: Initial test calibrations

Now it is time to determine the delays, and the bandpass solution (gain and phase).

Fig. XX6: The hifv_testBPdcals task page.

The plot on the main page (Fig. XX6) shows the flux density calibrator with the bandpass solution applied. The subpages show the delay, gain amplitude, gain phase, bandpass amplitude, and bandpass phase solutions for each antenna. Note that the phases will be close to zero for the reference antenna. When delays are more than +/-10ns it will be worth examining the data more closely. Some additional flagging may be needed. The gain apmlitude and phase solutions are derived per integration and they are used to correct for decorrelation for the spectral bandpass solutions. The latter are then determined over a full solution interval, usually for all bandpass scans together. Bandpasses should be smooth although they can vary substantially for wide frequency bands. The BP phases should capture the residuals after the delays are determined.

Fig. XX6delay1: The delays for ea16 - these are good solutions.

Fig. XX6delay2: The delays for ea21. The region of high scatter needs to be flagged.

Fig. XX6delay3: For delays for ea22. The large offset of -70ns might be calibratable (or not).

Example delays are shown in Fig. XX6delay: The delay for ea16 varies but is within a narrow range of only a few ns. These are good solutions. The delays for ea21 are fine except for the 33-35GHz frequency range where they scatter. The respective spws should be flagged if the following pipeline steps will not take do so. For ea22 the delays in the 35-37GHz range are excessive with a value of about -70ns. Maybe the pipeline will be able to calibrate these values correctly but one may need to flag the respective spws if not.

Fig. XX6gainamp1: The Gain Amplitude for ea03.

Fig. XX6gainamp2: The Gain Amplitude for ea04. The first few integrations with the elevated values should be flagged.

Fig. XX6gainamp3: The Gain Amplitude for ea18. Some low values are being derived, they are likely calibratable.

Fig. XX6gainamp4: The Gain Amplitude for ea25. The gain drops strongly as a function of time. May need to be flagged.

In Fig. XX6gainamp, we show some examples of the gain amplitude plots. Ea03 shows perfect solutions, whereas ea04 has elevated values until 8:06. Those should be flagged (but the pipeline may be able to detect and flag them in one of the following steps). Some of the baselines in ea18 also show low values, but they are constant in time. At this stage one can assume that they reflect the correct calibration values. It might still be worth to make a note and check if calibration dwonstream was applied correctly. The situation is different for ea25 which shows an extreme decrease of flux as a function of time. This antenna should be inspected carefully, there could be problem which will make it unusable (although the bandpass solutions seem to be ok).

Fig. XX6gainph1: The Gain Phase for ea03. The solutions are smooth.

Fig. XX6gainph2: The Gain Phase for ea09 - the reference antenna

Fig. XX6gainph3: The Gain Phase for ea21. Also good solutions.

Since the gain amp/phase steps are only performed to reduce decorrelation, the phase plots are the most important. We show a few solutions in Fig. XX6gainph. The reference antenna, ea09, has by definition all zero phases. The phase variations as a may change as a function of time for higher frequencies and longer baselines. Therefore both, ea03 and ea21 have good solutions. There are no jumps in the phases - remember that -180 and +180 are identical phase values and jumps between those values are only an issue of the plots, not the actual phase behavior.

Fig. XX6BPgain1: The BP Gain for ea17. Good bandpass solutions

Fig. XXBPgain2: The BP Gain for ea18. The variations in the bandpass in the 37-39GHz rage are large.

Fig. XX6BPgain3: The BP Gain for ea21. 33-35GHz is being flagged.

Fig. XX6BPgain4: The BP Gain for ea24. Some spikes in the bandpass.

Fig. XX6BPgain5: The BP Gain for ea25. The bandpass seems to be in order.

Now let's have a look at the bandpasses themselves (Fig. XX6BPGain). Ea17 shows a very good bandpass solutions. Since the spws are small compared to the entire frequency range, the edges of each spw dominate the variations. This is even more extreme for the 37-39GHz range of ea18. Although this could be just the behavior of the antenna, it will be worth to keep an eye on this portion of the data as it may require flagging. Some flagging was already done for the 33-35GHz range of ea21. This corresponds to the noisy delays. Ea24 shows a few high values. They usually are fine as they are also the edges of the spws. In particular if an spw edge coincides with a baseband edge, such spikes may occur. Kepp an eye on those although they are likely not a problem in the calibration. Finally, we show the bandpass of ea25. Although the Gain Amplitude showed decreasing values as a function of time (Fig. XX6gainamp), the bandpass itself does not look suspicious and can likely be used.

Fig. XX6BPph1: The BP Phase for ea09 - the reference antenna.

Fig. XX6Bpph2: The BP Phase for ea11.

Fig. XX6Bpph2: The Gain Phase for ea18.

Fig. XX6BPph3: The Gain Phase for ea24.

The BP phases show residual, channel by channel delays. Again the reference antenna ea09 only shows zero phases by definition. Ea11 is an example of perfect phases across a bandpass. Note that again the variations are dominated by residual phases at the edges of the spw. Some variations are larger than others, but they are all in a similar range. We already swa large scatter in the bandpass amplitude of ea18 37-39GHz and the pattern is repeated in the phases. So this portion may need extra care. Finally we show ea24 once more that also reflects higher values in the amplitudes in the phases, too. At this level, the solutions should be usable.

CHECK for: strong rfi and if it was eliminated later (especially a comparison with task 14). Also check for jumps in phase and or amplitudes. If there are jumps for all but the reference antenna, maybe a different choice for the reference antenna should be considered. Also watch out for extreme delays of tens of ns, and very noisy data. There may also be some unexpected spectral and temporal behaviors of the data.

7. hifv_flagbaddef: Flag bad deformatters The data inside the telescope is undergoing a formatting stage to convert it to an optical signal on the fiber links. On the correlator end it will be deformatted back to an electronic signal. Occasionally, the timing on the deformatter can be misaligned which results in a signal similar to a square sine, or a 'bouncing' signal across the a baseband for one polarization. This step tries to identify such deformatter errors and flag the respective baseband for the affected antenna and polarization. Similar deviations are being identified phases of the signals, but in those cases it is sufficient to flag individual spws and not the entire basebands.

Fig. XX6: The hifv_flagbaddef task page.

Here, no deformatter issues were detected in the data for the amplitudes, but the phases of a few spws are being flagged. It is always advisable to visually inspect the data, as sometimes deformatter problems are not being identified by this pipeline step. E.g. wide 'bounces', or values that don't approach zero may be missed by the deformatter problem detection algorithm. An example from a different dataset is provided in Fig. XX6example. The 'V' shape close to 5.3GHz with values close to zero are typical for a deformatter issue. If the pipeline does not detect and flags the affected baseband automatically, one should flag the affected antenna, poalrization for all spws of teh affected baseband for the entire dataset.

Fig. XX6example: An example for a bad deformatter.

CHECK for: 'bounces', ie very strong variations of amplitude with low values close to zero. The pattern can repeat a few times across a baseband but should be contained to a single baseband, antenna and polarization. All spws in a faulty baseband, however, are affected. Also check the phases that this step may have flagged.

8. hifv_checkflag: Flag possible RFI on BP calibrator using rflag

Rflag as part of flagdata is a threshold-based automatic flagging algorithm in CASA. In this step, rflag is being run on the bandpass calibrator to remove relatively bright rfi and to obtain an improved bandpass calibration later on.

CHECK for: nothing in particular on this page, but some cumbersome rfi may have been eliminated in preparation of the following steps.

9. hifv_semiFinalBPdcals: Semi-final delay and bandpass calibrations

Now, the step 6 is being repeated which results in better bandpass and delay calibration.

CHECK for: similar issues as in step 6.

10. hifv_checkflag: Flag possible RFI on BP calibrator using rflag

Once more rflag is being executed. After the bright rfi has been removed in step 8 and a new bandpass solution has been applied in step 9, a new threshold will also account for weaker rfi to be removed in this step 10.

CHECK for: removal of rfi in the following steps.

11. hifv_semiFinalBPdcals: Semi-final delay and bandpass calibrations

Again, having removed more rfi, new delay and bandpass solutions are being obtained here.

CHECK for: similar to step 6.

12. hifv_solint: Determine solint and Test gain calibrations

For the final calibration, the pipeline determines the shortest and longest applicable solution interval (solint). Typically they refer to the length of an integration and a scan.

Fig. XX12: The hifv_solint task page.

In our case (Fig. XX12) the integration time is 3 seconds which also corresponds the shortest solution interval. The longest solution interval is likely based the phase calibrator scans which typically last for ~85 minutes, minus the drive time and 'quack' flagging, the longest solution results in ~75s.

CHECK for: consistency with the data. The shortest solint should be close to the integration time and the longest to a calibration scan.

13. hifv_fluxboot: Gain table for flux density bootstrapping Now, the fluxes are bootstrapped from the flux calibrator to the complex gain (gain and phase) calibrator. To do so, spectral indices are computed for the secondary calibrator and the absolute fluxes are determined for each channel. They are then set via setjy and reported for each spw.

Fig. XX13: The hifv_fluxboot task page.

For our example, we derive fluxes between 0.61 and 0.68 Jy, as depending on frequency. The spectral behavior is reported as a declining spectral index of around -0.5 (Fig. XX13).

CHECK for: values that are close to the known fluxes of the calibrator. Check the VLA calibrator manual for consistency. Since most calibrator sources are time variable AGN, some differences to the VLA catalog are expected. In particular at higher frequencies they could be up to 50%.

14. hifv_finalcals: Final Calibration Tables The final calibration tables are now being obtained. Those are the most important ones as they are the ones that are being applied to the data in the following step. The tables are (one for each antenna): Final delay plots, BP initial gain phase, BP Amp solution, BP Phase solution, Phase (short) gain solution, Final amp time cal, Final amp freq cal, and Final phase gain cal. We have already inspected and discussed similar plots for the bandpass earlier. Let's now have a look at some of the complex gain/phase calibrator plots in Fig. XX13gains and XX13ph.

Fig. XX13gains: Temporal Gains for ea02.

Fig. XX13gains: Temporal Gains for ea04.

Fig. XX13gains: Temporal Gains for ea07.

Fig. XX13gains: Temporal Gains for ea09.

Fig. XX13gains: Temporal Gains for ea18.

Fig. XX13gains: Temporal Gains for ea23.

Fig. XX13gains: Temporal Gains for ea25.

The gains vary quite a bit for this observation. Typically, the gains stay within 10% around a normalized value of 1. Here, a few spws show

Fig. XX13ph: Temporal Phases for ea02.

Fig. XX13ph: Temporal Phases for ea04.

Fig. XX13ph: Temporal Phases for ea09.

Fig. XX13ph: Temporal Phases for ea13.

Fig. XX13ph: Temporal Phases for ea15.

Fig. XX13ph: Temporal Phases for ea17.

Fig. XX13ph: Temporal Phases for ea20.

CHECK for: issues similar to those described in step 6. Note that carefully checking calibrator tables in this step 14 is of particular importance as they are the final tables to be applied to the target source. Since this step also contains the phase cal solutions, they should be inspected in their temporal variations to be smooth and consistent for each calibrator.

15. hifv_applycals: Apply calibrations from context The calibration itself now concludes with the applycation of the derived calibration tables on the entire dataset. That includes all calibrators as well as the target sources. Note that there's no weighting of the caltables for the VLA (calwt=F) since the swithed power calibration is not being used at this time.

Fig. XX15a: The hifv_fluxboot task page.

Fig. XX15b: The hifv_fluxboot task page.

Fig. XX15c: The hifv_fluxboot task page.

In Fig. XX15, we show the results of this step. The first table shows what tables are being applied, what fields, spws, and antennas are being calibrated. The second table provides information on the flagging statistics. Failed calibration solutions result in flagged calibrator table entries and eventually the data will also be flagged as no calibration can be derived for such data. The following plots show the data of different calibrator source and spw in different plotting versions of phase and amplitude against frequency and uv-distance. To start with, the amp and phase as a function of frequency are being plotted for the complex gain/phase calibrator for each baseband. Next, the amplitudes as a function of uv-distance are plotted for the flux calibrator for each spw. They are followed by amp/time plots for all sources. Finally the amp and phases against time and amp against frequency of the target sources are being plotted for each baseband.

16. hifv_targetflag: Targetflag After the calibration was applied, the automated flagging routing rflag is run one more time on all sources to remove rfi and other outliers from the data.

17. hifv_statwt: Reweight visibilities

Since the VLA pipeline is currently not using the switched power calibration, there can be some sensitivity variations of the data over time, due to changes in opacity, elevation, temperature (gradients) of the antennas, etc. So it is usually advisable to weigh the data according to the inverse of its noise. This step is done via the CASA task statwt and will increase the signal-to noise ratio. Note that features such as rfi spikes and spectral lines will be part of the rms calculations and usually results in downweighting data that includes such features.

18. hifv_plotsummary: VLA Plot Summary

This task produces diagnostic plots of the final data. This includes a calibrator phase for all calibrators as a function of time, and all sources, including calibrators and target as amplitude against uv-distance.

Fig. XX18: The hifv_plotsummary task page.

Fig. XX18 shows that the calibration around 6:00 and 6:30 is still somewhat noise and maybe additional flagging of the calibrators may be required. Field 12, looks quite as expected and one may need to check why some values in field 0 are very low and others in field 11 are quite high. Those could correspond to individual antennas, spws, or polarizations. Also some individual Again, some editing may be required and the pipeline restarted.

19. hif_makeimlist: Compile a list of cleaned images to be calculated

Finally diagnostic images are being made for each spw of the phase calibrator. The images and basic parameters such as resolution (cell size) and image sizes are listed in this step and are available in the directory that the pipeline was run (usually where the SDM is located). At the moment images are being produced for each spw using the multi-frequency synthesis algorithm, ie. the continuum mode that will not have any spectral dependencies anymore but uses the improved uv-coverage due to the frequency bandwidth of each spw.

Fig. XX19: The hifv_makeimlist task page.

In Fig. XX19 the images are listed with 0.31" cell/pixel size ans 300 pixesl on each side. Names and phase centers are given for each spw.

20. hif_makeimages: Calculate clean products

The images from the previous step are shown in the final pipeline task.

Fig. XX20: The hifv_makeimages task page.

Imaging parameters are provided for each image. That contains beam characteristics, as well as image statistics and how they compare to the theoretical noise, the noise just based on bandwidth and integration time for typical VLA array parameters. As mentioned earlier, the score is not fully implemented in this version of the pipeline and should be ignored.

Assuming requirements are met, the pipeline: – Loads the data – Hanning smooths** – Retrieves information about the observing set-up from the data – Applies deterministic flags (online flags, shadowed data, end channels of subbands, etc.) – Identifies primary calibrators and loads models – Derives all prior calibrations (antenna position corrections, gain curves, atmospheric opacity, requantizer gains) – Iteratively determines initial delay and bandpass solutions, including running RFLAG (RFI flagging algorithm), and identifying other system (deformatter) problems – Derives initial gain solutions, does flux density bootstrapping and derives spectral index of all calibrators

• • May want to modify inputs and/or omit entirely for spectral line reductions

=

Heuristics (cont.): the pipeline: – Derives final delay, bandpass, and gain calibrations – Applies all calibrations to the MS – Runs RFLAG algorithm on all fields, including target** – Runs statwt to derive proper relative weights per antenna/spw**

• • May want to modify inputs and/or omit entirely for spectral line reductions

• Pipeline products and output – Flag and calibration tables – Calibrated MS (available for 15 days, not archived) – Logs, including weblog used by quality assurance (QA) staff and QA report if requested

Pipeline Outputs

The real-time pipeline produces a calibrated and flagged MS for download (follow the directions in the email from the data analysts) – You may request a QA2 report from the data analysts – If you are happy with the pipeline calibration, then: • Do further flagging if necessary • Split out your target and image – If you have the SDM or uncalibrated MS and the calibration and flag tables, instructions for applying flags and calibration tables may be found at https://science.nrao.edu/facilities/vla/data-processing/pipeline • In some cases the pipeline and/or the MS may need to be modified – Download the SDM from the archive plus pipeline scripts – Follow the directions at above link • In some cases the pipeline heuristics may not be appropriate for your data (e.g., some L-band set-ups do not work well with the pipeline yet) – Reduce data by hand

Known Issues and Workarounds

In general the pipeline does very well, but there are possible failure modes: – No flux density or gain calibrator intents defined, or flux density calibrator not one for which we have models • work around in scripted pipeline – Wrong scan intents • work around in scripted pipeline – Does not always identify deformatter problems (but does NOT usually have false positives – L-band may be an exception) • flag remaining bad spws – Calibrators are too weak for given spw bandwidth • heuristics have been developed and are currently being implemented

Incorrect scan intents – Best to use the scripted pipeline (otherwise have to edit SDM) – Can run through msinfo.py, then re-set the following string variables to refer to the correct scan and field IDs: flux_field_select_string='2' bandpass_scan_select_string='8' bandpass_field_select_string='4' delay_scan_select_string='8' delay_field_select_string='4' calibrator_scan_select_string='4,5,7,8,10,11,12' calibrator_field_select_string='1,2,3,4,5,6,7' phase_scan_list=[1,3,5,7,9,11,13,15] • If a standard flux density calibrator was not observed, you may still be able to use the pipeline IF you know the flux density and spectral index of one of your other calibrators, with a bit more work – contact the NRAO helpdesk

Accurate flux density bootstrapping – hifv_fluxboot uses medians to bootstrap flux densities: fairly robust, but in some cases (e.g., high frequencies with pointing, elevation dependent gains) you can do better by flagging the gain table used for the bootstrapping – In scripted pipeline, run pipeline through fluxgains.py – Flag gain table using “plotcal” – Similar heuristic for the CASA pipeline currently being tested – With care (match elevation of flux cal,

Spectral Line Data

Several steps in the real-time pipeline may not be appropriate for spectral line data: – Hanning smoothing (increases effective channel width) – Last run of RFLAG on target (may eliminate your line as interference!) – Statwt calculates rms based on scatter of channels per spw, per visibility; may want to run manually with channel selection turned on to eliminate use of channels containing line emission in calculating the rms • With the above modifications, the pipeline will work with spectral line data as long as the calibrators are strong enough

Mixed Correlator Setups

With the new WIDAR capabilities it is common to observe both wide and narrow spws to obtain both continuum and spectral line data simultaneously, or multiple receiver bands – A single heuristic (e.g., gain calibration solution interval) for entire dataset may not be appropriate • Solution: – Run pipeline through application of deterministic flags, including Hanning smoothing if you are going to use it – Split the MS by spw and/or scans – Run pipeline on split MSs WITHOUT Hanning smoothing (you have already applied it, if you are going to use it) – Warning: output flagging statistics may not be correct