VLA CASA Pipeline-CASA4.5.3: Difference between revisions

From CASA Guides
Jump to navigationJump to search
(Created page with "== Introduction == == Data == == Overview of the Pipeline procedures == == Pipeline Requirements == == Running the Pipeline == == Assessing the Weblog == == Pipeline Ou...")
 
Line 1: Line 1:
== Introduction ==
== Introduction ==
• With the start of Jansky VLA Full Operations (January 2013), we started a
new operational model:
– Deliver flagged and calibrated visibility data
– You will self-calibrate and image visibility data to meet science goals, using
resources at home institution or NRAO computing resources
• Automated pipeline should run correctly on all “standard” Stokes I science
SBs; “standard” means:
– 128 MHz spws, but may work on other set-ups as well
• Some constraints on strength of calibrators needed
– Contains correctly labeled and complete scan intents
• Current versions available:
– “scripted” pipeline is a collection of python scripts that use CASA tasks
wherever possible, but also uses toolkit calls; readable and easy to modify
– CASA integrated pipeline is compatible with ALMA pipeline infrastructure,
improved diagnostics in weblog, used as real-time pipeline since Sep 2015
=
• Real-time pipeline:
– Minimal human intervention
• Pipeline is run automatically on every science SB as it completes (not just
“continuum”)
– Pipeline output undergoes quality assurance checks by NRAO staff upon
request; reports generated are archived as pipeline products
• At your home institution:
– Instructions for installation and operation of the VLA CASA Calibration
Pipeline are available at
https://science.nrao.edu/facilities/vla/data-processing/pipeline
• Uses CASA 4.3.1, similar to current real-time pipeline
• CASA 4.5.2 currently being validated (you are helping with this!)
• Scripted pipelines for CASA versions through 4.5.0 also available
– Provides more flexibility in how to use the pipeline, options suitable for
spectral line datasets, mixed correlator set-ups, multi-band observations, etc.
– Working to incorporate these into the CASA integrated pipeline
=
“Standard” Stokes I science SB means:
– 128 MHz spws, but may work on other set-ups as well
• Can work for narrower BWs, depends on the strength of the calibrators
• Heuristics currently make some assumptions about the strength of the calibrators,
in particular, the delay calibrator
– Contains correctly labeled and complete scan intents
• And also that the observation has been set up correctly!
• Will the pipeline work for you?
– The pipeline successfully completes on ~95% of all science SBs observed on
the VLA; whether the output can be used for science depends on the science
goal, and whether the observation was correctly set up
• Pipeline includes Hanning smoothing, RFI flagging, and weight calculations that may
not be appropriate for spectral line projects (but can modify scripted pipeline)
• No polarization calibration (yet) but can use pipeline output as starting data for
pol. cal.
• Will probably work well for data taken since May 2012, may work for earlier EVLA
data, likely that extra flagging may be needed in these cases
=
Calibrator strength:
– Conservative limit on
strength of BP and
complex gain calibrators
can be derived from
requirement for initial gain
calibration to work at high
end of Q-band
– Heuristic for delay
calibration currently
requires the SNR=3 limit
on initial gain calibration
per integration


== Data ==
== Data ==

Revision as of 18:28, 2 May 2016

Introduction

• With the start of Jansky VLA Full Operations (January 2013), we started a new operational model: – Deliver flagged and calibrated visibility data – You will self-calibrate and image visibility data to meet science goals, using resources at home institution or NRAO computing resources • Automated pipeline should run correctly on all “standard” Stokes I science SBs; “standard” means: – 128 MHz spws, but may work on other set-ups as well • Some constraints on strength of calibrators needed – Contains correctly labeled and complete scan intents • Current versions available: – “scripted” pipeline is a collection of python scripts that use CASA tasks wherever possible, but also uses toolkit calls; readable and easy to modify – CASA integrated pipeline is compatible with ALMA pipeline infrastructure, improved diagnostics in weblog, used as real-time pipeline since Sep 2015

=

• Real-time pipeline: – Minimal human intervention • Pipeline is run automatically on every science SB as it completes (not just “continuum”) – Pipeline output undergoes quality assurance checks by NRAO staff upon request; reports generated are archived as pipeline products • At your home institution: – Instructions for installation and operation of the VLA CASA Calibration Pipeline are available at https://science.nrao.edu/facilities/vla/data-processing/pipeline • Uses CASA 4.3.1, similar to current real-time pipeline • CASA 4.5.2 currently being validated (you are helping with this!) • Scripted pipelines for CASA versions through 4.5.0 also available – Provides more flexibility in how to use the pipeline, options suitable for spectral line datasets, mixed correlator set-ups, multi-band observations, etc. – Working to incorporate these into the CASA integrated pipeline

=

“Standard” Stokes I science SB means: – 128 MHz spws, but may work on other set-ups as well • Can work for narrower BWs, depends on the strength of the calibrators • Heuristics currently make some assumptions about the strength of the calibrators, in particular, the delay calibrator – Contains correctly labeled and complete scan intents • And also that the observation has been set up correctly! • Will the pipeline work for you? – The pipeline successfully completes on ~95% of all science SBs observed on the VLA; whether the output can be used for science depends on the science goal, and whether the observation was correctly set up • Pipeline includes Hanning smoothing, RFI flagging, and weight calculations that may not be appropriate for spectral line projects (but can modify scripted pipeline) • No polarization calibration (yet) but can use pipeline output as starting data for pol. cal. • Will probably work well for data taken since May 2012, may work for earlier EVLA data, likely that extra flagging may be needed in these cases

=

Calibrator strength: – Conservative limit on strength of BP and complex gain calibrators can be derived from requirement for initial gain calibration to work at high end of Q-band – Heuristic for delay calibration currently requires the SNR=3 limit on initial gain calibration per integration

Data

Overview of the Pipeline procedures

Pipeline Requirements

Running the Pipeline

Assessing the Weblog

Pipeline Outputs

Re-running the pipeline

Applying Pipeline Results

Known Issues and Workarounds

Scripted Pipeline

Using the NRAO cluster batch processing