.. toctree:: :hidden: calibrator_old .. _calibrator_pipeline: Calibrator pipeline =================== .. note:: If you are running the deprecated genericpipeline version of the pipeline (**prefactor** 3.2 or older), please check the :doc:`old instructions page`. This pipeline processes the calibrator data in order to derive direction-independent corrections. It will take into account the correct order of distortions to be calibrated for. This chapter will present the specific steps of the calibrator pipeline in more detail. All results (diagnostic plots and calibration solutions) will be stored usually in the ``--outdir`` directory specified with your ``cwltool`` or ``toil`` command. .. image:: calibscheme.png Prepare calibrator, incl. "demixing" (``prep``) ----------------------------------------------- This part of the pipeline prepares the calibrator data in order to be calibration-ready. This mainly includes mitigation of bad data (RFI, bad antennas, contaminations from A-Team sources), selection of the data to be calibrated, and some averaging to reduce data size and enhance the signal-to-noise ratio. The user can specify whether to do raw data or pre-processed data flagging and whether demixing should be performed. The basic workflows are: - preparation of data (``prep``) - correcting for polarization alignment (``PA``) - correcting for Faraday Rotation (``FR``) - correcting for bandpass (``BP``) - correcting for ionospheric disturbances (``ion``) The workflow ``prep`` consists of: - determining suitable calibrator skymodel (steps ``find_skymodel_cal``, ``make_sourcedb``) - checking for nearby A-Team sources (step ``check_Ateam_separation``) - creating a model of A-Team sources to be subtracted (step ``make_sourcedb_ateam``) - basic flagging and averaging (subworkflow ``dp3_prep_cal``) - edges of the band (``flagedge``) -- only used if ``raw_data : true`` - statistical flagging (``aoflag``) -- only used in ``raw_data : true`` - baseline flagging (``flagbaseline``) - low elevation flagging (below 15 degress elevation) (``flagelev``) - low amplitude flagging (below 1e-30) (``flagamp``) - demix A-Team sources (``demix``) -- only used if specified ``demix : true`` - averaging of the data in time and frequency - wide-band statistical flagging (steps ``ms_concat`` and ``aoflag``) - write the calibrator skymodel into the ``MODEL_DATA`` column (step ``predict``) and perform direction-independent phase-only calibration (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``predict_cal``, baseline-dependent smoothing (step ``BLsmooth``) if specified ``do_smooth : true``) Calibration of the polarization alignment (``PA``) -------------------------------------------------- The phase solutions derived from the preparation step are now collected and loaded into **LoSoTo**. **LoSoTo** will derive the polarizion alignment and provide diagnostic plots under ``inspection/``: - ``polalign_ph_pol??.png``: matrix plot of the phase solutions for the XX and YY polarization .. image:: polalign_ph_polXX.png - ``polalign_ph_poldif.png``: matrix plot of the phase solutions from XX-YY .. image:: polalign_ph_poldif.png - ``polalign_rotangle.png``: matrix plot of the common rotation angle solutions .. image:: polalign_rotangle.png - ``polalign_amp_pol??.png``: matrix plot of the amplitude solutions for the XX and YY polarization .. image:: polalign_amp_polXX.png - ``polalign.png``: matrix plot of the derived polarization alignment between XX and YY .. image:: polalign.png - ``polalign_ph-res_pol??.png``: matrix plot of the residual phase solutions for the XX and YY polarization after subtraction the derived polarization alignment - ``polalign_ph-res_poldif.png``: matrix plot of the residual phase solutions for XX-YY after subtraction of the derived polarization alignment The workflow ``PA`` consists of: - deriving the polarization alignment from the calibration solutions (subworkflow ``PolAlign``) - creating diagnostic plots (steps ``losoto_plot``) - applying polarization alignment solutions (step ``applyPA``) and the element beam correction (step ``applybeam``) to the original data and re-calibrate (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``apply_calibrate_pa``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``) Calibration of the Faraday Rotation (``FR``) -------------------------------------------- The outcome of the re-calibration **after** correcting for the polarization alignment is again loaded into **LoSoTo** in order to derive corrections for Faraday Rotation. The following diagnostic plots are created: - ``fr_ph_pol??.png``: matrix plot of the phase solutions for the XX and YY polarization - ``fr_ph_poldif.png``: matrix plot of the phase solutions from XX-YY - ``fr_rotangle.png``: matrix plot of the common rotation angle solutions - ``fr_amp_pol??.png``: matrix plot of the amplitude solutions for the XX and YY polarization - ``fr``: matrix plot of the derived differential Rotation Measure from Faraday Rotation .. image:: fr.png The workflow ``FR`` consists of: - deriving the Faraday Rotation from the calibration solutions (subworkflow ``FaradayRot``) - creating diagnostic plots (steps ``losoto_plot``) - applying Faraday Rotation solutions (step ``applyFR``) and re-calibrate (diagonal terms + common rotation angle, step ``calib_cal``) (subworkflow ``apply_calibrate_fr``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``) Calibration of the Bandpass (``BP``) ------------------------------------ The outcome of the re-calibration **after** correcting for the polarization alignment and Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the bandpass. A robust flagging on the amplitude solutions as well as a Savitzky-Golay filter is applied in order to reject bad solutions and smooth the outcome. Frequency regimes up to a certain maximum width (parameter ``max2interpolate``) will be interpolated if flagged. The following diagnostic plots are created: - ``ampBFlag_??.png``: matrix plot of the amplitude solutions for the XX and YY polarization ``before`` flagging .. image:: ampBFlag_polXX.png - ``ampAFlag_??.png``: matrix plot of the amplitude solutions for the XX and YY polarization ``after`` flagging .. image:: ampAFlag_polXX.png - ``bandpass_pol??.png``: the derived bandpass of all stations in the XX and YY polarization - ``bandpass_time??.png``: matrix plot of the derived bandpass, where both polarizations are colorcoded .. image:: bandpass.png - ``bandpass_time??_pol??.png``: plot of the derived bandpass of the XX and YY polarization, where all stations are colorcoded .. image:: bandpass_polXX.png The workflow ``BP`` consists of: - deriving the bandpass from the calibration solutions (subworkflow ``bandpass``) - creating diagnostic plots (steps ``losoto_plot``) - transfer solutions for international stations for non-trusted calibrator sources (step ``transfer_solutions``), see parameters ``do_transfer`` and ``trusted_sources`` - applying polarization alignment solutions (step ``applyPA``), the bandpass (step ``applyBP``), the element beam correction (step ``applybeam``) and the Faraday Rotation solutions (``applyFR``) to the original data and calibrate (only single scalar phase, step ``calib_cal``) (subworkflow ``apply_calibrate_bp``, baseline-dependend smoothing (step ``BLsmooth``) if specified ``do_smooth : true``) - deriving final amount of flags applied to the data (step ``final_flags``) Calibration of the instrumental and ionospheric delays (``ion``) ---------------------------------------------------------------- The outcome of the re-calibration **after** correcting for the polarization alignment, the bandpass and the Faraday Rotation is loaded into **LoSoTo** in order to derive corrections for the instrumental and ionospheric delays. A robust flagging on the amplitude solutions is applied in order to reject bad solutions. These flags are applied to the phase solutions. These phase solutions should be mainly affected by instrumental (clock) and ionospheric (TEC) delays. This **LoSoTo** step will aim to separate both effects (clock-TEC separation). The following diagnostic plots are created: - ``ion_ph.png``: matrix plot of the phase solutions - ``clock.png``: matrix plot of the derived (instrumental) clock offsets in seconds .. image:: clock.png - ``tec.png``: matrix plot of the derived differential TEC in TECU .. image:: tec.png - ``ion_ph-res.png``: matrix plot of the residual phase solutions after the subtraction of the derived instrumental and ionospheric delays The workflow ``ion`` consists of: - performing clock-TEC-separation from the calibration solutions (subworkflow ``clocktec``) - creating diagnostic plots (steps ``losoto_plot``) - create a summary file (step ``summary``) .. note:: All solutions are written in the h5parm file format via the steps ``H5parm_collector`` and called during all the workflows. The solutions are stored in the final calibrator solution set ``cal_solutions.h5``. Further diagnostics ------------------- The results directory will contain all relevant outputs of the current **LINC** run, once the pipeline has finished: - logfiles in ``logs`` - summary file (JSON format) in ``??_LINC_calibrator_summary.json`` - calibration solutions in ``cal_solutions.h5`` - inspection plots in ``inspection`` ``Ateam_separation.png`` shows the distance and the elevation of A-Team sources with respect to the analyzed observation. You can also check the calibration solutions for more details:: $ losoto -i cal_solutions.h5 Summary of cal_solutions.h5 Solution set 'calibrator': ========================== Directions: 3c286 Stations: CS001HBA0 CS001HBA1 CS002HBA0 CS002HBA1 CS003HBA0 CS003HBA1 CS004HBA0 CS004HBA1 CS005HBA0 CS005HBA1 CS006HBA0 CS006HBA1 CS007HBA0 CS007HBA1 CS011HBA0 CS011HBA1 CS017HBA0 CS017HBA1 CS021HBA0 CS021HBA1 CS024HBA0 CS024HBA1 CS026HBA0 CS026HBA1 CS028HBA0 CS028HBA1 CS030HBA0 CS030HBA1 CS031HBA0 CS031HBA1 CS032HBA0 CS032HBA1 CS101HBA0 CS101HBA1 CS103HBA0 CS103HBA1 CS201HBA0 CS201HBA1 CS301HBA0 CS301HBA1 CS302HBA0 CS302HBA1 CS401HBA0 CS401HBA1 CS501HBA0 CS501HBA1 RS106HBA RS205HBA RS208HBA RS210HBA RS305HBA RS306HBA RS307HBA RS310HBA RS406HBA RS407HBA RS409HBA RS503HBA RS508HBA RS509HBA Solution table 'bandpass' (type: amplitude): 120 times, 11 freqs, 60 ants, 2 pols Flagged data: 0.000% Solution table 'clock' (type: clock): 120 times, 60 ants Flagged data: 0.000% Solution table 'faraday' (type: rotationmeasure): 60 ants, 120 times Flagged data: 0.222% Solution table 'polalign' (type: phase): 120 times, 60 ants, 40 freqs, 2 pols Flagged data: 0.000% For an overall summary it is advised to check the summary logfile:: $ cat logs/3c286_summary.log **************************************** *** LINC calibrator pipeline summary *** **************************************** Field name: 3c286 User-specified baseline filter: *& Additional antennas removed from the data: NONE A-Team sources close to the phase reference center: VirA Of which were demixed: NONE Of which were clipped: NONE Amount of flagged solutions per station and solution table: Station bandpass clock faraday polalign CS001HBA0 9.08% 100.00% 0.00% 0.00% CS001HBA1 9.08% 100.00% 0.00% 0.00% CS002HBA0 9.08% 100.00% 0.00% 0.00% CS002HBA1 9.08% 100.00% 0.00% 0.00% CS003HBA0 9.08% 100.00% 0.00% 0.00% CS003HBA1 9.08% 100.00% 0.00% 0.00% Amount of flagged data per station at a given state: Station initial final CS001HBA0 3.18% 3.96% CS001HBA1 2.98% 3.97% CS002HBA0 3.18% 4.42% CS002HBA1 2.95% 3.67% CS003HBA0 2.96% 3.94% CS003HBA1 3.10% 4.21% ********** Summary file is written to: 3c286_LINC_calibrator_summary.json Summary has been created. User-defined parameter configuration ------------------------------------ **Parameters you will need to adjust** *Location of the calibrator solutions* - ``msin``: location of the input calibrator data, for instructions look at the :doc:`configuration instructions` page **Parameters you may need to adjust** *Data selection and calibration options* - ``refant``: regular expression of the stations that are allowed to be selected as a reference antenna by the pipeline (default: ``CS00.*``) - ``flag_baselines``: DP3-compatible pattern for baselines or stations to be flagged (may be an empty list, i.e.: ``[]`` ) - ``process_baselines_cal``: performs A-Team-clipping/demixing and direction-independent phase-only self-calibration only on these baselines. Choose ``[CR]S*&`` if you want to process only cross-correlations and remove international stations (default: ``*&``) - ``filter_baselines``: selects only this set of baselines to be processed. Choose ``[CR]S*&`` if you want to process only cross-correlations and remove international stations (default: ``*&``) - ``do_smooth``: enable or disable baseline-based smoothing (default: ``false``) - ``rfistrategy``: strategy to be applied with the statistical flagger (`AOFlagger`_, default: ``$LINC_DATA_ROOT/rfistrategies/lofar-hba-wideband.lua``) - ``max2interpolate``: amount of channels in which interpolation should be performed for deriving the bandpass (default: 30) - ``fit_offset_PA``: assume that together with a delay each station has also a differential phase offset (important for old LBA observations, default: ``false``) - ``interp_windowsize``: size of the window over which a value is interpolated. Should be odd. (default: 15) - ``ampRange``: range of median amplitudes accepted per station (default: ``[0,0]``) - ``skip_international``: skip fitting the bandpass for international stations (this avoids flagging them in many cases, default: ``true``) - ``raw_data``: use autoweight, set to True in case you are using raw data (default: ``false``) - ``propagatesolutions``: use already derived solutions as initial guess for the upcoming time slot (default: ``true``) - ``flagunconverged`` : flag solutions for solves that did not converge (if they were also detected to diverge, default: ``false``) - ``maxStddev``: maximum allowable standard deviation when outlier clipping is done. For phases, this should value should be in radians, for amplitudes in log(amp). If None (or negative), a value of 0.1 rad is used for phases and 0.01 for amplitudes (default: ``-1.0``) - ``solutions2transfer``: provide own solutions from a reference calibrator observation in case calibrator source is not trusted - ``antennas2transfer``: DP3-compatible baseline pattern for those stations who should get calibration solutions from a reference solution set in case calibrator source is not trusted (default: ``[FUSPID].*``) - ``do_transfer``: enable solutions transfer for non-trusted calibrator sources (default: ``false``) - ``trusted_sources``: comma-separated list of trusted calibrator sources. Solutions are only transferred from a reference solution set in case the observed calibrator is not among them (default: ``3C48,3C147,3C196,3C295,3C380``) - ``ion_3rd``: take into account also 3rd-order effects for the clock-TEC separation (ionospheric calibration, default: ``false``) - ``clock_smooth``: only take the median of the derived clock solutions (enable this in case of non-joint observations, default: ``true``) - ``remove_phase_wraps``: detect and remove phase wraps in the clock-TEC separation (default: ``false``) A comprehensive explanation of the baseline selection syntax can be found `here`_. *Demixing options* (only used if demix step is added to the ``prep_cal_strategy`` variable) - ``demix_sources``: choose sources to demix (provided as list), e.g., ``[CasA,CygA]`` - ``demix_freqres``: frequency resolution used when demixing (default: 48.82kHz, which translates to 4 channels per subband) - ``demix_timeres`` : time resolution used when demixing in seconds (default: 10) - ``demix``: if ``true`` force demixing using all sources of ``demix_sources``, if ``false`` do not demix (default: ``null``, automatically determines sources to be demixed according to ``min_separation``) - ``lbfgs_historysize``: for the LBFGS solver: the history size, specified as a multiple of the parameter vector, to use to approximate the inverse Hessian (default: 10) - ``lbfgs_robustdof``: for the LBFGS solver: the degrees of freedom (DOF) given to the noise model (default: 200) *Further pipeline options* - ``min_separation``: minimum accepted distance to an A-team source on the sky in degrees (will raise a WARNING, default: 30) - ``tables2export``: choose which tables to export to the solutions file after the ionospheric calibration (default: ``clock``) **Parameters for pipeline performance** - ``max_dp3_threads``: number of threads per process for DP3 (default: 10) - ``memoryperc``: maximum of memory used for aoflagger in raw_flagging mode in percent (default: 20) - ``aoflag_reorder``: make aoflagger reorder the measurement set before running the detection. This prevents that aoflagger will use its memory reading mode, which is faster but uses more memory (default: false, see the `AOFlagger manual`_`) - ``aoflag_chunksize``: this will split the set into intervals with the given maximum size, and flag each interval independently. This lowers the amount of memory required (default: 2000) **Parameters you may want to adjust** *Skymodel directory* - ``calibrator_path_skymodel``: location of the calibrator skymodels - ``max_separation_arcmin``: maximum separation between phase center of the observation and the patch of a calibrator skymodel which is accepted to be chosen as a skymodel (default: 1.0) - ``A-Team_skymodel``: location of the A-Team skymodels *Averaging for the calibrator data* - ``avg_timeresolution``: final time resolution of the data in seconds after averaging (default: 4) - ``avg_freqresolution`` : final frequency resolution of the data after averaging (default: 48.82kHz, which translates to 4 channels per subband) - ``bandpass_freqresolution``: frequency resolution of the bandpass solution table (default: 195.3125kHz, which translates to 1 channel per subband) .. _here: https://dp3.readthedocs.io/en/latest/steps/Description%20of%20baseline%20selection%20parameters.html .. _AOFlagger: https://gitlab.com/aroffringa/aoflagger.git .. _AOFlagger manual: https://aoflagger.readthedocs.io/en/latest/using_aoflagger.html