[News] Report on the visit of Roeland to OAC
Roberto Silvotti
silvotti@na.astro.it
Wed, 14 May 2003 15:35:11 +0200
This is a multi-part message in MIME format.
--------------B3FA9336B5BCC4061037AD63
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Hi all,
Attached to this message you can find the report of the visit
of Roeland Rengelink to OAC last week (April 7-9, 2003).
Best regards,
Roberto
----------------------------------------
Dr. Roberto Silvotti
----------------------------------------
INAF (Istituto Nazionale di AstroFisica)
Osservatorio Astronomico di Capodimonte
via Moiariello 16, I-80131 Napoli, Italy
----------------------------------------
tel/fax: +39-081-5575583/456710
e-mail: silvotti@na.astro.it
web: http://www.na.astro.it/~silvotti
----------------------------------------
--------------B3FA9336B5BCC4061037AD63
Content-Type: text/plain; charset=us-ascii;
name="roeland_may03_out.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="roeland_may03_out.txt"
______________________________________________________________
Report on the visit of Roeland Rengelink to OAC, May 7-9, 2003
______________________________________________________________
Wednesday 7 afternoon
---------------------
Demonstrations:
1) Prereduction of WFI data with OACDF algorithms and Alfredo's tools:
bias, FF and fringing correction using sigmaclip and 2d fitting tool (cubic
spline).
2) Quality control: the sign test tool (works fine with uncorrelated noise).
Discussion on the two demo:
1) * calibration pipeline: what is still lacking and what needs to be improved;
* need of more detailed (and more concentrated) documentation on the
algorithms used.
It has been agreed that a small document containing the precise algorithms
and the list of parameters used in each step of the calibration pipeline
will be produced soon (Roeland et/or al.), following more or less the
scheme used for the sigmaclip. Although part of these info is already
present in the OmegaCAM docs, it will be very useful for the AW people
to have all these infos concentrated in one small document, which might
be included as a separate section in Manual.tex.
* Comparative tests to select the best strategies on FF: although for a
complete comparison we would need to measure the quality of the final
photometry, something may be yet done now (flatness, rms).
* Focus on fringing and in particular on the scaling coefficient calculation:
OAC has further improved the method used in the OACDF reductions.
In the new version the scaling factor is proportional to the background of
each scientific frame, after a rough source screening.
OAC will complete the tests on this method, produce the documentation and
deliver.
* Sigmaclip integration (see next session).
2) See last session (friday 9 afternoon).
Thursday 8 morning
------------------
Working together on the calibration pipeline: the sigmaclip has been introduced
into BIAS, dome flat and twilight flat procedures (Roeland, Alfredo, Mike)
Sigmaclip will be introduced into superflat too at Groningen next week by
Mike after discussion with Ewout (taking into account normalization etc. ...).
Thursday 8 afternoon
--------------------
General discussion on WP1. Items:
- pipeline installation: it is proposed to develop a script to help in the
pipeline installation (OAC, Mike); this script requires a formal descriptor
of the package dependencies. This means that all the changes in the external
packages must be reflected into this descriptor.
The goal of checking dependencies is partially covered by dependency_test.py,
which presently verifies only if all the needed software is present or not.
- High level scripts and automatization vs command line: it has been shown by
Roeland that it is possible, for example, to perform master BIAS with 3
command lines, including DB query on raw frames, retrieve the raw files, store
the master BIAS produced and update the DB.
It is clear however that at a certain point we will need high level scripts
in order to handle large sets of data.
- Interface between pipeline and storage system: an experimental version is
present in the pipeline.
- Parallelization: Roeland showed us that some mechanisms already exist to
submit tasks to the BEOWULF nodes. More detailed discussions are needed.
- List of attributes: a first version is present.
- Fine tuning or the reduction parameters by the end user (configuration file):
there are presently python scripts that concentrate all the parameters of one
reduction step (e.g. BIAS). We think that to concentrate the parameters in
one place will certainly help the fine tuning of the pipeline and we think
that this concept should be further developed.
Although it is reasonable to imagine that the total number of parameters that
should be modified by the end user will be quite small when the system will
reach its final configuration, we must consider that during the commissioning
phase many parameters will change oftenly.
- Photometric calibrations: chip by chip zeropoints vs global zero point.
It will be necessary to check whether the differences between the zero points
of each CCD are stable in time (as we can expect).
Detailed discussion on the photometric calibrations will be needed in the
near future.
- Astrometry: chip by chip vs. global astrometry.
Although we understand that it is easier at this stage to calculate the
astrometric solution chip by chip in order to maintain the parallel structure
of the pipeline, we think that this is a crucial point that must be discussed
in detail soon.
Friday 9 morning
----------------
Running the pipeline: unit tests etc. ...
Few errors on the local pipeline from regression_test.py are probably due to
the local python version (2.2.1). We will update to 2.2.2.
It will be necessary, at a certain point, to fix one python version for the
pipeline. From that point on, we will have to guarantee that all the future
versions of the pipeline are still compatible to that version of python.
During the implementation phase, it would be very useful to inform people
whenever it is necessary to install new versions of the software (everything
which is not included on a cvs update).
Of course this becomes difficult when new versions come out every week or so ...
The dependencies test scripts will help a lot to solve this problem.
Friday 9 afternoon: brain storm on QC
-------------------------------------
Different QC flags, single chip flags and global frame flags: one point for
further discussion is how to deal with frames having bad flags only in one
(or a few) single CCD(s).
Comparison between one frame and a previous frame (frame.compare) and between
two or more frames.
Sign test: it is extremely sensitive to many kind of problems (structures in a
wide range of frequencies) and works quite well provided that the pattern of
the calibration files is sufficiently constant on time. It may be applied to
the difference between one BIAS and a reference BIAS and/or to the ratio between
one flat and a reference flat.
We agreed to prepare here (Roberto) a list of possible problems for the
calibration images, at least for what concerns an astronomer's point of view.
Then the list will be circulated in order to add other possible problems,
also regarding electronics.
Based on this list, we will define a limited number of simple QC tools that
check the quality of the BIAS and the flat fields (both dome and twilight).
Every tool measures a particular well defined property and returns one flag.
Each raw frame will be accepted only if all the flags are ok.
For example, for the FFs we could have these different tools:
1) check that the FF is not saturated: average (or median) must be below
a fixed number of counts (already implemented).
2) check that the flatness is below a fixed threshold: this could be achieved
by a low degree polynomial (or cubic spline) fit. In this way we could put a
threshold on the parameters of the polynomial or put a threshold on the rms
of the fitting surface (which means to separate the shoot noise component in
the fitted frame from that one due to the nonflatness).
As a third test, we could compare with a reference FF.
Another possibility is to use a "feed-back" system where, after the production
of the master FF, each raw FF is divided by the master and we put constraints
on this ratio.
If the constraints are not satisfied for one (or more) ratio frame, then we
produce again the master without using the corresponding bad raw frame(s).
Although this system might be more expensive in terms of computing time, this
is not necessarily true, depending on how frequently it actually happens
that the master is recalculated.
--------------B3FA9336B5BCC4061037AD63--