[News] Some issues
Roberto Silvotti
silvotti at na.astro.it
Thu Nov 13 12:47:42 CET 2003
Hi Mark,
Thanks for your answer.
By the way we had problems with the network yesterday and that's
why I reply only now.
My impression is that we need real comparison between AW pipeline
and others (IRAF, Bonn, etc. ..), in particular for what concerns
pre-reduction (or calibration pipeline in AW terms) and photometric
pipeline (for this point we will better understand the situation in
the next week).
These are the goals of our comparisons between AW and IRAF reductions
on the OACDF (OAC Deep Field) data that we started with Philippe
Heraudeau.
On the other hand, for what concerns astyrometry, we are probably not
yet
in a position that allows real comparison as it is clear that
independent astrometric solutions for each CCD can not be good enough.
So we will have to wait until a global astrometric solution will be
implemented in the AW pipeline before being able to do real tests.
Concerning astrometry, I add a comment from Mario Radovich at the end
of this message.
Cheers and see you in Groningen,
Roberto
> Hi Roberto,
>
> here some answers to your questions:
>
> On Tue, 11 Nov 2003, Roberto Silvotti wrote:
>
> > Dear Mark,
> >
> > As we are planning to compare the AW pipeline results on the
> > Capodimonte Deep Field WFI data obtained by Philippe Héraudeau
> > with the results that we obtained here with IRAF,
> > I would like to ask you a few more details on your tests on AW
> > pipeline vs. Bonn pipeline vs. IRAF, in order to better understand
> > what you did in Munich and use at best all these tests during our
> > future discussions next week in Groningen.
> >
> >
> > > Hello All,
> > >
> > > As a test of data reduction routines (astro-wise vs. Bonn pipeline vs.
> > > IRAF) we have immersed ourselves (primarily Jan Snigula and
> > > Yuliana Goranova) in the astro-wise pipeline using our WFI data.
> > >
> > > Here some results/comments:
> > >
> > > - Dataset used for testing: WiFI- images taken over 6 nights
> > > (approx. 50GB).
> > >
> > > Time for testreduction: approx. 1 week (most of the delays were caused
> > > by unannounced pipeline changes occurring during the reduction.
> > >
> > > Details:
> > >
> > > - COSMICFILTER:
> > > We tried to implement our cosmic filtering routine (cosmicfits from
> > > Claus Goessl) in the pipeline. This attempt failed, due to the
> > > current handling of bad columns/pixels in the images and the
> > > existance of negative pixel values. The latter problem can be
> > > circumvented in the program call, but the bad columns/pixels would
> > > need to be replaced (PRIOR to executing our cosmic filter routine) with
> > > a well defined NaN value, e.g. 0. Essentially, our cosmic filter
> > > routine treats the bad columns as cosmic rays and spends an inordinate
> > > amount of time trying to detect/correct them. It would be easy to
> > > mask these as NaN's prior to running "cosmicfits".
> > >
> > > - BAD COLUMNS:
> > > Bad colums in general must be taken care of at an earlier stage of
> > > the pipeline than currently implemented (see example ps
> > > file). This was already discussed by Mark and Roeland.
> > >
> > > - SWARP:
> > > The pipeline currently uses a beta version of Swarp (swarp 2.0b),
> > > that fails to create usable weight images. Reasonable weight
> > > images are essential for obtaining a fast run-time with cosmicfits,
> > > and are crucial when the frames on which cosmic rays are detected
> > > have strong intensity slopes.
> > >
> > > - UPDATES:
> > > Unannounced pipeline changes that change object definitions, require
> > > changes to the database, that can be time consuming, given the fact,
> > > that we have to figure out the changes from the code, and guess
> > > the needed Database changes. Usually Danny sends out an e-mail
> > > explaining the required changes (usually after talking with Roeland
> > > about the changes in the pipeline), but these e-mail come about 2
> > > days after the changes, causing severe interruptions.
> > >
> > > Suggestion: Create a stable branch of the pipeline in the CVS with
> > > weekly? updates from a development branch, and the changes in these
> > > updates should be discussed with the DBAs before, so that updates to
> > > the pipeline cause as few interruptions as possible.
> > >
> > > - TESTS:
> > >
> > > - Bias:
> > >
> > > Subtracting resulting masterbias from its raw input frames
> > >
> > > rawbias file name median stddev average stddev
> > > seWFI.2003-01-05T20C54C42.283_4.fits -0.199997 18.876 -0.244914 18.8759
> > > seWFI.2003-01-05T20C55C44.983_4.fits -0.100006 18.9255 -0.161555 18.9254
> > > seWFI.2003-01-05T20C56C42.354_4.fits 0.100006 18.89 0.00905352 18.8897
> > > seWFI.2003-01-05T20C57C39.851_4.fits 0.100006 18.9164 -0.00907611 18.916
> > > seWFI.2003-01-05T20C58C34.232_4.fits 0 18.8818 -0.0418041 18.8818
> > > seWFI.2003-01-05T20C59C33.157_4.fits 0 18.881 -0.0651421 18.8809
> > > seWFI.2003-01-05T21C00C32.066_4.fits 0 18.8516 -0.0559038 18.8515
> > > seWFI.2003-01-05T21C01C33.362_4.fits 0 18.888 -0.0407198 18.888
> > > seWFI.2003-01-05T21C02C30.338_4.fits 0.100006 18.8894 0.000104276 18.8891
> > > seWFI.2003-01-05T21C03C29.919_4.fits 0.100006 18.9252 0.0489991 18.9252
> > >
> > > (Is there a numerical round-off occurring in the median computation?).
> >
> > I suppose that the master was calculated with the AW pipeline.
> > But it is not clear to me what these numbers tell us: just that each raw
> > frame
> > has a median or average very similar to the master ?
>
> Yes, the master was calculated with the AW pipeline. We were not looking
> for anything deeply philosophical in these numbers, but just wanted to
> test the pipeline computations. We wanted to be sure that the final
> master bias resulted in a median that was comparable to the input frames.
> The number confirm this.
>
> >
> > > - DomeFlat:
> > >
> > > Flatfielding biassubtracted raw domeflatframes using the processed
> > > domeflat. ( SEM = std. error)
> > >
> > > flatfielded raw flatfield filename median stddev average stddev
> > > dseWFI.2003-01-06T21C59C26.278_4.fits 17778.8 90.3731 17778.1 90.3702
> > > dseWFI.2003-01-06T22C00C53.969_4.fits 19056.3 94.5017 19055.6 94.4991
> > > dseWFI.2003-01-06T22C02C15.050_4.fits 19074.4 94.4472 19073.4 94.4418
> > >
> > > - TwilightFlats:
> > >
> > > Flatfielding biassubtracted raw twilightflatframes using the processed
> > > twilightflat. ( SEM = std. error)
> > >
> > > flatfielded raw flatfield filename median stddev average stddev
> > > dseWFI.2003-01-04T00C17C43.159_4.fits 17885.3 122.664 17886 122.662
> > > dseWFI.2003-01-04T00C19C14.085_4.fits 17161.4 134.724 17161.9 134.723
> > > dseWFI.2003-01-04T00C21C01.056_4.fits 16581.6 151.299 16582.4 151.297
> >
> > Here again I do not see what these numbers tell us.
> > The stddev are relatively small but, for example, we are not able
> > to say anything about flatness.
>
> Again, we just wanted to see that the numbers are reasonable. We were not
> looking for flatness.
>
> >
> > > - Astrometry:
> > > From visual inspection of stars from the USNO catalog overplotted
> > > on the image, the astrometric solution determined by the pipeline
> > > seems to be very accurate even out to the edges.
> > >
> > > Hard numbers:
> > > mean position differences in arcsec:
> > > RA DEC
> > > -0.0002 0.0001
> > >
> > > mean sigma of the position differences in arcsec:
> > > RA DEC
> > > 0.4199 0.4021
> >
> > For what concerns astrometry, the mean differences very close to zero
> > tell us
> > that there are no offsets in the RA and DEC.
>
> Exactly. Since the frames were dithered, these numbers are good. We
> just wanted to confirm that the astrometric solution produced by LDAC
> was good.
>
> > It is less clear the meaning of the mean sigmas:
> > 0.4 arcsec is an accuracy of the same order of the USNO catalogue
> > (~0.3).
> > So this values are normal if they represent absolute accuracy.
> > But they would be not good enough for our purposes if they represent the
> > relative
> > accuracy, for example the difference between one dithering and another.
> > So it would be interesting to known exactely what are these numbers.
>
> Clearly, the relative accuracy of our frames (the FIRST numbers) is
> extremely good, at levels of about 0.001 pixels. . . more than adequate
> for anything we need.
> The SECOND numbers are the absolute errors. Since the pmm 1 catalogue
> has rms errors of order 0.3 to 0.4 arcsec, this will be our limit in
> absolute terms.
>
> Cheers,
>
> Mark
As it concerns the astrometry, the mean position difference does not
give the
relative accuracy between the frames: it just gives the average offset,
one
could well have a very bad relative accuracy and very small offset.
The relative accuracy should be obtained by comparing the position of
sources
between two frames in the dithering and computing the RMS (again, not
the
offset). I expect that this will not be anyway much better than 0.3
arcsec,
as this is the accuracy of the USNO: this means that for only roughly
63% of
sources (assuming a Gaussian distribution) the overlap will be better
than
1 pixel, which of course would not be good for the PSF. Obviously, this
will
change when astrometry will be computed using positions from overlapping
sources and not from the USNO alone.
----------------------------------------
Dr. Roberto Silvotti
----------------------------------------
INAF (Istituto Nazionale di AstroFisica)
Osservatorio Astronomico di Capodimonte
via Moiariello 16, I-80131 Napoli, Italy
----------------------------------------
tel/fax: +39-081-5575583/456710
e-mail: silvotti at na.astro.it
web: http://www.na.astro.it/~silvotti
----------------------------------------
More information about the News
mailing list