[News] sigmaclip algorithm

[Roeland Rengelink]

Hi all,

I think the code that was recently submitted constitutes a considerable 
improvement on the previous version. I like the flexibility of the 
current implementation. Experience will teach us what parameters will 
give the best result.

The next step would be to incoprorate this work into the astro-wise 
framework. Let me discuss this in some detail for this particular case.

We work with several levels of abstraction with image processing

At the lowest level there is the eclipse C-code and the python-eclipse 
wrapper c_eclipse.py. This is the level at which the implemented the 
sigma-clipping algorithm was implemented.

At the next level there is the object-oriented wrapper of c_eclipse.py. 
At this level we define several objects (image, cube, header, pixelmap) 
whose methods define operations that can be performed on and with these 
objects. For example: One can add and subtract images, one can normalize 
them, extract subregions, perform statistics etc. etc. These are all 
operations on individual images. In addition there are operations on 
lists of images (cubes). These include arithmetic,normalization, 
averageing etc. etc.  It should come as no suprise that you can 
implement an average_sigmaclip() method on cubes that uses 
c_eclipse.cube_sigmaclip functon to perform the operation. I have made a 
prelimenary implemention of this method in the cube object in the 
eclipse.cube module in eclipse/lang/python/src/eclipse/cube.py

Note that the standard astro-wise interface to the low-level 
sigma-clipping functionality would be through this OO wrapper

At the next higher level we combine low-level processing algorithms to 
define data-reduction procedures for different data types. These include 
BiasFrame, DomeFlatFrame, etc. etc.  These procedures are invoked 
through the make() method of these objects. Taking BiasFrame as an 
example, we see that:

Given

bias = BiasFrame(...)           # a target 
                                   
bias.raw_bias_frames = [...]  # a list of dependencies
bias.make()                         # do it...

the following will happen

bias.make() will invoke bias.make_image(). bias.make_image() does the 
following:

1. construct a cube from a list of images created by extracting the trim 
section from each raw bias image.
2. average the images in the cube using cube.average()

or in code:

images = []
for f in raw_bias_frames:
     ima = eclipse.image.image(f)
     ima = ima.extract_region(...)
     images.append(ima)
c = eclipse.cube.cube(images)
bias_image = cube.average()

Note that only the last step needs to be replaced to implement bias 
production with sigma-clipping

bias_image = cube.average_sigmaclip()

Finally, at the highest level there are user-interfaces to the process 
targets. These are the recipes like the  bias recipe. Note that recipes 
are meant as interfaces to high-level operations, not low-level operations.

So, what are the next steps?

- I have implemented a cube.average_sigmaclip() method for your 
consideration. It would be good if you tested this, and fleshed out the 
interface. Things to consider:
  o the documentation
  o one method with flags as parameters to distinguish strategies, or
  o more methods to distinguish strategies?, e.g:
     cube.average_clip_measured_sigma(..., gain, rn)
     cube.average_clip_computed_sigma(...)
 
- Update recipes to use the sigma clipping averageing as specified by 
the CP. Questions to consider:
  o Which parameters to fix.
  o Which parameters to just give defaults for (through the 
BiasFrameParameters object) but allow the user to change
  o How to test the code (unittests)

Finally, as I remarked previously, the Recipes directory in opipe 
contains command-line user-interfaces to high-level pipeline operations. 
The sigma-clipping algorithm itself is not such a high-level operation, 
and, hence, the SigmaClip recipe does not belong there. If you feel that 
astro-wise should provide these interfaces to low-level functionality 
too, I'd like to have a separate discussion on that.

Cheers,

Roeland

Roberto Silvotti wrote:

>Hi all,
>
>The files containing the sigma clip algorithm have been 
>committed few days ago to the AW cvs.
> 
>There are 4 new files:
>- opipe/Recipes/SigmaClip.py
>- opipe/eclipse/src/iproc/cube_sigmaclip.c
>- opipe/eclipse/src/include/cube_sigmaclip.h
>- opipe/eclipse/src/include/eclipse.h
>
>The algorithm is quite general and can be used in different
>situations: creation of the master bias, but also for master 
>flats and scientific frames.
>Sigma is calculated considering a (squared) sum of poissonian
>noise + read-out noise, as it is done in IRAF.
>Although this method is certainly more robust, it is also 
>possible, if one prefers, to derive sigma directly from 
>the data (standard deviation). 
>More info are given in the gzipped ps file attached.
>
>You are kindly invited to test this algorithm and let us
>know your comments and suggestions.
>Please consider that you have to rebuilt eclipse 
>in order to include the new eclipse files.
>
>We also feel that it would be very usefull, to avoid 
>confusion,to define a general list of key for the parameters
>of all the python scripts.
>For example, if -o means output file, then the key -o should
>be used in all scripts (with the same meaning !).
>We can discuss on this in the next telekon.
>
>
>Best regards,
>               Roberto
>
>----------------------------------------
>         Dr. Roberto Silvotti
>----------------------------------------
>INAF (Istituto Nazionale di AstroFisica)
>Osservatorio Astronomico di Capodimonte
>via Moiariello 16, I-80131 Napoli, Italy
>----------------------------------------
>tel/fax:  +39-081-5575583/456710
>e-mail:   silvotti@na.astro.it
>web:    http://www.na.astro.it/~silvotti
>----------------------------------------
>