[News] Quality control- implementing subwindow statistics

[Edwin A. Valentijn]

Implementing subwindows in AstroWise QC

This note summarizes the basic QC structures in the AstroWise system 
and  how one can use these structures to implement the statistics of 
small subwindows as a tool for quality control evaluations.

There are two fundamentally different ways QC can be performed in the 
AstroWise system,
( relevant for the implementation of multiple subwindows as a means to 
assess quality of data):

Way one uses data committed to the db in the past. By quering the db on 
particular items, such as Calfiles or object attributes like image 
statistics on whole frames (imstat)  one can interactively study the 
behaviour of particular properties.  Interactively, because this is 
contrary to standard Pipelines.
When this is done as function of time we call this a form of trendanalysis.
In AstroWise this is mostly done  in the Awe (python) prompt or by means 
of the webinterface Objectviewer.
In special cases one could  make scripts for standardized analysis or 
tools, eg inspect trend of zeropoint .
For the subwindows, one could envisage  QC analysis on the statistical 
properties as function of time, or for a series of exposures.

The implementation of Way one is simply  provided by

    * the Class definition of SubWinStat  which has a BaseFrame
      dependency and a make() method.
    * and the requirement that for every  instantiation of a db object
      of type frame the SubsWinStat.make() button is pushed


Way two is the more elementary form of QC. It is performed on any 
instantiation of  an object created by the pipelines for which it is 
judged that a QC check has to be done. Here the QC is working on 
individual object creation;  the object QC flag is a property/attribute 
of the instantiated object. When the object is labeled "bad" by QC it 
will not be used by later Recipes which check on these flags.
The QC  itself is performed by (implemented) in one, two or three of the 
methods Verify, compare and inspect.
These methods can use subwindowstatistics and thus it is a requirement 
that the SubWinStat button is pushed before the verify, compare or 
inspect methods are instantiated.
Normally, verify would evaluate the statistics over different subwindows 
in one frame, while compare would compare  subwindows with a previous or 
a reference frame.

The implementation of Way two is provided by:

    * The Class definition of SubWinStat  which has a BaseFrame
      dependency and a make() method.
    * The detailed Python script of the usages of subwindows  in  each
      of the Classes ( the most relevant Classes are those who have a
      related  Recipe) for which it has been decided to push  the
      buttons verify and compare  ( eg Recipe Bias  -> Class BiasFrame
      ).    This is most of the work.


Note - Way three : On one hand we can look at SubWinStat as a property 
of a  BaseFrame object and thus we instantiate it any time a BaseFrame 
is created, on the other hand it would be nice to allow a later 
execution of SubWinStat on many already existing BaseFrames (for 
instance with different Subwindow dimensions)  and then SubWinStat 
should be threated as a Class in its own (and not as an attribute). The 
factory construct allows the combination of both properties (Class and 
attribute). Thus way three allows derivation of subwindowstatistics 
with  re-defined  dimensions of the SubWindows.

The implementation of Way three is provided by:

    * the SubWinStatFactory construct


<> Note - QC flag/index calculation : The calculation of the QC 
flag/index, and the qualification of the frame, is done in its verify() 
method, using the result from make_subwinstat().  For example, the check 
on the flatness of the DomeFlatFrame is implemented in the verify() of 
the DomeFlatFrame.


The scripts below provide an example of implementations at the various 
places.

<>-----------------------------------------------------------------------
<>





-----------------------------------------------------------------------
Rename the Class producing the subwindow objects:
class QCStatistics(DBObject): -> class SubWinStat(DBObject)

-----------------------------------------------------------------------
# Recipe Bias   example of QC structure in recipe
def make_masterbias(raw_bias_frames, read_noise, overscan = 0):

    '''This routine derives Calfile 541.

       Inputs :
           raw_bias_frames -- list of raw bias frames
           read_noise -- a read noise calibration 
           overscan -- overscan correction mode''' 

    filename = construct_filename(raw_bias_frames[0])
    masterbias = BiasFrame(pathname = filename)
    configure_process_parameters(masterbias, overscan)

    masterbias.raw_bias_frames = raw_bias_frames
    masterbias.read_noise = read_noise
    masterbias.make()              # rule: fixed order of execution for make, make_subwinstat, verify, compare, inspect
                                   # store, commit and subwinstat.commit
    masterbias.make_subwinstat()   # calculates statistics per subwindow and creates objects
    masterbias.verify()            # could use subwindows when desired - various windows within one frame
    masterbias.compare()           # could use subwindows when desired - windows of different frames
  # masterbias.inspect()           # the inspect() method does not necessarily have to be executed
 
    masterbias.store()
    masterbias.commit()            # commit CalFile in db
    masterbias.commit_subwinstat() # commit window stats in db
    return masterbias
-------------------------------

# add subwindow method to Baseframe

from SubWinStatFactory import create_subwinstat  # Only import the factory function from the SubWinStatFactory module
class BaseFrame:
   def make_subwinstat():
       self.subwinstat = create_subwinstat()     # to avoid circular dependencies
       self.subwinstat.qcframe = self            # SubWinStat object has to refer to this BaseFrame
        self.subwinstat.make()                    # compute the statistics
   def commit_subwinstat():
       if self.subwinstat:                       # commit the SubWinStat object for this BaseFrame
           self.subwinstat.commit()

------------------------------
# In module SubWinStatFactory

from SubWinStat import SubWinStat
def create_subwinstat():
   '''Instantiate sub-window statistics object.'''
   return SubWinStat()
------------------------------   
# astro.main.BiasFrame example of verify and compare methods

class BiasFrame:
def verify():                  
   ''' QC by Internal arithmetics using sub-windows 
       Example showing how to access the sub-window statistics for this BiasFrame'''
   for k in self.subwinstat.windows:
      if k.mean > 5:
         self.quality_flags = self.quality_flags | QC_BAD_SUBWINDOWFLAG

def compare():   
   ''' QC Trend-analysis using sub-windows
       Example showing how to access the sub-window statistics for this BiasFrame'''
   previous = BiasFrame.select_for_date((self.raw_bias_frames[0].DATE_OBS - DateTime.oneWeek).strftime('%Y-%m-%d')
   for x, y in zip(self.subwinstat.windows, previous.subwinstat.windows):
      if abs(x.mean - y.mean) > 0.1:
         self.quality_flags = self.quality_flags | QC_SUBWINDOW_CHANGED_FLAG

--------------------------------------------------------------------------------------------------------------

<><>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listman.astro-wise.org/pipermail/news/attachments/20041101/91ef80da/attachment.htm