<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
<b><font color="#a020f0"><font color="#000000">Implementing subwindows
in AstroWise QC</font></font></b><br>
<br>
This note summarizes the basic QC structures in the AstroWise system
and how one can use these structures to implement the statistics of
small subwindows as a tool for quality control evaluations.<br>
<br>
There are two fundamentally different ways QC can be performed in the
AstroWise system,<br>
( relevant for the implementation of multiple subwindows as a means to
assess quality of data): <br>
<br>
<b>Way one </b>uses data committed to the db in the past. By quering
the db on particular items, such as Calfiles or object attributes like
image statistics on whole frames (imstat) one can interactively study
the behaviour of particular properties. Interactively, because this is
contrary to standard Pipelines.<br>
When this is done as function of time we call this a form of
trendanalysis. <br>
In AstroWise this is mostly done in the Awe (python)
prompt or by means of the webinterface Objectviewer.<br>
In special cases one could make scripts for standardized analysis or
tools, eg inspect trend of zeropoint .<br>
For the subwindows, one could envisage QC analysis on the statistical
properties as function of time, or for a series of exposures.<br>
<br>
The implementation of Way one is simply provided by <br>
<ul>
<li>the Class definition of SubWinStat which has a BaseFrame
dependency and a make() method.</li>
<li>and the requirement that for every instantiation of a db object
of type frame the SubsWinStat.make() button is pushed<br>
</li>
</ul>
<br>
<b>Way two </b>is the more elementary form of QC. It is performed on
any instantiation of an object created by the pipelines for which it
is judged that a QC check has to be done. Here the QC is working on
individual object creation; the object QC flag is a property/attribute
of the instantiated object. When the object is labeled "bad" by QC it
will not be used by later Recipes which check on these flags.<br>
The QC itself is performed by (implemented) in one, two or three of
the methods <i>Verify, compare
and inspect.</i><br>
These methods can use subwindowstatistics and thus it is a requirement
that the SubWinStat button is pushed before the <i>verify, compare or
inspect </i>methods are instantiated. <br>
Normally, <i>verify</i> would evaluate the statistics over different
subwindows in one frame, while<i> compare</i> would compare subwindows
with a previous or a reference frame.<br>
<br>
The implementation of Way two is provided by:<br>
<ul>
<li>The Class definition of SubWinStat which has a BaseFrame
dependency and a make() method. </li>
<li>The detailed Python script of the usages of subwindows in each
of the Classes ( the most relevant Classes are those who have a
related Recipe) for which it has been decided to push the buttons <i>verify
</i>and <i>compare ( eg </i>Recipe Bias<i> </i>-> Class
BiasFrame ). This is most of the work.<br>
</li>
</ul>
<br>
<b>Note - Way three </b>: On one hand we can look at SubWinStat as a
property
of a BaseFrame object and thus we instantiate it any time a BaseFrame
is created, on the other hand it would be nice to allow a later
execution of SubWinStat on many already existing BaseFrames (for
instance with different Subwindow dimensions) and then SubWinStat
should be threated as a Class in its own (and not as an attribute). The
factory construct allows the combination of
both properties (Class and attribute). Thus way three allows derivation
of subwindowstatistics with re-defined
dimensions of the SubWindows.<br>
<br>
The implementation of Way three is provided by:<br>
<ul>
<li>the SubWinStatFactory construct<br>
</li>
</ul>
<br>
<><b>
Note - QC flag/index calculation </b>: The calculation of the QC
flag/index, and the qualification of the frame, is done in its verify()
method, using the result from make_subwinstat(). For example, the
check on the flatness of the DomeFlatFrame is implemented in the
verify() of the DomeFlatFrame.<br>
<br>
<br>
The scripts below provide an example of implementations at the various
places.<br>
<br>
<><b><font color="#a020f0"><font color="#000000">-----------------------------------------------------------------------</font></font></b><br>
<><b><font color="#a020f0"><font color="#000000"><br>
</font></font></b>
<pre><b><font color="#a020f0"><font color="#000000">
-----------------------------------------------------------------------
Rename the Class producing the subwindow objects:</font>
class</font></b> QCStatistics(DBObject): -> <font
color="#a020f0"><b>class</b> </font>SubWinStat(DBObject)
-----------------------------------------------------------------------
#<b> Recipe Bias example of QC structure in recipe</b>
<b><font color="#a020f0">def</font></b> <b><font color="#0000ff">make_masterbias</font></b>(raw_bias_frames, read_noise, overscan = 0):
<font
color="#bc8f8f"><b>'''This routine derives Calfile 541.
Inputs :
raw_bias_frames -- list of raw bias frames
read_noise -- a read noise calibration
overscan -- overscan correction mode'''</b></font>
filename = construct_filename(raw_bias_frames[0])
masterbias = BiasFrame(pathname = filename)
configure_process_parameters(masterbias, overscan)
masterbias.raw_bias_frames = raw_bias_frames
masterbias.read_noise = read_noise
masterbias.make() # rule: fixed order of execution for make, make_subwinstat, verify, compare, inspect
# store, commit and subwinstat.commit
masterbias.<font
color="#ff6666">make_subwinstat</font>() # calculates statistics per subwindow and creates objects
masterbias.verify() # could use subwindows when desired - various windows within one frame
masterbias.compare() # could use subwindows when desired - windows of different frames
# masterbias.inspect() # the inspect() method does not necessarily have to be executed
masterbias.store()
masterbias.commit() # commit CalFile in db
masterbias.<font
color="#ff6666">commit_subwinstat</font>() # commit window stats in db
<b><font
color="#a020f0">return</font></b> masterbias
-------------------------------
<b>
# add subwindow method to Baseframe
</b>
from SubWinStatFactory import create_subwinstat # Only import the factory function from the SubWinStatFactory module
<b><font color="#a020f0">class </font></b>BaseFrame:
def make_subwinstat():
self.subwinstat = create_subwinstat() # to avoid circular dependencies
self.subwinstat.qcframe = self # SubWinStat object has to refer to <b>this</b> BaseFrame
self.subwinstat.make() # compute the statistics
def commit_subwinstat():
if self.subwinstat: # commit the SubWinStat object for this BaseFrame
self.subwinstat.commit()
------------------------------
# In module SubWinStatFactory
from SubWinStat import SubWinStat
<b><font color="#a020f0">def </font><font color="#0000ff">create_subwinstat</font></b>():
'''Instantiate sub-window statistics object.'''
return SubWinStat()
------------------------------
<b># astro.main.BiasFrame</b> example of verify and compare methods
<b><font color="#a020f0">
class </font></b>BiasFrame:
<b><font color="#a020f0">def </font><font color="#0000ff">verify</font></b>():
''' QC by Internal arithmetics using sub-windows
Example showing how to access the sub-window statistics for this BiasFrame'''
for k in self.subwinstat.windows:
if k.mean > 5:
self.quality_flags = self.quality_flags | QC_BAD_SUBWINDOWFLAG
<b><font color="#a020f0">def</font></b> <b><font color="#0000ff">compare</font></b>():
''' QC Trend-analysis using sub-windows
Example showing how to access the sub-window statistics for this BiasFrame'''
previous = BiasFrame.select_for_date((self.raw_bias_frames[0].DATE_OBS - DateTime.oneWeek).strftime('%Y-%m-%d')
for x, y in zip(self.subwinstat.windows, previous.subwinstat.windows):
if abs(x.mean - y.mean) > 0.1:
self.quality_flags = self.quality_flags | QC_SUBWINDOW_CHANGED_FLAG
--------------------------------------------------------------------------------------------------------------
</pre>
<><>
</body>
</html>