<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type"
content="text/html; charset=ISO-8859-1">
<title>minutes-nov03</title>
<meta name="author" content="Edwin A. Valentijn">
</head>
<body>
<span style="font-weight: bold;"><br>
</span>
<hr style="width: 100%; height: 2px;"><span style="font-weight: bold;">Notes/Minutes/Action
Items </span>of the <span style="font-weight: bold;">ASTRO-WISE
Workshop on Photometric
Pipelines and quality control</span><br>
and tasksleader meeting<br>
<br>
17 - 20 November 2003 - Kapteyn Institute Groningen<br>
<br>
EV 27 Nov - 5 Dec 2002<br>
<hr style="width: 100%; height: 2px;"><br>
<br>
During the workshop the various components of the prototype of the
AstroWise Image and Calibration pipelines were reviewed. There
was particular focus on photometric
pipelines and quality control.<br>
As planned in the Awe milestone schedule at the end of Q4 of year two
(december 2003) we wish to stabilize the data model and Class
definitions, following experience
with the prototype.<br>
The workshop helped substantially to reach this goal, but also to
spread knowledge about the system and to review its merits.<br>
The programme is well on schedule, but we should realize that we are
only half way and that there is still a lot of work to be done.<br>
<br>
Below we only summarize the most important decisions taken, actions
items and steps to be taken for the near future.<br>
<br>
Action items are labelled <span style="font-weight: bold;">AI</span><br>
<hr style="width: 100%; height: 2px;"><br>
<big><span style="font-weight: bold;">Class Definitions<br>
</span></big>
<ul>
<li><span style="font-weight: bold;">The new schema of weight and
flag maps</span> (deleting flag maps) will be implemented</li>
<ul>
<li>Get rid of (Master)WeightFrame, (Master)FlagFrame. </li>
<li>replace ColdPixelFrame and HotPixelFrame with PixelMap
implementation </li>
<li>using 8 bit ints and on-the-fly (de-)compression to limit
file-size (using the eclipse built-in pixelmap) </li>
<li>Add a CosmicRayMap, based on the USM code </li>
<li>Produce weightmaps on the fly from the flatfield and the
bad-pixelmaps </li>
<li>Drop flag-maps (they are meaningless for co-added data
anyway). </li>
</ul>
</ul>
<div style="margin-left: 40px;">We end up with just
:<br>
</div>
<ul>
<ul>
<li>MasterFlatFrame</li>
<li>ColdPixelMap </li>
<li>HotPixelMap</li>
<li>CosmicRayMap <br>
</li>
</ul>
</ul>
<div style="margin-left: 40px;">It was also decided at the
workshop to add a <br>
<ul>
<li>SaturationPixelMap</li>
<li>Chi -SquaredFrame
Class <span
style="font-weight: bold;"> AI </span>M. Neeser will look into
this<br>
</li>
</ul>
<br>
</div>
<div style="margin-left: 40px;"> <span style="font-weight: bold;">AI
RR+EH: </span>will implement this scheme, and will update the top
level
descriptions in Odoco and manuals<br>
<span style="font-weight: bold;">Danny</span> will update db
schema<br>
<br>
Open issue is to address the question whether to assign persistency to
the Userlogs<span style="font-weight: bold;"> AI QC RR+ E. Deul</span><br>
</div>
<ul>
<li><span style="font-weight: bold;">Illumination Correction</span>-
The severity of this problem again became apparent at this workshop.
Although OmegaCAM is taking optimal precautions (design telescope +
camera) and zero-point determination on <span
style="font-style: italic;">individual </span>chips there is
still a threat for systematic effects on large scales. <br>
</li>
<ul>
<li>An illumination correction map (gain map) will be added to the
data model (CALfile 548) which will be applied in the image
pipeline. This will be<span style="font-style: italic;">
effectively </span>done on an image on which the
skybackground is subtracted (background is 0), which is then
multiplied with Cal548. Optionally the sky is again added to the image.
This way the sky background will be flat on the resultant illumination
corrected image. Note that the sky background must somehow be retained
for
use in the photometric pipeline.<span style="font-weight: bold;"> </span></li>
</ul>
</ul>
<div style="margin-left: 80px;"><span style="font-weight: bold;">AI
RR+RV </span>:discuss details (seq order in image pipeline) ,
implement in pipeline and put descriptions in Odoco.<br>
<br>
<span style="font-weight: bold;">AI EV</span> - Study alternative/new
methods to characterize the illumination correction<br>
</div>
<ul>
<li><span style="font-weight: bold;">Sourcelist and
sAssociatelist - <br>
</span></li>
</ul>
<ul>
</ul>
<ul>
<ul>
<li>Proposed Class definitions are <a
href="../Presentations/lists.html">SourceList and AssociateList</a>.</li>
</ul>
</ul>
<div style="margin-left: 80px;">Near future (weeks) developments
involve <span style="font-weight: bold;">AI KGB:</span><br>
<ul>
<li>Installation Client/Server</li>
<li>Documentation - Introduction Python use - Cookbook <span
style="font-weight: bold;">- AI RV</span> <br>
</li>
<li>Documentation in code/ code clean-up <span
style="font-weight: bold;">- AI RV</span> <br>
</li>
</ul>
</div>
<ul>
<ul>
<li>Scalabilty will be further
explored
<span style="font-weight: bold;">AI KGB+MT</span></li>
<li>new functionality: make Sourcelist from Associatelist <span
style="font-weight: bold;">AI KGB + EH</span><br>
</li>
</ul>
</ul>
<span style="font-weight: bold;">
</span>
<ul>
<li><span style="font-weight: bold;">Quality Control</span></li>
<ul>
<li>For Calibration pipeline OAC will define/review complete
set of persistent parameters in combination with the set of Verify,
Compare and Inspect method which are forming together the
integrated quality control machinery</li>
</ul>
</ul>
<div style="margin-left: 80px;"><span style="font-weight: bold;">AI RS
etal </span>make sure the set of QC methods is complete, integrated
and with the right definitions of persistent objects<br>
</div>
<ul>
<ul>
<li>QFITS and image pipeline QC Result of Splinter DB,
AV, EH <br>
</li>
<ul>
<li>useful components are:</li>
<ul>
<li>Thumbnails <br>
</li>
<li>various histograms/visualizations <br>
</li>
<li>the generation of XML output <br>
</li>
</ul>
<li>qualityFITS can be used stand-alone and has been made aware
of Astro-Wise FITS keywords during the meeting.</li>
<li>The Astro-Wise pipeline should include the ORIGIN keyword
when writing FITS headers.<br>
</li>
</ul>
</ul>
</ul>
<div style="margin-left: 80px;"><span style="font-weight: bold;">AI RS
+ AV</span> Check out the QFITS package and see what could be used as
idea or as module for quality control tools<br>
</div>
<br>
<ul>
<li><span style="font-weight: bold;">Photometric redshifts - Splinter
</span>MN, KGB, DB</li>
<ul>
<li>The SourceList and AssociateList classes are available from CVS
(currently opipe/Experimental) - see also the <span
style="font-weight: bold;">Sourcelist and
sAssociatelist</span> bullet.<br>
</li>
<li><span style="font-weight: bold;">AI JSnigula</span> will
continue implementation and connect this to Sourcelist sAssociate Class</li>
</ul>
</ul>
<ul>
<li><span style="font-weight: bold;">Co-addition</span></li>
</ul>
<ul>
<ul>
<li><span style="font-weight: bold;">AI RV </span>facilitate
the choice of input files for co-addition. The issue of finding the
correct
data for a specific purpose extends to various calibration pipelines.</li>
</ul>
</ul>
<ul>
<li><span style="font-weight: bold;">Cosmics</span></li>
<ul>
<li>Klaus will put cosmic ray removal programme in CVS to be
verified by <span style="font-weight: bold;">AI RR<br>
</span></li>
<li><span style="font-weight: bold;">C</span>urrent speed is 1 min
for a 2x4k chip- this should be optimized <span
style="font-weight: bold;">AI Klaus</span></li>
</ul>
<li><span style="font-weight: bold;">Fringe Maps</span></li>
<ul>
<li>prepare for extra method using small windows for determining
the fringe amplitude <span style="font-weight: bold;">AI EH</span><br>
</li>
</ul>
</ul>
<br>
<br>
<hr style="width: 100%; height: 2px;"><br>
<big><span style="font-weight: bold;">VISUALISATION<br>
</span></big>
<div style="margin-left: 40px;">Frederic Magnard gave two presentions
on Visualization tools developed at Terapix. <br>
</div>
<ul>
<li><span style="font-weight: bold;">Panorapix</span></li>
<ul>
<li><span style="font-weight: bold;">X11</span></li>
<li>schedule for OpenGL implementation made at Taksleader meeting -
delivery 1 July 2004<span style="font-weight: bold;"> AI FM</span><br>
</li>
</ul>
</ul>
<ul>
<li><span style="font-weight: bold;">QFITS</span></li>
<ul>
<li>input splinter - see Quality control <br>
</li>
</ul>
</ul>
<big><span style="font-weight: bold;"><br>
</span></big>
<hr style="width: 100%; height: 2px;"><big><span
style="font-weight: bold;"><br>
DOCUMENTATION<br>
</span><small><br>
Current Documentation scheme is:<br>
</small></big>
<ul>
<li>Odoco .tex -> req's -> .pdf for URD
, CP and DRS<br>
</li>
<li>cookbook</li>
<li>separate reports (eg Bias, flatfield, illumination, photometry)</li>
<li>Doc string in .py -> PyDoc</li>
<li>.py source code + Py doc string in CVS <br>
</li>
</ul>
<big><span style="font-weight: bold;"><br>
</span><small>Make documentation better accessable by adding<br>
</small></big>
<ul>
<li>vividly update the Calibration analysis, pseudo code etc section
in Odoco- being the highest in the three of the source of documentation
information <span style="font-weight: bold;">AI- all-
specifically for those who updated modified methods</span><br>
</li>
<li><big><small>odoco -> data reduction/progammers
manual <span style="font-weight: bold;">- AI RR, RV EV</span><br>
</small></big></li>
<li>html refs in .py doc string to Base class, and to odoco
etc - <span style="font-weight: bold;">AI RV</span><br>
</li>
<li><big><small>odoco in html</small> - AI EV +DB </big><br>
</li>
</ul>
<big><span style="font-weight: bold;"></span></big><br>
<hr style="width: 100%; height: 2px;"><br>
<big><span style="font-weight: bold;">PHOTOMETRIC PIPELINE AND WFI DATA
PROCESSING </span><small>splinter
RS, PH, ED, HM, RV</small><span style="font-weight: bold;"> </span><br>
</big><br>
It was agreed that best short cuts to maintain WFI photometry in
pipeline environment is by including:<br>
<ul>
<li>use sky background for determining chip to chip zeropint
variations<br>
</li>
<li>use standard ESO extinction coeff.</li>
</ul>
<br>
The photometric pipeline needs further testing (and possible
improvements) for what
concerns:
<ul>
<li>colour term (constant vs determined from the fit of standard
transformation equations)</li>
<li>zero points: needs more robust procedure in order to exclude
possible bad points when averaging the zero points of each standard
star.</li>
<li>sky background vs. photometry of PG1525 standard field are
ongoing.</li>
</ul>
In order to solve these open questions and to test the whole pipeline,
the comparison Napoli-Groningen (<span style="font-weight: bold;">PH</span>
in Groningen) using the Capodimonte Deep Field data
reductions (IRAF vs AW pipeline) will continue.
<br>
<br>
At the end of this phase, a report will be produced <span
style="font-weight: bold;">(AI RS et al.)</span><br>
<br>
<hr style="width: 100%; height: 2px;"><br>
<big><span style="font-weight: bold;">FEDERATION<br>
</span></big>
<ul>
<li>The filename convention. <br>
The full filename description was
re-iterated and a consesus was reached to implement the proposed and
slightly modified convention as soon as possible. The modifications
with respect to the orginal proposal are:</li>
<ul>
<li>Inclusion of the MJD of the start of the processing sequence</li>
<li>Extension .XXX to denote the type of the file</li>
</ul>
<li>The dataservers at the different sites are to be connected. <span
style="font-weight: bold;">AI KGB, JS, MP</span><br>
Note that first two dataservers will be connected in Groningen. <span
style="font-weight: bold;">AI KGB</span></li>
<li>A database taskforce has been formed consisting of Danny
Boxhoorn, Fedor Getman and Joel Poels to implement a prototype
federation that is based on Oracle Streams. The taskforce will report
at least once a week about progress and problems. The <a
href="http://listman.astro-wise.org/mailman/listinfo/wp3">WP3 mailing
list</a> is the main communication channel.<br>
Jan Snigula will join this taskforce.</li>
<li>The taskforce have agreed on the following sequence of steps to
be taken <span style="font-weight: bold;">AI DB, FG, JP, JS</span><br>
<ol>
<li>Have two separate databases at each of our sites.<br>
Make sure that Oracle Streams works from one database to the other for
a simple table.</li>
<li>Get the Streams Messaging example in Chapter 18 of the
Streams guide working on each of our sites.</li>
<li>Get Streams Messaging working for a simple object table on
each of our sites. For example the Astro-Wise "Chip" object table.<br>
Note that we do not care about DDL. Streaming DML has to work, and if
we get Streaming DDL to work that is a bonus.</li>
<li>Get Streams Messaging working for a more complex object
containing references on each of our sites. "BiasFrame" is probably a
nice example.</li>
<li>Activate Messaging from the Python code or via triggers.</li>
<li>Connect our databases and enable peer-to-peer messaging.<br>
</li>
</ol>
</li>
<li>The databases and dataservers have to be connected through
firewalls and over secure connections. This has to be investigated. <span
style="font-weight: bold;">AI FG</span></li>
<li>The identity of the database users has to be the same for all
participating sites in the federation. <span style="font-weight: bold;">AI
DB</span><br>
</li>
</ul>
<br>
<hr style="width: 100%; height: 2px;"><br>
<span style="font-weight: bold;">TASKLEADER Meeting
ASTRO-WISE </span>
19 November 2003<br>
<br>
Kapteyn Room Second floor
Kapteyn Institute Groningen<br>
<br>
<br>
A G E N D A<br>
<br>
<br>
1) Latest news OAC, Terapix, USM, OmegaCEN<br>
<br>
2) Status<br>
- vs Contract<br>
- vs minutes last PI meeting
nov 03<br>
- vs Current AIs<br>
<br>
- WP1 <br>
- WP2<br>
- WP3 Federation See
bullets agenda workshop<br>
- CVS<br>
- Oracle db<br>
- Federation<br>
<br>
- WP4 - coordination
ingestion of data<br>
- WP6<br>
<br>
3) Science collaborations? GT, Public Suveys<br>
<br>
4) Financial<br>
<br>
5) staff and partner plans<br>
<br>
6) aob<br>
<hr style="width: 100%; height: 2px;"><br>
Most issues touched at the taskleader/PI meeting are included in the
minutes/notes above.<br>
In addition:<br>
<br>
WP1 <br>
<ul>
<li>test optimize quality control -OAC</li>
<li>photometric pipeline review - Mark + Roberto</li>
<li>implement lensing software shapelets - Ole Muller (OmegaCEN)</li>
<li>cm diagram of globular clusters on WFI data AI Konrad</li>
<li>USM will put cosmics module in CVS<br>
</li>
</ul>
WP2<br>
Terapix/ Panorapix <br>
<ul>
<li>Multi Extension Fits- almost done (december)<br>
</li>
</ul>
<ul>
<li>new person will start in Januari - 6 month and work in parallel on</li>
</ul>
<ul>
<ul>
<li>new tools - see ADD<br>
</li>
<li>implementation in open GL s - see ADD</li>
</ul>
<li>another person will start working for 6 month on masking</li>
<li>Emmanuel Bertin increased speed of re-gridding in SWARP by a
factor 4<br>
</li>
</ul>
<br>
WP4<br>
<span style="font-weight: bold;">AI- PIs </span>give info on
requirement privalages proprietary data in Awe system<br>
eg WFI data ingested at USM etc<br>
<br>
Henry Joy McCracken- new Astrowiser - Terapix expert on
photometry on large scales<br>
<hr style="width: 100%; height: 2px;"><br>
</body>
</html>