[Issues] Re: [Fwd: changing process parameters in dpu]
Edwin A. Valentijn
valentyn at astro.rug.nl
Fri Jul 7 19:58:54 CEST 2006
Dear all,
this discussion is important but apparantly leads to quite incorrect
perceptions of the behaviour of the system and conclusions.
Obviously, it would be desirable to give the dpu a similar flexibilty to
set process parameters as both the Awe prompt and the target processor
has. This issue has nothing to do with data lineage or tracebility
which is maintained in all cases, (even in the case of providing own
versions of the code).
I'll discuss the dpu with Kor when he is back and inform you after that.
best regards
Dave Wilman wrote:
>Thanks Ewout, yes, I think this is something which will have to be
>changed, since the results in the database will become practically
>untracable and unrepeatable if we have different versions of code floating
>around. Actually this is probably even more important when it comes to the
>SExtractor/SWARP stages than in the basic reduction, since these tasks are
>even more intensive and will need to be run using the dpu. I hope someone
>has an idea about how to implement this change.
>
>Thanks,
>Dave (on behalf of the Munich node)
>
>David Wilman
>Max Planck Institute for Extraterrestrial Physics
>Garching bei Muenchen
>Germany
>Tel: +49 (0)893 0000 +3117
>Fax: +49 (0)893 0000 +3495
>
>On Fri, 7 Jul 2006, Ewout M. Helmich wrote:
>
>
>
>>Hi,
>>
>>I've looked a little bit into the way parameters are handled, and I
>>don't think it will be trivial to change all the parameters (especially
>>Sextractor/Swarp parameters) in the way you suggest. I agree that it is
>>preferable to have as much control as possible through parameters, which
>>is something we can trace, as opposed to control through changes in the
>>code. There's definitely room for improvement here.
>>
>>Regards,
>>Ewout
>>
>>
>>
>>
>>Dave Wilman wrote:
>>
>>
>>
>>>Dear Ewout & everybody,
>>>
>>>We are seriously concerned that this does not sound like a feasible
>>>solution in the long term. In particular, surely it could get way out of
>>>hand if everyone has his/her own version of astrowise code? Could it not
>>>be included in the main AstroWise code (i.e. in the Reduce.py routine
>>>there is a dictionary **kw passed in, which is not made use of: Perhaps
>>>naively we would not expect it to be too difficult to make use of the kw
>>>later on to pass any parameters). In particular, with federation of the
>>>system over various nodes it is surely imperitive that there are not many
>>>conflicting versions of code hanging around! Nonetheless, there are often
>>>going to be occasions where it is necessary to change parameters, and yet
>>>the job should be able to run remotely using the dpu object.
>>>
>>>Thanks,
>>>Dave, Johannes & Jan
>>>
>>>David Wilman
>>>Max Planck Institute for Extraterrestrial Physics
>>>Garching bei Muenchen
>>>Germany
>>>Tel: +49 (0)893 0000 +3117
>>>Fax: +49 (0)893 0000 +3495
>>>
>>>On Thu, 6 Jul 2006, Ewout M. Helmich wrote:
>>>
>>>
>>>
>>>
>>>
>>>>Hi Johannes, Dave,
>>>>
>>>>Unfortunately, no. Almost none of the parameters can be changed in the
>>>>call to the dpu:
>>>>
>>>>I.e.
>>>>
>>>>dpu.run('Bias', i='WFI', d='2000-04-28', oc=8)
>>>>
>>>>This is pretty much as far as the control at this level goes. I.e. "oc"
>>>>(OVERSCAN_CORRECTION) is a regular (but very important) process parameter.
>>>>
>>>>The typical way to change the process parameters in this case is to
>>>>change the DBRecipe, in your case DBRecipes/HotPixels.py and add the
>>>>following (the second) line after the instantiation of the HotPixelMap
>>>>class:
>>>>
>>>>hotpixelmap = HotPixelMap()
>>>>hotpixelmap.process_params.MAXIMUM_HOTPIXELCOUNT = 70000
>>>>
>>>>Since the user's own code is sent to the processing cluster (!), this
>>>>approach works, but it requires more knowledge from the user.
>>>>
>>>>Regards,
>>>>Ewout
>>>>
>>>>
>>>>
>>>>Gijs Verdoes Kleijn wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>Hi Ewout and Willem-Jan,
>>>>>
>>>>>Dave and Johannes (see cc for emails) are preparing an Astro-Wise
>>>>>tutorial to be given next week in Munich. Could you help them out with
>>>>>their question about th dpu below? Perhaps it is useful to include the
>>>>>on-line "target processor" in this reply as well.
>>>>>
>>>>>
>>>>>Thanks and cheers, Gijs
>>>>>ps. please cc me in the reply.
>>>>>
>>>>>
>>>>>-------- Original Message --------
>>>>>Subject: changing process parameters in dpu
>>>>>Date: Thu, 06 Jul 2006 10:23:22 +0200
>>>>>From: Johannes Koppenh?fer <koppenh at mpe.mpg.de>
>>>>>Organization: http://freemail.web.de/
>>>>>To: Gijs Verdoes Kleijn <g.verdoes at astro.rug.nl>
>>>>>
>>>>>
>>>>>
>>>>>Hi Gijs,
>>>>>
>>>>>we are working on the tutorials at the moment and wonder if it is
>>>>>possible to change process parameters in dpu. It seems that there is
>>>>>an optional keyword list **options you can specify in the dpu-call and
>>>>>there is also an optional **kw list in Reduce.py but nothing is done
>>>>>with it. So is it yet possible to change parameters in dpu? This is
>>>>>important because else we would have to run any task with non-standard
>>>>>parameters on the local cpu which would be pointless. E.g. we want to
>>>>>change the MAXIMUM_HOT_PIXELS parameters in order to make the task
>>>>>work on WFI images (where there are a lot of bad pixels). I wonder if
>>>>>some of you have faced the same problem before.
>>>>>
>>>>>Could you comment quickly on this issue?
>>>>>
>>>>>Regards,
>>>>>Johannes & Dave
>>>>>_____________________________________________________________________
>>>>>Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
>>>>>http://smartsurfer.web.de/?mc=100071&distributionid=000000000071
>>>>>
>>>>>
>>>>>
>>>>>
>>>>--
>>>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>>>Drs. Ewout Helmich <><>
>>>>Kapteyn Astronomical Institute <><> Astro-Wise/OmegaCEN
>>>>Landleven 12 <><>
>>>>P.O.Box 800 <><> email: helmich at astro.rug.nl
>>>>9700 AV Groningen <><> tel : +31(0)503634548
>>>>The Netherlands <><>
>>>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>--
>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>Drs. Ewout Helmich <><>
>>Kapteyn Astronomical Institute <><> Astro-Wise/OmegaCEN
>>Landleven 12 <><>
>>P.O.Box 800 <><> email: helmich at astro.rug.nl
>>9700 AV Groningen <><> tel : +31(0)503634548
>>The Netherlands <><>
>>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>
>>
>>
>>
>
>
>_______________________________________________
>Issues mailing list
>Issues at astro-wise.org
>http://listman.astro-wise.org/mailman/listinfo/issues
>
>
--
----------------------------------------------------------------------
Prof. Dr. Edwin A. Valentijn
Head OmegaCEN www : www.astro.rug.nl/omegacen
Kapteyn Astronomical Institute tel : +31 (0)50 3634011
P.O.Box 800 mobile: +31 (0)6 48276416
NL-9700 AV Groningen e-mail: valentyn at astro.rug.nl
The Netherlands www : www.astro.rug.nl/~valentyn
----------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listman.astro-wise.org/pipermail/issues/attachments/20060707/413d33ca/attachment-0001.html
More information about the Issues
mailing list