[Issues] GASTROM: ORA-28576: lost RPC connection to external procedure agent, ORA-02393: exceeded call limit on CPU usage
Philippe Heraudeau
heraud at astro.uni-bonn.de
Fri Sep 28 11:33:26 CEST 2007
Hi John,
1) All the files have valid astrometry but this failed again:
dpu.run('GAstrom',object='DEEP3a*',instrument='WFI',filter='#844',C=1)
...
GAstromTask ERROR: ORA-28576: lost RPC connection to external procedure
agent
[foucault] 16:19:35 - [node199] 15:40:35 - Outer bounding box:
(171.54369,-21.98915,170.83598,-21.98923,170.83631,-21.39761,171.54347,-21.39781
)
[foucault] 16:19:35 - [node199] 15:43:03 - Copied 78263 sources to
scratch table
[foucault] 16:19:35 - Error executing 'execute'
[foucault] 16:19:35 - Exception: ORA-28576: lost RPC connection to
external procedure agent
[foucault] 16:19:35 -
2) With only chips 50 is also now failed but it worked before, I even
coadded them...
dpu.run('GAstrom',object='DEEP3a*',instrument='WFI',filter='#844',chip='ccd50',C=0)
cx_Oracle.DatabaseError: ORA-02393: exceeded call limit on CPU usage
[foucault] 11:07:51 - Starting GAstromTask
[foucault] 11:07:51 - [node100] 17:30:59 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6672057-429c0a89f68d6253015beed4407e80cf200a4ce7.fits.slist,
SLID = 193551, number of sources = 221
[foucault] 11:07:51 - [node100] 17:31:00 - Outer bounding box:
(171.54328,-21.71374,171.29374,-21.71396,171.29322,-21.39813,171.54306,-21.39799
)
[foucault] 11:07:51 - [node100] 17:31:04 - Copied 10388 sources to
scratch table
[foucault] 11:07:51 - [node100] 17:32:39 - Pairing query found 108947 pairs
[foucault] 11:07:51 - [node100] 17:32:39 - Looking for closest pairs
[foucault] 11:07:51 - [node100] 17:32:48 - After filtering 102925 pairs
remain
[foucault] 11:07:51 - [node100] 17:38:15 - Needed 5 loops to combine
102925 pairs into 645 associations
[foucault] 11:07:51 - Error executing 'execute'
[foucault] 11:07:51 - Exception: ORA-02393: exceeded call limit on CPU
usage
3) To get the inspect plot, I used only 5 frames, locally and got the
same error "ORA-02393: exceeded call limit on CPU usage again
awe>
list=['WFI.2000-03-30T05:48:39.916_1.fits','WFI.2000-03-30T06:01:44.931_1.fits','WFI.2000-03-30T06:29:29.466_1.fits','WFI.2000-03-30T06:23:02.880_1.fits','WFI.2000-03-30T06:16:37.269_1.fits']
awe> task =
GAstromTask(raw_filenames=list,object='DEEP3a*',instrument='WFI',filter='#844',C=1,inspect=1)
awe> task.execute()
[foucault] 15:33:20 - Querying database for instances of class
ReducedScienceFrame...
[foucault] 15:33:24 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6617829-fbda44a69c1dab3d46296b5f53be22beb5acb1c7.fits.slist,
SLID = 192831, number of sources = 393
[foucault] 15:33:24 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6660892-57d1125fe9e52b29153deb64c1e9f89964b8805f.fits.slist,
SLID = 192461, number of sources = 368
[foucault] 15:33:24 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6680245-a7e712330177956fb64c879787b36fd9e8f23e70.fits.slist,
SLID = 194471, number of sources = 410
[foucault] 15:33:24 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6610351-22282b08173d88fe9125e059a64d5243db58daa8.fits.slist,
SLID = 192391, number of sources = 363
[foucault] 15:33:24 - Using SourceList
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6689499-0626e44428195810666447b24320eae844222463.fits.slist,
SLID = 192171, number of sources = 374
[foucault] 15:33:25 - Outer bounding box:
(171.53478,-21.70295,171.35693,-21.70308,171.35675,-21.40999,171.53454,-21.40985
)
[foucault] 15:34:02 - Copied 2237 sources to scratch table
[foucault] 15:34:11 - Pairing query found 4219 pairs
[foucault] 15:34:11 - Looking for closest pairs
[foucault] 15:34:12 - After filtering 3916 pairs remain
[foucault] 15:34:17 - Needed 3 loops to combine 3916 pairs into 453
associations
[foucault] 15:45:16 - Created Matched AssociateList with ALID = 70041,
name = GAS-DEEP3a-FR-023;DEEP3a-FR-025;DEEP3a-FR-033-54370.5648709 and
453 associates!
[foucault] 15:45:18 - RUNNING: asctoldac -i
tmp1190900004.519876.LbCP_s.asctoldac.txt -o
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6680245-a7e712330177956fb64c879787b36fd9e8f23e70.fits.slist.out
-c tmp1190900004.519876.LbCP_s.asctoldac.defs -t OBJECTS -b 1
[foucault] 15:45:18 - RUNNING: asctoldac -i
tmp1190900004.519876.LbCP_s.asctoldac.txt -o
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6617829-fbda44a69c1dab3d46296b5f53be22beb5acb1c7.fits.slist.out
-c tmp1190900004.519876.LbCP_s.asctoldac.defs -t OBJECTS -b 1
[foucault] 15:45:19 - RUNNING: asctoldac -i
tmp1190900004.519876.LbCP_s.asctoldac.txt -o
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6689499-0626e44428195810666447b24320eae844222463.fits.slist.out
-c tmp1190900004.519876.LbCP_s.asctoldac.defs -t OBJECTS -b 1
[foucault] 15:45:19 - RUNNING: asctoldac -i
tmp1190900004.519876.LbCP_s.asctoldac.txt -o
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6660892-57d1125fe9e52b29153deb64c1e9f89964b8805f.fits.slist.out
-c tmp1190900004.519876.LbCP_s.asctoldac.defs -t OBJECTS -b 1
[foucault] 15:45:19 - RUNNING: asctoldac -i
tmp1190900004.519876.LbCP_s.asctoldac.txt -o
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6610351-22282b08173d88fe9125e059a64d5243db58daa8.fits.slist.out
-c tmp1190900004.519876.LbCP_s.asctoldac.defs -t OBJECTS -b 1
[foucault] 15:45:19 - RUNNING: ldacpaste -i
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6680245-a7e712330177956fb64c879787b36fd9e8f23e70.fits.slist.out
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6617829-fbda44a69c1dab3d46296b5f53be22beb5acb1c7.fits.slist.out
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6689499-0626e44428195810666447b24320eae844222463.fits.slist.out
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6660892-57d1125fe9e52b29153deb64c1e9f89964b8805f.fits.slist.out
GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6610351-22282b08173d88fe9125e059a64d5243db58daa8.fits.slist.out
-o tmp1190900004.519876.LbCP_s.ccd1.cat1 -t OBJECTS
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
line 138, in execute
self.make_global_astrometric_solution()
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
line 177, in make_global_astrometric_solution
gastrom.make()
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
line 81, in make
self.make_global_astrom()
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
line 218, in make_global_astrom
incats, pairscat = self._make_astrom_cats()
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
line 251, in _make_astrom_cats
pairscat = self._retrieve_astrom_pairs()
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
line 379, in _retrieve_astrom_pairs
al_data_dict = al.associates.get_data(attrlist=attrlist)
File
"/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
line 907, in get_data_on_associates
c.execute(Ocommand)
cx_Oracle.DatabaseError: ORA-02393: exceeded call limit on CPU usage
These error messages do not speak to me...
And here are (some of) the astrometric parameters for these 5 files:
print q.is_valid,q.quality_flags, q.creation_date,q.CRVAL1,q.CRVAL2,
q.SIG_DRA, q.SIG_DDEC ,q.SIG_DRA_OVERLAP, q.SIG_DDEC_OVERLAP ,
q.N_OVERLAP,q.filter.name,q.reduced.OBJECT
... else: print 'NO VALID ASTROMETRY'
...
1 0 2007-09-21 13:33:23 171.2189 -21.70839 0.230177544969 0.22680029227
0.146810706385 0.115117787134 7207 #844 DEEP3a-FR-023
1 0 2007-09-21 13:33:28 171.25 -21.70524 0.225270016809 0.207282298011
0.185773325732 0.179577929405 6346 #844 DEEP3a-FR-025
1 0 2007-09-21 13:33:38 171.2478 -21.69674 0.249895167384 0.218377969391
0.187278549452 0.176399650447 6181 #844 DEEP3a-FR-034
1 0 2007-09-21 13:33:36 171.2272 -21.72005 0.256060120165 0.22070956672
0.178420261574 0.12062212583 7011 #844 DEEP3a-FR-033
1 0 2007-09-21 13:33:32 171.2169 -21.70266 0.239051612926 0.228794998717
0.161104576299 0.112454404906 7235 #844 DEEP3a-FR-032
Could you maybe try the last example with only 5 frames on your local
computer?
Any other idea?
Thanks,
Philippe
John P. McFarland wrote:
> Hi Philippe,
>
> After consultation with our DB experts, it appears that the RPC connection
> was to the HTM library indicating that there was something unusual with the
> astrometry of your data. This error likely caused the pickling error later
> on. Please check that your astrometry is reasonable and that your
> grid_targets are consistent and try again.
>
> Cheers,
>
>
> -=John
>
>
> On Mon, 24 Sep 2007, John P. McFarland wrote:
>
>
>> Hi Philippe,
>>
>> I think the error messages are somewhat terse, but nonetheless say exactly
>> what happened: a network error and an exceeded resource limit. What exactly
>> caused them and how to fix it, I can't be sure, but we will look into it get
>> back to you as soon as possible.
>>
>> Cheers,
>>
>>
>> -=John
>>
>>
>> On Fri, 21 Sep 2007, Philippe Heraudeau wrote:
>>
>>
>>> Hi ,
>>>
>>> I don't know what are these error messages. Could you please help?
>>> Cheers,
>>> Philippe
>>>
>>> This worked:
>>> dpu.run('GAstrom',object='DEEP3a*',instrument='WFI',filter='#844',C=1,chip='ccd50')
>>> # 32 chips
>>>
>>> This fails:
>>> dpu.run('GAstrom',object='DEEP3a*',instrument='WFI',filter='#844',C=1)
>>> # 32 * 8 chips
>>> .....
>>> [foucault] 14:15:17 - [node199] 13:48:48 - Using SourceList
>>> GAS-Sci-PHERAUDEAU-WFI-------#844-ccd57-Red---Sci-54360.6615498-16e57bd5703860af7f4cb7e5285f069f9568e633.fits.slist,
>>> SLID = 193791, number of sources = 313
>>> [foucault] 14:15:17 - [node199] 13:48:48 - Using SourceList
>>> GAS-Sci-PHERAUDEAU-WFI-------#844-ccd50-Red---Sci-54360.6639484-e47cf7e2ee094d029b0eef34268c3f6cbcb804a0.fits.slist,
>>> SLID = 192251, number of sources = 313
>>> [foucault] 14:15:17 - [node199] 13:48:48 - Using SourceList
>>> GAS-Sci-PHERAUDEAU-WFI-------#844-ccd55-Red---Sci-54360.6707366-78399516189335c491317ba037e58e500dca5055.fits.slist,
>>> SLID = 193341, number of sources = 166
>>> [foucault] 14:15:17 - [node199] 13:48:48 - Using SourceList
>>> GAS-Sci-PHERAUDEAU-WFI-------#844-ccd53-Red---Sci-54360.6638464-f9ac838d2ace470ba8637f42326b0001746d8c9b.fits.slist,
>>> SLID = 192351, number of sources = 247
>>> [foucault] 14:15:17 - [node199] 13:48:59 - Outer bounding box:
>>> (171.54369,-21.98915,170.83598,-21.98923,170.83631,-21.39761,171.54347,-21.39781
>>> )
>>> [foucault] 14:15:17 - [node199] 13:49:26 - Copied 78254 sources to
>>> scratch table
>>> [foucault] 14:15:17 - Error executing 'execute'
>>> [foucault] 14:15:17 - Exception: ORA-28576: lost RPC connection to
>>> external procedure agent
>>> [foucault] 14:15:17 -
>>> [foucault] 14:15:17 - File
>>> "/home/rugwn201/omegacen/awehome/opipe/current/astro/net/dpu_runner.py",
>>> line 174, in main
>>> [foucault] 14:15:17 - job.execute()
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
>>> line 138, in execute
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
>>> line 177, in make_global_astrometric_solution
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
>>> line 80, in make
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
>>> line 179, in make_associate
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 145, in make
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 200, in associate_sourcelists
>>> [foucault] 14:15:17 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 1617, in make_matched_associatelist
>>> [foucault] 14:15:17 - Error pickling job
>>> [foucault] 14:15:17 - Exception: can't pickle function objects
>>> [foucault] 14:15:17 - File
>>> "/home/rugwn201/omegacen/awehome/opipe/current/astro/net/dpu_runner.py",
>>> line 191, in main
>>> [foucault] 14:15:17 - pjob = pickle.dumps(job)
>>> [foucault] 14:15:17 - File
>>> "/home/rugwn201/omegacen/local/x86_64/lib/python2.3/copy_reg.py", line
>>> 69, in _reduce_ex
>>> [foucault] 14:15:17 - raise TypeError, "can't pickle %s objects" %
>>> base.__name__
>>>
>>>
>>> And with the same command I aslo got another error message:
>>> [foucault] 16:23:20 - [node199] 16:09:48 - Using SourceList
>>> GAS-Sci-PHERAUDEAU-WFI-------#844-ccd57-Red---Sci-54360.6608829-d98bcf15ac57f72b5ed27d823843a897b6e38c20.fits.slist,
>>> SLID = 193271, number of sources = 381
>>> [foucault] 16:23:20 - [node199] 16:09:58 - Outer bounding box:
>>> (171.54369,-21.98915,170.83598,-21.98923,170.83631,-21.39761,171.54347,-21.39781
>>> )
>>> [foucault] 16:23:20 - [node199] 16:10:17 - Copied 78254 sources to
>>> scratch table
>>> [foucault] 16:23:20 - Error executing 'execute'
>>> [foucault] 16:23:20 - Exception: ORA-02393: exceeded call limit on CPU
>>> usage
>>> [foucault] 16:23:20 -
>>> [foucault] 16:23:20 - File
>>> "/home/rugwn201/omegacen/awehome/opipe/current/astro/net/dpu_runner.py",
>>> line 174, in main
>>> [foucault] 16:23:20 - job.execute()
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
>>> line 138, in execute
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/DBRecipes/GAstrom.py",
>>> line 177, in make_global_astrometric_solution
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
>>> line 80, in make
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/GAstrometric.py",
>>> line 179, in make_associate
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 145, in make
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 200, in associate_sourcelists
>>> [foucault] 16:23:20 - File
>>> "/net/hercules01/data/users/heraud/ASTROWISE/opipe/astro/main/AssociateList.py",
>>> line 1617, in make_matched_associatelist
>>> [foucault] 16:23:20 - Error pickling job
>>> [foucault] 16:23:20 - Exception: can't pickle function objects
>>> [foucault] 16:23:20 - File
>>> "/home/rugwn201/omegacen/awehome/opipe/current/astro/net/dpu_runner.py",
>>> line 191, in main
>>> [foucault] 16:23:20 - pjob = pickle.dumps(job)
>>> [foucault] 16:23:20 - File
>>> "/home/rugwn201/omegacen/local/x86_64/lib/python2.3/copy_reg.py", line
>>> 69, in _reduce_ex
>>> [foucault] 16:23:20 - raise TypeError, "can't pickle %s objects" %
>>> base.__name__
>>> [foucault] 16:23:20 -
>>>
>>>
>>>
>>>
>> _______________________________________________
>> Issues mailing list
>> Issues at astro-wise.org
>> http://listman.astro-wise.org/mailman/listinfo/issues
>>
>>
>>
>> ** CRM114 Whitelisted by: From: "John P. McFarland" <mcfarland at astro.rug.nl **
>>
>> ** ACCEPT: CRM114 Whitelisted by: From: "John P. McFarland" <mcfarland at astro.rug.nl **
>>
>>
>>
> _______________________________________________
> Issues mailing list
> Issues at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/issues
>
>
>
--
==========================================================================
Dr. Philippe Heraudeau
Argelander-Institut für Astronomie Email: heraudeau at astro.uni-bonn.de
Auf dem Hügel 71 Tel.: +49 (0)228 733673
D-53121 Bonn Fax.: +49 (0)228 731775
Germany
=========================================================================
More information about the Issues
mailing list