Re: [thredds] Java heap problems with large ASCII downloads

I do believe THREDDS does all f its work in memory, if so you have to greatly increase the java hep settings. We run at 1500Mb and still run into this. I would be curious to know if even with a very large heap size if file sof tha size can be successfully transferred. 

-Roy

On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:


Hi,

I use thredds as opendap server for some large netcdf-files (CF-1.0,
size > 1.1GB). When selecting to download the whole file with the 'Get 
ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.

I'm currently using a default memory setup, i.e. -Xmx512MB for tomcat.
How much memory will I need to avoid this problem. Does DODS read the
whole file at once or variable after variable, or even slice after
slice? Or are there some adjustments I can make to reduce memory
consumption?

Best regards,

Heiko




Details:
DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError: Java 
heap space
ReqState:
 serverClassName:    'thredds.server.opendap.NcDODSServlet'
 dataSet:            'XXXX.nc'
 requestSuffix:      'ascii'
 CE:                 ''
 compressOK:          true
 InitParameters:
   DebugOn: ''
   maxNetcdfFilesCached: '10'

java.lang.OutOfMemoryError: Java heap space
       at ucar.nc2.N3raf.readData(N3raf.java:77)
       at ucar.nc2.N3iosp.readData(N3iosp.java:159)
       at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
       at ucar.nc2.Variable._read(Variable.java:923)
       at ucar.nc2.Variable.read(Variable.java:618)
       at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
       at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
       at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
       at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
       at
opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java: 1630) 
       at
thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
...



Thredds-Version:
Built-By: caron
Built-On: 2008-02-20 22:56:05
Implementation-Title: THREDDS Data Server
Implementation-Version: 3.16.33
Implementation-Vendor: UCAR/Unidata

Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"