Re: [thredds] Java heap problems with large ASCII downloads

[Roy Mendelssohn <Roy.Mendelssohn@xxxxxxxx>]

Yes, but in reality it doesn't appear to work on even that big a  
setting.  I believe some people have THREDDS working in 64-bit  (see  
for example http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2008-March/001008.html) 
 ,  but we do not.  Our experience is that is takes a reasonably  
larger amount of heap than the file size to deal with it.

Happy Thanksgiving Nathan,

-Roy

On Nov 27, 2008, at 10:01 AM, Nathan Potter wrote:

> Isn't the JVM heap a capped at 2GB for 32-bit operating systems/JVM's?
>
>
>
>
>
>
> On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:
>
> > I do believe THREDDS does all f its work in memory,  if so you have  
> > to
> > greatly increase the java hep settings.  We run at 1500Mb and still
> > run into this.  I  would be curious to know if even with a very large
> > heap size if file sof tha size can be successfully transferred.
> >
> > -Roy
> >
> > On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:
> >
> > > Hi,
> > >
> > > I use thredds as opendap server for some large netcdf-files (CF-1.0,
> > > size >  1.1GB). When selecting to download the whole file  with the
> > > 'Get
> > > ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.
> > >
> > > I'm currently using a default memory setup, i.e. -Xmx512MB for  
> > > tomcat.
> > > How much memory will I need to avoid this problem. Does DODS read  
> > > the
> > > whole file at once or variable after variable, or even slice after
> > > slice? Or are there some adjustments I can make to reduce memory
> > > consumption?
> > >
> > > Best regards,
> > >
> > > Heiko
> > >
> > >
> > >
> > >
> > > Details:
> > >
> > > DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError:
> > > Java
> > > heap space
> > > ReqState:
> > > serverClassName:    'thredds.server.opendap.NcDODSServlet'
> > > dataSet:            'XXXX.nc'
> > > requestSuffix:      'ascii'
> > > CE:                 ''
> > > compressOK:          true
> > > InitParameters:
> > >  DebugOn: ''
> > >  maxNetcdfFilesCached: '10'
> > >
> > > java.lang.OutOfMemoryError: Java heap space
> > >      at ucar.nc2.N3raf.readData(N3raf.java:77)
> > >      at ucar.nc2.N3iosp.readData(N3iosp.java:159)
> > >      at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
> > >      at ucar.nc2.Variable._read(Variable.java:923)
> > >      at ucar.nc2.Variable.read(Variable.java:618)
> > >      at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
> > >      at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
> > >      at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
> > >      at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
> > >      at
> > > opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
> > >      at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
> > > 1630)
> > >      at
> > > thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
> > >      at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
> > >      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > ...
> > >
> > >
> > >
> > > Thredds-Version:
> > > Built-By: caron
> > > Built-On: 2008-02-20 22:56:05
> > > Implementation-Title: THREDDS Data Server
> > > Implementation-Version: 3.16.33
> > > Implementation-Vendor: UCAR/Unidata
> > >
> > > Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.
> > >
> > > --
> > > Dr. Heiko Klein                              Tel. + 47 22 96 32 58
> > > Development Section / IT Department          Fax. + 47 22 69 63 55
> > > Norwegian Meteorological Institute           http://www.met.no
> > > P.O. Box 43 Blindern  0313 Oslo NORWAY
> > > _______________________________________________
> > > thredds mailing list
> > > thredds@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe,  visit: 
> > > http://www.unidata.ucar.edu/mailing_lists/
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> > Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > 1352 Lighthouse Avenue
> > Pacific Grove, CA 93950-2097
> >
> > e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
> > voice: (831)-648-9029
> > fax: (831)-648-8440
> > www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit: 
> > http://www.unidata.ucar.edu/mailing_lists/
>
> = = =
> Nathan Potter                        ndp at opendap.org
> OPeNDAP, Inc.                        541.752.1852

**********************
"The contents of this message do not reflect any position of the U.S.  
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"