[thredds] THREDDS 3.16: Memory issues

[Carlos Valiente <carlos.valiente@xxxxxxxxx>]

Hi! We are having memory exhaustion issues in our THREDDS setup (using 
THREDDS 3.16.35 running under Tomcat 6.0.10 with Java 1.6.0_04-b12 on 
SLES 9 Linux x86_64).

We have aggregated the following NetCDF files:

http://ensembles.ecmwf.int/thredds/catalog/demeter-non-agg/catalog.html

by variable:

http://ensembles.ecmwf.int/thredds/demeter/variables.html

using the following XML catalogue snippets:

[catalog.xml]
..
        <datasetScan name="DEMETER Data - Non-aggregated"
                ID="demeterNonAggregated"
                path="demeter-non-agg"
                location="/vol/data/demeter/"
                serviceName="thisDODS">

                <filter>
                        <include wildcard="*.nc"/>
                </filter>
        </datasetScan>

        <catalogRef name="DEMETER data - Aggregated variables"
                xlink:href="variables.xml"
                xlink:title="DEMETER data - Aggregated variables" />
..

[variables.xml]
..

        <dataset name="Geopotential"
                ID="demeter-geopotential"
                urlPath="demeter/g">

                <serviceName>thisDODS</serviceName>

                <netcdf 
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
                        <aggregation dimName="time" type="joinExisting">
                                <scan location="/vol/data/demeter/129/"
                                        suffix=".nc" />
                        </aggregation>
                </netcdf>
        </dataset>
..

        <dataset name="Total precipitation accumulated in the previous 
24 hours"
                ID="demeter-rain"
                urlPath="demeter/rain">

                <serviceName>thisDODS</serviceName>

                <netcdf 
xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
                        <aggregation dimName="time" type="joinExisting">
                                <scan location="/vol/data/demeter/228/"
                                        suffix=".nc" />
                        </aggregation>
                </netcdf>
        </dataset>
..

We are able to retrieve data from the individaul NetCDF files, and also 
from some aggregated variables, like 'Total precipitation accumulated in 
the previous 24 hours':

http://ensembles.ecmwf.int/thredds/dodsC/demeter/rain.html

However, with some other variables like 'Geopotential':

http://ensembles.ecmwf.int/thredds/dodsC/demeter/g.html

,the download fails, and on catalina.out we get the following stack trace:

DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError: Java 
heap space
ReqState:
  serverClassName:    'thredds.server.opendap.NcDODSServlet'
  dataSet:            'demeter/g'
  requestSuffix:      'dods'
  CE:                 ''
  compressOK:          false
  InitParameters:
    DebugOn: ''
    maxNetcdfFilesCached: '10'

java.lang.OutOfMemoryError: Java heap space
        at ucar.ma2.ArrayFloat.<init>(ArrayFloat.java:86)
        at ucar.ma2.ArrayFloat$D5.<init>(ArrayFloat.java:335)
        at ucar.ma2.ArrayFloat$D5.<init>(ArrayFloat.java:327)
        at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:52)
        at ucar.ma2.ArrayFloat.factory(ArrayFloat.java:36)
        at ucar.ma2.Array.factory(Array.java:130)
        at ucar.ma2.Array.factory(Array.java:79)
        at ucar.nc2.ncml.Aggregation.read(Aggregation.java:604)
        at ucar.nc2.ncml.Aggregation.read(Aggregation.java:659)
        at ucar.nc2.dataset.VariableDS._read(VariableDS.java:277)
        at ucar.nc2.Variable.read(Variable.java:618)
        at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
        at opendap.dap.Server.SDArray.serialize(SDArray.java:426)
        at opendap.dap.Server.CEEvaluator.send(CEEvaluator.java:275)
        at 
opendap.servlet.AbstractServlet.doGetDAP2Data(AbstractServlet.java:805)
        at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:1626)
        at 
thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
[..]

We're starting up Tomcat with -Xmx406m, in order to give it 4 GB of 
memory, but that does not seem to be enough. We've tried raising that 
limit to 6 GB, which is the total memory (physicall + virtual) available 
on that box, but it did not make any difference.

Are THREDDS memory requirements for aggregation higher than those 
figures, or are we perhaps doing something not too clever with our setup?

Cheers,

Carlos Valiente <carlos.valiente@xxxxxxxxx>