Re: [thredds] tomcat-crash with thredds4: Too many open files

[Heiko Klein <Heiko.Klein@xxxxxx>]

I haven't enabled remote monitoring yet, but the problem disappeared 
since I set the low-watermark of the files-cache to 0, so after 10min 
the cache is completely cleared.

I think the culprit are files being removed from another 
cleanup-process, while still being in the cache. (i.e. forecast-files).

With a cache of only 10min, instead of 200 files forever, the problem is 
not completely solved, but doesn't seem to appear any longer (stable for 
a week yet). And we didn't recognize any performance degration.

Heiko


Richard Signell wrote:
> Heiko,
>
> Did you ever figure this out?  We just had our
> thredds4.0.26/tomcat6.0.18 crash with "Too man open files" also.
>
> -Rich
>
> On Fri, Sep 18, 2009 at 4:53 AM, Heiko Klein <Heiko.Klein@xxxxxx> wrote:
> > Hi,
> >
> > we just had a crash of the thredds4.0.25/tomcat server and I was able to
> > analyse a bit the system before the operators restarted it.
> >
> > Error message was: 'Too many open files' and checking the open files with
> > lsof or /proc/PID/fd showed 1021 open netcdf-files for the tomcat-pid. We
> > haven't made any changes to the thredds4 configuration concerning the
> > file-caches, so they should be between 200-400. We're using WCS, too, so
> > this might add some more files, but still not up to 1021.
> >
> > We also use some aggregations of netcdf files, so one 'thredds-aggregation'
> > might consist of some hundred netcdf-files. Maybe this is counted as one
> > file in the fileCache?
> >
> > I currently try to avoid further crashes by increasing the ulimit -n from
> > 1024 to 16384, but I would rather be able to tune the file-cache properly
> > instead of throwing more resources at it.
> >
> >
> > Best regards,
> >
> > Heiko
> >
> > --
> > Dr. Heiko Klein                              Tel. + 47 22 96 32 58
> > Development Section / IT Department          Fax. + 47 22 69 63 55
> > Norwegian Meteorological Institute           http://www.met.no
> > P.O. Box 43 Blindern  0313 Oslo NORWAY
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> > http://www.unidata.ucar.edu/mailing_lists/

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY