Re: [netcdfgroup] simultaneous NetCDF writes to same file

I'm not quite sure what you mean by the "application layer"? My goal was to have 15 different processors process 15 segments of a satellite orbit, where each processor would write to the same NetCDF file in the most disk space efficient manner possible without any problems with simultaneous NetCDF writes. I had previously done the compression with the "nf_def_var_chunking" function call in non-parallel NetCDF. As this function does not seem to be available in parallel NetCDF, I'd be interested in alternative suggestions to accomplish my goal. Sorry I am more of the scientist type and am not a software engineer, so I may absorb some of these concepts a little slower than others. 

Thanks for the help,
Kris

On Jun 5, 2012, at 11:13 AM, Rob Latham wrote:


On Tue, May 29, 2012 at 02:29:12PM -0600, Russ Rew wrote:

Hi Kristopher,


I am processing a large volume of satellite data where multiple
processes could be simultaneously writing data to the same netcdf
file.   This has not been supported in previous NetCDF versions and
I've gotten fatal errors when two simultaneous writes conflicted.  I
now understand that recent NetCDF versions do support this
functionality. Could someone tell me or provide an example of what I 
need to do (i.e. new
function calls, options in netcdf open, etc...) to make this work for me? I've tried the pnetcdf package does not support chunking which 
I need to internally compress these files.


No, sorry, it's not supported in current netCDF versions either.
NetCDF-4 uses HDF5 as its storage layer, and HDF5 does not support
compression with parallel access, as explained here:


Is there any chance you can compress at the application layer?  Each
processor takes it's local hunk of data, compresses it, then writes to
the file.

I admit, you will quickly find out why parallel writes with
compression is not already implemented in these parallel I/O
libraries!

However, it's possible that at your application level, there may be
ways to simplify the parallel, compressed writes problem that a
general purpose library cannot use.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA


=========================================================
Kristopher Bedka
Science Systems & Applications, Inc. @ NASA Langley Research Center
Climate Science Branch
1 Enterprise Parkway, Suite 200
Hampton, VA 23666
Phone:  (757) 951-1920
Fax: (757) 951-1902
Kristopher.m.bedka@xxxxxxxx
=========================================================