http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco/nco (Source Code, Issues, Releases)
What's new?
Version 5.4.0 contains a raft of features and fixes:
Some notable features:
All operators allow setting the netCDF4 metadata block size.
Many operators now optionally preserve orphaned dimensions.
ncclimo can now climatologize simulation ensembles.
ncremap tolerates absent hybrid p0/P0 variables.
Users interested in these features should upgrade to 5.4.0.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncclimo now generates ensemble monthly climatologies.
Simply feed the files from all ensemble members to ncclimo
using any supported input method. Redirecting standard input
is often most concise. Consider three ensemble members:
# Example: Three Member Decadal-mean Ensemble Climatology
drc_top=/big/doe/computer/E3SMv3
caseid1=v3.LR.historical_0051
caseid2=v3.LR.historical_0101
caseid3=v3.LR.historical_0151
fl_nsm1=${drc_top}/${caseid1}/run/${caseid1}.elm.h0.185?-??.nc
fl_nsm2=${drc_top}/${caseid2}/run/${caseid2}.elm.h0.185?-??.nc
fl_nsm3=${drc_top}/${caseid3}/run/${caseid3}.elm.h0.185?-??.nc
ls ${fl_nsm1} ${fl_nsm2} ${fl_nsm3} | ncclimo --fml_nm=ensemble ...
This approach scales to any number of ensemble members.
All the usual climo-mode options, such as regridding, work.
For N-member ensembles, ncclimo averages N-times
as many months as a single member climo to get the mean
January, February, etc. This takes about N-times longer.
Seasonal and annual means are computed from monthly means so
their speed is unchanged, as is regridding speed.
For now, the output statistics are climatological means.
Future work could support outputting standard deviations.
Let us know what you would like to see in future versions.
B. ncclimo --mth_end now defaults to 11 months after --mth_srt.
Formerly, users specifying a non-January start month (--mth_srt)
were required to also explicitly provide --mth_end, even though in
climatology mode the end month must always be 11 months after the
start month. Now --mth_end is automatically set to mth_srt+11 (mod
12) when it is not specified by the user. For example, specifying
--mth_srt=5 now automatically sets --mth_end=4 (May through April
annual cycle). Explicitly-specified --mth_end values are still
accepted and take precedence. In timeseries mode, --mth_end may
still be explicitly specified when the range of months spans a
non-integral number of years. Alternatively, one can now avoid
the --mth_srt and --mth_end options entirely even with non-January
start months. Do this by specifying the years in YYYYMM format:
ncclimo -s 2013 -e 2014 --mth_srt=5 ... # mth_end automatically set to 4
ncclimo -s 2013 -e 2014 --mth_srt=5 --mth_end=4 ... # same, explicit
ncclimo -s 201305 -e 201404 ... # Omit --mth_srt/end options.
http://nco.sf.net/nco.html#yr_srt
http://nco.sf.net/nco.html#mth_srt
http://nco.sf.net/nco.html#mth_end
C. The --rad (retain_all_dimensions) option is now supported by many
data operators: ncap2, ncecat, nces, ncflint, ncpdq, and ncwa.
Previously only ncks supported --rad. When --rad is specified,
dimensions that are not referenced by any extracted variable (orphaned
dimensions) are retained in the output file rather than silently
dropped. The E3SM ELM model, for example, uses orphan dimensions to
trigger different physics treatments. When ncks is invoked in printing
mode, --rad also causes orphaned dimensions to be printed
(they are not printed by default). Synonyms --retain_all_dimensions,
--orphan_dimensions, and --rph_dmn are also accepted. Thanks to Jon
Wolfe (LANL) for prompting this feature.
ncap2 --rad -s 'one=1' in.nc out.nc # Keep all dimensions
nces --rad in.nc out.nc # Keep all dimensions
ncks --rad -m -v T in.nc # Print all dimensions
http://nco.sf.net/nco.html#rad
D. All data operators now support the newish netCDF ability
to set the metadata block size in netCDF4 files. Option --blk_sz
allows users to specify sizes other than the default 2048 bytes.
Large files chunked files often require much more space because
the metadata block is where HDF5 stores the location of every
chunk. If this list grows too long it must point to another
metadata block and HDF5 must follow the trail, possibly through
thousands of metadata blocks, just to find a single chunk.
CMIP7 submissions will need to take advantage of larger sizes.
One megabyte is a good size for large, chunked, netCDF4 files.
Setting blk_sz=0 will reset the block size to the default value
of the active netCDF4/HDF5 library (which may increase beyond
2048 B in the future). This option only affects HDF5 files.
ncks -4 -O ~/nco/data/in.nc ~/foo.nc # Default 2048 B
ncks -4 -O --blk=0 ~/nco/data/in.nc ~/foo.nc # Default
ncks -4 -O --blk=1048576 ~/nco/data/in.nc ~/foo.nc # 1 MB
http://nco.sf.net/nco.html#blk_sz
E. ncremap now gracefully tolerates the absence of the reference
surface pressure variable in hybrid sigma-pressure coordinate
datasets. Previouly these variables (usually named p0 or P0) had to be
contained in the the vertical grid of the input data. However, the
de facto convention is that p0=P0=100000 Pa. Now when the reference
pressure is missing (as is the case with older versions of EAMxx),
ncremap supplies this value internally and completes without failing.
F. ncremap now supports regridding MODIS MOD10CM datasets.
The dimensions in those dataset have been added to ncremap's
internal database so explicitly specifying which is latitude
and which is longitude is unnecessary.
G. ncclimo and ncremap now use the srun "--exclusive" option
to prevent compute nodes from sharing their resources with
concurrently running others jobs. This been shown to substantially
increase throughput of some NCO-based analysis workflows such as
LIVVkit. Thanks to Mike Kelleher (ORNL) for suggesting this.
H. In ncremap the --alg option is now a synonym for --alg_typ.
Just a small change to reduce typing.
I. ncremap and ncclimo have been significantly refactored.
They now run in "set -eou" mode and use modernized expression
evaluation and test constructs that fail on any Bash-reported errors.
This makes debugging easier. Both codes have now been tested with
the most common use-cases. However, we might have missed some
"unbound variables". Please send us an error report if you notice
this. The workaround is to run with an earlier version of NCO.
BUG FIXES:
A. Previous versions of the BitGroom algorithm could produce
incorrect answers when single-precision variables were
quantized with NSD=7 significant digits. The maximum value
now accepted is NSD=6. Thanks to Ed Hartnett for catching this.
B. Certain combinations of climo-mode and input-mode could
cause the CASEID to be prepended to filenames twice.
The workaround is to use a different input method.
The solution is to upgrade.
Full release statement at http://nco.sf.net/ANNOUNCE
--
Charlie Zender, Dept. of Earth System Science
University of California, Irvine 949-891-2429