Daryl and Mike - I did follow up with NOAA on the Canadian RADAR data. As
a reference, a clip from the answer:
"The general flow of Canadian data as I understand it from its source ->
NCEI works like this...
- *Canada -> NCEP/NWSTG -> NCF -> SBN -> Unidata*
My group (*redacted for safety purposes*) is essentially the operators of
the NWSTG who receives data from Canada and transmits data to the NCF where
it is uploaded to the NWS SBN (Satellite Broadcast Network).
August 19th was an important date for the NCF/SBN as that was when the
final major realignment of datasets and channels on the SBN took place,
with my group making upstream dataflow changes as a result of the
realignment plan.
An NWS SCN describing the SBN changes was issued and is available here:
https://www.weather.gov/media/notification/pdf_2025/scn25-59_channel_realignment_to_SBN_no3.pdf
As the above SCN indicates, one of the major changes planned was to move
all US/Canadian/Caribbean radar data on the SBN to the EXP channel. After
moving the Canadian and Caribbean radar products to the new SBN channel
though, there was enough bandwidth improvement on the original NMC channel
(which has highest priority) to keep US radar data there as described in
this SCN update:
https://www.weather.gov/media/notification/pdf_2025/scn25-59%20_channel_realignment_to_SBN_number_3_aaa
.
With the Canadian radar data moving to the SBNs' EXP channel, it is
possible that change caused it to default to a different LDM feedtype
(IDS|DDSPLUS) when the downstream Unidata SBN ingestor processes put the
data into their LDM."
This, along with the Great Lakes wind/wave data output (^L) are ending up
with a feedtype that doesn't aline with expectations because the SBN
Product header is being utilized differently than designed or has been used
in the past, such that the noaaportIngester won't know that the Canadian
RADAR is RADAR or the wave/wind model output as NGRID or HDS.
That is why, for the most part, the NEXRAD L3 data (also called NIDS for us
older people) being moved from the NWSTG/NMC SBN PID to the EXP PID simply
worked - it is coded as NEXRAD at the level of the SBN product header. It
is the inclusion of new data that is problematic due to the fact that a lot
of resources that knew all of these moving parts at the encoding stage have
retired, passed away, or were forced to leave due to budget concerns.
My initial thought was to simply update the LDM noaaportIngester to catch
the errant data, but was given an immediate brake-check by the car ahead of
me that the fix would have to be propagated across the entirety of the IDD
simultaneously and across all versions that happen to running at the same
time. So, the solution has be one that can be utilized by all versions and
permutations - and thus the regex filtering on the user side that doesn't
touch the insertion point.
As you are both knowledgeable in the area of constructing regular
expressions to the benefit of your operations, I post here for the benefit
of others as a reminder that monitoring data and constructing regular
expressions is your best defense against overutilization of storage. The
filtering string that *may* be used:
DDPLUS|IDS ^([^LS]|S[^D]|SD[^C]|SDC[^N])
My response does not address the metadata archaeology that has to be
performed on the unidentifiable GRIB - that is a whole separate issue. For
those that have students in data labs, this is an opportunity to provide
those students with experience by giving them the task of hunting down the
identity of these new products and providing clarity on the correct
decoding.
-
Stonie Cooper, PhD
Software Engineer III
NSF Unidata
cooper@xxxxxxxx
On Thu, Jan 22, 2026 at 1:09 PM Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
wrote:
> Daryl,
>
> Fascinating, Discipline 78 isn't an official discipline as defined in any
> grib2 tables, so no wonder wgrib2 spits that out.
>
> KWBN is indeed the NDFD source as you noted, but center 7 is NCEP, center
> 8 is NDFD...
>
> If I look at current grib2 tables and ignore the discipline, here are what
> seem like top candidates matching your other values:
>
> NCEP:
>
> https://github.com/NOAA-EMC/wgrib2/blob/7bce4ba3baf96c85fb05a334c3ea8fbe01b41367/wgrib2/gribtables/ncep/gribtable.dat
> {0,0,0,255,7,1,1,192, "CRAIN", "Categorical Rain", "-"},
> {1,0,0,255,7,1,1,192, "CPOZP", "Probability of Freezing Precipitation",
> "%"},
> {2,0,0,255,7,1,1,192, "CANL", "Cold Advisory for Newborn Livestock", "-"},
> {3,0,0,255,7,1,1,192, "USCT", "Scatterometer Estimated U Wind Component",
> "m/s"},
> {10,0,0,255,7,1,1,192, "OMLU", "Ocean Mixed Layer U Velocity", "m/s"},
>
> NDFD:
>
> https://github.com/NOAA-EMC/wgrib2/blob/7bce4ba3baf96c85fb05a334c3ea8fbe01b41367/wgrib2/gribtables/ndfd/NDFD_gribtable.dat
> {0,1,0,255,8,1,1,192, "WX", "Weather information", "WxInfo"},
>
> Between that and the non-standard L..... product code, I think you're
> right if the timing aligns with that PNS your example is closest to a
> "Ocean Mixed Layer U Velocity" variable in a provisional/exper status.
> While that "mixed layer" isn't listed in that PNS, discipline 10 such as
> that product would be an oceanographic product if it were in prod. I've
> set up a pqact to capture new products and I'll take another look when more
> come in.
>
> Opinion: IF I'm close to right on this, my two cents are it'd merit a
> message to those contact points as I don't think them landing in IDS|DDPLUS
> is intentional. HDS I'd buy, but not here. Although, with Can. Radar
> persisting in this feedtype too would two data points make a trend? If so
> there could be a larger issue looming...
>
> Thanks for the head-scratcher, this is worth keeping an eye on.
>
> -Mike
>
> On Thu, Jan 22, 2026 at 10:12 AM Herzmann, Daryl E [AGRON] <
> akrherz@xxxxxxxxxxx> wrote:
>
>> Hi Mike,
>>
>> Thanks for the response and interest. The WMO source is KWBN, which is
>> the NDFD. My guess is that the products are related to this:
>>
>>
>> https://www.weather.gov/media/notification/pdf_2025/pns25-81_Experimental_OPC_High_Seas_Gridded_Fcsts_NDFD.pdf
>>
>> wgrib2 isn't very helpful as my grib tables must be old. An example:
>>
>> 1:28:d=2026012112:var discipline=78 center=7 local_table=1 parmcat=1
>> parm=192:surface - surface:69 hour fcst:
>>
>> daryl
>>
>> ________________________________________
>> From: Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
>> Sent: Wednesday, January 21, 2026 4:27 PM
>> To: Herzmann, Daryl E [AGRON]
>> Cc: LDM
>> Subject: Re: [ldm-users] Grib data in the IDS|DDPLUS feed
>>
>> Hey Daryl,
>>
>> I have to ask, did you bother to peak at the data to see what it actually
>> is? My hunch is model output but L products are strangely omitted from the
>> following table: https://www.weather.gov/tg/tablea So now I'm curious.
>>
>> -Mike
>>
>>
>> On Wed, Jan 21, 2026 at 3:48 PM Herzmann, Daryl E [AGRON] <
>> akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
>> Greetings,
>>
>> I was reviewing my IDS|DDPLUS archival and noticed that my hourly files
>> have a large variance in size.
>>
>> 26M 2026012104.txt
>> 390M 2026012105.txt
>> 27M 2026012106.txt
>> ...snipped...
>> 30M 2026012116.txt
>> 387M 2026012117.txt
>> 27M 2026012118.txt
>>
>> So I had a look at my LDM telemetry logging and see the the following
>> byte counts for IDS|DDPLUS for the 17 UTC hour today
>>
>> wmo_ttaaii | sum
>> ------------+------------
>> SDCN01 | 1700051333
>> LDUD00 | 34844596
>> LDUD06 | 34771861
>> LDUD12 | 34731012
>> LDUE00 | 34702011
>> LDUD18 | 34668540
>> LDUE06 | 34664366
>> LDUE12 | 34593737
>> LDUH12 | 34560734
>> LDUE18 | 34527322
>> LDUH06 | 34485257
>> LDUF06 | 34466943
>> LDUF00 | 34458010
>> LDUF12 | 34452545
>> LDUF18 | 34412543
>> LDUH00 | 34405475
>> LDUG18 | 34382563
>> LDUG00 | 34381983
>> LDUG06 | 34376450
>> LDUG12 | 34366267
>> LDUH18 | 16845611
>>
>>
>> Of course, the Canadian RADAR (SDCN01) is well known fun that I was
>> already excluding from the file write, but what is LDUD00? Well, it is grib
>> data:
>>
>> LDUD00 KWBN 211200
>> GRIB^@^@^A^B...
>>
>> So folks may wish to update their pqact's to exclude these as well!
>>
>> daryl
>> _________________________________________________________
>> NOTE: All exchanges posted to NSF Unidata maintained email lists are
>> made publicly available through the web. Users who post to any of the
>> lists we maintain are reminded to remove any personal information that
>> they do not want to be made public.
>>
>> NSF Unidata ldm-users Mailing List
>> (ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>)
>> For list information, to unsubscribe, or change your membership options,
>> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>>
> _________________________________________________________
> NOTE: All exchanges posted to NSF Unidata maintained email lists are
> made publicly available through the web. Users who post to any of the
> lists we maintain are reminded to remove any personal information that
> they do not want to be made public.
>
> NSF Unidata ldm-users Mailing List
> (ldm-users@xxxxxxxxxxxxxxxx)
> For list information, to unsubscribe, or change your membership options,
> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>