Well, this is embarrassing, but I'm just going to own it. ;-)
I had...
try:
with open(f, 'wb') as fh:
shutil.copyfileobj(sys.stdin.buffer, f)
except Exception as e:
print("ERROR ", e)
Note the copyfileobj is referencing *f* rather than *fh*
I must have looked at this code over a dozen times and didn't see it until
this morning. Also, it must not have ever worked with this error, although,
I did see a couple of occasions where it appeared to work. :shrug:
Anyway, it's working consistently now that I fixed this simple error.
Thank you to everyone who chimed in to help!
Matt
On Fri, Apr 24, 2026 at 4:13 PM Stephen Adams <sadams@xxxxxxxx> wrote:
> Matt,
>
> Hopefully this is stupid question but...
> are you writing to a local disk, or a network one?
>
> *Steve Adams | AWIS Weather Services*
> 888.798.9955 | sadams@xxxxxxxx | www.awis.com
>
>
> On Fri, Apr 24, 2026 at 3:00 PM Matthew Foster - NOAA Affiliate <
> matthew.foster@xxxxxxxx> wrote:
>
>> Yet more fun!
>>
>> I just discovered that it doesn't fail every time. The Python script
>> reads the entire product from stdin, writes it to disk, and then sends a
>> message to QPID (AWIPS).
>>
>> I think the read/write part is failing, as I have a zero-byte file
>> resulting from a failed run. I might just put a bash script in front of it
>> that executes "cat > filename", then call the Python.
>>
>> Matt
>>
>>
>> On Thu, Apr 23, 2026 at 10:08 PM Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
>> wrote:
>>
>>> Matt,
>>>
>>> I'll level with you, at this point I'm stumped too. In my experience
>>> this doesn't sound like a LDM issue, especially if it started spontaneously
>>> and is not resolved by remaking the product queue and either an LDM or an
>>> OS restart. There must be some clue though, either by invoking the script
>>> manually in different ways (e.g. cat, xargs), using different python
>>> interpreters or something else to recreate the issue manually and checking
>>> the exit code that way. Or by wrapping the PQACT PIPE command in a shell
>>> script to capture/log any error output to a designated log file. There must
>>> be another breadcrumb somewhere...
>>>
>>> Best,
>>> -Mike
>>>
>>> On Thu, Apr 23, 2026 at 4:12 PM Matthew Foster - NOAA Affiliate <
>>> matthew.foster@xxxxxxxx> wrote:
>>>
>>>> I have the python interpreter hard-coded and explicit in the script
>>>> shebang line.
>>>>
>>>>
>>>> On Thu, Apr 23, 2026 at 3:10 PM Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> I'm old school - I usually say
>>>>>
>>>>> which python
>>>>>
>>>>> at the command prompt where it works, which should return something
>>>>> like
>>>>>
>>>>> /usr/bin/python
>>>>>
>>>>> or
>>>>>
>>>>> ~/miniforge3/bin/python
>>>>>
>>>>> and then hard code that into my script on the top line - e.g.
>>>>>
>>>>> #!/home/poker/miniforge3/bin/python
>>>>>
>>>>> Pete Pokrandt - System Engineer IV
>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>> ------------------------------
>>>>> *From:* Michael Taylor - NOAA Affiliate <michael.c.taylor@xxxxxxxx>
>>>>> *Sent:* Thursday, April 23, 2026 2:59 PM
>>>>> *To:* Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>> *Cc:* Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>; Matthew Foster - NOAA
>>>>> Affiliate <matthew.foster@xxxxxxxx>; LDM <ldm-users@xxxxxxxxxxxxxxxx>
>>>>> *Subject:* Re: [ldm-users] Stumped by PIPE action
>>>>>
>>>>> I've had issues using a Python script in a pact with the following
>>>>> shebang:
>>>>>
>>>>> #!/usr/bin/env python3
>>>>>
>>>>> The script won't have access to STDOUT or STDERR
>>>>>
>>>>> Mike Taylor
>>>>>
>>>>> On Thu, Apr 23, 2026 at 1:23 PM Pete Pokrandt <poker@xxxxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>> Is it running/finding the correct version of python? Sometimes if the
>>>>> script runs at the command line but not from cron or spawned by the LDM
>>>>> it's because it's running under a different environment and using a
>>>>> different version of python that doesn't have the modules that you need.
>>>>> Just another thing to check.
>>>>>
>>>>> Pete
>>>>>
>>>>> Pete Pokrandt - System Engineer IV
>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>>>>> 608-262-3086 - poker@xxxxxxxxxxxx
>>>>> ------------------------------
>>>>> *From:* ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of
>>>>> Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
>>>>> *Sent:* Thursday, April 23, 2026 1:18 PM
>>>>> *To:* Matthew Foster - NOAA Affiliate <matthew.foster@xxxxxxxx>
>>>>> *Cc:* LDM <ldm-users@xxxxxxxxxxxxxxxx>
>>>>> *Subject:* Re: [ldm-users] Stumped by PIPE action
>>>>>
>>>>> Okay if we ruled those out I've got a few more thoughts...
>>>>>
>>>>> Broken pipes are usually a sign that the script is exiting before it
>>>>> reads from STDIN. Sometimes that means it's not executable, or the
>>>>> shebang
>>>>> path is incorrect, but I'm assuming if it's been working for you recently
>>>>> it's not a code issue. If it works when you run it manually then it could
>>>>> be environmental; how do you run it manually, and are you sure it's using
>>>>> the same python binary/environment as LDM would? IIRC you could also see
>>>>> this if the destination of that script is unreachable, e.g. downed mount,
>>>>> but again if it works manually that doesn't sound likely.
>>>>>
>>>>> Something else you can do to help debug this is wrap the python
>>>>> script's execution in a shell script which logs error output, and adjust
>>>>> your pqact to invoke that. That way you can see exactly how that script
>>>>> is
>>>>> failing by using that intended LDM pipeline.
>>>>>
>>>>> Hope this helps,
>>>>> -Mike
>>>>>
>>>>> On Thu, Apr 23, 2026 at 2:06 PM Matthew Foster - NOAA Affiliate <
>>>>> matthew.foster@xxxxxxxx> wrote:
>>>>>
>>>>> Thanks for the quick reply Mike.
>>>>>
>>>>> pqactcheck returns "syntactically correct"
>>>>>
>>>>> I've restarted LDM a couple of times this week in the process of
>>>>> trying to figure this out. We use a systemd script for stopping/starting
>>>>> that recreates the queue on start, so the queue is fresh. This machine
>>>>> (AWS
>>>>> instance, actually) has 16 CPU cores and 64 GB of RAM. RAM, CPU usage and
>>>>> disk space all look normal.
>>>>>
>>>>> I've administered LDMs for many years, and I've never had a pqact
>>>>> issue stump me like this one.
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>> On Thu, Apr 23, 2026 at 12:49 PM Mike Zuranski <
>>>>> mike@xxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>> Hi Matt,
>>>>>
>>>>> Some basic troubleshooting questions:
>>>>>
>>>>> Does ldmadmin pqactcheck show any errors? When was the last time LDM
>>>>> was restarted or the PQ remade? Does the health of your system (e.g. disk
>>>>> space, memory, cpu) appear normal?
>>>>>
>>>>> -Mike
>>>>>
>>>>> On Thu, Apr 23, 2026 at 1:44 PM Matthew Foster - NOAA Affiliate <
>>>>> matthew.foster@xxxxxxxx> wrote:
>>>>>
>>>>> I have a PIPE action in our pqact.conf that was working...until it
>>>>> stopped. I can't figure out what changed. Putting pqact in DEBUG yielded
>>>>> the following log messages...
>>>>>
>>>>> 20260423T171759.226677Z pqact[3286699]
>>>>> palt.c:processProduct:1332 INFO
>>>>> 928481a80643522717d70b2fa139c28e
>>>>> 35526969 20260423171755.509410 NOTHER 1130006 TICY70 KNES 231707
>>>>> 20260423T171759.226808Z pqact[3286699]
>>>>> palt.c:prodAction:1217 DEBUG pipe: {cmd: "-close
>>>>> /usr/local/ldm/agents/storeAndIngestFile.py
>>>>> /data_store/polar/20260423/17/TICY70_KNES_23170
>>>>> 7_1130006.2026042317 TICX99", ident: "TICY70 KNES 231707"}
>>>>> 20260423T171759.227206Z pqact[3286699]
>>>>> filel.c:pipe_open:1854 DEBUG 6 3295768
>>>>> 20260423T171759.227247Z pqact[3286699]
>>>>> filel.c:pipe_prodput:2174 DEBUG 6 TICY70 KNES 231707
>>>>> 20260423T171759.227414Z pqact[3295768]
>>>>> filel.c:pipe_open:1819 INFO Executing decoder
>>>>> "/usr/local/ldm/agents/storeAndIngestFile.py"
>>>>> 20260423T171759.292967Z pqact[3286699]
>>>>> filel.c:fl_removeAndFree:421 DEBUG Deleting failed PIPE entry:
>>>>> cmd="-close /usr/local/ldm/agents/storeAndIngestFile.py
>>>>> /data_store/polar/20260423/
>>>>> 17/TICY70_KNES_231707_1130006.2026042317 TICX99", pid=3295768
>>>>> 20260423T171759.293001Z pqact[3286699]
>>>>> filel.c:pipe_close:1901 DEBUG 6, 3295768
>>>>> 20260423T171759.293050Z pqact[3286699]
>>>>> pbuf.c:pbuf_flush:111 ERROR Broken pipe
>>>>> 20260423T171759.293070Z pqact[3286699]
>>>>> pbuf.c:pbuf_flush:111 ERROR Couldn't write to pipe: fd=6,
>>>>> len=4096, cmd="-close /usr/local/ldm/agents/storeAndIngestFile.py
>>>>> /data_store/polar/20260423/17/TICY70_KNES_231707_1130006.2026042317
>>>>> TICX99"
>>>>> 20260423T171759.293088Z pqact[3286699]
>>>>> palt.c:prodAction:1229 ERROR Couldn't process product:
>>>>> feedtype=NOTHER, pattern="^(TICY70) (KNES) (..)(..)(..)", action=pipe,
>>>>> args="-close /usr/local/ldm/agents/storeAndIngestFile.py
>>>>> /data_store/polar/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
>>>>> TICX99"
>>>>> 20260423T171759.293385Z pqact[3286699] filel.c:reap:3056
>>>>> WARN Child 3295768 exited with status 1
>>>>>
>>>>> The pqact.conf entry is...
>>>>> ANY<TAB>^(TICY70) (KNES) (..)(..)(..)
>>>>> <TAB>PIPE<TAB>-close /usr/local/ldm/agents/storeAndIngestFile.py
>>>>> /data_store/polar/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H TICX99
>>>>>
>>>>> I also ran the Python script from the command line, and it runs
>>>>> without errors.
>>>>>
>>>>> I'm thinking a clue might be the presence of "-close" in the cmd
>>>>> entries above, but I can't see anything wrong with the pqact.conf entry.
>>>>>
>>>>> Also, multiple products/patterns that use this PIPE action and script
>>>>> are all failing with these errors.
>>>>>
>>>>> I appreciate any input!
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>> --
>>>>> Matt Foster
>>>>> Sr. Application Development Met.
>>>>> NOAA Affiliate - KBR
>>>>>
>>>>>
>>>>> _________________________________________________________
>>>>> NOTE: All exchanges posted to NSF Unidata maintained email lists are
>>>>> made publicly available through the web. Users who post to any of the
>>>>> lists we maintain are reminded to remove any personal information that
>>>>> they do not want to be made public.
>>>>>
>>>>> NSF Unidata ldm-users Mailing List
>>>>> (ldm-users@xxxxxxxxxxxxxxxx)
>>>>> For list information, to unsubscribe, or change your membership
>>>>> options,
>>>>> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>>>>> <https://urldefense.com/v3/__https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/__;!!Mak6IKo!OKwA_h0qESJpH-obji8ljVM3JW2BNySVouiad5FB8A2C4a1QjLB_kQJsnN1Pkewxplb_FcdtvN_vhrfiNsS_$>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Matt Foster
>>>>> Sr. Application Development Met.
>>>>> NOAA Affiliate - KBR
>>>>>
>>>>>
>>>>> _________________________________________________________
>>>>> NOTE: All exchanges posted to NSF Unidata maintained email lists are
>>>>> made publicly available through the web. Users who post to any of the
>>>>> lists we maintain are reminded to remove any personal information that
>>>>> they do not want to be made public.
>>>>>
>>>>> NSF Unidata ldm-users Mailing List
>>>>> (ldm-users@xxxxxxxxxxxxxxxx)
>>>>> For list information, to unsubscribe, or change your membership
>>>>> options,
>>>>> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>>>>> <https://urldefense.com/v3/__https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/__;!!Mak6IKo!KREilj5LBJgGxyGqqj0ZS2l_I96HF_8USOaKMxDK6hfc0wqJZ_Wam5ccgn4t-pgejbz6GO7UWhn98WdfQVyHNKhjhXQccw$>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Matt Foster
>>>> Sr. Application Development Met.
>>>> NOAA Affiliate - KBR
>>>>
>>>>
>>>>
>>
>> --
>> Matt Foster
>> Sr. Application Development Met.
>> NOAA Affiliate - KBR
>>
>>
>> _________________________________________________________
>> NOTE: All exchanges posted to NSF Unidata maintained email lists are
>> made publicly available through the web. Users who post to any of the
>> lists we maintain are reminded to remove any personal information that
>> they do not want to be made public.
>>
>> NSF Unidata ldm-users Mailing List
>> (ldm-users@xxxxxxxxxxxxxxxx)
>> For list information, to unsubscribe, or change your membership options,
>> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>>
>
--
Matt Foster
Sr. Application Development Met.
NOAA Affiliate - KBR