I've had issues using a Python script in a pact with the following shebang:
#!/usr/bin/env python3
The script won't have access to STDOUT or STDERR
Mike Taylor
On Thu, Apr 23, 2026 at 1:23 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
> Is it running/finding the correct version of python? Sometimes if the
> script runs at the command line but not from cron or spawned by the LDM
> it's because it's running under a different environment and using a
> different version of python that doesn't have the modules that you need.
> Just another thing to check.
>
> Pete
>
> Pete Pokrandt - System Engineer IV
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
> ------------------------------
> *From:* ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of Mike
> Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
> *Sent:* Thursday, April 23, 2026 1:18 PM
> *To:* Matthew Foster - NOAA Affiliate <matthew.foster@xxxxxxxx>
> *Cc:* LDM <ldm-users@xxxxxxxxxxxxxxxx>
> *Subject:* Re: [ldm-users] Stumped by PIPE action
>
> Okay if we ruled those out I've got a few more thoughts...
>
> Broken pipes are usually a sign that the script is exiting before it reads
> from STDIN. Sometimes that means it's not executable, or the shebang path
> is incorrect, but I'm assuming if it's been working for you recently it's
> not a code issue. If it works when you run it manually then it could be
> environmental; how do you run it manually, and are you sure it's using the
> same python binary/environment as LDM would? IIRC you could also see this
> if the destination of that script is unreachable, e.g. downed mount, but
> again if it works manually that doesn't sound likely.
>
> Something else you can do to help debug this is wrap the python script's
> execution in a shell script which logs error output, and adjust your pqact
> to invoke that. That way you can see exactly how that script is failing by
> using that intended LDM pipeline.
>
> Hope this helps,
> -Mike
>
> On Thu, Apr 23, 2026 at 2:06 PM Matthew Foster - NOAA Affiliate <
> matthew.foster@xxxxxxxx> wrote:
>
> Thanks for the quick reply Mike.
>
> pqactcheck returns "syntactically correct"
>
> I've restarted LDM a couple of times this week in the process of trying to
> figure this out. We use a systemd script for stopping/starting that
> recreates the queue on start, so the queue is fresh. This machine (AWS
> instance, actually) has 16 CPU cores and 64 GB of RAM. RAM, CPU usage and
> disk space all look normal.
>
> I've administered LDMs for many years, and I've never had a pqact issue
> stump me like this one.
>
> Matt
>
>
> On Thu, Apr 23, 2026 at 12:49 PM Mike Zuranski <mike@xxxxxxxxxxxxxxxxxxx>
> wrote:
>
> Hi Matt,
>
> Some basic troubleshooting questions:
>
> Does ldmadmin pqactcheck show any errors? When was the last time LDM was
> restarted or the PQ remade? Does the health of your system (e.g. disk
> space, memory, cpu) appear normal?
>
> -Mike
>
> On Thu, Apr 23, 2026 at 1:44 PM Matthew Foster - NOAA Affiliate <
> matthew.foster@xxxxxxxx> wrote:
>
> I have a PIPE action in our pqact.conf that was working...until it
> stopped. I can't figure out what changed. Putting pqact in DEBUG yielded
> the following log messages...
>
> 20260423T171759.226677Z pqact[3286699]
> palt.c:processProduct:1332 INFO 928481a80643522717d70b2fa139c28e
> 35526969 20260423171755.509410 NOTHER 1130006 TICY70 KNES 231707
> 20260423T171759.226808Z pqact[3286699] palt.c:prodAction:1217
> DEBUG pipe: {cmd: "-close
> /usr/local/ldm/agents/storeAndIngestFile.py
> /data_store/polar/20260423/17/TICY70_KNES_23170
> 7_1130006.2026042317 TICX99", ident: "TICY70 KNES 231707"}
> 20260423T171759.227206Z pqact[3286699] filel.c:pipe_open:1854
> DEBUG 6 3295768
> 20260423T171759.227247Z pqact[3286699]
> filel.c:pipe_prodput:2174 DEBUG 6 TICY70 KNES 231707
> 20260423T171759.227414Z pqact[3295768] filel.c:pipe_open:1819
> INFO Executing decoder
> "/usr/local/ldm/agents/storeAndIngestFile.py"
> 20260423T171759.292967Z pqact[3286699]
> filel.c:fl_removeAndFree:421 DEBUG Deleting failed PIPE entry:
> cmd="-close /usr/local/ldm/agents/storeAndIngestFile.py
> /data_store/polar/20260423/
> 17/TICY70_KNES_231707_1130006.2026042317 TICX99", pid=3295768
> 20260423T171759.293001Z pqact[3286699]
> filel.c:pipe_close:1901 DEBUG 6, 3295768
> 20260423T171759.293050Z pqact[3286699] pbuf.c:pbuf_flush:111
> ERROR Broken pipe
> 20260423T171759.293070Z pqact[3286699] pbuf.c:pbuf_flush:111
> ERROR Couldn't write to pipe: fd=6, len=4096, cmd="-close
> /usr/local/ldm/agents/storeAndIngestFile.py
> /data_store/polar/20260423/17/TICY70_KNES_231707_1130006.2026042317 TICX99"
> 20260423T171759.293088Z pqact[3286699] palt.c:prodAction:1229
> ERROR Couldn't process product: feedtype=NOTHER,
> pattern="^(TICY70) (KNES) (..)(..)(..)", action=pipe, args="-close
> /usr/local/ldm/agents/storeAndIngestFile.py
> /data_store/polar/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H TICX99"
> 20260423T171759.293385Z pqact[3286699] filel.c:reap:3056
> WARN Child 3295768 exited with status 1
>
> The pqact.conf entry is...
> ANY<TAB>^(TICY70) (KNES) (..)(..)(..)
> <TAB>PIPE<TAB>-close /usr/local/ldm/agents/storeAndIngestFile.py
> /data_store/polar/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H TICX99
>
> I also ran the Python script from the command line, and it runs without
> errors.
>
> I'm thinking a clue might be the presence of "-close" in the cmd entries
> above, but I can't see anything wrong with the pqact.conf entry.
>
> Also, multiple products/patterns that use this PIPE action and script are
> all failing with these errors.
>
> I appreciate any input!
>
> Matt
>
>
> --
> Matt Foster
> Sr. Application Development Met.
> NOAA Affiliate - KBR
>
>
> _________________________________________________________
> NOTE: All exchanges posted to NSF Unidata maintained email lists are
> made publicly available through the web. Users who post to any of the
> lists we maintain are reminded to remove any personal information that
> they do not want to be made public.
>
> NSF Unidata ldm-users Mailing List
> (ldm-users@xxxxxxxxxxxxxxxx)
> For list information, to unsubscribe, or change your membership options,
> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
> <https://urldefense.com/v3/__https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/__;!!Mak6IKo!OKwA_h0qESJpH-obji8ljVM3JW2BNySVouiad5FB8A2C4a1QjLB_kQJsnN1Pkewxplb_FcdtvN_vhrfiNsS_$>
>
>
>
> --
> Matt Foster
> Sr. Application Development Met.
> NOAA Affiliate - KBR
>
>
> _________________________________________________________
> NOTE: All exchanges posted to NSF Unidata maintained email lists are
> made publicly available through the web. Users who post to any of the
> lists we maintain are reminded to remove any personal information that
> they do not want to be made public.
>
> NSF Unidata ldm-users Mailing List
> (ldm-users@xxxxxxxxxxxxxxxx)
> For list information, to unsubscribe, or change your membership options,
> visit: https://mailinglists.unidata.ucar.edu/listinfo/ldm-users/
>