Hello Kevin,
Thank you! FYI: I am going to run this against this script which will tell me
which ones are missing based on a current hourly METAR file. Change the
directory and the name of the filing convention. It should make work for
getting missing new stations in a lot easier for you.
#!/bin/bash
#
# mcidas_missing_stations.sh - report station IDs present in raw SA/METAR data
# but absent from the McIDAS station database.
#
# Usage:
# ./mcidas_missing_stations.sh [options] [file...]
#
# With no file arguments, uses the file for the previous UTC hour, computed
# from the clock rather than by picking the newest name -- a METAR with a bad
# timestamp can create a file dated into the future.
#
# Options:
# -c FILE path to station database
# (default: /home/mcidas/data/final_station_database_mcidas.txt)
# -A count any entry as known, flagged for METAR or not
# (default: an entry must carry DATATYPE=...METAR)
# -d DIR raw data directory (default: /home/data/sao)
# -b N end N hours back from now, UTC (default: 1)
# -n N use N consecutive hours ending there (default: 1)
# -m N minimum report count to list a station (default: 1)
# -x REGEX exclude IDs matching REGEX (awk regex, matched against the ID)
# -q IDs only, no counts or header (for piping)
# -h this help
#
# Examples:
# ./mcidas_missing_stations.sh # previous UTC hour
# ./mcidas_missing_stations.sh -b 0 # current hour (partial)
# ./mcidas_missing_stations.sh -n 24 -m 20 # last 24 hours,
persistent only
# ./mcidas_missing_stations.sh -A # ignore the METAR flag
# ./mcidas_missing_stations.sh -n 48 -x '^KQ' -m 20
# ./mcidas_missing_stations.sh -q -n 24 > /tmp/to_add.ids
# ./mcidas_missing_stations.sh /home/data/sao/2026083118.sao
#
set -u
CTY=/home/mcidas/data/final_station_database_mcidas.txt
DATADIR=/home/data/sao
BACKHOURS=1
NFILES=1
MINCOUNT=1
EXCLUDE=""
QUIET=0
ALLMODE=0
while getopts "c:d:b:n:m:x:Aqh" opt; do
case "$opt" in
c) CTY="$OPTARG" ;;
d) DATADIR="$OPTARG" ;;
b) BACKHOURS="$OPTARG" ;;
n) NFILES="$OPTARG" ;;
m) MINCOUNT="$OPTARG" ;;
x) EXCLUDE="$OPTARG" ;;
A) ALLMODE=1 ;;
q) QUIET=1 ;;
h) sed -n '2,35p' "$0" | sed 's/^# \{0,1\}//' ; exit 0 ;;
*) echo "try -h" >&2 ; exit 2 ;;
esac
done
shift $((OPTIND - 1))
# ---------------------------------------------------------------------------
# With no files named, derive filenames from the clock.
#
# Deliberately NOT "newest file in the directory": a METAR carrying a bad day
# or hour makes pqact create a file dated into the future, and that bogus file
# sorts last. Working back from the current UTC hour avoids it entirely.
# ---------------------------------------------------------------------------
DEFAULTED=0
SELDESC=""
if [ "$#" -eq 0 ]; then
if [ ! -d "$DATADIR" ]; then
echo "$(basename "$0"): no data directory: $DATADIR (try -h)" >&2
exit 2
fi
sel=()
absent=()
i=0
while [ "$i" -lt "$NFILES" ]; do
back=$(( BACKHOURS + i ))
stamp=$(date -u -d "$back hours ago" +%Y%m%d%H) || exit 1
f="$DATADIR/$stamp.sao"
if [ -r "$f" ]; then
sel=( "$f" "${sel[@]}" ) # prepend: oldest ends up first
else
absent=( "${absent[@]}" "$stamp" )
fi
i=$(( i + 1 ))
done
if [ "${#sel[@]}" -eq 0 ]; then
echo "$(basename "$0"): none of the expected files exist in $DATADIR"
>&2
echo "looked for: ${absent[*]}" >&2
echo "(is the clock UTC-correct, or the feed stalled? try -h)" >&2
exit 2
fi
if [ "${#absent[@]}" -gt 0 ] && [ "$QUIET" -eq 0 ]; then
echo "$(basename "$0"): no file for hour(s): ${absent[*]}" >&2
fi
first=$(basename "${sel[0]}" .sao)
last=$(basename "${sel[$(( ${#sel[@]} - 1 ))]}" .sao)
if [ "${#sel[@]}" -eq 1 ]; then
SELDESC="${first}Z"
else
SELDESC="${first}Z .. ${last}Z"
fi
set -- "${sel[@]}"
DEFAULTED=1
fi
if [ ! -r "$CTY" ]; then
echo "$(basename "$0"): cannot read station database: $CTY" >&2
exit 1
fi
TMP=$(mktemp -d) || exit 1
trap 'rm -rf "$TMP"' EXIT
# ---------------------------------------------------------------------------
# Station IDs known to McIDAS.
#
# Records are keyword=value, not columnar, and the ID field is frequently
# empty (synoptic-only and VOR entries carry a numeric IDN but no station
# ID). Two traps:
#
# * "IDN=" also contains "ID", so the match must require whitespace before
# "ID=" to avoid keying off the numeric station number.
# * On an empty ID the next token is LATLON=, and a bare [A-Z0-9]* match
# happily returns "LATLON" as the ID. Anchoring the far side on LATLON=
# makes an empty ID come out empty, and it is then dropped.
# ---------------------------------------------------------------------------
extract_ids() {
sed -n
's/.*[[:space:]]ID=[[:space:]]*\([A-Za-z0-9]*\)[[:space:]]*LATLON=.*/\1/p' \
| tr 'a-z' 'A-Z' \
| grep -E '^[A-Z0-9]{3,8}$' \
| sort -u
}
# Every entry, flagged or not.
extract_ids < "$CTY" > "$TMP/cty_all.ids"
# Only entries McIDAS will actually use for METAR. DATATYPE is the last
# keyword on the record, so [^=]* cannot run past it into another field --
# that keeps a station NAME containing "METAR" from matching.
grep -E 'DATATYPE=[^=]*METAR' "$CTY" | extract_ids > "$TMP/cty_metar.ids"
if [ "$ALLMODE" -eq 1 ]; then
cp "$TMP/cty_all.ids" "$TMP/cty.ids"
else
cp "$TMP/cty_metar.ids" "$TMP/cty.ids"
fi
if [ ! -s "$TMP/cty.ids" ]; then
echo "$(basename "$0"): parsed no station IDs from $CTY -- wrong file or
format?" >&2
exit 1
fi
# A wrong -c file can parse "successfully" and make every station look
# missing. A real McIDAS station database has thousands of entries.
if [ "$(wc -l < "$TMP/cty.ids")" -lt 100 ]; then
echo "$(basename "$0"): only $(wc -l < "$TMP/cty.ids") IDs parsed from
$CTY" >&2
echo "that is far too few for a McIDAS station database -- wrong file?" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Station IDs actually reporting.
#
# Matches "XXXX ddhhmmZ" anywhere on the line, which catches both the
# "METAR KORD 301753Z" form and the bare "KORD 301753Z" form used inside
# collective bulletins. Requiring the trailing Z keeps WMO abbreviated
# headers ("SAMX55 MMMY 301800") out of the results.
# ---------------------------------------------------------------------------
cat -- "$@" 2>/dev/null \
| tr -d '\r' \
| grep -Eo '\b[A-Z][A-Z0-9]{3} [0-9]{6}Z' \
| awk '{print $1}' \
| sort > "$TMP/obs.ids"
if [ ! -s "$TMP/obs.ids" ]; then
echo "$(basename "$0"): no observations found in input files" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Difference, counted by how often each unknown station reported.
# ---------------------------------------------------------------------------
grep -Fxv -f "$TMP/cty.ids" "$TMP/obs.ids" \
| uniq -c \
| sort -rn > "$TMP/missing.cnt"
if [ -n "$EXCLUDE" ]; then
awk -v ex="$EXCLUDE" '$2 !~ ex' "$TMP/missing.cnt" > "$TMP/missing.flt"
mv "$TMP/missing.flt" "$TMP/missing.cnt"
fi
awk -v min="$MINCOUNT" '$1 >= min' "$TMP/missing.cnt" > "$TMP/missing.out"
# ---------------------------------------------------------------------------
# Split the misses. A station absent from the file needs a new record; one
# that is present but lacks the METAR datatype only needs its DATATYPE line
# edited. Different fixes, so report them apart.
# ---------------------------------------------------------------------------
awk '{print $2}' "$TMP/missing.out" | sort > "$TMP/missing.ids"
comm -12 "$TMP/missing.ids" "$TMP/cty_all.ids" > "$TMP/unflagged.ids"
comm -23 "$TMP/missing.ids" "$TMP/cty_all.ids" > "$TMP/absent.ids"
pick() { # $1 = id list; emit "count id" lines from missing.out, count-sorted
awk 'NR==FNR { want[$1]; next } $2 in want' "$1" "$TMP/missing.out"
}
# ---------------------------------------------------------------------------
# Report.
# ---------------------------------------------------------------------------
if [ "$QUIET" -eq 1 ]; then
awk '{print $2}' "$TMP/missing.out"
exit 0
fi
known=$(wc -l < "$TMP/cty.ids")
seen=$(sort -u "$TMP/obs.ids" | wc -l)
miss=$(wc -l < "$TMP/missing.cnt")
shown=$(wc -l < "$TMP/missing.out")
if [ "$ALLMODE" -eq 1 ]; then
printf 'station database : %s (%s IDs, METAR flag ignored)\n' "$CTY"
"$known"
else
printf 'station database : %s (%s METAR-flagged IDs)\n' "$CTY" "$known"
fi
if [ "$DEFAULTED" -eq 1 ]; then
if [ "$#" -eq 1 ]; then
printf 'input file : %s (hour %s)\n' "$1" "$SELDESC"
else
printf 'input files : %s (hours %s)\n' "$#" "$SELDESC"
fi
else
printf 'input files : %s\n' "$#"
fi
printf 'IDs reporting : %s\n' "$seen"
printf 'IDs not usable : %s\n' "$miss"
if [ "$MINCOUNT" -gt 1 ] || [ -n "$EXCLUDE" ]; then
printf 'listed below : %s (min count %s%s)\n' \
"$shown" "$MINCOUNT" "${EXCLUDE:+, excluding /$EXCLUDE/}"
fi
if [ ! -s "$TMP/missing.out" ]; then
printf '\nnothing missing.\n'
exit 0
fi
if [ "$ALLMODE" -eq 1 ]; then
printf '\n%8s %s\n' "REPORTS" "ID"
awk '{printf "%8d %s\n", $1, $2}' "$TMP/missing.out"
exit 0
fi
if [ -s "$TMP/absent.ids" ]; then
printf '\nnot in database at all -- need a new record (%s):\n' \
"$(wc -l < "$TMP/absent.ids")"
printf '%8s %s\n' "REPORTS" "ID"
pick "$TMP/absent.ids" | awk '{printf "%8d %s\n", $1, $2}'
fi
if [ -s "$TMP/unflagged.ids" ]; then
printf '\npresent but not flagged METAR -- need DATATYPE edited (%s):\n' \
"$(wc -l < "$TMP/unflagged.ids")"
printf '%8s %s\n' "REPORTS" "ID"
pick "$TMP/unflagged.ids" | awk '{printf "%8d %s\n", $1, $2}'
fi
Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01DD3D65.11EDC7D0]
From: Kevin C Baggett <kevinb@xxxxxxxxxxxxx>
Sent: Saturday, September 5, 2026 3:54 PM
To: Sebenste, Gilbert <sebensteg@xxxxxxx>
Cc: mcidas-x@xxxxxxxxxxxxxxxx
Subject: [External] Re: [mcidas-x] Text METAR database for McIDAS?
CAUTION: This email originated from outside of COD’s system. Do not click
links, open attachments, or respond with sensitive information unless you
recognize the sender and know the content is safe.
Gilbert,
I uploaded what we use for McIDAS-X just before creating our binary file
STNDB.CORE to
[cid:image002.png@01DD3D65.11EDC7D0]
final_station_database<https://bin.ssec.wisc.edu/pub/incoming/final_station_database.txt>
Text Document · 4.5
MB<https://bin.ssec.wisc.edu/pub/incoming/final_station_database.txt>
We’ve been adding the stations you’ve provided recently, but I might be missing
a couple.
Kevin
On Sep 5, 2026, at 2:26 PM, Sebenste, Gilbert
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>> wrote:
Hello everyone,
I am pretty sure I am still missing METAR sites, both domestic and
international, in McIDAS. Is there a text-based version of the database
somewhere that I can cross-check and see which ones are missing? Or can I make
one?
Gilbert Sebenste
Meteorology Support Analyst
<image001.png>
_________________________________________________________
NOTE: All exchanges posted to NSF Unidata maintained email lists are
made publicly available through the web. Users who post to any of the
lists we maintain are reminded to remove any personal information that
they do not want to be made public.
NSF Unidata mcidas-x Mailing List
(mcidas-x@xxxxxxxxxxxxxxxx<mailto:mcidas-x@xxxxxxxxxxxxxxxx>)
For list information, to unsubscribe, or change your membership options,
visit:
https://urldefense.com/v3/__https://mailinglists.unidata.ucar.edu/listinfo/mcidas-x/__;!!Mak6IKo!MrAdR1WNIobz4nrJrxyFnhQDsdqOfg8cMvH0YlXKegoIaHRUYjyDCgdYfRTCkZMyvfulThl6pS2K-AStv_mtfQ$
