Dear Peter,
I read all the e-mails from this thread and checked the statistics of
your machines, including the latencies.
ldmingest01.nwc.ou.edu [6.8.1]
<http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?ldmingest01.nwc.ou.edu>
ldmingest02.nwc.ou.edu [6.8.1]
<http://www.unidata.ucar.edu/cgi-bin/rtstats/siteindex?ldmingest02.nwc.ou.edu>
See that you are losing data, more on ingest02 than ingest01, and that
the latencies on both machines reach a delay of 3600s many times, when
your LDM "gives up", forget about the old products and request for the
newer ones. Everytime this happen you lose data.
To me seems more a situation where the LDM data-requests are more than
the current capability of your network to serve your LDMs.
I am used to similar situations. So, I suggest:
1) check the network connection to idd.unidata.ucar.edu, your feeder.
Traceroute is ok, but I prefer to use the "mtr":
mtr idd.unidata.ucar.edu
keep watching for 5 minutes. If any "packet loss" appears on the path,
you found where the problem is. These things will keep your network
working, perhaps a bit slower, but only you (data hunger) will notice
anything...
2) Your LDM machines are getting a lot of data. Someone in your path
could have set a kind of "packet shaping", "QoS" or anything else to
reduce the priority of who is getting too much data (you). For the
others the network will be nice and faster, and only your LDMs will suffer.
If this is the case, there are tricks to cheat these things, like to
replace one request line for several ones. This worked fine to improve
the massive LDM feeds to South America.
replace:
REQUEST ANY ".*" idd.unidata.ucar.edu
By:
REQUEST NEXRAD2 ".*" idd.unidata.ucar.edu
REQUEST CONDUIT ".*" idd.unidata.ucar.edu
REQUEST FNMOC ".*" idd.unidata.ucar.edu
REQUEST NGRID ".*" idd.unidata.ucar.edu
REQUEST NEXRAD3 ".*" idd.unidata.ucar.edu
REQUEST HDS ".*" idd.unidata.ucar.edu
and so on...
Check to see if this improves the output. At this moment your statistics
are:
Data Volume Summary for ldmingest01.nwc.ou.edu
Maximum hourly volume 12967.169 M bytes/hour
Average hourly volume 3520.793 M bytes/hour
Data Volume Summary for ldmingest02.nwc.ou.edu
Maximum hourly volume 5016.503 M bytes/hour
Average hourly volume 845.772 M bytes/hour
Cheers,
Waldenio.
--
Waldenio Gambi de Almeida
Instituto Nacional de Pesquisas Espaciais
Centro de Previsão de Tempo e Estudos Climáticos
Divisão de Operacoes
Rodovia Pres. Dutra, km 40
Cachoeira Paulista/SP - Brasil
12630-000
Phone : (12)3186-8541
e-mail : waldenio.almeida@xxxxxxxxxxxxx