I submitted a bunch of files to OPUS this morning. There was (and still is, as I write this) a banner warning that OPUS is experiencing high demand and that processing will take much longer than usual. Not a problem, I thought, I can wait a few hours.
A few hours turned into a bunch of hours, and late this afternoon I began checking the progress chart -- hovering over it tells you how many submissions failed, how many completed, and how many are waiting. When I first started checking I was seeing numbers like 3834 completed and 24 waiting. "Great," I thought, "Mine will be done shortly." Awhile later it was 3965/25, and a few minutes ago it was 4014/22. So how is it that someone keeps getting in front of me? Or are the "waiting" numbers not actually representative of the backup? Insights, anyone?
An hour and 40 minutes after my last post, the numbers now show 4347 completed and 23 waiting. I'm beginning to get the idea that the "waiting" number isn't accurate.
It's now 1:15 a.m. EDT (stats started over at midnight), and we're at 490/54. *Maybe* I'll get my results by morning...
No luck. As of 10:26 a.m. EDT there are 62 completed and 98 waiting. I wonder what's going on?
I just submitted a single, 3 hour session.
Solution in about 3 minutes
Jim, It may be just my paranoia, but who have you offended at NOAA?
I've certainly been giving the OPUS Projects team a lot of headaches over the last few months, but I didn't think that's the issue.
My solutions have finally been dribbling in for the last hour or so. I think there are about 30 of them.
I appear to be on the OPUS good-guy list this evening. The queue graphic has shown a backup all day, but tonight the dozens of files I submitted have all sailed through.
My suspicion is that the data files from the CORS sites near your position may be slow to download.
Hey @Norman_Oklahoma , orbit availability does not affect the queueing of solutions. The OPUS servers will not know if there is orbit data available to process your data until your data makes it thru the queue and is being processed.
If there are no orbits are available, you will get an "OPUS aborting" message with code 3003: An IGS orbit (precise, rapid or ultra-rapid) was not available at the time of data submission. Please re-submit the data in a day or two.
If only the Ultra-Rapid orbits are available, OPUS-RS (Rapid Static) will deliver a solution but with code 2005: NOTE: The IGS precise and IGS rapid orbits were not available at processing time. The IGS ultra-rapid orbit was/will be used to process the data. I have not seen OPUS-S (Static) use this code/message, but there may be circumstances in which it does.
@jhframe If you see this, but no response to your overall question, then I am still typing 🙂
First thing to note is that the OPUS servers utilize 2 separate queues.
Second thing to note is that no one is manually filtering OPUS uploads based on email addresses of users who have frustrated them. The OPUS queuing, processing, and delivering is all automated, there is only human intervention when things go wrong.
There is the Regular queue that the average user upload goes into, like when you submit 3-5 files in a morning of post-processing. These are processed in the order they are received. Note that whenever the OPUS servers have to translate (like when uploading a .DAT instead of RINEX) or decimate (like uploading a 5 s logging rate file... which is cut down to 30 s logging rate anyway) an uploaded file, that adds not just to your processing time but also to everyone else's queueing time. I believe this is a part of the reason that future versions of OPUS will only accept RINEX uploads.
Along with the Regular queue, there is the Overflow queue that handles the "power user" uploads. These are also processed in the order they are received, but this queue can end up much longer than the Regular. How do the OPUS servers decide what is a "power user" to send to the Overflow? There is a 50 upload threshold per day; once an you exceed the magic number of 50 uploads, all of your solutions are migrated from the Regular to the Overflow queue. I do not know if the "day" is defined by UTC midnight-to-midnight, or a previous rolling 24 hour period. The Overflow queue has at times been upwards of 30k files. And just like in the Regular queue, if Overflow users are uploading files that OPUS servers need to translate or decimate before processing... well then, that can get pretty backed up.
Unfortunately, I don't have an answer to which queue the OPUS Today graphic is illustrating (i.e. Regular, Overflow, both?) but I will ask for details.
Did you go over the 50 solution limit on the day you had delays? That's my best guess, but I was wrong once before.
Hope this helps clarify.
Thanks for the explanation! I didn't go over 50 that day, I don't think, so I expect I was in the regular queue. I just happened to be submitting when the backup was substantial enough that for a couple of days there was a banner noting that processing times would be much longer than usual. That banner isn't there now, so whatever unusual activity had been occurring appears to have abated.
Now, if I can just figure out why OPUS Projects won't respond to my adjustment submittals...
