Maybe these facts may clear things up a bit...
- I have levelling notes from May of 2021 coming from #800 (elev = 900.687) to #822 (elev = 961.865m) that close back to 800 to within 16mm
- If I run ONLY?ÿmy most recent static data in StarNet holding the elevation of #800, I get an elevation to #822 of 961.860m. This is using UTM-11N, NAD83, GRS80. Geoid HT2_2010v70.
- If I run ONLY?ÿmy conventional data holding the elevation of #800 and turning off geoid modelling I get an elevation to #822 of 961.873m. Now if I turn on my geoid modelling and leave everything else the same I get an elevation to 822 of 961.752m
- Today I spoke to one of the guys that work at Microsurvey (StatNet) and he himself told me the software is incorrectly applying geoid modelling to my conventional data. He is going to speak with the coders about having this issue fixed.
- I dont know if it matters geoid wise, but this is a traverse in the Rockies.
?ÿ
I get the feeling you may not be familiar with StarNet? And may be applying your thinking to how the software you use works. I don't mean that in a rude way at all, but if Im holding 800 then why would my vertical datum matter when comparing total station values to those from levelling? Maybe I wasn't clear enough about that before. It clear as day that the geoid modelling is being applied to my conventional data in a improper way.
?ÿ
"I think I should steer this conversation back or track a bit with some observations I've made. Lets forget horizontal for now and focus on elevations.?ÿ 1. The GPS observations have nothing to do with the problems I'm seeing."
To be fair, it most likely is ALL about your GPS observations. GPS is always 3-D, and the various "heights" (ellipsoid/geoid/orthometric) are always the fly in the ointment.
?ÿ
Tying into a published "national" system with GPS technology has certain requirements that go along with it to make all the terrestrial data sources happy with each other and with GPS data sources. Mismatch anything not specificaly meant to go together, and you can get answers that are very close to the variation we're seeing now, or something so obvious we don't need to spend days going down a rabbit hole of minutia to figure out what the problem is.
You never shared the metadata associated with the sources of your record survey data, data collection, or desired final output. That information is probably the first and one of the most important pieces of information to identify and consider for planning and execution.
For example:
You shared a screen shot showing the StarNet Project Options Adjustment tab, and it had a greyed out Coordinate System of "UTM83-11", because you were adjusting to "local". Okay, fine. (UTM 11 North NAD83(CSRS) v7-2010 appears to be the most current, but "UTM83-11" isn't detailed enough to know if they are the same.)
In UTM Zone 11 (N), there are at least 11 published horizontal datums (each being slightly different for various reasons) for Canada, there are at least 7 published geoid models for Canada (and a heights transformation tool), and there are at least 2 published vertical datums to consider as well. That's a lot of things to dial in correctly to get all this fancy tech to spit out meaningul answers.
This note is also prominently displayed on the 5 benchmark datasheets, and the 1 CORS datasheet I could find (that was within about 250km of your site): "The CGVD2013 height reported in the Vertical Data table is approximate because it is calculated from historical levelling data and not from the combination of GNSS and a geoid model."
Benchmarks:
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
?ÿ
CORS:
?ÿ
?ÿ
?ÿ
Height Tool:
?ÿ
?ÿ
Vertical Datum Transformations:
?ÿ
?ÿ
NAD83(CSRS) v7:
?ÿ
?ÿ
?ÿ
Why might all of this matter?
It appears as though you may be using the wrong vertical datum with the wrong geoid model (with/without the height transformation tool?) and the wrong geometric datum that links all these things together so they work properly. All of these compounding factors will likely cause the the LSA to blow up when combining all 3 data sets in a homogenous single adjustment.
Basically, your terrestrial optical data sources will coincide with each other no matter what the vertical datum because it's all local, and it appears that they do by what you've shared:?ÿ "2. If I run the exact same data with the exact same project settings but change the coordinate system to local (vs UTM) and use my scaled local coordinates (vs UTM) I get results that agree very closely to the old levelling data I was provided. ...And on the east side of the traverse (extended with todays data) im seeing a difference in elevation of about 0.16m between the same point in each project."
"I am unfamiliar with vertical deflection modelling, how does one go about finding the correct .vdf file? When I run the conventional on its own for a check point I get 15mm off the given value. When I run the GPS on it's own I get 30mm off the given value. ...But the weird thing is when I run them together I get about 50mm off the given value.
What vertical datum is/are your level run(s)? The vertical datum difference between GCVD28 (leveling) and CGVD2013 (leveling) is in the -0.457m range for the benchmarks. The vertical datum difference between GCVD28 (leveling) and CGVD2013 (GNSS) is in the -0.132m range for the 1 CORS. Those numbers were fairly close to the differences causing the problem(s), no?
If this simple mistake (which we've all made) is the culprit, adjusting the various error estimates in an attemp to feather out a blunder/systematic error is not really the appropriate way to go about it.?ÿ You can polish a turd, and put lipstick on a pig, but they both still stink.
Everyone on this message board has been burned exactly once with mismatching data types/sources before they learned never to do it again.
So, despite the uncertainty between the numerous combinations of reference frames, epoch dates, vertical datums, geoid models, and no source(s)/date(s) of your survey/record data, you received a nice crash-course in geodesy and a lesson in Least Squares Adjusment.
I dont know if it matters geoid wise, but this is a traverse in the Rockies
That may be very significant. The geoid model could very possibly be haywire in such a place.?ÿ
Thanks for some of the background info on your data, it is helpful.
I, like lots of folks here, have used various software over the years (StarNet, StarLev, LevProc, GPSurvey, TGO, TBC, LGO and any number of others from the MS-DOS days, to early Windows 95 versions). I will admit that I have not used StarNet in the recent past, but it appears to look/behave similar to the way it did years ago. StarNet was a great independent 3rd party LSA package that could accept and adjust almost any manufacturer's data format, given the proper converter. And I agree with others that say the StarNet user interface wasn't the most intuitive, and I could be wrong, but it appears that they have not progressed away from an ascii.txt input style, and a coordinates only .txt file "seed value" approach. (Which was both a blessing and a curse, because it was the source of it's great ability to accept most 3rd party data formats, but also made it vulnerable to easily using incorrect/outdated "seed" values by missing a step during the recalculate/recompute phase of data reduction, or not selecting the most current .dat file reflecting those changes for subsequent processing. There were many manual steps that had to be perfect to get StarNet output correct. When everything was correct, StarNet was amazing. But all those manual steps made me less confident while reviewing the adjustment results of others when troubleshooting control issues and the .lst output file did not display a copy of the text of each file used in the sequence of calculations. You often referred to the .csv file coming out of the data collector. I'm not sure how you used that file, but hearing things like that make me a bit nervous. In my opinion, when doing this type of work, it is a big step backward to use just a coordinate file coming out of a data collector. In my experience, robust and complete raw sensor data into a full software package is key to success. To each their own.
Other manufacturers have automated/refined the processing routines such that any updates to the dataset is instantly propagated throughout the rest of the correlated data; one less thing to worry about. (Which is why I think many people commented that they've switched to x,y,z manufacturer software that matches their hardware.) You've said you're pretty comfortable with StarNet. Cool, it's a good software to know. But since you have Trimble hardware and the software, I encourage you to get more comfortable with TBC for data QA/QC and LSA. Once you get the hang of it, I think you'll discover it's a cleaner, easier, and smoother process.
Over the course of the thread, many suggested you separate your conventional and gps adjustments. This makes it easier to isolate problems. According to your screenshots, you kind of did that by using different point ranges for traverse and gps, but both appeared to be in the same StarNet project. This may be a problem because the datasets are not 100% isolated from each other and you can't test changes in one dataset without potentially affecting the other, and vice versa. (Going through all the steps to "turn off" this gps data, or that optical data for testing iterations would drive me nuts.)
You are correct, the vertical datum should not matter when comparing just total station values to levels. If the marks were stable, and the measurements were good, the propagation of those measurments will be consistent with each other and the results will be shifted up or down based on the delta between the two datums. In this particular case, the orthometric elevations would shift by about 0.458m.
By all accounts, your field techniques appear to be effective. New level loop hits existing point = 0.016m; Traverse to existing point holding same beginning ortho. height = 0.008m. (But doing manual data entry of point id's and HI's for static observation processing is always a pain, and ripe with opportunity for error.)
They may have said it, but I wouldn't hang my hat on "the Microsurvey guys said the software is incorrectly applying the geoid modelling to my conventional data." If you're combining levels, total statation, and gps datasets into one adjustment on a mapping projection, then there may some merit to this statement. I don't know, I didn't write the software.
"If I run ONLY my conventional data holding the elevation of #800 and turning off geoid modelling I get an elevation to #822 of 961.873m. Now if I turn on my geoid modelling and leave everything else the same I get an elevation to 822 of 961.752m"
If you only have optical data (levels and conventional traverse), without regard to a geometric datum (which is what you are telling StarNet to do; be on a 1.0 scale grid, 5000/5000 type system), then using a geoid model is improper, and you would be incorrectly applying geoid modeling to your conventional data. The software is not incorrectly applying geoid modelling, the user is.
In a true "local" system, your coordinate values have absolutely no relation to a published datum/projection. Good practice is to keep those coordinate values substatially different than any published datum/projection values. Your local system site is usually so far away from the real world site, any attempt to use those incompatible things will be obvious. If the software allows you to attach a geoid model to said local system, the routine will do what it can to accomodate the request. Good software should alert the user that this is not possible, or a bad idea, or similar.
In both use cases cited above, you're fixing one end of the traverse in ortho height on a local system (no mapping projection/datum). (Think fulcrum point on a lever.) First Case: Run the traverse without a geoid model, and you hit your leveling elevation. Rightfully so; no pressure on the lever. Second Case: Run the traverse with a geoid model, and you miss by almost the exact amount of difference between CGVD28 (leveled) and CGVD2013(2010.0) (gnss) at the end of the traverse.
This is because you told StarNet to hold the ortho height of a local (non-geodetic position) point fixed, use a geoid model to compute the vertical difference associated at that point, and then try to apply that answer along your traverse. Normally, this would be nonsense. That is because when using a geoid model, the vertical componenet of your conventional data is a function of the horizontal poistion, and your local system would not fall in the realm of 3d space encompassed by the projection.
Your case is unique becuase you are using horizontal positions derived from a 3d source relative to the target geometric datum/projection as your "local grid coordinates", which are very close (horizontally) to reasonable UTM grid coordinates. So it does give an answer, but a wrong anwer.
That's because your "local system" technically has only one point of reference available to the geoid model for fixing the ortho height, geoid height, ellipsoid height and their position within its geoid model universe. Remember, in theory, this local system has no relation or knowledge of the geodetic datum/projection or associated hz/vt parameters in the geoid model.
Your local traverse is being rotated about the beginning fixed point (imparting a slope across the change in horizontal position along the similar orientation of the geoid model with respect to the datum/projection) such that the endpoint of your traverse is off by almost exactly the difference in the geoid model correction at your fixed point plus the measured vertical change between the two points at the beginning/end of the traverse. If you run the same process using different starting ortho heights (different vertical datum), this scenario is exacerbated by the difference between the 2 vertical datums.
In a typical GPS scenario, fixing a single ortho height of a point causes the geoid model to shift up or down to match H (ortho), and generate N (geoid). Next, does this procedure come up with a reasonable ellipsoid height that matches your gps measuement at that point? If so, great. Does this procedure compute the ortho heights of nearby points such that they are close to their leveled ortho heights? Yes? Even better! If all these factors are aligned correctly, fixing more ellipsoid heights on CORS and fixing more ortho heights for a given control network should not blow up an LSA.
I see that the site is located in the mountains, but I didn't see a change of 300+ meters from one end of the site to the other. There's only so much vertical relief you can design into a road per short segments, etc. Your site appears to be (effectively) flat along the route (at least as far as adjacent optical measurements would be concerned), which is probably why others commented about the deflection of the vertical being suspect as significant error source.
Obviously Canada has unique issues with topography, geoid models, projections, etc. And it appears that Canada is actively working much of the time to improve and give their CACS users the best info/data they can. I glanced back through the data sheets, but I didn't see/can't easily tell if #800 or #822 are either of the actual Canada leveled benchmarks. If not, hopefully, the elevation of those 2 points came from a level loop originating and closing on any number of the 5 benchmarks along your project corridor. It is more likely than not all of those remaining benchmarks (that are still present today) were included in the original level run for CGVD28 and then again in CGVD2013(2010.0).
Almost beer-thirty on a Friday!
I have not taken the time to read all of your post in detail. I see that you have a control point off some distance in the east direction but none from the west.
Linear projects can be very prone to exaggerating the slope because so much of the data is close to the central line of the project. Perhaps another control point removed some distance from the project to the west will allow the two distant points to act similar to outriggers on a canoe stabilizing the vertical exaggeration.
There well may be another cause that I haven't seen for lack of detailed reading, but from experience, this jumped out to me.
Also, check your weighting. The vertical of the GNSS will not likely be as good as leveling or even good total station work. Don't let your GNSS vertical overweigh other more precise observations.
Best wishes,
JAC
I have had practically no processing experience to date but looking to learn. This post answered a number of questions, simple questions to most in this field I suppose, that I have been curious about. You have a way of explaining, perhaps the detail or common language, that I really connected with. Thank you for that, and will look for your future posts.
@jbw?ÿ
Thanks. I'm glad it helped.
I was fortunate early in my career to learn some valuable lessons from Alan Akman and Jessie Hummel (Caice Co-Founder and Caice/Electronic Field Book (EFB)/Florida DOT). Those two guys were a huge inspiration to me for wanting to learning the ins-and-outs of how surveying works.
Many aspects of surveying/geodesy are intense:
Learning the concepts? Okay, I'm listening.
Learning the Math? Uh, what did you say?
Learning to code? OMG, have you seen that new TikTok dance?
Wanting to talk about what you do for a living with other people in the same boat because our clients and everyone else outside of our little club don't know anything about what we do? Bingo! That's the glue that holds us together.
There are many in this field (and related fields) that are smart, or know what they're doing, or execute well, or effectively communicate with others. Put all of those together in a single package, and you have a winning combination.
The beauty of these message boards is that each of us is on a path, and no matter how far along the path you are, there are people in front of you and people behind you.?ÿLearning is a life-long endeavor, welcome aboard.
@michigan-left?ÿ I don't disagree with much of what you wrote, but though I'd point out a couple of things in reference to your Star*Net comments:
made it vulnerable to easily using incorrect/outdated "seed" values by missing a step during the recalculate/recompute phase of data reduction, or not selecting the most current .dat file
The new version of Star*Net -- which I only reluctantly bought when my v6 dongle failed -- monitors the .dat file(s?) and pops up a window asking if you want to use the current version whenever a change is detected.?ÿ I find this annoying, but maybe others find it useful.
and the .lst output file did not display a copy of the text of each file used in the sequence of calculations
As you probably know, including observation data in the listing file is a user-selectable option.?ÿ I don't select it, as it just makes the file larger and therefore harder to navigate, and I know where to find the obs data if I need to review it.
?ÿ
Going through all the steps to "turn off" this gps data, or that optical data for testing iterations would drive me nuts.
It's not that big a deal if your data is organized well.?ÿ For small projects it doesn't matter much, but for larger ones segregating data into separate .dat files (e.g. by field crew, by instrument, by date, etc.) makes it easy to test scenarios by just turning on or off selected files.
Star*Net aside:
But since you have Trimble hardware and the software, I encourage you to get more comfortable with TBC for data QA/QC and LSA. Once you get the hang of it, I think you'll discover it's a cleaner, easier, and smoother process.
I think this makes sense for most shops.?ÿ I'm not a big fan of Trimble optical gear, and as a small operator their business model doesn't work very well for me, so I run GeoMax and Javad (except for the occasional campaign network, in which I still use a bunch of Trimble 4000SSi receivers).?ÿ For pure GNSS projects I adjust in TBC, but for terrestrial and GNSS/terrestrical combinations I use Star*Net.?ÿ I'd like to try TBC with terrestrial data just to see what it's like, but I haven't yet figured out a good way of getting my SurvCE data into a Trimble-digestible format.?ÿ Maybe someday I'll get around to writing a converter...