Drone Software Accuracy/Inaccuracy for Digitial Terrain Modeling

[chris-mills]

The critical action is to ensure that for each survey random ground checks are also taken across the site, BEFORE THE FLIGHT. If temporary marks are left at some of these points then they can be added as control, if the first model created shows significant errors. It’s no good making them all control points initially, as then you have nothing to compare with. Once the error range is known then they can be added in to tighten up the final model as necessary (or accept that the photography is poor and walk the whole job with GPS!)

Key is to remember traditional survey theory, apply good geometry to the control with fixed ground points and don’t believe statistics – real measured checks have far more significance. This is a “blanket” technology which relies on large numbers of “near-enough” points rather than many fewer “spot-on”s. Typical accuracies of 5cm. should be attainable as long as care is taken.

[Yuriy Lutsyshyn]

I was wondering if anyone tried Agisoft PhotoScan, how is it ?. I does lens calibration too, this should add towards accuracy.

[chris-mills]

Agisoft works quite well, although you need a lot of computing power if you have several stages open at the same time and the models are big (say 500 plus photos). For high quality level image manipulation you need really serious computing power. (That probably applies to any package, but others may have better data handling. Very large jobs need cutting into smaller chunks. Biggest pain is on high overlap jobs where each control point might appear on 10-15 photos. Although it will identify the relevant photos and mark the points, each needs to be adjusted manually on each photo for best quality results.

The stats are a little suspect (compared to real life) but I assume they represent internal residuals rather than the fit to control. Providing the image GPS coords does speed the process up and reduces the number of images it is unable to fit. As with any software it generally struggles with low texture areas and has, on occasion, managed to match up a sliver of the gimbal (which appeared in some frames by accident – incorrectly mounted camera) and create some interesting artefacts in the orthophoto.

It does have quite good facilities for masking out dubious sections of images. It copes well with vertical photography and the semi-obliques captured around each turn.

[Yuriy Lutsyshyn]

Thank you, I tried demo and it died on my current old computer, so I could not even try anything. I noticed from this program you can print out circular bar code markers (may be on a plastic sheets), and the program will automatically recognize the markers on all images they are present, I imagine this would reduce amount on manual work significantly.

[Frank Willis]

chris mills, post: 346192, member: 6244 wrote: The critical action is to ensure that for each survey random ground checks are also taken across the site, BEFORE THE FLIGHT. If temporary marks are left at some of these points then they can be added as control, if the first model created shows significant errors. It’s no good making them all control points initially, as then you have nothing to compare with. Once the error range is known then they can be added in to tighten up the final model as necessary (or accept that the photography is poor and walk the whole job with GPS!)

Key is to remember traditional survey theory, apply good geometry to the control with fixed ground points and don’t believe statistics – real measured checks have far more significance. This is a “blanket” technology which relies on large numbers of “near-enough” points rather than many fewer “spot-on”s. Typical accuracies of 5cm. should be attainable as long as care is taken.

Chris, even when you do what you suggest, the surface model is a fuzzy group of many points. Random points might match the surface, but they can just as easily miss it by feet. I agree that the check point should not be control points, because the model will adjust or warp to the control points. That is not the problem though. The problem is that the digital terrain surface is the average of thousands of points, which are only so accurate, period.

[pdop 1.0]

Here is a link to a very recent article in a south african survey & geomatics publication were they did a 6km road strip survey using traditional ground survey with GNSS versus a photogrammetry survey using a UAV, no lidar used.

Each survey was done by two different companies and the results interpreted independently, I have only briefly read the article, so I don’t have an opinion on it yet.

The article only compares the accuracies and precisions, I would also be interested in the costs comparision per km or hectare between the two methods.

I recently sub contracted out a 126 hectare farm survey out to some aerial guys, the time saving was in the order of 40 days and the cost saving about 50 %, as to the results… well its looks good and met all the checks and balances . But if I was going to use it for detailed engineering services and planning, I think I might go shoot it conventional, not the whole farm, but just areas of interest.

http://www.ee.co.za/article/rpas-vs-ground-survey.html#.VlfsqF9XenM

Here is a photo showing the type of terrain and the ground control we placed.

[chris-mills]

Frank is correct in saying the surfaces are fuzzy groups of points, but Photoscan has an option which allows you to set a search and slope parameter. Using this it will find the lowest point in the search area (say 1 foot) and eliminate all other points which lie at an angle of more than the slope parameter. This not only thins down the model substantially, but removes most of the outliers (either random, or plants/bricks, etc.). There still needs to be manual inspection and selection for feature lines etc. but on subjects with not much detail the time saved is large. To date, as long as a rigorous approach is taken, nothing within the parameter area falls outside a 30cm. (1 foot) band once the automatic thinning out has been done. Go outside the control and the errors spiral away.

PDOP 1.0 is right about taking engineering detail conventionally. I tend to do this using a total station, even when the rest of the site has been surveyed by GPS. The aerial method is best suited to work where access is difficult or the surface is very irregular, when a surveyor is having to make significant judgements about what are “typical” points, or large areas with little of interest – ie. bulk spots.

Incidentally, Photoscan have just released a new version, which has a number of improvements – on orthophotos it shows the boundaries of the areas used from each photo and it will display contours on the imagery (very useful for a quick check on any rogue points left in the model)

I haven’t seen the bar code markers that Yurly mentions being used, but I imagine they would have to be quite large to be read satisfactorily from 400 feet. For terrestrial work from a range of a few tens of feet smaller markers like that would be satisfactory. The photo by PDOP 1.0 shows a very large marker. For SUA work I’ve never needed to try anything bigger than 60cm (2 ft) square, although for conventional LIDAR I put out 120cm (4ft) boards.

[warren-smith]

Ha ha – looks like a beer leg target!
Don’t you shoot the point before laying out the plastic sheet?

[Yuriy Lutsyshyn]

the markers print out is in Tools->Markers->Print Markets, but I do not know if one can print on a larger sheet than A4 (Letter size)

[pdop 1.0]

Warren Smith, post: 346290, member: 9900 wrote: Ha ha – looks like a beer leg target!
Don’t you shoot the point before laying out the plastic sheet?

We used 3 polypropylene woven sand bags filled with few rocks or stones to keep them in place , positioned to form a triangle in the middle , then wack in a spike and record , cost for 3 bags about US $1

[Yuriy Lutsyshyn]

pdop 1.0 I was just wondering what kind of a radio antenna are you using with your R? (5800) in the picture above, do you have p/n ? Sorry for bugging. Intuitively looks like it has a good gain antenna, and the fiberglass pogo does not interfere much with the reception unlike regular metallic pogos.

[pdop 1.0]

Yuriy Lutsyshyn, post: 346322, member: 2507 wrote: pdop 1.0 I was just wondering what kind of a radio antenna are you using with your R? (5800) in the picture above, do you have p/n ? Sorry for bugging. Intuitively looks like it has a good gain antenna, and the fiberglass pogo does not interfere much with the reception unlike regular metallic pogos.

We are on a VRS network , no radio antenna plugged into the R4 receiver, what looks like an antenna is the bipod leg that we attach to the top of the pole , so we can lay it down on the ground while fixing the ground control

[jhframe]

Yuriy Lutsyshyn, post: 346322, member: 2507 wrote: I was just wondering what kind of a radio antenna are you using with your R? (5800) in the picture above

I think you may be mistaking a 12-inch bipod for a radio antenna. It looks to me like he has the Seco bipod pictured below mounted just underneath the receiver.

[pdop 1.0]

Spot on Jim, that’s it

[jhframe]

That bipod is one of my favorite tools. I use it with an 8-foot Seco carbon-fiber adjustable pole for both GPS and total station work.

[cptdent]

Yuriy Lutsyshyn, post: 346194, member: 2507 wrote: I was wondering if anyone tried Agisoft PhotoScan, how is it ?. I does lens calibration too, this should add towards accuracy.

Lens calibration on a plastic lens?? Really? All internally? Really??
I’ll bet you a dollar right now that if the government maintains the same lens calibration criteria that they use on “Real” Aerial cameras, that there is not ONE light weight digital camera made, that a drone can lift, that would pass such testing. I have a Leica digital camera that takes awesome photos, but Leica will not state that it meets the U. S. Government calibration criteria.
Many shady claims are bring made on GIS systems that just don’t hold water. I can see where such systems are great for initial planning, but when you are talking comps that will affect what someone gets paid for a service, you better be ready to spend a LOT of time in court.
It’s bad science. The physics calculations just don’t verify the claims.
Remember back to the early gps field units. We had a salesman tell us not only would this new gear give sub-meter accuracy, BUT he laid a dime on the ground and said that was the x/y accuracy we could expect and then he stood it on edge and told us that was the Z accuracy we could expect. Really???
Nothing has changed when it comes to new stuff sales claims. Not in over 1,00 years. That’s why we have the OLD saying “Caveat Emptor”.

[u1011919]

We have used UAV’s to great success with photogrammetry for certain applications (large civil ptojects) – consistently achieving an average of 40mm in verified area’s when the site is stripped to bare earth.

We have determined that Lidar is a better option when there is long grass , foliage etc. There are circumstances where conventional RTK will be essential such as near vertical cuts/excavations. I think that UAV’s could be considered the best solution for quarry volumes and the like

In my opinion with the correct verification procedure UAV’s are indispensable in today’s large scale volume surveys if one is not to fall behind the 8 ball. In saying that – we have had subcontractors that we have commissioned for UAV monthly volumes that have failed dismally when verified.

[Yuriy Lutsyshyn]

thanks, I really thought this is some kind of a special high gain antenna

[chris-mills]

Going back to Frank’s original posting.

There is a real concern that button pushers will leap in, produce rubbish and then get the technique a bad name. Manufacturers rely on selling the things, something which is less likely if they explain that it takes months of hard work to get your head around how the systems best operate and even longer before you have some degree of certainty as to how each site needs to be tackled in order to get an acceptable result. The difficulties start when non-surveyors can’t analyse the results they produce and recognise obvious problems.

It’ll be a long, long while before “press and go” comes as a reliable SUA system. In the meantime surveyors need to do what they have always done best: think it through logically, check and then check again.

I think the lens calibration matter is a bit of a red herring – Photoscan matching is good, especially when the forward and side overlaps are high and the sheer weight of data seems to get the result. Any daft points are easily screened out once the initial cloud has been generated.

A4 size targets are about the minimum useful size from 400 ft.

[Frank Willis]

Well said, Chris.

Regarding a previous post, I too have have reproduced points on UAV work to an average of 40 mm, only later go back and fly it with the same camera an hour later, with approximately the same overlap, flying height, etc, to find a significantly different average, and to find some individual points (not control points) off vertically by more than 2 feet. During some of the various UAS seminars I have presented over last 4 years, I constantly ran into surveyors who found very substantial errors in DTM that they cannot explain. I have seen the same thing. The consensus is to check and double check. When does checking render the process non cost effective?

An average of 40 mm of error doesn’t provide enough data for analysis. What range do 95% of the points fall within? 68%? Is the histogram symmetrical? Are there patches of bad data from an independently mis-computed drone position? How does one explain the variation found in re-shooting a stockpile or quarry pile with the same equipment and techniques? How much checking is necessary? How should it be done, and where? Where are the hidden glitches or anomalies? What is tolerable risk? Where is it? Who cares?

I am certainly not saying close-range photogrammetry is not a useful tool. I am saying that is a terribly dangerous tool in the wrong hands, and the 3d imagery it produces exacerbates the possibility of using unverified data with catastrophic results.

In my opinion, standard stereophotogrammetry is a completely different process and is more reliable for digital terrain work on civil jobs than the close-range photogrammetry that is being deployed on drones at present. But the standard stereo photogrammetry process requires a flying height high enough so that the change in relief is a small percentage of the flying height (to provide acceptable scale error), and that the photos be corrected for tip and tilt.

I believe that UAS photogrammetry work is best implemented on small jobs or jobs that are easy to use standard methods to literally check the dickens out of–way more than checking check points or running a few sections if accuracy anywhere near 40 mm is required, or even a foot or more.

I also believe that POB and others are chasing the money of advertising, which seems to be GIS, and the novelty and fun of drones. I do not fault them for that. But it greatly exacerbates problem, as it relates to the profession of surveying. If one reads archived POB magazines over the last 20 years, the movement has followed new technology, but doesn’t do enough to present the dangers of what happens when it gets into the hands of untrained and unqualified GIS surveyors, etc. RTK and scanning have evolved similarly. But RTK and GIS are so expensive that it keeps GIS drone jockeys out of the picture. A GIS drone jockey can get into the drone surveying business for $700.

Advances in technology make our profession more fun, a lot easier, less time consuming, but it attracts people who do not have an appreciation for how important it is to measure the earth correctly and in accordance with Minimum Standards that have evolved through experience in our profession. The non professionals also have no one to answer to if their work is incompetent. Advanced technology is a razor sharp double edged sword.

I gave keynote speeches in two states this year about the double edge sword of technology, and called upon NSPS to try to help our profession to address and handle the mess. At one state, the NSPS state director was sitting on the front row during my 15 minute speech, dozing off about 2 minutes into the speech, and he literally went to sleep. I started to say something.

So here we sit, with a million drones that will be bought this year, many with cameras, and all kinds of software claiming miracles, catalyzed by the novelty of it all–a dangerous mix for land surveying and the industrial public. The financial risk to the public related to surveying is substantial. I am of the opinion that the work falls within the definition of land surveying in most states, and it should be regulated where it violates the law. And NSPS reps ought to make an attempt not to fall asleep when someone brings it up. Trade magazines are not necessarily our friends–they have one goal–to sell advertisements, and I don’t blame them. Our BORs and NSPS and states societies should get busy to adopt policies that protect the public per the law–now.

The time is coming when surveyors who have been doing asbuilt surveys or quarry surveys will be told that their services are no longer needed because someone with a drone is doing the work for pennies on the dollar. The loss is not nearly as much the loss of business as it is the fact that the industries and clients who are convinced to their own detriment that drone surveying in the hands of unqualified people is the cat’s meow.