Leveling Math Errors: Leica LS10 vs. Excel Spreadsheet Template

Several questions.

First, what precision is your level reading out to? Second, what precision are you writing in your field book?

Third, are you actually a Leica shop or a Trimble shop?

If Leica shop, why are you not using the Leica program to do the levels instead of putting them in an Excel spreadsheet?  Seems like a waste of time.

And if you're a Trimble shop, TBC will also read a leica level file.

I have seen a lot of digital level users record all of the data in a field book and then manually enter it into a spreadsheet or manually compute it. To me this is totally ignoring one of the main benefits of a digital level. 

I use a Trimble Dini (a lot) but have also processed data from Leica and Topcon. Relatively simple to create a program that reads the data file and creates a field book style listing as well as an input file for least squares (Geolab or Star*Net). 

I would hope that any recent graduate of a 2 or 4 year degree program has the ability to write a simple program to parse data. 

This reminds me of a professor I had a few classes from back in the day.  He actually believed he could read a quality slide rule to five places.  Maybe six places.  Ridiculous presumed accuracy.

I have seen state agencies require that the digitally collected data be transcribed to afield book just it was done in 1955.

Long ago I worked for a guy who beleived the field book was the only way. I tried to introduce him to electronic data collection, and he allowed it, but insisted we write everything in the field book as well. After a period of time he was disappointed because he said electronic data collection wasn't faster. In fact, he said it was taking longer. I couldn't bring myself to respond to that so I moved on.

Posted by: @bobwesterman

↑

First, what precision is your level reading out to? Second, what precision are you writing in your field book?

Level reading to the nearest thousandth. Writing rod readings to the nearest thousandth in the field book.

Posted by: @field-dog

↑

Posted by: @bobwesterman

↑

First, what precision is your level reading out to? Second, what precision are you writing in your field book?

Level reading to the nearest thousandth. Writing rod readings to the nearest thousandth in the field book.

Well there's your problem then.  According to Leica:

Data Recording Precision

Height Values:

Stored to (0.01{ mm}) ((0.00003 {ft}))

Distance Values:

Stored to (1{ mm}) ((0.003{ ft})

Those are the values that the level is working with when it does it's computations.  As long as the level computations are using those extra decimal places you're going to get some rounding errors.

How is it that your are able to read a level to the nearest thousandth? Do you estimate? Are you doing 3 wire readings? Or are you just taking the same readings that the digital level spits out? 

If the last, then the difference in the thousandths after using the spreadsheet can't be very far off from the Leica results. Do you really do work that relies on the thousandth instead of the hundreth? Has the difference ever been greater than .01'?

Sounds like your supervisor is chasing meaningless ghosts. If you were to compare the raw data and do the calcs by hand I'm betting you'll find it is nothing more than rounding differences that I don't believe are material anyway.

The OP said it was a digital level. There is still the problem of writing down something different than what the level displays, like transposing digits, etc. 

In the days before digital levels, I ran two types of methods for precise levels.

One method was using a dual scale invar rod with a micrometer on the level. I ran a 130 mile second order level line using this method that was bluebooked. I programmed an HP41 to be a "data collector" for this, and the program checked the high/low readings as well as the upper and lower hair readings, then stored the data in the HP41 to be downloaded in the office using HP-IL. This worked very well, as there were several checks on the data.

The other method was three wire leveling. In this, you read all three wires to 0.001 feet. The note keeper needs to check to make sure the difference between upper and middle and lower and middle fall within a tolerance. I had a party chief run many miles of levels this way, he was only checking the thousandths digit, and blew a bunch of hundredths and tenths. A lot of that had to be re-run. 

Posted by: @bobwesterman

↑

Those are the values that the level is working with when it does it's computations.

So you’re saying the level reads the rod to the nearest 0.00003?

Posted by: @bobwesterman

↑

If Leica shop, why are you not using the Leica program to do the levels instead of putting them in an Excel spreadsheet?

We did not purchase Leica Infinity.

Posted by: @john-hamilton

↑

Relatively simple to create a program that reads the data file and creates a field book style listing as well as an input file for least squares (Geolab or Star*Net). 

I’ve been working slowly to create such a program.

Posted by: @field-dog

↑

Posted by: @bobwesterman

↑

Those are the values that the level is working with when it does it's computations.

So you’re saying the level reads the rod to the nearest 0.00003?

That's what Leica says.

It could be that your level is applying curvature and refraction corrections.

Posted by: @field-dog

↑

So you’re saying the level reads the rod to the nearest 0.00003?

That's the standard deviation of the mean. The least count is generally the nearest thousandths of a foot, but you can set any number of decimals points you care to.