System precision and local accuracy test results suggest that the point clouds of the Elios 3, when processed with GeoSLAM Connect, compare well with the ZEB Revo and ZEB Horizon, the leading mobile mapping systems on the market.

In this article, we focus on system precision and local accuracy, as tested in an office environment. You will discover here the results of tests we did for global accuracy and georeferencing in a large warehouse.

Test results in a nutshell

• What was tested?
  System precision and local accuracy of 3D models created with LiDAR data collected by Flyability’s Elios 3 and processed with GeoSLAM Connect.
• Who conducted the test?
  GeoSLAM 3D mapping experts and Flyability’s product team.
• Wat tests were carried out?
  A plane-to-plane analysis for local accuracy and a Range Noise Analysis for precision.
• Reference model
  The reference model used for the tests was created with a TLS (Terrestrial Laser Scans) Riegl VZ-400. The registration process was performed with RiScan Pro V2.14.1.
• Test results local accuracy
  All comparisons fell within +/- 16 mm, and the mean absolute normal distance between the Elios 3 and the reference model was 8 mm.
• Test results accuracy analysis
  The standard deviation of all surfaces fell within 15 mm, and the mean standard deviation between the Elios 3 and the reference model was 8 mm to 1-sigma.

In recent years, LiDAR data has rapidly become one of the most reliable bases for accurate and precise 3D modelling.

Industries such as mining, construction and infrastructure use these models to perform routine inspections, make safety decisions, track asset changes over time and support project planning.

The results professionals in these industries get from 3D models created with LiDAR data include:

• Detailed digital twins
• Accurate 2D and 3D measurements
• The ability to determine the location of defects in action
• The ability to export data to common 3D point cloud file extensions such as *.e57, *.las, *.laz and *.ply
• The ability to merge multiple georeferenced 3D models to track changes in assets over time

Regardless of industry or output, the quality of the model is essential to its usefulness. If the data is not precise and accurate, it may not reflect reality well enough to provide valuable insights.

This article discusses the results of tests conducted by experts at GeoSLAM. They identified findings from 3D models created with the Elios 3 and GeoSLAM Connect:

• Precision of the system
• Local accuracy
  
  

WWhy we tested the Elios 3’s precision and accuracy

Flyability’s Elios 3 comes with Ouster’s OSO-32 LiDAR sensor and the ability to perform SLAM (simultaneous localization and mapping). This allows it to create 3D models in real-time during flight.

After the flight, Elios 3 users can process the collected LiDAR data with GeoSLAM Connect. And thus, create precise and accurate 3D models. These new capabilities add accurate 3D mapping to Flyability’s proven ‘collision tolerance technology’ for work in confined spaces. This allows effective mapping of inaccessible environments as well as giving users more information about conditions in industrial assets.

However, LiDAR scanners can be sensitive. Changes in the way they are worn, for example, can alter the quality of the data they collect. It is therefore logical that potential users of the Elios 3 have questions about what happens when it is equipped with a LiDAR payload.

• Will the drone’s vibrations or environmental factors such as dust or moisture affect the accuracy of the resulting 3D models?
• What do the resulting 3D models look like and how usable are they?

To answer these questions, we conducted a thorough analysis of the precision and local accuracy of 3D models captured with the Elios 3 and processed with GeoSLAM Connect. All tests were carried out in a way that was both representative and repeatable.

Defining our terms: local accuracy and precision in 3D mapping

When discussing 3D models made with LiDAR data, the terms precision and accuracy have specific definitions.

Accuracy is usually defined as the degree of agreement of a measured quantity with its true (reference) value. For example, if you measure the distance of 100 mm (3.9 inches) in your point cloud, but the actual or known distance is 500 mm (19.7 inches) … then your measurement is not accurate.

Precision – or noise or repeatability – is in turn defined as the extent to which further measurements give the same result. For example, if you measure the distance between two points on a 3D model five times and get 100 mm each time … then your measurements on the model are precise.

The precision requirements for a 3D model are usually determined by the industry in question. For stock measurements in mining, a precision of a few centimetres may be sufficient because the statistical error smoothes itself out. Other applications such as construction may require even higher precision. It should be noted, however, that precision puts a lower limit on the accuracy of some measurements. In particular, when measurements are made by picking features on the point cloud. The lower the precision, the higher the noise in the point cloud and the more difficult it is to choose the point that corresponds exactly to the right feature.

Accuracy, on the other hand, is important in a 3D model because it ensures that the model reflects reality. This includes considerations such as the correct shape of corners and avoiding walls being on top of each other. When we talk about local accuracy, we are talking about the distance between two points in a point cloud, when the object can be viewed from one position. A good example is the dimensions of one room. As noted above, this article only contains test data for local accuracy with the Elios 3 and GeoSLAM Connect.

Assessing the precision and local accuracy of the system with the Elios 3

To evaluate the precision and local accuracy of the Elios 3 with GeoSLAM Connect, GeoSLAM 3D mapping experts conducted the following:

• A plane-to-plane analysis
• A Range Noise Analysis

Establishing a benchmark

To evaluate the accuracy of a system, you need to use a second measurement system to provide the benchmark value (control). This second system should have greater accuracy than the system being tested.

In the case of a Mobile Mapping solution like the Elios 3, a Total Station (TPS) or a Terrestrial Laser Scanner (TLS) are used as a control by default. They have accuracy that exceeds that of a Mobile Mapping solution. This is because they capture data from a single, stationary position where multiple positions are recorded together using point coupling algorithms.

By comparison, a Mobile Mapping solution like the Elios 3 moves continuously while collecting data, recording it at multiple positions as the drone moves through the environment it is mapping.

Collecting the data

GeoSLAM experts captured data from a flat indoor environment using both the Elios 3 and a TLS standard. For this, they used a Riegl VZ-400 TLS as a benchmark. The accuracy of this benchmark is determined from a single position with a defined confidence level. The manufacturer of the Riegl VZ-400 claims an accuracy of 5 mm at 1-sigme. This means that 68% of all measurements must be within a range of 5 mm. From Riegl’s point cloud, the reference model was then created to serve as a known ground control.

Aligning the Elios 3’s point cloud to the reference model

To effectively compare the Elios 3’s point cloud with the TLS reference model, the former was aligned with the latter.

The alignment changed the position and orientation of the point cloud data and brought it into the coordinate system of the TLS reference model. GeoSLAM used the PolyWorksInspector MRS2019 IR3 software for this purpose. This is a 3D analysis and quality control solution used to assess the accuracy of products.

The steps below were followed to perform the alignment:

• Manual alignment: is used in PolyWorks for an initial rough alignment of the equation point cloud (that of Elios 3) with the reference model.
• Calculated transformation matrix. Once completed, the Best-Fit* automatic alignment function was used to create a calculated transformation matrix between the point cloud and the reference model.
• Application of the transformation to point cloud.Once the rigid transformation matrix was calculated, it was applied to the point cloud to align the Elios 3 data with the reference model.

*Best-Fit: a surface-based alignment tool that iteratively transforms the position and orientation of the comparison point cloud data to minimize the deviation of the point cloud from the reference model.

Assessment of local accuracy – plane-to-plane analysis

The plane-to-plane analysis was performed by fitting planes to both the Elios 3 data and the measured reference model. The normal distance between the surfaces was then evaluated.

That normal distance was calculated by finding the difference between the extracted plane in the Elios 3’s point cloud and the corresponding plane in the reference model. This is done using an automated workflow in PolyWorks MRS2019 IR3. This assessment indicated that the local accuracy of the point cloud and any variations in the point cloud were identified.

Assessment of system precision – Range Noise Analysis

To assess the precision of the Elios 3, a Range Noise Analysis was performed.

Range Noise is the difference between each range measurement (point) and the average range value within the selected area. The areas chosen to assess Range Noise are the flat surfaces extracted for the plane-to-plane analysis.

The Range Noise is presented as a standard deviation of the mean point of the plane. The standard deviation is thus a measure of the precision of the system and is equal to 1-sigma. The standard deviation was calculated using PolyWorks MRS2019 IR3.

Testomgeving

Om de nauwkeurigheid en precisie van de Elios 3 met GeoSLAM Connect te evalueren, zijn de gegevens opgenomen in een standaard kantooromgeving met 6 vlakke oppervlakken, ongeveer 1 meter in het vierkant, die op regelmatige afstanden rond de scan zijn geplaatst.

GeoSLAM-deskundigen plaatsten laserscanreferentiebollen met een diameter van 145 mm (5,7 inch) rond de omgeving om de Terrestrial Laser Scans te registeren en zo het referentiemodel te creëren met behulp van de LAZ-uitvoer van RiSCAN Pro.

Na het creëren van het referentiemodel vloog een dronepiloot met de Elios 3 volgens de aanbevolen richtlijnen voor mappingvluchten, waarbij de vlucht op dezelfde locatie begon en eindigde.

De piloot vloog één volledige lus van het kantoor en een extra kleine lus waar de twee gangen samenkomen. Na de vlucht werden de door de LiDAR-sensor van de Elios 3 (Ouster OSO-32) met GeoSLAM Connect v2.1.0 verzamelde gegevens verwerkt, gefilterd om uitschieters te verwijderen en geëxporteerd in het LAZ-bestandsformaat.

Test results for system precision and local accuracy

These are the test results for local accuracy and system precision.

Assessment of local accuracy

Experts assessed the local accuracy of the LiDAR data from the Elios 3 using plane-to-plane analysis.

The normal distances between the surfaces in the reference model and the surfaces of the Elios 3 with GeoSLAM Connect data are shown in table 1.

The results show that all comparisons are within +/- 16 mm, and the average absolute normal distance between the Elios 3 and the reference model was 8 mm.

Assessment of system precision

The results of the Range Noise Analysis calculated using the standard deviation of the comparator planes in the Elios 3 data are given below.

The results of the Elios 3 precision analysis show that the standard deviation of all planes is within 15 mm and the average standard deviation between the Elios 3 and the reference model was 8 mm to 1-sigma.

Conclusion

The results of this test by GeoSLAM suggest that the point clouds from the Elios 3 processed with GeoSLAM Connect compare well with a traditional TLS and with the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems on the market.