A. Terminology
The table below lists terms and acronyms that appear in this document.
Table 1: Terms and acronyms
Term | Description |
ROI | Region of Interest |
FOV | Field of View |
GT | Ground Truth |
RMS | Root of Mean Square |
STD | Standard Deviation |
B. Quality Metrics
Assessing depth quality involves quantifying various aspects of the depth data and comparing them against desired performance criteria or ground truth measurements. Some common metrics for quantitative depth quality assessment include:
- Depth accuracy assesses the absolute error by calculating the median deviation from depth data.
- Spatial Precision assesses spatial noise by calculating the root mean squared error of the depth data.
- Temporal Precision assesses the uniformity of depth values over time.
- Fill Rate assesses the percentage of valid depth pixels across an image.
Quantitative metrics demand a thorough understanding and diligent assessment of depth data quality. Establishing a controlled and comprehensive testing environment is important for accurately assessing the capabilities and limitations of depth sensors, as well as for determining their suitability for specific applications or scenarios. For the sake of maintaining consistency in depth quality assessment, it is advisable to utilize a high-reflectivity, flat surface characterized by uniform diffuse reflection attributes, thereby avoiding intricate, uncontrolled real-world scenarios. This refers to a planar reference target, placed at a known distance from the capturing device, serving as an essential benchmark for the evaluation. If such conditions are lacking, a flat wall surface or white board may also serve as an appropriate test target. Each metric may be computed over any desired ROI within the camera’s FOV.
Figure 1: Reference plane for depth quality assessment.
B1. Depth accuracy
Depth Accuracy refers to the degree to which the measured depth values correspond to the true distances between objects and the capturing device. High accuracy implies that the depth measurements closely match the GT, minimizing errors in distance estimation. As previously mentioned, a reference plane placed at a known distance from the sensor is recommended in order to simplify the GT of depth data for evaluation, and the distance between the plane and the sensor can be precisely determined using common measuring tools such as laser distance meter or tape measure.
Table4: Depth accuracy (Signed)
Item | Description |
Category | Depth accuracy |
Purpose/Goal | Measure the accuracy of depth images within a specified ROI relative to the GT. |
Target | Reference plane\Wall |
Analyzed Image | Depth image |
ROI | For instance, an 81% ROI corresponds to 90% of the image width multiplied by 90% of the image height. |
Description | This metric measures the per-pixel depth accuracy relative to the GT in a single frame depth image, excluding errors due to camera placement. GT is obtained by mesuring the distance from the depth origin to the reference target. Depth accuracy can be calculated using the following formula:
Depth_accuracy_Signed = Median(CP_Length – GT + PP_Dist) where CP_Dist represents the actual length between the depth origin and a fitted plane along the GT measuring direction, and PP_Dist denotes the per-pixel distance of the point cloud to the fitted plane. |
Pseudo-Code | INPUT: Depth_Image, GT
REMOVE all zeros in Depth_Image COMPUTE Point_Cloud (X,Y,Z matrix) from Depth_Image COMPUTE Fitted_Plane to Point_Cloud REMOVE outliers: all Z values that below percentile 0.5% and above percentile 99.5% COMPUTE the actual length between the depth origin and the Fitted_Plane along the GT vector as CP_Dist COMPUTE distance of all points to the Fitted_Plane as PP_Dist COMPUTE Depth_Error = CP_Dist – GT + PP_Dist COMPUTE Median_Value = 50% percentile value of Depth_Error OUPUT: Depth_accuracy_Signed = 100 * Median_Value / GT |
Output Value | Depth_accuracy_signed |
Output Units | [%] |
B2. Spatial Precision
While accuracy concerns the systematic deviation from the true depth, spatial precision relates to the consistency of depth measurements. Spatial precision indicates the noise level of depth data across a planar target. A high spatial precision depth data exhibits low variance in depth values for identical or similar surfaces within the same scene. To assess spatial precision, one can perform measurements of a reference plane at known or controlled locations, calculate the RMS error of the depth values obtained, and compare them against desired precision thresholds or specifications.
RMS error can be expressed in units of distance or disparity, but considering the coupling between depth and noise, it can be normalized by the distance between the camera and the planar target and represented in percentage (%) units.
Table 2 : Spatial Precision
Item | Description |
Category | spatial precision |
Purpose/Goal | Measure the noise level of depth values relative to the best-fit plane. |
Target | Reference plane\Wall |
Analyzed Image | Depth Image |
ROI | For instance, an 81% ROI corresponds to 90% of the image width multiplied by 90% of the image height. |
Description | Calculate the RMS of the distance from each valid pixel to the best-fit plane, normalized by the distance from the optical center to the plane, with the RMS Error output expressed as a percentage. |
Pseudo-Code | INPUT: Depth_Image
REMOVE all zeros in Depth_Image COMPUTE Point_Cloud (X,Y,Z matrix) from Depth_Image COMPUTE Fitted_Plane to Point_Cloud REMOVE outliers: all Z values that below percentile 0.5% and above percentile 99.5% COMPUTE distance between the depth origin and the Fitted_Plane as CP_Dist COMPUTE distance of all points to the Fitted_Plane as PP_Dist COMPUTE RMSE = RootMeanSquare of PP_Dist OUPUT: Spatial_precision = 100 * RMSE / CP_Dist |
Output Value | Spatial_precision |
Output Units | [%] |
B3. Temporal Precision
Temporal Precision measures the variation in depth values over time within a ROI. The quality of a depth image can be assessed based on its temporal consistency, high-quality depth images should exhibit smooth and stable over time. This method is defined as the STD of depth values across a specific number of frames (for example, 30). The quantification of temporal noise is carried out on a per-pixel basis, followed by calculating the STD of each pixel over a specified time.
Table 3: Temporal Precision
Item | Description |
Category | Depth temporal noise |
Purpose/Goal | Measure the extent of variation in depth values per pixel over time. |
Target | wall |
Analyzed Image | N sequential depth images |
ROI | For instance, an 81% ROI corresponds to 90% of the image width multiplied by 90% of the image height. |
Description | Per-pixel temporal noise is defined by measuring the STD of depth values for each pixel across N consecutive frames, resulting in a STD matrix. Temporal Noise is defined as the median of the STD matrix as a percentage of the target distance Z. This metric excludes NaN and zero depth data. |
Pseudo-Code | INPUT: N images of Depth_Image, GT
LOOP for each of N Depth_Image to create Depth_Tensor ASSIGN Depth_Tensor = (Depth_Image,N) //dimension is (width,height,N) REMOVE all zeros from Depth_Tensor COMPUTE STD_Matrix = STD of Depth_Tensor(width,height,all) COMPUTE Median_STD = 50% percentile value of STD_Matrix Output: 100 * Median_STD / Z |
Output Value | The median value of the STD matrix, divided by distance to wall |
Output Units | [%] |
B4. Fill Rate
A high-quality depth image should cover the entire field of view uniformly, without missing regions. Fill Rate measures the proportion of non-zero pixels in a depth image, representing the depth coverage. High fill rate indicates that most scene points have been successfully captured.
Table 5: Fill Rate
Item | Description |
Category | Fill rate |
Purpose/Goal | Measure the proportion of valid depth values within a specified ROI. |
Target | Reference plane\Wall |
Analyzed Image | Depth image |
ROI | For instance, an 81% ROI corresponds to 90% of the image width multiplied by 90% of the image height. |
Description | Calculate the ratio of pixels with valid depth values to all pixels within a specified ROI. |
Pseudo-Code | INPUT: Depth_Image
COMPUTE Total_Num = size(Depth_Image) ASSIGN Valid_Image = Depth_Image REMOVE all zeros from Valid_Image COMPUTE Valid_Num = size(Valid_Image) COMPUTE Fill_Rate = 100 * Valid_Num / Total_Num Output Fill_Rate |
Output Value | The proportion of valid pixels within the specified ROI. |
Output Units | [%] |