Depth From Stereo on APQ8096 (VOXL1)
See the top-level Depth From Stereo page for an overview of depth from stereo on VOXL. This page covers the APQ8096-specific implementation which is distributed as the package apq8096-dfs-server.
DFS on APQ8096 uses the GPU for the majority of the intensive block matching algorithm.
Source
Source code is available on Gitlab
Table of contents
voxl-configure-dfs
Like all MPA services, voxl-dfs-server has a basic configure script to enable/disable the service as well as to reset the config file back to default for factory resets.
yocto:/$ voxl-configure-dfs -h
Start wizard with prompts:
voxl-configure-dfs-server
Shortcut configuration arguments for scripted setup.
factory_enable will reset the config file to factory defaults
before enabling the service.
voxl-configure-dfs-server disable
voxl-configure-dfs-server factory_disable
voxl-configure-dfs-server factory_enable
voxl-configure-dfs-server enable
show this help message:
voxl-configure-dfs-server help
Note that calling voxl-configure-dfs-server factory_enable to enable the service will wipe any changes you’ve made to the /etc/modalai/voxl-dfs-server.conf config file. Use voxl-configure-dfs-server enable or systemctl enable voxl-dfs-server to enable the service without touching the config file.
You can also run voxl-configure-dfs without arguments to start the wizard:
yocto:/$ voxl-configure-dfs
Starting Wizard
Do you want to reset the config file to factory defaults?
1) yes
2) no
#? 1
wiping old config file
Creating new config file: /etc/modalai/voxl-dfs-server.conf
do you want to enable or disable voxl-dfs-server
1) enable
2) disable
#? 1
enabling voxl-dfs-server systemd service
Created symlink from /etc/systemd/system/multi-user.target.wants/voxl-dfs-server.service to /etc/systemd/system/voxl-dfs-server.service.
starting voxl-dfs-server systemd service
Done configuring voxl-dfs-server
DFS Server Configuration File Tuning
The default configuration file has settigns that are a good starting point for our OV7251 stereo cameras. Note that you will want to fine tune these for your specific application. For example, when using DFS for obstacle avoidance you will want to tune these to reduce false-positives, wheras for 3D mapping you may want to tune these for more complete data in lower-contrast regions.
yocto:/$ cat /etc/modalai/voxl-dfs-server.conf
/**
* This file contains configuration specific to voxl-dfs-server.
*
* skip_n_frames
* how many frames to skip between processed frames. For 30hz
* input frame rate, we recommend skipping 5 frames resulting
* in 5hz disparity and depth data.
*
* n_disparity_levels
* number of disaprity levels to check, must be positive
* but doesn't need to be a multiple of 16 like opencv.
* For VGA resolution 48 works up to less than a meter away
* but is a bit slow. 24 is faster but only works at longer
* distances.
*
* sobel_cutoff
* regions of the image with gradients below this value will
* be ignored. Increase this to reduce false-positives, but
* doing so will also reduce the amount of depth reported
* across the image. 5-10 is a reasonable range.
*
* first_median_filter_size
* must be an odd number. This filter serves to remove
* speckles and noise after calculating pixel disparity.
*
* en_second_median_filter & second_median_filter_size
* The second median filter can be smaller and serves
* to clean up the edges of objects as well as remove
* any remaining speckles the first filter left
*/
{
"skip_n_frames": 5,
"n_disparity_levels": 48,
"sobel_cutoff": 5,
"first_median_filter_size": 17,
"en_second_median_filter": true,
"second_median_filter_size": 7
}
skip_n_frames
APQ8096 can consistently acheive 5hz DFS on VGA stereo pairs with 48 disparity levels. Assuming 30hz original frame rate, this means voxl-dfs-server should skip 5 frames and process every 6th frame to achieve this. It is capable of slightly faster processing. Since it will automatically skip frames if still processing the current one, it is safe to set skip_n_frames to 0 and let it automatically process data as fast as it can. Note that this will result in a faster but less consistant frame rate.
n_disparity_levels
The number of disparity levels that the GPU will check in configurable. GPU processing time scales with this value so reducing this from the default of 48 down to 36 or even 24 will speed up processing time significantly. However, this will reduce the minimum distance detectable which is also a function of the baseline (distance between cameras). This should be tuned for your application. For example, if using DFS to track distant objects, a max disparity value of 16 may be sufficient.
sobel_cutoff
The algorithm performs a sobel filter on the input images and has a configurable cutoff that serves to ignore parts of the image that don’t contain sufficient contrast or gradients to do block matching. The default is 5. If you experience a lot of noise or false positives, try increasing this to 10 or 15. We recommend tuning this using a live image stream in voxl-portal to achieve a disparity_scaled image that you are happy with.
Median Filters
DFS block matching algorithms usually generate a lot of high-frequency noise called “speckles”. Usually a median filter to used to clean this up. A quote from the (opencv DFS docs)[https://docs.opencv.org/3.4/dd/d53/tutorial_py_depthmap.html]:
Block-based matchers often produce "speckles" near the boundaries of objects, where the matching window catches the foreground on one side and the background on the other. In this scene it appears that the matcher is also finding small spurious matches in the projected texture on the table.
apq8096-dfs-server uses a configurable two-stage median filtering process. The first stage is very large (default block size of 17) to clean up edges and most small speckles. An optional second stage is smaller and helps to reduce false positives. You can disable the second stage if you feel it is unnecessary. However, if you experience a lot of false positives, you can make the second stage more agressive by increase the block size from 7 to 13 or even 17.
We recommend tuning this using a live image stream in voxl-portal to achieve a disparity_scaled image that you are happy with.
Troubleshooting
Ensuring pipeline is active
Make sure the voxl-dfs-server is running using voxl-inspect-services
If voxl-dfs-server is not running it is either not enabled, in which case you should run voxl-configure-dfs-server enable or systemctl enable voxl-dfs-server to enable the service.
If the service is enabled but not running, the most likely cause is that there is no stereo calibration file.
Calibration
Accurate depth from stereo requires cameras to be focused, rigidly mounted, and well calibrated. The voxl-dfs-server service won’t even start until the specified camera calibration file is missing. Depth from stereo is hyper-sentitive to the camera mounting and calibration.
The following things will require the stereo cameras to be recalibrated:
- When cameras are removed and reinstalled in the mount.
- When a camera mount is removed and installed on a new airframe (e.g. when installing a flight deck on a new airframe.)
- When refocusing the cameras.
- When reversing cameras left/right or rotating the image 180 degrees in software.
- After a hard crash.
When experiencing problems with depth from stereo, the first step should always be to recalibrate the stereo pair.
Reducing False Positives
After calibrating, if you see a lot of flickering in the live dfs_disparity_scaled image in voxl-portal or a lot of false-positive object detections when flying with VOA, then you need to tune the parameters in /etc/modalai/voxl-dfs-server.conf as described above. Speficially, look at increasing the sobel_cutoff and the second median filter size. Tune each parameter using the live feed of the dfs_disparity_scaled image in voxl-portal before doing flight testing with VOA.