ModalAI Vision Lib
Table of contents
The ModalAI Vision Lib provides Visual Inertial Odometry (VIO), April Tag Detection, and Depth from Stereo on VOXL Hardware.
1) Make sure you have an account at developer.modalai.com
2) Down the Qualcomm Machine Vision and ModalAI Vision Lib IPK packages from https://developer.modalai.com/asset
3) Download the Qualcomm Community License File (snapdragon-flight-license.bin) from https://developer.qualcomm.com/software/machine-vision-sdk/tools
You will need to create an account with developer.qualcomm.com and wait 1-2 days for this file to be emailed to you. This page offers downloads for the Machine Vision IPK package as well, however at time of writing these links are outdated. The version made available at developer.modalai.com is what is used and tested internally at ModalAI.
4) Push the Qualcomm Community License file to the voxl board. This command will do it via ADB:
adb shell mkdir -p /opt/qcom-licenses/ adb push snapdragon-flight-license.bin /opt/qcom-licenses/
5) Check which version of imu_app is installed on VOXL. Unless you have a very early unit, it should be pre-installed with V0.0.2. If it is not installed, download from https://developer.modalai.com/asset
adb shell opkg list imu_app imu_app - 0.0.2
6) Push the Machine Vision and ModalAI Vision Lib ipks to VOXL. Also push imu_app if it’s not installed already.
adb shell mkdir -p /home/root/ipk/ adb push mv_1.2.8_8x96.ipk /home/root/ipk/ adb push modalai-vl_0.0.1_8x96.ipk /home/root/ipk/ adb push imu_app_0.0.2_8x96.ipk /home/root/ipk/ #OPTIONAL
7) Log into VOXL via ADB or SSH to install these packages. imu_app, if not already installed, needs to be installed with the –force-overwrite flag
james@silverstone:~/git/documentation$ adb shell / $ bash # adb starts an sh shell, start a bash shell manually /home/root$ opkg install ipk/mv_1.2.8_8x96.ipk /home/root$ opkg install ipk/modalai-vl_0.0.1_8x96.ipk /home/root$ opkg install --force-overwrite ipk/imu_app_0.0.2_8x96.ipk #OPTIONAL
1) While still logged into VOXL, run the configuration script. You will be prompted by the first question:
/home/root$ bash /etc/snav/configure_modalai_vl.sh Which camera channel is the tracking camera connected to? 0 for DFT-only configuration 1 for stereo configuration 2 for TOF configuration
The wide-angle tracking camera will enumerate as camera ID 0, 1, or 2 depending on what other cameras are connected. This not NOT the physical port the camera is connected to, this is the camera ID that it is enumerated as in software. The 3 most common configurations and their corresponding camera ID are listed in the command prompt and repeated above for reference.
If you are using another configuration refer to the Camera Configuration page. You can always check the cameras before this step with the camera-test app built into the VOXL image or with the voxl-cam-ros node. You can also go back later and change the camera channel used by the ModalAI Vision Lib by updating the “override_cam_id” value in /etc/snav/camera.downward.xml
2) Next you will be prompted with the second question:
Which physical configuration is this for? 1) red_tray 3) voxlcam_upside_down_tracking_cam 2) voxlcam_upright_tracking_cam
This configures the position of the IMU relative to the wide-angle tracking camera for the Visual Inertial Odometry function. Most use cases will fall into these 3 options. For custom use cases refer to the camera imu coordindate frames page.
3) Now we need to calibrate the IMU. Set up your VOXL in whichever orientation you will consider roll/pitch to be zero. Ideally this will be done with VOXL mounted to the quadcotper/robot and sitting still with no vibration or human contact. For VOXLCam this orientation should be with VOXLCam upright and looking forward with no tilt. For a red tray mount this is with the tray level.
the next step is static IMU calibration Make sure the quadcopter or VOXLCAM is flat/upright (roll/pitch=0) be patient, this will take 20-30 seconds PRESS ENTER to start static IMU calibration
This process will take 20-30 seconds, be patient. You can ignore any systemctl warnings that show up. When it’s done you should see:
 Static accel calibration was completed successfully Stopping SNAV Done starting imu_app DONE
This will now start the imu_app service. Note that this systemd service is enabled automatically on installation of of the ModalAI Vision Lib and takes control of the IMU. If you need direct access to the IMU (eg, not through the imu_app server) for your development you will want to stop or disable the imu_app systemd service when not using the ModalAI Vision Lib.
You can now test the vision lib with voxl-test-vision-lib
/home/root/ipk# voxl-test-vision-lib -h usage: voxl-test-vision-lib [opts] at least one option must be provided -h print this message. -v print version information. -i Set IP address of the VOXL for video streaming -p Set port number for the video stream, default 5556 -o enable visual inertial odometry -a enable April Tag detection
Run with odometry turned on to see VIO data
/home/root/ipk# voxl-test-vision-lib -o [72785.678483] SNAV INFO: modalai_vl is now running... VIO: points: 39 X: -0.009 Y: 0.012 Z: 0.023 R: -0.024 P: -0.000 Y: -0.008 VIO: points: 39 X: -0.009 Y: 0.012 Z: 0.024 R: -0.024 P: 0.000 Y: -0.008 VIO: points: 39 X: -0.009 Y: 0.012 Z: 0.023 R: -0.024 P: 0.000 Y: -0.008 VIO: points: 39 X: -0.009 Y: 0.012 Z: 0.023 R: -0.023 P: 0.000 Y: -0.008 VIO: points: 39 X: -0.009 Y: 0.012 Z: 0.023 R: -0.023 P: 0.000 Y: -0.009
Print out an Apriltag from https://github.com/AprilRobotics/apriltag-imgs and test the AprilTag detector. Note that the translation values will be incorrectly scaled until you set up the “tag_size” parameter in /etc/snav/app_params.snav_dft_vio_apriltag_app.excelsior.xml This parameter is the length of the side of the tag in meters.
/home/root/ipk# voxl-test-vision-lib -a [72902.784510] SNAV INFO: modalai_vl is now running... Detected 1 AprilTags(s) Apriltag ID: 0: X: 0.966 Y: -0.384 Z: 1.300 R: -0.049 P: -0.897 Y: -3.125 Detected 1 AprilTags(s) Apriltag ID: 0: X: 0.348 Y: -0.558 Z: 1.425 R: -0.544 P: -0.751 Y: -2.862 Detected 1 AprilTags(s) Apriltag ID: 0: X: 0.217 Y: -0.546 Z: 1.331 R: -0.404 P: -0.331 Y: -2.976 Detected 1 AprilTags(s) Apriltag ID: 0: X: 0.226 Y: -0.563 Z: 1.369 R: -0.439 P: -0.419 Y: -2.940 Detected 1 AprilTags(s) Apriltag ID: 0: X: 0.255 Y: -0.583 Z: 1.348 R: -0.573 P: -0.476 Y: -2.862
If you want, check the the video stream by passing the ip address of the VOXL to the voxl-test-vision-lib via the -i argument. This should be its on the network interface to be used for the vieo streeam. Probably wlan0.
View the video stream with the image_viewer.py script, passing it the same ip address that you gave to voxl-test-vision-lib.
Look inside /usr/include/snav/modalai_vl.h and /usr/include/snav/sn_interface_plugin.h for the functions available to use.
A good example of how to use this library is voxl-vision-px4.
Copyright (c) 2019 ModalAI Inc.