ModalAI Vision Lib

Overview

The ModalAI Vision Lib provides Visual Inertial Odometry (VIO), April Tag Detection, and Depth from Stereo on VOXL Hardware.

Installation

The ModalAI Vision Lib is included in the VOXL Software Bundle.

Configuration

1) While still logged into VOXL, run the configuration script. You will be prompted by the first question:

/home/root$ bash /etc/snav/configure_modalai_vl.sh

Which camera channel is the tracking camera connected to?
0 for DFT-only configuration
1 for stereo configuration
2 for TOF configuration

The wide-angle tracking camera will enumerate as camera ID 0, 1, or 2 depending on what other cameras are connected. This not NOT the physical port the camera is connected to, this is the camera ID that it is enumerated as in software. The 3 most common configurations and their corresponding camera ID are listed in the command prompt and repeated above for reference.

If you are using another configuration refer to the Camera Configuration page. You can always check the cameras before this step with the camera-test app built into the VOXL image or with the voxl-cam-ros node. You can also go back later and change the camera channel used by the ModalAI Vision Lib by updating the “override_cam_id” value in /etc/snav/camera.downward.xml

2) Next you will be prompted with the second question:

Which physical configuration is this for?

1) red_tray
3) voxlcam_upside_down_tracking_cam
2) voxlcam_upright_tracking_cam

This configures the position of the IMU relative to the wide-angle tracking camera for the Visual Inertial Odometry function. Most use cases will fall into these 3 options. For custom use cases refer to the camera imu coordindate frames page.

3) Now we need to calibrate the IMU. Set up your VOXL in whichever orientation you will consider roll/pitch to be zero. Ideally this will be done with VOXL mounted to the quadcotper/robot and sitting still with no vibration or human contact. For VOXLCam this orientation should be with VOXLCam upright and looking forward with no tilt. For a red tray mount this is with the tray level.

the next step is static IMU calibration
Make sure the quadcopter or VOXLCAM is flat/upright (roll/pitch=0)
be patient, this will take 20-30 seconds
PRESS ENTER to start static IMU calibration

This process will take 20-30 seconds, be patient. You can ignore any systemctl warnings that show up. When it’s done you should see:

[100] Static accel calibration was completed successfully
Stopping SNAV
Done
starting imu_app
DONE

This will now start the imu_app service. Note that this systemd service is enabled automatically on installation of of the ModalAI Vision Lib and takes control of the IMU. If you need direct access to the IMU (eg, not through the imu_app server) for your development you will want to stop or disable the imu_app systemd service when not using the ModalAI Vision Lib.

Testing

You can now test the vision lib with voxl-test-vision-lib

/home/root/ipk# voxl-test-vision-lib -h

usage: voxl-test-vision-lib [opts]
at least one option must be provided
-h  print this message.
-v  print version information.
-i  Set IP address of the VOXL for video streaming
-p  Set port number for the video stream, default 5556
-o  enable visual inertial odometry
-a  enable April Tag detection

Run with odometry turned on to see VIO data

/home/root/ipk# voxl-test-vision-lib -o
[72785.678483] SNAV INFO: modalai_vl is now running...
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.024 P: -0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.024 R: -0.024 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.024 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.023 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.023 P:  0.000 Y: -0.009

Print out an Apriltag from https://github.com/AprilRobotics/apriltag-imgs and test the AprilTag detector. Note that the translation values will be incorrectly scaled until you set up the “tag_size” parameter in /etc/snav/app_params.snav_dft_vio_apriltag_app.excelsior.xml This parameter is the length of the side of the tag in meters.

/home/root/ipk# voxl-test-vision-lib -a
[72902.784510] SNAV INFO: modalai_vl is now running...
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.966 Y: -0.384 Z:  1.300 R: -0.049 P: -0.897 Y: -3.125
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.348 Y: -0.558 Z:  1.425 R: -0.544 P: -0.751 Y: -2.862
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.217 Y: -0.546 Z:  1.331 R: -0.404 P: -0.331 Y: -2.976
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.226 Y: -0.563 Z:  1.369 R: -0.439 P: -0.419 Y: -2.940
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.255 Y: -0.583 Z:  1.348 R: -0.573 P: -0.476 Y: -2.862

If you want, check the the video stream by passing the ip address of the VOXL to the voxl-test-vision-lib via the -i argument. This should be its on the network interface to be used for the vieo streeam. Probably wlan0.

View the video stream with the image_viewer.py script, passing it the same ip address that you gave to voxl-test-vision-lib.

Next Steps

Look inside /usr/include/snav/modalai_vl.h and /usr/include/snav/sn_interface_plugin.h for the functions available to use.

A good example of how to use this library is voxl-vision-px4.