Link Search Menu Expand Document

ModalAI Vision Lib

This package is deprecated.

Please refer to VOXL SDK for our current software bundle.

Overview

The ModalAI Vision Lib provides Visual Inertial Odometry (VIO), April Tag Detection, and Depth from Stereo on VOXL Hardware.

Installation

The ModalAI Vision Lib is included in the VOXL Software Bundle.

Configuration

1) While still logged into VOXL, run the configuration script. You will be prompted by the first question. It is important the camera configuration matches what cameras are connected to the VOXL. The system will not work if there is a mismatch.

yocto:/# voxl-configure-modalai-vl

Which camera configuration are you using?
1 Tracking + Stereo (default)
2 Tracking Only
3 Hires + Stereo + Tracking
4 Hires + Tracking
5 TOF + Tracking
6 Hires + TOF + Tracking
7 TOF + Stereo + Tracking
8 None

If you are using another configuration refer to the Camera Configuration page. You can always check the cameras before this step with the voxl_cam_ros node. You can also go back later and change the camera configuration by running voxl-configure-cameras.

Note: Option 3 is typical for m500 and Flight Deck as they both typically ship with Hi-res + Stereo + Tracking installed.

2) Next you will be prompted with the second question:

Which physical configuration is this for?
This step is purely to define the relationship between VOXL's IMU and tracking camera.

1) voxl_tray                 4) ModalAI_M500_and_Flight_Deck
2) voxlcam_upright_tracking_cam      5) ModalAI_seeker
3) voxlcam_upside_down_tracking_cam  6) Inspired_Flight_IF750

This configures the position of the IMU relative to the wide-angle tracking camera for the Visual Inertial Odometry function. Most use cases will fall into these 3 options. For custom use cases pick any option for now and refer to the camera imu coordindate frames page on how to update this configuration.

3) Now we need to tell it how far the camera is from the ground IF it is mounted in a drone pointed 45 degrees downward. This helps the VIO algorithm converge quickly on first movement. The value does not have to be perfect. Select number 6 to disable this feature for other configurations.

How far off the ground is the tracking camera?
This is a rough number to help VIO initialize, not critical.
Pick whichever option is closest to your platform.
If you REALLY want to enter a custom value, pick the closest value for now
and see https://docs.modalai.com/modalai-vision-lib/ for more details.

1) 2.5cm_dragonfly   3) 10cm_M500_short_legs  5) 28cm_IF750
2) 4cm_seeker        4) 19cm_M500_tall_legs   6) not_applicable

That’s it for questions!

starting imu_app
DONE configuring ModalAI Vision Lib

This will now start the imu_app service. Note that this systemd service is enabled automatically on installation of of the ModalAI Vision Lib and takes control of the IMU. If you need direct access to the IMU (eg, not through the imu_app server) for your development you will want to stop or disable the imu_app systemd service when not using the ModalAI Vision Lib.

Testing

You can now test the vision lib with voxl-test-vision-lib

/home/root/ipk# voxl-test-vision-lib -h

usage: voxl-test-vision-lib [opts]
at least one option must be provided
-h  print this message.
-v  print version information.
-i  Set IP address of the VOXL for video streaming
-p  Set port number for the video stream, default 5556
-o  enable visual inertial odometry
-a  enable April Tag detection

Run with odometry turned on to see VIO data

/home/root/ipk# voxl-test-vision-lib -o
[72785.678483] SNAV INFO: modalai_vl is now running...
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.024 P: -0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.024 R: -0.024 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.024 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.023 P:  0.000 Y: -0.008
VIO: points: 39 X: -0.009 Y:  0.012 Z:  0.023 R: -0.023 P:  0.000 Y: -0.009

Print out an Apriltag from https://github.com/AprilRobotics/apriltag-imgs and test the AprilTag detector. Note that the translation values will be incorrectly scaled until you set up the “tag_size” parameter in /etc/modalai/vio_apriltag_app.xml This parameter is the length of the side of the tag in meters.

/home/root/ipk# voxl-test-vision-lib -a
[72902.784510] SNAV INFO: modalai_vl is now running...
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.966 Y: -0.384 Z:  1.300 R: -0.049 P: -0.897 Y: -3.125
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.348 Y: -0.558 Z:  1.425 R: -0.544 P: -0.751 Y: -2.862
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.217 Y: -0.546 Z:  1.331 R: -0.404 P: -0.331 Y: -2.976
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.226 Y: -0.563 Z:  1.369 R: -0.439 P: -0.419 Y: -2.940
Detected 1 AprilTags(s)
Apriltag ID:  0: X:  0.255 Y: -0.583 Z:  1.348 R: -0.573 P: -0.476 Y: -2.862

If you want, check the video stream by passing the ip address of the VOXL to the voxl-test-vision-lib via the -i argument. This should be its on the network interface to be used for the video stream. Probably wlan0.

View the video stream with the image_viewer.py script, passing it the same ip address that you gave to voxl-test-vision-lib.

Next Steps

Look inside /usr/include/snav/modalai_vl.h and /usr/include/snav/sn_interface_plugin.h for the functions available to use.

A good example of how to use this library is voxl-vision-px4.