Link Search Menu Expand Document

VOXL Camera Server

Table of contents

  1. Overview
    1. HAL3
  2. Output of Camera Server
    1. preview Output
    2. stream Output
      1. Supported Resolutions
    3. record Output
    4. snapshot Output
  3. Exposure Control
    1. Manually Setting Exposure and Gain
  4. Camera sensor options and capabilities
    1. Inspecting
  5. Configuration
    1. Changing the Resolution of an Image Sensor
    2. Parameter definitions
    3. Default parameters
  6. Streaming Options
    1. Stream Over Web/HTTP using VOXL Portal
    2. Stream Over RTSP
    3. Stream to ROS
  7. Source

Overview

VOXL Camera Server is built around the Modal Pipe Architecture (using libmodal_pipe) to serve up MIPI cameras to multiple MPA clients. Serving to multiple MPA clients - this means publishing image frames for consumption by MPA clients such as: voxl-portal, ROS, or rtsp streams.

HAL3

VOXL Camera Server uses Google’s HAL3 API to access raw camera data from qualcomm’s mm-qcamera-daemon pipeline.

Output of Camera Server

voxl-camera-server supports 4 output modes, each published via Modal Pipe Architecture (using libmodal_pipe):

ModePurposeFormat
previewOutput format of the ISP, uncompressed, for processing with computer vision algorithmsNV12 or RAW8
streamCompressed using hardware acceleration, OMX, for streaming over a networkh.264
recordCompressed using hardware acceleration, OMX, for saving to flashh.264
snapshot  

The frames from camera server are published to /run/mpa via libmodal_pipe. The data is sent in a binary form with a metadata struct followed by the raw image data (in a format defined by the metadata). The metadata struct is currently source:

/**
 * The metadata for the camera image. One of these is sent before every frame
 */
typedef struct camera_image_metadata_t
{
    uint32_t magic_number;  ///< set to CAMERA_MAGIC_NUMBER
    int64_t  timestamp_ns;  ///< timestamp in apps-proc clock-monotonic of beginning of exposure
    int32_t  frame_id;      ///< iterator from 0++ starting from first frame when server starts on boot
    int16_t  width;         ///< image width in pixels
    int16_t  height;        ///< image height in bytes
    int32_t  size_bytes;    ///< size of the image, for stereo this is the size of both L&R together
    int32_t  stride;        ///< bytes per row
    int32_t  exposure_ns;   ///< exposure in microseconds
    int16_t  gain;          ///< ISO gain (100, 200, 400, etc..)
    int16_t  format;        ///< raw8, nv12, etc
    int32_t  reserved;
} __attribute__((packed)) camera_image_metadata_t;

The complete list of image types that can be sent through a camera pipe can be found here.

voxl-inspect-cam output with multiple output types

voxl-streamer-arch

preview Output

The default output for each sensor using voxl-camera-server is the preview channel output. voxl-streamer is then capable of taking those data streams as an input, compressing them, and then streaming over IP as RTSP. voxl-qvio-server takes this output as its input for the tracking sensor, performs VIO processing, and publishes its qvio-overlay in this format via MPA.

  1. preview_width
  2. preview_height

stream Output

voxl-camera-server V1.4.5 and above has the ability to compress image frames using OMX for a no-copy, hardware accelerated and memory optimized data path to publish compressed data. This then can be leveraged by voxl-streamer v0.5.0 and above to input the h264 as input and stream over IP as an RTSP stream. The following is how a user can leverage this ability in voxl-camera-server:

The above is enabled by default on hires camera. The name of the pipe this encoded data is being sent to is: hires_stream The parameters pertinent to this feature are:

  1. stream_width
  2. stream_height
  3. stream_bitrate

With the previous parameters populated, the user can now stream encoded 720p, 2k, and 4K video to voxl-streamer to be viewed in VLC or QGC.

The following video shows how to setup 720p encoded video stream to voxl-streamer using a VOXL2 and IMX412.

###INSERT VIDEO###

Supported Resolutions

VOXL 2, there are only a subset of three (3) resolutions that with OMX:

  1. 4096x2160
  2. 2048x1536
  3. 1024x768

The following configurations have been tested

Image SensorResolutionSupported
IMX2141024x768 
IMX2142048x1536 
IMX2144096x2160 
IMX4121024x768V1.4.5+
IMX4122048x1536 
IMX4124096x2160NO
IMX6781024x768 
IMX6782048x1536 
IMX6784096x2160 

record Output

Like streaming low latency over h264 at 720p/4K, there is also the capability now of doing the same thing via this record feature. This once again, has the ability to compress image frames using OMX for a no-copy hardware accelerated and memory optimized data path to publish this compressed and encoded data. This encoded video is then saved on disk using this record feature. The following parameters are how a user can leverage this ability in voxl-camara server:

This output is enabled by default on hires camera. The name of the pipe this encoded data is being sent to is: hires_record The parameters pertinent to this feature are:

  1. record_width
  2. record_height
  3. record_bitrate

Please refer to the ‘stream Output’ Supported Resolutions for resolution details.

snapshot Output

The Snapshot output…

Exposure Control

VOXL Camera Server primarily uses ModalAI’s libmodal_exposure for camera auto exposure/gain values, but this can also be changed to use the ISP’s (Image Signal Processing) exp/gain settings, or can be disabled altogether. ISP is Additionally, you can use voxl-send-command to manually send commands to update the exposure/gain of a specific camera. If a camera was using auto exposure/gain when such a command is sent, the AE will be disabled until the start_ae command is sent and the camera will sit at the requested exposure/gain.

Manually Setting Exposure and Gain

When an image sensor is configured to use a ModalAI exposure algorithm, auto_exposure_mode: modalai, one can manually set exposure and gain values from the command line. This can be useful for debugging image sensor performance.

voxl-send-command hires set_exp_gain <exposure> <gain>

For example:

voxl-send-command hires set_exp_gain 200 150

Camera sensor options and capabilities

For the available camera configurations as well as the accepted FPS/Dimensions associated to each camera sensor, please follow the link here.

Inspecting

The voxl-inspect-cam tool provides a simple way of inspecting camera frames in-terminal and the source serves as a good example for how to read from camera pipes on an embedded process.

Configuration

Whenever cameras are plugged/unplugged (which should only happen when the board is powered off), make sure to run voxl-configure-cameras. This script will generate a file at /etc/modalai/voxl-camera-server.conf containing the default settings for that camera configuration. If you want to change some aspects of the camera behavior such as disabling/framerate/resolution(for hires)/exposure setpoint you can do so by modifying that file and restarting camera server with systemctl restart voxl-camera-server.

When voxl-configure-cameras is run (without specifying camera config in the executable), the user will be prompted with a selection of pre-existing options of camera combinations. These include a combination of stereo cameras, TOF sensors, tracking cameras, and highres cameras. When running after determining the what setup of cameras the user has, please use voxl-configure-cameras #NUMBEROFCAMERACONFIGURATION. This will then generate the voxl-camera-server.conf mentioned above specific to the camera setup the user has. At this point, once again restart the voxl-camera-server with the command mentioned above.

Changing the Resolution of an Image Sensor

The configuration of specific parameters for each image sensor can be achieved by editing the configuration file /etc/modalai/voxl-camera-server.conf. Most of the services that subscribe to the voxl-camera-server are leveraging the preview stream. The preview stream can be thought as a low latency minimal resoluation stream to the user (aka voxl-portal) that would be live of what the drone’s cameras are seeing. To change the resolution output of the preview stream, edit the preview_width and preview_height parameters for the specific sensor. The supported resolutions for each sensor can be seen by running voxl-camera-server -l. There is another option for something called a “snapshot” that is basically the ability to take a photo or in this case a snapshot (at high resolution) from the preview stream. This gets stored as a traditional JPG on the drone. The snapshot option shows up in the stream in voxl-portal.

Parameter definitions

“name”: Name of camera being used. E.G. tracking camera has a name of “tracking”
“flip”: If the camera is flipped so the frame is rotated
“enabled”: Is this camera to be enabled by the service - if set to true, then it will be used or streamed
“frame_rate”: The framerate associated to the camera
“type”: This is the actual hardware sensor name - in the case of the tracking camera used (provided by ModalAI), it is the ov7251
“camera_id”: This is the the id of the camera
“ae_desired_msv”: This is the desired mean sample value, a.k.a. the average value of pixels that the auto exposure algorithm should try to achieve in frame
“ae_mode”: Type of autoexposure mode chosen (such as modal algo, isp, lme_hist, lme_msv)
“ae_k_p_ns”: This is the desired p_ns for the exposure algorithm
“ae_k_i_ns”: This is the desired k_i for the exposure algorithm
“ae_max_i”: This is the desired max_i for the exposure algorithm
“ae_filter_alpha”: This is a low-pass filter constant that filters the calculated MSV to slow down responses - the filter used is an IIR filter
“ae_ignore_fraction”: This is maximum percentage of saturated (255) pixels that will be used in calculation of MSV. If there are more saturated_pixels / total_pixels, then additional saturated pixels are not used to calculate MSV. This helps prevent image getting too dark if there are large blobs of very bright light
“ae_slope”: This is a ratio that specifies how much gain vs exposure should be changed when trying to achieve desired MSV. Both gain and exposure linearly affect the pixel brightness, but gain and exposure have different effects on the image quality - mostly in the sense that gain affects granularity and that exposure affects motion blur.
“ae_exposure_period”: This controls the duration where the cells of the camera sensor are exposed to light
“ae_gain_period”: The gain period associated to the auto exposure (think of this as an amplification factor of the pixels)
“preview_width”: This is the preview width of the frame being sent to VOXL-PORTAL from the camera
“preview_height”: This is the preview height of the frame being sent to VOXL-PORTAL from the camera
“snapshot_width”: This is the snapshot width of the frame being taken as a photo/snapshot from the camera
“snapshot_height”: This is the snapshot height of the frame being taken as a photo/snapshot from the camera
“stream_width”: This is the stream width of the frame streamed to voxl-streamer
“stream_height”: This is the stream height of the frame streamed to voxl-streamer
“stream_bitrate”: This is the stream bitrate of the frame streamed to voxl-streamer
“record_width”: This is the record width of the frame being saved onboard in a .avi file
“record_height”: This is the record height of the frame being saved onboard in a .avi file
“record_bitrate”: This is the record bitrate of the frame being saved onboard in a .avi file
“independent_exposure”: This is the independent exposure value for a stereo pair

Default parameters

{
    "version":  0.1,
    "cameras":  [{
            "name": "tracking",
            "enabled":  true,
            "frame_rate":   30,
            "type": "ov7251",
            "camera_id":    1,
            "ae_desired_msv":   60,
            "ae_filter_alpha":  0.600000023841858,
            "ae_ignore_fraction":   0.20000000298023224,
            "ae_slope": 0.05000000074505806,
            "ae_exposure_period":   1,
            "ae_gain_period":   1
        }, {
            "name": "hires",
            "enabled":  true,
            "frame_rate":   30,
            "type": "imx214",
            "camera_id":    0,
            "preview_width":    1280,
            "preview_height":   720,
            "snapshot_width":   1920,
            "snapshot_height":  1080
        }, {
            "name": "stereo",
            "enabled":  true,
            "frame_rate":   30,
            "type": "ov7251",
            "camera_id":    2,
            "camera_id_second": 3,
            "independent_exposure": false,
            "ae_desired_msv":   60,
            "ae_filter_alpha":  0.600000023841858,
            "ae_ignore_fraction":   0.20000000298023224,
            "ae_slope": 0.05000000074505806,
            "ae_exposure_period":   1,
            "ae_gain_period":   1
        }]
}

Streaming Options

Stream Over Web/HTTP using VOXL Portal

VOXL Portal provides a simple, easy-to-use web interface for viewing images from camera server or any other service providing an image on VOXL. This can be accessed by hitting the IP address of the modalAI device running the voxl-portal service (which can be enabled with a systemctl enable voxl-portal). Once on this page, you will be able to see the preview stream of whatever camera sensors are plugged into the device.

Stream Over RTSP

The VOXL Streamer application can be used to stream video from voxl-camera-server over RTSP.

Stream to ROS

The VOXL MPA to ROS node can be used to provide data coming out of camera server to ROS.

Source

The source code for VOXL Camera Server can be found here