VOXL Camera Drivers

Table of Contents

1. VOXL Camera Drivers
   1. Components of a Camera Driver
   2. Sensormodule File
   3. Sensor File
   4. Tuning File
2. Camera Frame Rotation for Tracking Cameras
3. Camera Frame Rotation for Hires Cameras
   1. ISP Processing Path
   2. MISP Processing Path
   3. voxl-camera-server integration

Components of a Camera Driver

There are three main components of the camera drivers on VOXL2:

• sensormodule (.bin)
• sensor (.so)
• tuning file (.bin)

Sensormodule File

• binary file (not human readable), located in /usr/lib/camera/
• a sensormodule is required for each camera and the slot id has to match actual port where the camera is connected
  • the slot id in the sensormodule is the number at the end of the bin, such as: com.qti.sensormodule.imx412_fpv_1.bin – slot 1
  • note that the number in the name has no effect on the value – the slot id is also stored inside the .bin and changing filename will not change the slot number
• example name: com.qti.sensormodule.imx412_fpv_1.bin
• contains the following information
  • camera’s hardware ID (slot ID), i2c address, and WHOAMI register information for camera probing
    • a different sensormodule is required if an alternate i2c slave address is used for a camera
  • power on/off sequencing order
  • specification of all camera modes, including resolution, fps, bayer type, bit depth, etc
  • a list of camera configuration registers for each streaming mode
  • the name of the tuning file to load (has an effect on ISP pipeline only)
  • …

Sensor File

• shared library, located in /usr/lib/camera/
• example file names: com.qti.sensor.imx412_fpv.so, com.qti.sensor.imx664.so
• contains functions for converting the exposure / control information between real units and camera registers
• the sensor name (e.g. imx412_fpv) has to match the sensor name that is stored in the sensormodule, otherwise the sensor.so will not be loaded correctly
• if the sensor.so is not found, the camera pipeline will not fail, but it will use built-in default functions for exposure / gain conversion and those will most likely not work correctly

Tuning File

• applies to the ISP pipeline only
• located in /usr/lib/camera/
• example file names : com.qti.tuned.default.bin, com.qti.tuned.imx412_fpv.bin
• upload loading of sensormodule, the camera pipeline will see which tuning file is associated with the sensormodule and tries to load that tuning file
  • if the specific tuning file is not found, the default tuning file is loaded com.qti.tuned.default.bin
  • if neither the spefic nor the default tuning files are present, the camera will appear not detected

Camera Frame Rotation for Tracking Cameras

• applies to AR0144
• all camera server versions use en_rotate flag in voxl-camera-server.conf to speficy whether the camera should be set to reverse the readout direction
• bayer type does not matter because the AR0144 tracking camera is monochrome

Camera Frame Rotation for Hires Cameras

• applies to IMX412 and IMX664 cameras
• image rotation (180 degrees) can be achieved by reversing the readout direction from camera (both X and Y)
  • this results in zero processing overhead, however usually changes the bayer pattern (reversed) from RGGB to BGGR (etc)

ISP Processing Path

• applies to the following streams, if enabled: small_video, large_video
• also applies to preview stream if en_raw_preview is set to false
• Qualcomm ISP is used to process the image to generate YUV and encoded videos
• sensormodule bin file contains the register setting for the readout direction as well as the bayer pattern (RGGB, BGGR, etc)
• the ISP will use the bayer pattern specification from the sensormodule file to properly debayer the image
  • incorrectly debayered image will have R and B colors swapped

MISP Processing Path

• applies to all MISP streams when en_raw_preview is set to false (MISP requires it)
• MISP does not use the bayer specification from sensormodule, unlike the ISP
• the bayer pattern is determinted by the default patter type (stored in the camera server) and the en_rotate flag, which reverses the readout and the bayer pattern.

voxl-camera-server integration

• camera server before v2.2.12
  • does not control the image readout direction for imx412 and imx664 cameras
  • en_rotate option in voxl-camera-server.conf does not change camera registers, but it does change interpretation of bayer for MISP pipeline (RGGB vs BGGR)
• camera server v2.2.12 and later uses en_rotate option from voxl-camera-server.conf to override the rotation setting in the sensormodule
  • this is done by setting the readout direction register after the camera pipeline initializes the camera using the registers from sensormodule
  • the relevant camera registers are written in the cci_direct_helpers file / functions
  • en_rotate option also controls the interpretation of colors in the MISP pipeline
  • this means that rotation register in sensormodule no longer has effect because it is overridden
  • since camera server can override the readout direction, but it cannot change the bayer pattern specification in the sensormodule, the ISP may not correctly interpret the image, unless the camera server sets the rotation register that is consistent with the sensormodule
• Configuration that works for all versions of voxl-camera-server
  • non-rotated image:
    • use sensormodule without _flip_
    • set en_rotate to false in voxl-camera-server.conf - this will allow MISP to interpret colors correctly (and set the readout direction register for version >= v2.2.12)
    • ISP pipeline automatically determines bayer pattern from sensormodule and it will be consistent with camera registers
  • rotated image:
    • use sensormodule with _flip_
    • set en_rotate to true in voxl-camera-server.conf - this will allow MISP to interpret colors correctly (and set the readout direction register for version >= v2.2.12)
    • ISP pipeline automatically determines the reversed bayer pattern from sensormodule and it will be consistent with camera registers