Electronic Image Stabilization

Electronic Image Stabilization

Overview

Electronic Image Stabilization (EIS) enables software-based image (video) stabilization. EIS uses IMU data (accelerometer and gyroscope) to help with removing unwanted rotation of the image.

On VOXL2, the EIS processing is performed using a combination of algorithms running on the CPU and GPU.

Hardware Requirements

VOXL2 or VOXL2 Mini

Supported Cameras and Input Resolutions

- IMX412 (M0107 or M0161)
  - 4040x3040 - calibration file resolution 4040x3040 or 2020x1520 up to 60FPS
  - 1996x1520 - calibration file resolution 1996x1520 or 2020x1520 up to 120FPS
- IMX664 (M0186)
  - 2704x1540 - calibration file resolution 2704x1540 or 1352x770  up to 120FPS

Note that output (stabilized) resolution can be any resolution equal to or smaller than the input resolution
If intrinsic calibration files are not provided, ideal defaults will be used (OK for testing without intrinsics cal)

Software Requirements

EIS implementation will be part of voxl-camera-server starting VOXL2 SDK 1.5.? (TBD)

Latest Source Code

voxl-camera-server
- build from source or use latest nightly build : dev packages
voxl-portal
- need to build the latest package manually in order to use the latest version (not required for basic testing)

Minimal Setup For Testing

Minimal Software Setup

Use a recent VOXL2 SDK (1.4.5 or later)
Use standard VOXL2 and camera orientation
- VOXL2 right side up, X axis pointing forward
- Camera right side up, facing straight forward
Use latest camera drivers
- For IMX412: test drivers
- For IMX664: included with the SDK
use latest voxl-camera-server from nightly or build from dev branch
Update voxl-imu-server settings : /etc/modalai/voxl-imu-server.conf
- set imu0_fifo_poll_rate_hz to 250
- confirm imu0_sample_rate_hz is 1000
Confirm:
- in /vendor/etc/camera/camxoverridesettings.txt … maxRAWSizes=20
- this allows the camera pipeline to use any of the raw image sizes for the preview stream

Minimal Camera Server Configuration

{
    "version":  0.1,
    "cameras":  [
    {
            "type":             "imx412",
            "name":             "hires",
            "enabled":          true,
            "camera_id":        0,
            "fps":              30,
            "en_preview":       true,
            "en_raw_preview":   true,
            "en_misp":          true,
            "preview_width":    4040,
            "preview_height":   3040,

            "misp_width":       1280,
            "misp_height":      720,
            "misp_awb":         "auto",

            "misp_zoom":        1.5,
            "en_eis":           true,
            "eis_mode":         "horizon-level",
            "eis_view":         "pip",
            "eis_follow_rate":  0.01,

            "en_large_video":   false,
            "en_small_video":   false,
            "en_snapshot":      false,
            "ae_mode":          "lme_msv",
            "en_rotate":        false,
            "misp_venc_enable": true,
            "misp_venc_mode":   "h264",
            "misp_venc_br_ctrl":    "cbr",
            "misp_venc_Qfixed": 30,
            "misp_venc_Qmin":   15,
            "misp_venc_Qmax":   51,
            "misp_venc_nPframes":   29,
            "misp_venc_mbps":   3,
            "ae_desired_msv":   100,
            "exposure_min_us":  25,
            "exposure_max_us":  15000,
            "gain_min":         100,
            "gain_max":         32000,
            "exposure_soft_min_us": 5000,
            "snapshot_jpeg_quality":    75
        }],
    "fsync_en": false,
    "fsync_gpio":   109
}

Relevant Camera Server Config parameters

Please note the parameter names may change during development cycle

- "type"            : should be one of: "imx412", "imx412-fpv-misp", "imx664", "imx664-fpv"
- "en_raw_preview"  : has to be set to "true" for EIS to work.
- "en_misp"         : enable Modalai ISP processing, has to be set to "true"
- "misp_width"      : the output size of the stabilized image. Can be arbitrary size
- "misp_height"     : the output size of the stabilized image. Can be arbitrary size
- "misp_awb"        : white balance mode, should be set to "auto"
- "en_eis"          : enable image stabilization ("true" or "false")
- "misp_zoom"       : initial digital zoom for eis output (start with 1.5)
- "eis_mode"        : type of behavior of the stabilized view
  - "horizon-level" : will completely stabilize roll, but stabilize and follow yaw and pitch
  - "full-follow"   : will stabilize and follow all three axes
- "eis_view"        : type of output (stabilized) image view
  - "eis-only"      : only show the stabilized view
  - "pip"           : stabilized view + full frame and ROI outline in PIP (picture-in-picture)
  - "side-by-side"  : stabilized view and full frame + ROI outline side by size (wide image)
- "eis_follow_rate" : parameter that specifies convergence rate of the ROI to the IMU (0.0 = no convergence, 1.0 = instant)

Running EIS

For best results, make sure the cpu is in perf mode : voxl-set-cpu-mode perf
start voxl-camera-server in foreground and confirm that it does not report a critical error
enable adb forwarding adb forward tcp:8080 tcp:80
open browser to access voxl-portal at localhost : http://localhost:8080
select hires_misp_color in cameras drop down menu
you should see the stabilized camera view as well as the debug view to the right of the stabilized image

Calibration Files

Intrinsic Calibration Files

If intrinsic calibration file is not found, an ideal calibration is assumed for the specific camera / mode
for each camera, calibration file with the following file name format needs to be present in /data/modalai/
The <camera_name> needs to match the camera name property in voxl-camera-server.conf

/data/modalai/<camera_name>_intrinsics.yml

if the intrinsics file name is not found, a default one for the camera type will be loaded, which will result in suboptimal stabilization quality
calibration files have very specific requirements, related to the resolutions supported by EIS, see Supported Cameras and Resolutions section

Extrinsic Calibration

If the extrinsics transform is not found, a default transform will be used, assuming the camera is right side up and points straight forward. Specifically:
- the camera Z axis is aligned with IMU X axis (forward)
- the camera X axis is aligned with IMU Y axis (right)
- the camera Y axis is aligned with IMU Z axis (down)
A common file with all extrinsic calibration parameters should already be located in /etc/modalai/extrinsics.conf
For each camera, the following entry must be present, where matches the camera `name` property in `voxl-camera-server.conf`
The translational part (T_child_wrt_parent) is currently not used by EIS

The rotational part can be approximated based on the design, if exact calibration is not available

{
  "parent": "imu_apps",
  "child":  "<camera_name>",
  "T_child_wrt_parent": [0.0, 0.0, 0.0],
  "RPY_parent_to_child":    [0, 90, 90]
}

Extrinsic rotation examples:
- Refer to docs: VOXL2 IMU and VOXL2 Mini IMU
- For these examples we will assume that the VOXL2 / VOXL2 Mini is nominal orientation (X=Forward, Y=Right, Z=Down)
  - Camera is right side up its optical axis (z) is aligned with VOXL2 X axis (camera points straight forward)
    "RPY_parent_to_child": [0, 90, 90]
  - Similar to above, but instead of pointing forward, the camera is pitched down 45 degrees
    "RPY_parent_to_child": [0, 45, 90]
  - Similar to above, but instead of pointing forward, the camera is pitched straight down (90 degrees)
    "RPY_parent_to_child": [0, 0, 90]

Using EIS with MISP Channels

3 additional output channels are now supported in addition to the main MISP channel
currently, EIS stabilization parameters are shared for all output channels
eis on/of can be selected on per channel basis (“eis_mode” = “off” will disable EIS on this channel, otherwise will enable. This parameter will be extended)

sample “misp_channels” entry is provided below (this should go in voxl-camera-server.conf)

...
"eis_follow_rate":      0.01,
"misp_channels":        [{
                        "enable":       1,
                        "name": "default_misp",
                        "width":        1280,
                        "height":       720,
                        "fps":  30,
                        "mbps": 2,
                        "nPframes":     29,
                        "codec":        "h265",
                        "osd":  0,
                        "eis_mode":     "on"
                }, {
                        "enable":       1,
                        "name": "long_range_misp",
                        "width":        960,
                        "height":       540,
                        "fps":  30,
                        "mbps": 0.8,
                        "nPframes":     29,
                        "codec":        "h265",
                        "osd":  0,
                        "eis_mode":     "off"
                }, {
                        "enable":       1,
                        "name": "low_latency_misp",
                        "width":        960,
                        "height":       540,
                        "fps":  60,
                        "mbps": 1.5,
                        "nPframes":     29,
                        "codec":        "h265",
                        "osd":  0,
                        "eis_mode":     "off"
                }],
...

Other Info

EIS includes image undistortion according to the camera calibration parameters
Correct camera calibration will produce better stabilization results
Rolling shutter compensation is not yet available
For best output quality, use maximum input (preview) resolution (4040x3040 for IMX412), up to 60FPS is supported.
For reduced rolling shutter effect, use lower (2x2 binned) input resolution of (1996x1520) - this resolution has faster readout time = reduced rolling shutter effect
debayering process currently adds image smoothing, so using 2x2 binned resolution will result in some loss of image crispness

Tips

In dark environments, force lower exposure by setting exposure_max_us parameter to 5000-10000 us to avoid motion blur
Use H265 encoder for best results (CBR or CQP, depending on desired behavior)
- see link

Supported Control Commands via `voxl-portal`

Make sure the latest voxl-portal is running from the eis-integration branch
Make sure the advanced control panel is active (click on the check box at the bottom left corner of the image view)
Zoom : use mouse wheel while hovering over the image to zoom in / out
Click : click either in stabilized view or full frame view to center the ROI on the clicked point
Drag : click and drag the image to move the ROI (either stabilized or unstabilized)
Manual exposure / gain controls. Note that once manual exposure / gain is selected, the auto exposure does not come back until camera server is restarted (will be fixed soon)
Enable / Disable EIS : checkbox (coming soon)

Performance

up to 3840x2160 @ 60 FPS encoded (h264/h265). GPU utilization about 30% (no pip or side-by-side view)
use of h265 encoder is strongly encouraged (not h264), especially for larger size output streams
viewing larege YUV images in voxl-portal will result in significant voxl2 cpu usage and dropped frames in the feed (due to mjpg encoding)
initial testing recommended at lower output resolutions (easier streaming and viewing… 1280x720, 1920x1080)

Calibrating Camera at Half Resolution

set the preview size to full frame (4040x3040 for IMX412)
set misp_width and misp_height to half resolution (2020x1520 for IMX412)
disable EIS (config file: "en_eis": false)
set max exposure to 5ms to reduce motion blur (in camera config)
use a pattern displayed on a PC screen (suggested 10x7 checkerboard)
run voxl-cameera-server
run voxl-calibrate-camera hires_misp_grey -f -m -s 10x7 -l 0.1