Asynchronous & Mixed-Shutter Cameras

Classical multi-camera VIO assumes hardware-synchronized cameras: every camera exposes at the same instant, so one filter clone serves all of them. voxl-open-vins-server removes that requirement. Cameras may free-run at different rates with arbitrary phase, and each camera may be global- or rolling-shutter — including rigs that mix a rolling-shutter hires sensor with a global-shutter tracking camera. Synchronized rigs run the identical code path, unchanged.

This page explains the three mechanisms that make it work: the lock-free ingest, epoch-anchored cloning with the ACI² preintegration bridge, and the per-row rolling-shutter measurement model.

Table of contents

1. Why unsynchronized cameras are hard
2. Lock-free asynchronous ingest
3. Epoch-anchored cloning and the ACI² bridge
4. Rolling-shutter support
5. Mixed-shutter rigs and the fpvhires configuration
6. Estimation quality on asynchronous rigs
7. Configuration reference
   1. Per-camera declarations
   2. Asynchronous multi-camera keys
   3. Rolling-shutter / temporal calibration keys
8. Further reading
9. Appendix — mathematical reference
   1. A.1 Time model and epoch binding
   2. A.2 ACI² preintegration bridge
   3. A.3 Rolling-shutter Jacobians
   4. A.4 Ingest ordering and the period EMA
   5. A.5 Quality metric

Why unsynchronized cameras are hard

Feeding unsynchronized cameras to a stock MSCKF filter fails in one of two ways:

1. Snapping timestamps (pretending frames are synchronized) injects a systematic timing error of up to half a frame period. At a focal length of 450 px and a rotation rate of 1 rad/s, a 7 ms phase error is a ~3 px systematic reprojection bias against a ~1 px noise model. The filter cannot tell timing error from calibration error, so it slowly corrupts its own calibration and bias estimates until it diverges.

2. Cloning at every camera’s frame time fragments the filter’s sliding window. With N cameras, each camera’s feature tracks only span 1/N of the window, so tracks hit the marginalization horizon early, never reach the length that promotes them to SLAM landmarks, and triangulation loses its baseline. The estimator starves.

The resolution is to keep one clone timeline — anchored to a single reference camera — and to attach every other camera’s measurements to that timeline exactly, using the IMU itself.

Lock-free asynchronous ingest

Every camera owns a lock-free single-producer/single-consumer ring inside the estimator. Camera pipe callbacks push frames and return immediately — no locks, no fusion thread, no pairing logic. The IMU callback (the single VIO thread) releases frames when three conditions hold:

• IMU coverage — inertial data spans past the frame time (plus a small guard), so propagation never extrapolates;
• global order — every other live camera has either delivered a frame at or beyond this time, or has been silent long enough (relative to its own measured frame period) that it cannot block;
• per-stream order — frames that arrive out of order or too late are dropped, and every dropped frame is released back to its owner exactly once.

A dead or throttled camera therefore adds bounded latency instead of stalling VIO, and a healthy rig holds all drop counters at zero.

The staleness gate is self-tuning: each stream maintains an exponentially-weighted moving average of its own frame period, updated on every arrival (90/10 blend — see Appendix A.4). The declared per-camera frame rate (camN_fps, below) seeds this EMA, so the ordering gates run at design width from the very first frame after boot and after every reset instead of converging over a warm-up window.

Epoch-anchored cloning and the ACI² bridge

With epoch_mode enabled, stochastic clones are created only at the reference camera’s frame times — the epochs. When another camera’s frame arrives at time t_f, its known offset from the nearest epoch, δ = t_f − t_e, is bridged with an ACI²-style closed-form preintegration of the actual IMU samples spanning [t_e, t_f]: a relative rotation ΔR and position/velocity preintegrals α, β that express the camera pose at t_f exactly in terms of the epoch clone at t_e.

Two properties make this a first-class filter update rather than an approximation:

• Exactness — the bridge integrates real IMU data in closed form; there is no first-order “pose interpolation” error, no matter where in the frame interval the measurement falls.
• Analytic bias Jacobians — the bridge carries ∂(ΔR, α, β)/∂(b_g, b_a), so camera residuals on bridged frames still correct the IMU biases through the bridge, preserving the estimator’s observability structure and consistency (first-estimate Jacobians throughout).

The reference camera must be the fastest camera. A slower reference forces the faster camera to deliver more than one frame per epoch; the extras fall back to private clones and the window fragments — exactly the failure epoch mode exists to prevent. The server warns at boot if the declared rates violate this.

Deferred marginalization. An epoch’s measurements arrive in installments — the reference frame first, bound frames up to one binding horizon later. The oldest clone is marginalized only when its epoch completes, so every camera’s tracks span the full clone window and SLAM landmark promotion behaves exactly as on a synchronized rig.

Rolling-shutter support

A rolling-shutter sensor exposes row by row: a pixel on row m of M was imaged at

t(m) = t_frame + t_offset + t_readout · m/M

The estimator applies this per measurement row: each observation is transported to its true row time using the clone’s angular and linear velocity. Two rules keep this safe:

• Value vs. estimation. The readout value is always applied when nonzero. The readout is estimated online (calib_cam_readout) only for cameras explicitly declared "rolling" — a global-shutter camera’s readout stays pinned at zero and never becomes a spurious estimated state.
• Degenerate-motion gating. Temporal parameters (time offsets, readout) are only observable when the motion varies. Under hover or constant-velocity cruise the estimator freezes their estimation (values remain applied) rather than letting them drift on noise.

A wrong readout value is worse than none. The row-averaged part of a readout error is indistinguishable from a camera time offset and will silently bias it, while the row-dependent part corrupts calibration. Measure the readout of the streamed sensor mode (rows × line time) rather than guessing from a datasheet; if you cannot measure it yet, declaring the camera rolling with readout 0 runs the sensor deliberately unmodeled with a loud boot warning — a legitimate bring-up state.

Mixed-shutter rigs and the fpvhires configuration

The mechanisms above compose: a rolling-shutter hires camera and a global-shutter tracking camera, at different rates, with no sync line, share one clone window and both contribute MSCKF and SLAM updates. That is the fpvhires configuration:

voxl-configure-open-vins fpvhires

which points the server at the /usr/share/modalai/voxl-open-vins/VoxlConfig/fpvhires estimator configuration — IMX664 hires (rolling shutter, 60 fps, epoch reference) plus AR0144 tracking (global shutter, 30 fps) with epoch_mode enabled — and sets sync_config: true, so the OpenVINS calibration YAMLs are regenerated from your system calibration at every startup. (sync_config only controls this calibration synchronization; frame ingest is always the lock-free asynchronous path, and whether frames share timestamps is a property of the rig, not a configuration switch.)

Because fpvhires synchronizes calibration from the system at startup, your system calibration must be complete and correct before enabling it:

1. Intrinsics — both cameras calibrated, with their calibration files present in /data/modalai/ (see Camera Calibration).
2. Extrinsics — /etc/modalai/extrinsics.conf must contain the body → imu and camera relationships for both the hires and the tracking camera (see Configure Extrinsics).
3. /etc/modalai/vio_cams.conf — edit it so exactly these two cameras are enabled: the hires camera’s pipe_for_tracking (e.g. the hires grey/misp stream) and the tracking camera’s pipe, each with the correct cal_file, and both referencing the same imu.

With any of these missing or stale, the regenerated calibration will be wrong and the estimator will misbehave in ways that look like tracking problems.

The classic synchronized configurations (fpv, starling2_2cam, …) are untouched and keep running the synchronized path.

Estimation quality on asynchronous rigs

The published quality score (see the Pipe Interface page) needs special care on asynchronous rigs. Updates arrive per camera — the reference camera opens an update, the other camera’s features join up to a binding horizon later — and MSCKF features only surface in an update when their tracks end. Scoring any single update therefore sees a sparse, noisy sample of tracking health: half the rig on reference ticks, nothing at all on zero-velocity ticks. (A synchronized rig never noticed, because every update carried both cameras at once.)

The server solves this with a windowed union + EMA design:

1. Pool every update’s used-feature list from the last 120 ms and de-duplicate by feature id — this reconstructs the full per-camera track set that a synchronized batch would have carried (and it is the horizon over which a dead camera’s evidence naturally ages out);
2. Score each camera over the pooled set with the standard scorer — a 5×5 image-coverage grid combined with per-feature weights (SLAM landmarks by covariance and track quality, MSCKF tracks by quality and track length);
3. Sum across cameras and clamp to 100 — identical fusion to the synchronized path;
4. Smooth the published value with a 250 ms time-constant EMA: a single weak update moves the score by a couple of points, while a real tracking collapse still reaches the reset thresholds in about half a second.

The quality state is invalidated on every reset (sensor time is continuous across resets, so smoothed values from the previous estimation episode can never pre-charge the new one), and the score is capped low during initialization so a freshly built covariance cannot report false confidence. Exact formulas in Appendix A.5.

Configuration reference

All keys live in the selected estimator_config.yaml. Per-camera declarations tell the estimator what each physical camera is; they are validated at boot and the server fails fast on contradictions.

Per-camera declarations

Asynchronous multi-camera keys

Rolling-shutter / temporal calibration keys

Runtime telemetry: with debug verbosity the estimator prints a per-update [EPOCH] line — epochs snapped vs. fallbacks, plus ingest drop counters (late / ring-full / bogus). On a well-configured rig fallbacks and all drop counters stay at zero; a climbing fallback count almost always means the reference camera is not the fastest declared camera.

Further reading

The full mathematical specification — the bridge closed forms and bias Jacobian recursions, the rolling-shutter Jacobians, the measurement-noise analysis, and the consistency validation — lives with the source in ov_msckf/README_ASYNC.md in the ModalAI OpenVINS fork. Server-side integration is in the voxl-open-vins-server repository.

Appendix — mathematical reference

Notation follows OpenVINS: JPL quaternions, with \({}^{I}_{G}R\) the rotation taking a vector from the global frame into the IMU frame, \({}^{G}p_I, {}^{G}v_I\) position and velocity, \(b_g, b_a\) the gyro/accel biases, and \(\lfloor\cdot\rfloor\) the skew operator.

A.1 Time model and epoch binding

A pixel on row \(m\) of \(M\), in a frame stamped \(t_{f,i}\) by camera \(i\) with time offset \(t_{d,i}\) and readout \(t_{r,i}\), was imaged at

With epochs at the reference camera’s frame times \(t_e\), a frame’s epoch residual and the binding rule are

with \(\beta\) = epoch_bind_factor and \(T_{\mathrm{ref}}\) the reference frame period. One frame per (camera, epoch); a frame outside the horizon falls back to its own clone.

A.2 ACI² preintegration bridge

The bridge relates the IMU pose at the true imaging time \(t_f = t_e + \delta\) to the epoch clone by integrating the actual IMU samples over \([t_e, t_f]\) into relative terms \(\Delta R, \alpha, \beta\) (all expressed in the epoch frame):

Per IMU step of length \(\Delta t\) (bias-corrected midpoint signals \(\hat\omega, \hat a\); step rotation \(R_{s} = \mathrm{Exp}(-\hat\omega\,\Delta t)\); \(A = \Delta R^{\top}\); \(\Xi_{1..4}\) the ACI² closed-form integrals):

The bridge is built once at the bias linearization \(\bar b\) and corrected to first order at update time via analytic bias Jacobians, transported per step as

so that at update time, with \(\delta b = \hat b - \bar b\):

These Jacobians become columns of the measurement matrix on \(b_g, b_a\) (epoch_bridge_bias_cols), so bridged camera residuals correct the IMU biases; all linearization points follow the first-estimate-Jacobian convention.

A.3 Rolling-shutter Jacobians

For a measurement on row fraction \(u = m/M\), the time transport beyond the clone is \(\Delta = \Delta t_{d,i} + t_{r,i}\, u\), linearized with the clone kinematics. The readout columns are, on the observing clone \(I\) and (for anchored representations) the anchor \(I_a\):

The same transport with \(u = 1\) generates the paired per-camera time-offset columns (\(+\) on \(t_{d,i}\), \(-\) on the reference \(t_{d,r}\)) — which is exactly the structure that leaves the common clock shift unobservable (a gauge freedom) while keeping the relative offsets observable. Readout columns exist only for declared-rolling cameras; all temporal columns are skipped (values still applied) when the window motion is degenerate.

A.4 Ingest ordering and the period EMA

Each stream \(j\) tracks its own frame period with an exponentially-weighted moving average, seeded from the declared rate \(f_j\) (camN_fps):

The merge head at time \(t_h\) (the minimum staged timestamp) is released when inertial data covers it and no live stream can still precede it:

with \(g\) = async_guard and \(s\) = async_stale_factor. A silent camera can therefore delay release by at most \(s\,\hat T_j\) before it is treated as stale and bypassed.

A.5 Quality metric

Let \(U(t)\) be the union of all used-feature lists from updates in \([t - W,\ t]\), de-duplicated by feature id (window \(W = 0.12\) s, capacity 256). Each feature carries a weight by class:

where \(\lambda_{\max}\) is the largest eigenvalue of the landmark’s marginal covariance, \(\sigma_{\mathrm{ref}} = 1.4\), \(q_{\mathrm{feat}}\) the tracker’s per-feature quality, \(n_{\mathrm{meas}}\) the feature’s measurement count and \(C = 11\) the clone window. Per camera, features are binned into a \(5\times 5\) image grid; a cell with \(n \ge 2\) features contributes its mean weight, a sparser cell is discounted by \(\sqrt{n/2}\). With \(\rho\) the fraction of occupied cells, the camera score combines coverage and content:

The published value is the time-constant EMA of the fused score (\(\tau = 0.25\) s, \(\Delta t\) the time since the previous update):

The EMA state is discarded on every reset, and the published value is floored at 10 while the filter runs (collapse is reported through the hysteresis machine and the auto-reset path, not by this metric alone).

The asynchronous multi-camera architecture — the lock-free ingest, epoch-anchored cloning, the ACI² preintegration bridge — and the zbft_sfm initialization solver are authored by Joao Leonardo Silva Cotta, PhD, building on the OpenVINS estimator by Patrick Geneva, Guoquan Huang, and the OpenVINS contributors.