FLIR Hadron 640R Support on VOXL2 and VOXL2 Mini

Overview

Overview

VOXL2 and VOXL2 mini support interfacing with FLIR Hadron 640R. Modalai interface board and connectors are required.

Required Hardware

FLIR Hadron 640R (not included)
M0159 + M0181 adapters
MCBL-00084-180 (qty 2) micro coax cables

WARNING

Mating M0159 with M0181 needs to be done with care
- ❗ Even though the coax cables are the same for both connections between M0159 and M0181, the order cannot be swapped ❗
- M0159 J2 (5L) <-> M0181 J1 (5L)
- M0159 J3 (HR) <-> M0181 J2 (HR)
- HR connector has to conect to HR and 5L to 5L. HR and 5L are labels next to the connectors on M0159 and M0181
❗Incorrectly connecting M159 to M0181 will likely result in damage of Hadron 640R due to incompatible power rails ❗
Shipped assemblies have been correctly connected and tested

Interfacing Options

Once M0159 adapter is plugged into the back of Hadron 640R, usb port on M0159 can be used to power Hadron and interface / configure the IR sensor via the USB interface.
In order to use MIPI connection for the IR sensor, it must be configured to enable MIPI output, see SW instructions in https://docs.modalai.com/m0153/
- Note that for some versions of Hadron firmware, switching from 60 to 30 FPS is required, as described in the document above. This is because frame timing of 60 FPS may be incompatible with VOXL2 camera ISP, resulting in stopoped stream shortly after start.
- Hadron firmware 3.0.26416 has been tested to work properly at 60FPS. If other firmware is used, 30 FPS may need to be selecting using the frame skip feature in Hadron configuration.
It is safe to have the Hadron plugged into VOXL2 as well as USB - make sure all connections are made while the power to VOXL2 is off

Hardware Setup

Connect M0159 Adapter to Hadron

If necessary, remove any interface boards plugged into Hadron 640R in order to allow M0159 to plug in flush with the back of Hadron 640R
Make sure the connector of M0159 aligns with Hadron
See image below for an example of a mated connection

Connect M0181 to Voxl2 Mini J7 or Voxl2

Make sure pin alignment on connectors is correct. the mounting holes should align and the tab of M0181 will hover above the main SoC
See attached images below
The tab can be removed from M0181 if not needed
J7 is officially supported. Other connectors (J6, J8) may be used, but a Kernel change is required (not recommended)

Software Setup

Use Voxl2 mini (VOXL2 also supported)
Install sdk 1.4.0 (tested) or later
Change apt source to dev : /etc/apt/sources.list.d/modalai.list
deb [trusted=yes] http://voxl-packages.modalai.com/ ./dists/qrb5165/dev/binary-arm64/
```
apt update
apt install libmodal-pipe voxl-portal voxl-camera-server
```

Confirm sensormodule drivers exist

in chi-cdk directory where all the camera drivers are kept (but not used)

ls /usr/share/modalai/chi-cdk/ov64b40 : com.qti.sensormodule.ov64b40_0.bin ... 5.bin
ls /usr/share/modalai/chi-cdk/boson:    com.qti.sensormodule.boson_0.bin ... 5.bin

Copy correct sensormodule drivers

cp /usr/share/modalai/chi-cdk/boson/com.qti.sensormodule.boson_2.bin /usr/lib/camera/
cp /usr/share/modalai/chi-cdk/ov64b40/com.qti.sensormodule.ov64b40_3.bin /usr/lib/camera/

Confirm these exist (also part of camera sernsor drivers):

/usr/lib/camera/com.qti.sensor.ov64b40.so
/usr/lib/camera/com.qti.sensor.boson.so

Configure CCI Mux on M0159 via GPIO

Use gpio 6 on J7 of VOXL2 Mini to configure the i2c mux to connect J7U cci to Hadron’s cam cci
this needs to be done each time after reboot, before running camera server
```
voxl-gpio -m 6 out && voxl-gpio -w 6 1
```

Testing

Detect cameras

voxl2-mini:/$ voxl-camera-server -l
DEBUG:   Attempting to open the hal module
DEBUG:   SUCCESS: Camera module opened on attempt 0
DEBUG:   ----------- Number of cameras: 2

DEBUG:   Cam idx: 0, Cam slot: 2, Slave Address: 0x00D4, Sensor Id: 0x00FF
DEBUG:   Cam idx: 1, Cam slot: 3, Slave Address: 0x006C, Sensor Id: 0x6442
DEBUG:   Note: This list comes from the HAL module and may not be indicative
DEBUG:      of configurations that have full pipelines

DEBUG:   Number of cameras: 2
...

Minimum config

paste into /etc/modalai/voxl-camera-server.conf:

{
  "version":  0.1,
  "fsync_en": false,
  "fsync_gpio":   109,
  "cameras":  [
      {
          "type": "boson",
          "name": "boson",
          "enabled":  true,
          "camera_id":    0,
          "fps":  30,
          "en_preview":   true,
          "en_misp":  false,
          "preview_width":    640,
          "preview_height":   512,
          "en_raw_preview":   true,
          "en_small_video":   false,
          "en_large_video":   false,
          "ae_mode":  "off"
      },
      {
          "type":             "ov64b",
          "name":             "hires",
          "enabled":          true,
          "camera_id":        1,
          "fps":              30,
          "en_preview":       true,
          "en_misp":          false,
          "preview_width":    1920,
          "preview_height":   1080,
          "en_raw_preview":   false,
          "en_small_video":   false,
          "en_large_video":   false,
          "en_snapshot":  false,
          "ae_mode":  "isp",
          "gain_min": 54,
          "gain_max": 32000
      }
      ]
}

Start Voxl Camera Server

run voxl-camera-server in foreground to make sure everything is going right and you can view streams using voxl-portal
without voxl-portal, camera streams can be tested using voxl-inspect-cam tool:

voxl2-mini:/$ voxl-list-pipes 
boson
hires_color
hires_grey

voxl2-mini:/$ voxl-inspect-cam boson hires_color
|   Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
|       boson |  327680 |  640 |  512 |  0.00 |  799 |      456 |     11.2  | 60.0 |  157.3 | RAW8
| hires_color | 3110400 | 1920 | 1080 | 16.64 |  324 |      224 |     40.0  | 29.8 |  741.8 | NV12

note that very high resolution streams viewed as YUVs (which are transferred as MJPG by voxl-portal) will take a lot of CPU to encode to MJPG and also a lot of bandwidth to stream.

Supported Resolutions and Modes

IR sensor supports 640x512 resolution
OV64B currently supports the following modes
these are the RAW modes that camera can be configured to (by the camera pipeline)
- the camera pipeline picks the best mode based on the requested resolution and fps
- note that if you request 3840x2160 30FPS, the camera pipeline actually selects 4624x3472 because it matches the desired FPS (30)
- actual selected mode can be checked using logcat | grep -i selected before starting camera server, the selected resolution will be printed for each camera.
```
# (Mbps per csi lane)
Mode0_9248x6944_10fps_2500.8Mbps
Mode1_4624x3472_30fps_PD_1136x860_1502Mbps
Mode4_3840x2160_60fps_2500.8Mbps
Mode5_1920x1080_240fps_2500.8Mbps
Mode6_1920x1080_30fps_800Mbps
```

Max Resolution

9248x6944 9fps

voxl-camera-server.conf setting:

"preview_width":    9248,
"preview_height":   6944,

Note that at this >8K resolution, the Qualcomm ISP runs in dual VFE mode, which means you cannot use any other non-RAW camera that uses the ISP.
MISP approach does not use Qualcomm ISP and will support more cameras (documentation coming soon)

Images and Diagrams

M0159 Adapter Plugged into Hadron 640R

m0159-plugged-into-hadron

M0181 Adapter

M0181 TOP m0181-top

M0181 BOTTOM m0181-bottom

Image of M0181 Plugged into VOXL2 Mini J7

voxl2-mini-hadron-setup

Connection Diagram

the primary use case is to use M0159 + M0181
M0159 + M0155 + M0162 is also possible

voxl2-hadron-connection-options