FLIR Hadron 640R Support on VOXL2 and VOXL2 Mini

Overview

Overview

VOXL2 and VOXL2 mini support interfacing with FLIR Hadron 640R. Modalai interface board and connectors are required.

Required Hardware

FLIR Hadron 640R (not included)
M0202 (or M0159) + M0181 adapters
MCBL-00084-180 (qty 2) micro coax 180mm cables (shorter options available)

❗ M0202 is a newer version of M0159 with identical functionality and slightly better fit into Hadron. Any references to M0159 below also apply to M0202.

WARNING

Mating M0159 with M0181 needs to be done with care
- ❗ Even though the coax cables are the same for both connections between M0159 and M0181, the order cannot be swapped ❗
- M0159 J2 (5L) <-> M0181 J1 (5L)
- M0159 J3 (HR) <-> M0181 J2 (HR)
- HR connector has to conect to HR and 5L to 5L. HR and 5L are labels next to the connectors on M0159 and M0181
❗Incorrectly connecting M159 to M0181 will likely result in damage of Hadron 640R due to incompatible power rails ❗
Shipped assemblies have been correctly connected and tested

Interfacing Options

Once M0159 adapter is plugged into the back of Hadron 640R, usb port on M0159 can be used to power Hadron and interface / configure the IR sensor via the USB interface.
In order to use MIPI connection for the IR sensor, it must be configured to enable MIPI output, see SW instructions in https://docs.modalai.com/M0153/
- Note that for some versions of Hadron firmware, switching from 60 to 30 FPS is required, as described in the document above. This is because frame timing of 60 FPS may be incompatible with VOXL2 camera ISP, resulting in stopped stream shortly after start.
- Hadron firmware 3.0.26416 has been tested to work properly at 60FPS. If other firmware is used, 30 FPS may need to be selecting using the frame skip feature in Hadron configuration.
It is safe to have the Hadron plugged into VOXL2 as well as USB - make sure all connections are made while the power to VOXL2 is off
The USB connection can be used to interface to the IR sensor via the FLIR GUI as well as viewing IR images using standard Linux UVC driver

Hardware Setup

Connect M0159 Adapter to Hadron

If necessary, remove any interface boards plugged into Hadron 640R in order to allow M0159 to plug in flush with the back of Hadron 640R
Make sure the connector of M0159 aligns with Hadron
See image below for an example of a mated connection

Connect M0181 to VOXL2 or VOXL2 Mini

Make sure pin alignment on connectors is correct. the mounting holes should align and the tab of M0181 will hover above the main SoC
See attached images below
The tab can be removed from M0181 if not needed
VOXL2 Mini supports Hadron connected to J6 or J7 (using default kernel)
- For VOXL2 Mini J6, use sensormodules com.qti.sensormodule.boson_0.bin and com.qti.sensormodule.ov64b40_1.bin
- For VOXL2 Mini J7, use sensormodules com.qti.sensormodule.boson_2.bin and com.qti.sensormodule.ov64b40_3.bin
VOXL2 supports Hadron connected to J7 or J8
- For VOXL2 J7, use sensormodules com.qti.sensormodule.boson_2.bin and com.qti.sensormodule.ov64b40_3.bin
  - ❗ Must use kernel variant 0
- For VOXL2 J8, use sensormodules com.qti.sensormodule.boson_4.bin and com.qti.sensormodule.ov64b40_5.bin
  - ❗ Must use kernel variant 1 (for M0173)
- VOXL2 J6 may be used, but a Kernel change is required (not recommended)

Software Setup

Use VOXL2 or VOXL2 Mini
Install VOXL2 SDK 1.4.0 or later
Optionally update packages to use the latest version
- Change apt source to dev : /etc/apt/sources.list.d/modalai.list
- deb [trusted=yes] http://voxl-packages.modalai.com/ ./dists/qrb5165/dev/binary-arm64/
```
apt update
apt install libmodal-pipe voxl-portal voxl-camera-server
```

Confirm sensormodule drivers exist

in chi-cdk directory where all the camera drivers are kept (but not used)

ls /usr/share/modalai/chi-cdk/ov64b40 : com.qti.sensormodule.ov64b40_0.bin ... 5.bin
ls /usr/share/modalai/chi-cdk/boson:    com.qti.sensormodule.boson_0.bin ... 5.bin

Copy correct sensormodule drivers

Select correct sensormodule index based on which camera connector on VOXL2 the Hadron is actually connected to. The example lists sensormodules used for VOXL2 J7.

cp /usr/share/modalai/chi-cdk/boson/com.qti.sensormodule.boson_2.bin /usr/lib/camera/
cp /usr/share/modalai/chi-cdk/ov64b40/com.qti.sensormodule.ov64b40_3.bin /usr/lib/camera/

Confirm these exist (also part of camera sernsor drivers):

/usr/lib/camera/com.qti.sensor.ov64b40.so
/usr/lib/camera/com.qti.sensor.boson.so

Configure CCI Mux on M0159 via GPIO

For the VOXL2/mini J7 option, use GPIO 6 on J7 of VOXL2 Mini (or VOXL2) to configure the CCI mux on M0159 to connect J7U CCI to Hadron’s cam CCI
- For J6, J8 options see pin configuration right below
this needs to be done each time after reboot, before running camera server
Hadron module has only a single CCI bus, while two VOXL2 (mini) camera ports (one for IR and one for RGB camera) use two CCI buses
After enabling the mux, the CCI bus for J7L and J7U will be physically connected together
Since camera probing happens sequentially at the start of voxl-camera-server, and VOXL2 does not normally communicate with IR sensor after start of streaming, there is no issue with bridging the two CCI buses

# VOXL2 / VOXL2 Mini J6
voxl-gpio -m 30 out && voxl-gpio -w 30 1

# VOXL2 / VOXL2 Mini J7
voxl-gpio -m 6 out && voxl-gpio -w 6 1

❗VOXL2 J8: The pin for enabling the MUX when using J8 is GPIO 14. However, this pin is mapped to UART5_TX (/dev/ttyHS0) and cannot be used as GPIO (see J8 Pinout). Luckily, when UART is idle, the TX is set to a high state, which happens to enable the MUX. Therefore, there is no need to set GPIO 14 to output high (it will not have any effect) - just DO NOT USE UART /dev/ttyHS0!

Testing

Detect cameras

voxl2-mini:/$ voxl-camera-server -l
DEBUG:   Attempting to open the hal module
DEBUG:   SUCCESS: Camera module opened on attempt 0
DEBUG:   ----------- Number of cameras: 2

DEBUG:   Cam idx: 0, Cam slot: 2, Slave Address: 0x00D4, Sensor Id: 0x00FF
DEBUG:   Cam idx: 1, Cam slot: 3, Slave Address: 0x006C, Sensor Id: 0x6442
DEBUG:   Note: This list comes from the HAL module and may not be indicative
DEBUG:      of configurations that have full pipelines

DEBUG:   Number of cameras: 2
...

Minimum config

paste into /etc/modalai/voxl-camera-server.conf:

{
  "version":  0.1,
  "fsync_en": false,
  "fsync_gpio":   109,
  "cameras":  [
      {
          "type":             "boson",
          "name":             "boson",
          "enabled":          true,
          "camera_id":        0,
          "fps":              30,
          "en_preview":       true,
          "en_misp":          false,
          "preview_width":    640,
          "preview_height":   512,
          "en_raw_preview":   true,
          "en_small_video":   false,
          "en_large_video":   false,
          "ae_mode":          "off"
      },
      {
          "type":             "ov64b",
          "name":             "hires",
          "enabled":          true,
          "camera_id":        1,
          "fps":              30,
          "en_preview":       true,
          "en_misp":          false,
          "preview_width":    1920,
          "preview_height":   1080,
          "en_raw_preview":   false,
          "en_small_video":   false,
          "en_large_video":   false,
          "en_snapshot":      false,
          "ae_mode":          "isp",
          "gain_min":         100,
          "gain_max":         32000
      }
      ]
}

Start Voxl Camera Server

run voxl-camera-server in foreground to make sure everything is going right and you can view streams using voxl-portal
without voxl-portal, camera streams can be tested using voxl-inspect-cam tool:

voxl2-mini:/$ voxl-list-pipes 
boson
hires_color
hires_grey

voxl2-mini:/$ voxl-inspect-cam boson hires_color
|   Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
|       boson |  327680 |  640 |  512 |  0.00 |  799 |      456 |     11.2  | 60.0 |  157.3 | RAW8
| hires_color | 3110400 | 1920 | 1080 | 16.64 |  324 |      224 |     40.0  | 29.8 |  741.8 | NV12

note that very high resolution streams viewed as YUVs (which are transferred as MJPG by voxl-portal) will result in high CPU usage to encode to MJPG and also a lot of bandwidth to stream.

Supported Resolutions and Modes

IR sensor supports 640x512 resolution
OV64B currently supports the following modes
these are the RAW modes that camera can be configured to (by the camera pipeline)
- the camera pipeline picks the best mode based on the requested resolution and fps
- note that if you request 3840x2160 30FPS, the camera pipeline actually selects 4624x3472 because it matches the desired FPS (30)
- actual selected mode can be checked using logcat | grep -i selected before starting camera server, the selected resolution will be printed for each camera.
```
# (Mbps per CSI lane)
Mode0_9248x6944_10fps_2500.8Mbps
Mode1_4624x3472_30fps_PD_1136x860_1502Mbps
Mode4_3840x2160_60fps_2500.8Mbps
Mode5_1920x1080_240fps_2500.8Mbps
Mode6_1920x1080_30fps_800Mbps
```

Max Resolution

9248x6944 9fps

voxl-camera-server.conf setting:

"preview_width":    9248,
"preview_height":   6944,

Note that at this >8K resolution, the Qualcomm ISP runs in dual VFE mode, which means you cannot use any other non-RAW camera that uses the ISP.
MISP approach does not use Qualcomm ISP and will support more cameras (documentation coming soon)

Images and Diagrams

M0159 Adapter Plugged into Hadron 640R

m0159-plugged-into-hadron

M0181 Adapter

M0181 TOP m0181-top

M0181 BOTTOM m0181-bottom

Image of M0181 Plugged into VOXL2 Mini J7

voxl2-mini-hadron-setup

Connection Diagram

the primary use case is to use M0159 + M0181
M0159 + M0155 + M0162 is also possible

voxl2-hadron-connection-options

USB Connector on M0159 (J4)

M0159 has a 6-pin USB connector (J4), connector type BM06B-SRSS-TBT
J4 Connector pin-out is listed below, pin 1 is marked with a dot on silkscreen, also marked with an arrow in the image below

m0159-j4

Pin#	Signal	Notes/Usage
1	5V_VDC	5V power input from USB
2	GND	GND
3	SYNC_GPIO	Unused
4	GND	GND
5	USB_D_P	USB 2.0 Data P
6	USB_D_N	USB 2.0 Data N