libmodal_pipe
libmodal_pipe is a C helper library to make publishing and subscribing to data via POSIX named pipes easy and flexible without adding any extra overhead. This serves as the foundation for Modal Pipe Architecture.
POSIX pipes are a fast, portable, and reliable method of inter-process communication. However, they only practically support transferring data in one direction from a single process to another. libmodal_pipe offers more flexible functionality on top of this along with features such as automatically connecting/disconnecting, and bidirectional communication.
Table of contents
Server & Client Relationship
An MPA server advertises and publishes data. An MPA client opens the pipe created by a server and reads data from it. A single process can be a server and a client to multiple pipes.
When a server wishes to advertise data, it creates a new pipe. In this context, a ‘pipe’ is actually a directory in the file system consisting of multiple POSIX pipes and an information file. These directories can live anywhere in the file system chosen by the server, but typically live in /run/mpa/ since this is a non-permanent directory that is not saved to disk.
Within each directory lives:
- Request pipe. A client will request their own dedicated POSIX pipe by sending a request to the server through the request pipe.
- Control pipe: Clients can send commands back to the server through the control pipe, for example to put voxl-imu-server into calibration mode or to request voxl-qvio-server to reset.
- An info file.
- An individual POSIX pipe for each connected client to read from.
Info File
When a Server creates a new pipe with pipe_server_create(), an info file is created advertizing basic information about the pipe in json format. The first 6 json fields are automatically populated when the server creates a new pipe. Additional fields may be added as needed, for example to publish camera lens calibration data, or list available commands that can be sent to the control pipe.
yocto:/$ cat /run/mpa/imu0/info
{
"name": "imu0",
"location": "/run/mpa/imu0/",
"type": "imu_data_t",
"server_name": "voxl-imu-server",
"size_bytes": 131072,
"server_pid": 4338,
"available_commands": ["start_calibration", "stop_calibration"]
}
The ‘type’ field is typically the name of the C-struct that is sent over the pipe. It is used by clients to ensure that the pipe is publishing the correct data type that it expects before subscribing. It is also used by bash-autocompletion so that tools like voxl-inspect-cam and voxl-inspect-imu can tab-complete available pipes that they can subscribe to.
For example you can type out voxl-inspect-imu {TAB} and bash will suggest and auto-complete any pipes publishing imu data, typically both imu0 and imu1 are being published.
yocto:/$ voxl-inspect-imu {TAB}
imu0 imu1
The server_pid field is used so that the server publishing a particular pipe can be identified and shut down with the voxl-kill-pipe tool.
The available_commands field is optional but should be set if text commands can be sent to the server as this enables bash-autocompletion for the voxl-send-command tool.
Pipe Size
POSIX Pipes act as FIFO buffers and the server initially sets their size. It is important that the server set a pipe size reasonable for the data being transferred to prevent overflowing the buffer. For example, 4k color images require larger pipes than imu data. The largest pipe size varies from one platform to another, but 256MB is a typical max.
For each data type defined in modal_pipe_interfaces.h a recommended pipe size is provided.
If a client expects to have long processing time between pipe reads, then it may elect to increase the size of the pipe for its specific application. Functions are available in both the client and server APIs for reading the current size of a pipe, the amount of data waiting to be read in a pipe, and for setting the size of a pipe. Generally these are not necessary and it is sufficient for the server to set a reasonable safe size for pipes from the beginning.
Client Interface Features
The client-side interface defined in modal_pipe_client.h has the most features, options, and configurability.
Client Helpers
A basic call to pipe_client_open() will request a new dedicated POSIX pipe for a client to read data from. You can then retrieve a file descriptor and read from the pipe however you wish. However, we HIGHLY recommend using one of the client helpers by adding one of these flags to the call to pipe_client_open()
#define CLIENT_FLAG_EN_SIMPLE_HELPER (1<<0)
#define CLIENT_FLAG_EN_CAMERA_HELPER (1<<1)
#define CLIENT_FLAG_EN_POINT_CLOUD_HELPER (1<<2)
All of these helpers will start a background thread to handle automatic opening/closing/reopening of the pipe as the server starts, stops, and restarts. They also handle allocating memory for read buffers and reading data from the pipe. The client simply needs to set callback functions to be set when the helper thread either connects, disconnects, or has read data from the pipe. See the examples for how to use the helpers.
Pause & Resume
After opening a pipe with pipe_client_open() it can be paused and resumed again with pipe_client_pause() and pipe_client_resume() which disconnect and reconnect to the server but the channel remains claimed and configured such that calls to `pipe_client_get_next_available_channel()’ won’t grab a paused channel, and you don’t need to pass in pipe name and flag arguments when reconnecting.
This feature is primarily used when a process wants to preemptively open any pipes that it will need, then quickly resume to start getting data only once needed. This sort of behavior can take advantage of simply opening the pipe in a paused state from the beginning, just just calling pipe_client_resume() when data is needed. voxl-mpa-to-ros is a good example of this.
To open a pipe in a paused state, simply used the CLIENT_FLAG_START_PAUSED flag when calling pipe_client_open().
Claiming Channels
Typical clients will subscribe to a known fixed number of pipes, such as voxl-qvio-server which only subscribes to one IMU and one camera pipe. In this case it’s easy to just pre-define which pipe channel numbers correspond to which data stream like this:
#define IMU_CH 0
#define CAM_CH 1
pipe_client_set_connect_cb(IMU_CH, _imu_connect_cb, NULL);
pipe_client_set_disconnect_cb(IMU_CH, _imu_disconnect_cb, NULL);
pipe_client_set_simple_helper_cb(IMU_CH, _imu_helper_cb, NULL);
pipe_client_set_connect_cb(CAM_CH, _cam_connect_cb, NULL);
pipe_client_set_disconnect_cb(CAM_CH, _cam_disconnect_cb, NULL);
pipe_client_set_camera_helper_cb(CAM_CH, _cam_helper_cb, NULL);
For clients like voxl-mpa-to-ros which must dynamically subscribe to an unknown number of pipes, it can be helpful to call pipe_client_get_next_available_channel() to claim the next available channel. Any channel that has has already been opened is also considered claimed.
Sinks
Whereas the server interface allows multiple clients to read data from one server, a sink is intended for one or multiple processes to send data to a single process. For example, voxl-vision-px4 creates a sink into which one or more processes can pass in relocalization data for it to use. This is a typical use for for POSIX pipes and the sink interface does little more than create a single POSIX pipe along with a read helper thread. This is useful for cases where the full server/client interface is unnecessary.
API Reference and Examples
The libmodal_pipe headers are thoroughly self-documented and human readable. Please read them carefully for details on using the API.
https://gitlab.com/voxl-public/modal-pipe-architecture/libmodal_pipe/-/tree/master/library/include
Examples of how to create Server, Client, and Sink interfaces can be found here:
https://gitlab.com/voxl-public/modal-pipe-architecture/libmodal_pipe/-/tree/master/examples