Fripuck devlog n°4: Static FlatBuffers
Now that I am back from vacation, I managed to get a few things done ! Here is a list of the key areas I managed to make progres in: 1. Project managment, to have a better overview of what was done and what I need to achieve; 2. A smarter packaging system to manage multiple sensors at once; 3. Reducing memory usage by implementing static FlatBuffers using a custom emitter.
1. Project Managment
First, I finally took the time to look into project managment software, and stumbled on Kaneo, a rather minimal kanaban application that also includes a backlog and a Gantt chart. I wanted something that was easy to self-host (there is a Dokploy template for it) and a small enough featureset I wouldn't waste time learning how to use it. Now, if you want to follow the project's development, you can take a look at my kanban board !
Visually writing down what has to be done and has already been done helps me to prioritize certain features over others, and helps me stay motivated by seing the steady progress, so I am glad I finally took the time to configure everything.
2. Smarter packaging
Before, every single sensor provided its own _pack() function. This function was used to take as all the data that the sensor had accumulated since the last _pack() call and add it to the common Flatbuffer.
Then, a central telemetry task would call the _pack() functions sequentially and then finally send out the final, serialized packet.
While this worked for testing purposes, it came with several issues: + It required to write extremely similar code each time I added a new sensor; + There was no check wether the packaged data exceeded the remaining available space; + Starvation was guaranteed since the sensors would alwas package in the same order.
The first issue was kind of fixed by using a large macro that could automatically generate the _pack() function for you, but the two other issues remained.
To fix them, I decided to throw away this implementation to switch to a sensor-agnostic implementation with a priority system.
Generalizing the packaging
In FlatBuffer (specifically flatcc), you would typically use specialized functions for each datatype defined in your .fbs schema.
To be able to handle any sensor, we use the underlying flatcc_ functions.
These typically abstract away the datatype to an id, alignment and size variables.
This enables us to register a sensor to the telemetry system by providing only three values, instead of providing a whole function, simplifying the work of the user.
Now we have a generic_pack() function to use instead of the ~10 individual pack functions.
Since this generic function is provided by the telemetry task and not the sensor task, it has also much more context allowing for more granular control of how the data gets packaged.
Data from the sensors is appended using this (simplified workflow):
flatcc_builder_start_vector(B, size, alignment, max_count); // prepare to build scrap vector in heap
flatcc_builder_append_vector(B, byte_buf + (start_idx * size), count); // copy section of sensor data to the scrap vector
flatcc_builder_ref_t vec_ref = flatcc_builder_end_vector(B); // get offset of built scrap vector
flatcc_builder_ref_t* table_slot_ref = flatcc_builder_table_add_offset(B, id); // get offset of the sensor's final vector
*table_slot_ref = vec_ref; // map the scrap vector to the sensor's final vector
// repeat the above steps for each sensor, each with its own size, alignment and id
FripuckProtocol_Sensors_SensorBatch_end_as_root(B); // finaly commit the full FB
Currently those three values have to be digged up in the generated flatcc files, which is not ideal.
In theory, these values can be calculated or referenced in the generated code, which would be ideal in the long run.
Priority System
To reduce starvation, and make sure time sensitive data gets priority, a priority system was made.
A priority and age is assigned to each sensor when registering them, to determine the order in which the sensors get packaged, and thus which one can package the most data.
That process is rather simple, simply adding age_step to the age of the sensor at each iteration to track how much time has passed since last data was sent, and then adding the age to the priority for the final priority.
The sensor which gets the highest priority gets picked next.
This system prevents starvation but isn't optimal. Ideally, taking into account how much data the sensor has collected or what would optimally fill the FB packet would also be interesting metrics to determine the priority, but this simple system currently is sufficient for my needs.
Puting it all together
With these two new systems as well as a limit tracker, we can now create a simple packaging loop to package and send our data:
void pack_loop(flatcc_builder_t* builder, uint32_t budget) {
for (; budget > 0;) {
// Priority check
struct sensor_info* s = update_age_and_pick_sensor();
if (s == NULL) continue;
// Managing the remaining space and packaging the sensor data.
uint32_t bytes_written = 0;
generic_pack(builder, &s->fb_data, budget, &bytes_written);
budget -= bytes_written;
...
Adding to that a simple timeout that will not block the pack_loop when only few or slow sensors are running, and we have a much better system than before !
However, this doesn't solve our initial problem of high memory use, which requires the next adaptation:
3. Static FlatBuffers
First, we need to understand how FlatBuffers works. Here is what it is according to the docs:
FlatBuffers is an efficient cross platform serialization library [...].
To achieve this serialization, flatcc (the C implementation of FlatBuffers) first creates the individual objects and vectors that go into the final FB object, and in a second step, it compacts down those individual blocks into a single, continuous memory block that is ready to be sent.
A unique twist on the serialization is that instead of growing the sequential block in only one direction, it starts in the center of the available space (offset 0), and grows data to the left (negative offset) and metadata (the position and type of the data) to the right (positive) offset.
┌───────┬────────┐
│ FB buffer │ Emitter ctx
└───────┼────────┘
Data │ Vtable (metadata)
- negative offset ◄───── │ ─────► + positive offset
│
offset 0
One major issue when using the default flatcc configuration, is that it builds this serialized block in the heap, meaning we need to allocate a huge chunk of data on the heap when the serialization starts. Since we only have so much heap space to begin with, this is a no go.
Luckily for us, the flatcc developers go us coverd and provide a custom initialization command which enables us to provode our own emitter (the object responsible for building the serialized block). This way, we can assign a static buffer to it, which prevents fragmentation and doesn't reaquire a bunch of free heap space. It is also possible to define a custom allocator, which is used when creating the temporary vectors and objects before serializing them, but this is overkill for our needs.
The custom emitter is divided into three parts:
– The emitter context (ctx), which holds all the data the emitter function needs to build the serialized buffer;
– The emitter function (custom_builder_emit_fun), which is called each time an object or vector needs to be added to the final buffer, and appends the data and the metadata to the respective ends of the buffer;
– And finally the buffer itself, where the data will be added.
We define the buffer as simply a static array of bytes, with a “center”.
Typically the center (or offset 0) won't be perfectly in the center of the static buffer, since flatcc generally produces much more data (left side) than metadata/Vtable (right side)
#define STATIC_FB_BUFFER_SIZE (2 * 1024) // 2KB static payload buffer
#define FB_BUFFER_OFFSET_ZERO (STATIC_FB_BUFFER_SIZE * 3 / 4) // Index of offset 0
static uint8_t fb_buffer[STATIC_FB_BUFFER_SIZE] = {0};
Then, we need to define the context which will be passed through different function calls. To be able to build the buffer and correctly determine its size, we need to store a reference to the buffer, its capacity, the offset of the center, the min and max offsets.
typedef struct {
uint8_t* buf;
size_t capacity;
flatbuffers_soffset_t zero_offset; // Center (offset +0) of the buffer
flatbuffers_soffset_t min_offset; // Tracks the lowest negative offset emitted
flatbuffers_soffset_t max_offset; // Tracks the highest positive offset emitted
} static_emitter_context_t;
Then begins the hard part, the emitter function. It has to strictly follow the following function definition:
int custom_builder_emit_fun(void* emit_context,
const flatcc_iovec_t* iov,
int iov_count,
flatbuffers_soffset_t offset,
size_t len);
Let's go through the different arguments:
– emit_context is a pointer to the context we defined earlier.
– iov is an array containing data+size pairs of information we need to copy to the buffer.
– iov_count is the number of elements in that array.
– offset is the offset (positive or negative) where we need to write the data to.
– len is the combined length of all the iov entries.
Then, the workflow is pretty simple.
1. We loop through the iov entries,
2. We calculate the final address in the buffer where the data needs to be copied to,
3. Update the offset before writing the next entry.
In code, this translates to the following (omitting bound checks and details):
for (int i = 0; i < iov_count; ++i) {
size_t elem_len = iov[i].iov_len;
if (elem_len == 0) continue;
// Calculate the final address to write to
uint8_t* dest = &ctx->buf[ctx->zero_offset + offset];
// copy
memcpy(dest, iov[i].iov_base, elem_len);
// update the entry
offset += (flatbuffers_soffset_t)elem_len;
}
Then, simply pointing to the lowest negative offset gives you a valid FB, no additional steps !
Wrappingn it up
Now that I’m back from vacation, I’ve managed to make progress on several key areas: 1. Project management, to get a clearer overview of what’s done and what’s left to achieve; 2. A smarter packaging system to handle multiple sensors more efficiently; 3. Reducing memory usage by implementing static FlatBuffers with a custom emitter.
1. Project Management
First, I finally took the time to explore project management tools and came across Kaneo, a minimal Kanban application that also includes a backlog and a Gantt chart. I wanted something easy to self-host (there’s a Dokploy template for it) with a small enough feature set that I wouldn’t waste time learning how to use it.
If you want to follow the project’s progress, you can now check out my Kanban board!
Visually tracking what needs to be done and what’s already completed helps me prioritize features and stay motivated by seeing steady progress. I’m glad I finally set this up.
2. Smarter Packaging
Previously, each sensor had its own _pack() function. This function would take all the data the sensor had accumulated since the last _pack() call and add it to a shared FlatBuffer (FB). A central telemetry task would then call these _pack() functions sequentially and send out the final serialized packet.
While this worked for testing, it had several issues: – It required writing nearly identical code for each new sensor. – There was no check to ensure the packaged data didn’t exceed the available space. – Starvation was inevitable since sensors were always processed in the same order.
The first issue was partially fixed using a macro to auto-generate the _pack() function, but the other two remained. To address them, I decided to scrap this approach and switch to a sensor-agnostic implementation with a priority system.
Generalizing the Packaging
When using FlatBuffers, you typically use specialized functions for each datatype defined in your .fbs schema. To handle any sensor, we use the underlying flatcc_ functions, which abstract the datatype into an ID, alignment, and size. This allows us to register a sensor in the telemetry system by providing just these three values, instead of an entire function, simplifying the user’s work.
Now, we have a single generic_pack() function replacing the ~10 individual pack functions. Since this generic function is provided by the telemetry task rather than the sensor task, it has more context, enabling finer control over how data is packaged.
Sensor data is appended using this simplified workflow:
flatcc_builder_start_vector(B, size, alignment, max_count); // Prepare to build scrap vector in heap
flatcc_builder_append_vector(B, byte_buf + (start_idx * size), count); // Copy section of sensor data to the scrap vector
flatcc_builder_ref_t vec_ref = flatcc_builder_end_vector(B); // Get offset of built scrap vector
flatcc_builder_ref_t* table_slot_ref = flatcc_builder_table_add_offset(B, id); // Get offset of the sensor's final vector
*table_slot_ref = vec_ref; // Map the scrap vector to the sensor's final vector
// Repeat the above steps for each sensor, each with its own size, alignment, and ID
FripuckProtocol_Sensors_SensorBatch_end_as_root(B); // Finally commit the full FlatBuffer
Currently, these three values (ID, alignment, size) must be manually extracted from the generated flatcc files, which isn’t ideal. In theory, they could be calculated or referenced in the generated code, which would be a better long-term solution.
Priority System
To prevent starvation and ensure time-sensitive data gets priority, I implemented a priority system. Each sensor is assigned a priority and an age when registered. These determine the order in which sensors are processed and how much data they can package. The process is straightforward: at each iteration, age_step is added to the sensor’s age to track the time since its last data was sent, and then the age is added to the priority to compute the final priority. The sensor with the highest priority is selected next.
This system prevents starvation but isn’t perfect. Ideally, we’d also consider how much data a sensor has collected or how to optimally fill the FlatBuffer packet. However, this simple system currently meets my needs.
Putting It All Together
With these two new systems and a limit tracker, we can now create a simple packaging loop to package and send data:
void pack_loop(flatcc_builder_t* builder, uint32_t budget) {
for (; budget > 0;) {
// Priority check
struct sensor_info* s = update_age_and_pick_sensor();
if (s == NULL) continue;
// Manage remaining space and package sensor data
uint32_t bytes_written = 0;
generic_pack(builder, &s->fb_data, budget, &bytes_written);
budget -= bytes_written;
...
}
}
Adding a simple timeout ensures the pack_loop doesn’t block when only a few or slow sensors are running. This results in a much better system than before. However, it still doesn’t address the initial problem of high memory usage, which brings us to the next adaptation:
3. Static FlatBuffers
First, let’s recap how FlatBuffers works. According to the docs:
FlatBuffers is an efficient cross-platform serialization library for C++, Java, Python, Go, and more.
Unlike traditional serialization formats (e.g., JSON or Protobuf), FlatBuffers does not serialize or deserialize data in the conventional sense. Instead, it directly constructs a binary buffer in memory, where data is organized for zero-copy access. This means you can read fields directly from the buffer without parsing or deserialization steps.
To build this buffer, flatcc (the C implementation of FlatBuffers) follows a two-step process:
- It first creates intermediate vectors and tables (temporary structures) for each object defined in your
.fbsschema. - It then compacts these into a single, contiguous memory block ready for transmission. A unique aspect of FlatBuffers is its bidirectional growth mechanism. Instead of growing the buffer in one direction, it starts at the center (offset 0) and grows:
- Data (e.g., vectors, scalars) toward negative offsets (left).
- Metadata (e.g., VTables, which store field layouts for tables) toward positive offsets (right). This minimizes alignment gaps and ensures the final buffer is tightly packed.
┌───────┬────────┐
│ FB buffer │ Emitter ctx
└───────┼────────┘
Data │ Vtable (metadata)
- negative offset ◄───── │ ─────► + positive offset
│
offset 0
But why does it matter ?
By default, flatcc allocates this final buffer on the heap, which is problematic for embedded systems with limited memory. Fortunately, flatcc allows us to provide a custom emitter—a callback function that redirects buffer construction to a static memory region of our choosing. This avoids dynamic allocation entirely. It’s also possible to define a custom allocator for temporary vectors and objects before serialization, but that’s overkill for our needs.
The custom emitter consists of three parts:
– The emitter context (ctx), which holds all the data the emitter function needs to build the serialized buffer.
– The emitter function (custom_builder_emit_fun), called each time an object or vector needs to be added to the final buffer. It appends data and metadata to their respective ends of the buffer.
– The buffer itself, where the data is stored.
We define the buffer as a static byte array with a “center.” Typically, the center (offset 0) isn’t perfectly centered in the static buffer, as flatcc usually generates more data (left side) than metadata/vtable (right side):
#define STATIC_FB_BUFFER_SIZE (2 * 1024) // 2KB static payload buffer
#define FB_BUFFER_OFFSET_ZERO (STATIC_FB_BUFFER_SIZE * 3 / 4) // Index of offset 0
static uint8_t fb_buffer[STATIC_FB_BUFFER_SIZE] = {0};
Next, we define the context passed through function calls. To build the buffer and correctly determine its size, we need to store a reference to the buffer, its capacity, the center offset, and the min/max offsets:
typedef struct {
uint8_t* buf;
size_t capacity;
flatbuffers_soffset_t zero_offset; // Center (offset +0) of the buffer
flatbuffers_soffset_t min_offset; // Tracks the lowest negative offset emitted
flatbuffers_soffset_t max_offset; // Tracks the highest positive offset emitted
} static_emitter_context_t;
Now, the tricky part: the emitter function. It must strictly follow this definition:
int custom_builder_emit_fun(void* emit_context,
const flatcc_iovec_t* iov,
int iov_count,
flatbuffers_soffset_t offset,
size_t len);
Let’s break down the arguments:
– emit_context: A pointer to the context we defined earlier.
– iov: An array containing data+size pairs of the information to copy to the buffer.
– iov_count: The number of elements in that array.
– offset: The offset (positive or negative) where the data should be written.
– len: The combined length of all iov entries.
The workflow is straightforward:
1. Loop through the iov entries.
2. Calculate the final address in the buffer where the data should be copied.
3. Update the offset before writing the next entry.
In code, this looks like (omitting bounds checks and minor details):
for (int i = 0; i < iov_count; ++i) {
size_t elem_len = iov[i].iov_len;
if (elem_len == 0) continue;
// Calculate the final address to write to
uint8_t* dest = &ctx->buf[ctx->zero_offset + offset];
// Copy
memcpy(dest, iov[i].iov_base, elem_len);
// Update the offset
offset += (flatbuffers_soffset_t)elem_len;
}
Once all data is written, the lowest negative offset points to the start of a valid FlatBuffer—no further processing needed!
Wrapping Up
While I haven’t been able to work on the project as much as I’d have liked, I’m glad to see it moving forward again. I’m also happy to see my memory issues disappear, hopefully for good 😅.
Over the next month, I’ll try to dedicate as much time as possible to the project. Right now, I’m focusing my efforts on getting the camera to work and laying the initial foundations for the laptop → e-puck commands. This involves coordinating the radio and controller chip, as well as the Lua VM I’ve started implementing.
C you next month!
Project Github page: https://github.com/Uhrbaan/fripuck2
