Crossing the Language Barrier: nosqlite-ffi and the C ABI
Series: nosqlite devlog, Part 5
You know that feeling when your Rust code compiles on the first try with zero warnings?
You sit back in your chair, look at the terminal text, take a sip of your tea or cold coffee, and for five whole minutes you genuinely think you are the smartest programmer alive.
That was me right after finishing part 4. The document storage was actually working. BSON data was getting packed into 4096-byte pages. Linked lists were connecting pages on disk. The catalog was tracking collections. I opened the shell, inserted some documents, closed it, killed the terminal, and opened it again. Everything was still right there.
I was so happy. I felt like I had built something really solid.
Then, after a few minutes, reality hit me.
I had built a whole database engine in Rust that only people who write Rust can actually use.
Think about it for a second. If you are building a web backend in Go, or writing Python scripts for machine learning, or building a mobile app with Flutter, nosqlite is completely useless to you. You cannot just open Python and type import nosqlite. Python has no idea what Rust is.
To the rest of the programming world, my database was sitting on its own lonely island.
If I wanted anyone to actually care about this project, I had to fix this. I had to let other languages talk to my Rust code.
And that meant leaving the comfortable, safe world of Rust and stepping directly into the wild west of C and raw pointers.
Why Languages Cannot Talk to Each Other
Now, a very fair question is: why can’t programming languages just talk to each other directly?
When you compile Rust, it becomes machine code. When you compile Go or C++, it also becomes machine code. Your CPU does not care if you wrote your logic in Rust or Python. At the end of the day, a CPU just moves numbers between registers and executes instructions.
The hardware is not the problem. The rules of each language are the problem.
When a function is called, where do the arguments go? Which CPU register holds the first value? Who cleans up the stack memory after the function returns: the caller or the function itself? How are fields arranged inside a struct in RAM?
This set of rules is called the Application Binary Interface, or ABI.
Every high-level language has its own ABI rules, and they do not match at all. To make things even more fun, Rust does not even have a stable ABI. The Rust team decided that the compiler should be free to rearrange fields inside your struct between different compiler versions just to make memory access faster.
The result? Two binaries compiled with slightly different versions of the Rust compiler cannot even reliably talk to each other unless they agree on a specific format.
So how does anything in software ever connect together?
The answer is C.
C is very old, but because it is so old, everyone in the world agreed to follow C rules. Python knows how to call C functions. Go knows how to call C functions. Dart knows how to call C functions.
So the plan was simple: if I make my Rust code behave like an old C library, any programming language on earth can use it.
The Cloakroom Token Trick
Once I decided to write a C interface, I got hit with my first big headache: what do I actually give the caller?
In safe Rust, my Database struct has a pager, file handles, and a lot of internal logic. It relies on the borrow checker and automatic cleanup.
If I try to hand that Database struct directly to Python or Go, those runtimes will have no idea what to do with it. They do not know how big the struct is, and they do not know how to free its memory.
The solution is an old systems trick called the opaque pointer.
I like to think of it like a cloakroom counter at a railway station or a wedding hall.
When you walk up and hand over your heavy bag, the person behind the counter does not give you a map of the storage room. They do not say: your bag is on shelf number three, second row from the left.
They just hand you a small metal token with a number on it, like token number 42.
You take that token and keep it in your pocket. You do not know where your bag is. You cannot see inside the room. But whenever you want something from your bag, you walk back to the counter, show that token, and the person handles it for you.
In nosqlite, that token is called DBHandle:
pub struct DBHandle {
inner: Database,
last_error: Option<CString>,
}The DBHandle lives on the heap. When a foreign language opens a database, we do not give it the struct. We give it a raw memory address: *mut DBHandle.
To Python or Go, this is just a raw number pointing somewhere in RAM. They cannot look inside inner. They cannot touch any field. All they can do is hold that pointer and give it back to us whenever they want to insert a document or run a query.
Tearing Down the Safety Gates
Let us look at nosqlite_open from crates/nosqlite-ffi/src/lib.rs. This is the function that foreign languages call to start the database:
#[unsafe(no_mangle)]
pub extern "C" fn nosqlite_open(path: *const c_char) -> *mut DBHandle {
let result = panic::catch_unwind(|| {
if path.is_null() {
return ptr::null_mut();
}
let path = unsafe {
match CStr::from_ptr(path).to_str() {
Ok(v) => v,
Err(_) => return ptr::null_mut(),
}
};
match Database::open(path) {
Ok(db) => {
let handle = DBHandle {
inner: db,
last_error: None,
};
Box::into_raw(Box::new(handle))
}
Err(_) => ptr::null_mut(),
}
});
result.unwrap_or(ptr::null_mut())
}There is so much going on in these few lines.
First, look at #[unsafe(no_mangle)].
Normally, the Rust compiler changes your function names behind your back. If you write a function called open, the compiler turns it into some crazy long name like _ZN12nosqlite_ffi13nosqlite_open17h8a2c3f9d4b5e6f7aE. This is called name mangling. Rust does this so that functions with the same name in different modules do not crash into each other.
That is fine for Rust, but if a Go or C program looks for nosqlite_open in the compiled library, it will not find it because of that crazy scrambled name. So no_mangle tells Rust: please leave the name alone so other languages can find it.
Next is extern "C".
This tells Rust: stop using your own calling rules here. Use the standard C calling convention. Put the arguments and return values in the exact CPU registers that C expects.
And then comes the part that gives every Rust developer trust issues: path: *const c_char.
In safe Rust, strings are civilized. A &str has a clear length and is guaranteed to be valid UTF-8.
In C, a string is just a raw pointer pointing to a byte in memory, and you just pray that if you keep reading forward, you will eventually find a zero byte before your program crashes into protected RAM.
What if the caller passes a null pointer? If you try to read memory address zero, your operating system will kill your program immediately with a segmentation fault. No helpful error message. Just an instant crash.
That is why we check path.is_null() first.
And once we know it is not null, we have to write the keyword: unsafe.
For weeks, the Rust compiler behaves like an overprotective mom. It checks your lifetimes. It warns you about every small mistake. It refuses to compile if you borrow two mutable things at once. And then you write unsafe, and suddenly the mom walks out of the room, shrugs, and hands you a running chainsaw.
unsafe does not mean the code is broken. It just means the compiler is saying: I cannot verify this memory pointer, so you are responsible for it.
CStr::from_ptr(path) reads the raw C pointer until it hits the zero byte, and .to_str() checks if the characters are valid UTF-8. Once both pass, we finally have a safe Rust string slice.
The Panic Airlock
Here is another trap that can easily ruin your week: panics.
Get Priyanshu Verma’s stories in your inbox
Join Medium for free to get updates from this writer.
In Rust, when something goes badly wrong, like accessing an array index that does not exist, the code panics. A panic starts unwinding the stack, cleaning up variables as it walks backward through function calls.
Now ask yourself: what happens if a Rust panic tries to unwind through a function written in C or Go?
C has no idea what Rust stack unwinding is. The moment a Rust panic tries to jump over a C stack frame, your program aborts instantly.
Imagine someone using your database inside a big Go backend, and because of one small bug in your Rust code, their entire Go server disappears from the screen without even printing an error log. Good luck debugging that.
That is why the whole function is wrapped in panic::catch_unwind:
let result = panic::catch_unwind(|| {
// all database code runs in here
});
result.unwrap_or(ptr::null_mut())catch_unwind is like an airlock at an international border. If anything blows up inside the Rust code, catch_unwind catches the panic, stops it from escaping to C, and lets us safely return a null pointer.
The foreign program does not crash. It just gets a null pointer, checks for an error, and stays alive.
Error Handling When C Has No Result Type
In Rust, error handling is so clean. We return Result<T, Error> and use the question mark operator.
C has none of that. C does not have Result. C does not even have enums that can hold values.
So how do we tell Go or Python what went wrong?
We use the same old pattern that SQLite has used forever: return an integer status code from functions, and save the actual error message inside the database handle.
fn set_last_error(db: &mut DBHandle, err: impl ToString) {
db.last_error = CString::new(err.to_string()).ok();
}Whenever an operation fails inside the core engine, the FFI layer catches the error, turns it into a null-terminated CString, and stores it inside last_error. The function itself returns an integer like 1 or 2.
Then we give the caller a small function to read that error:
#[unsafe(no_mangle)]
pub extern "C" fn nosqlite_last_error(db: *mut DBHandle) -> *const c_char {
if db.is_null() {
return ptr::null();
}
let handle = unsafe { &*db };
match &handle.last_error {
Some(err) => err.as_ptr(),
None => ptr::null(),
}
}Look at the memory rule here. The error string is allocated by Rust and stays inside DBHandle. The foreign caller can read the characters and print them, but they must never try to free that memory. Rust created it, and Rust will clean it up.
The Memory Free Trap
And that brings us to the nastiest trap in all of FFI: memory allocators.
Every language runtime has its own memory allocator. Rust has one, Go has its own garbage collector, and Python has pymalloc.
What happens if Rust allocates some memory, gives the pointer to Python, and Python tries to free that pointer using C’s free()?
Total chaos.
If Python hands a Rust pointer to a different allocator, the memory bookkeeping gets corrupted. Your program might not crash right away. It might run fine for two hours, and then randomly crash in an unrelated background task.
The golden rule of FFI is very simple: whoever allocates the memory must be the one to free it.
If Rust allocates, Rust must free. If C allocates, C must free.
Look at how nosqlite_open creates the handle:
Box::into_raw(Box::new(handle))Box::into_raw is the only time Rust lets you leak memory with a smile. It tells the compiler: put this DBHandle on the heap, forget about cleaning it up automatically, turn it into a naked pointer, and let it float away.
And here is its matching twin:
#[unsafe(no_mangle)]
pub extern "C" fn nosqlite_close(db: *mut DBHandle) {
if !db.is_null() {
unsafe { drop(Box::from_raw(db)) };
}
}Box::from_raw(db) tells Rust: remember that naked pointer we gave to the foreign language? It is back. Take ownership again.
And then drop runs.
Because Rust owns the box again, its automatic cleanup fires. The DBHandle drops. The underlying Database struct drops. The .lock file closes, releasing the process lock. The file buffers flush their pages to disk and close cleanly.
Every single into_raw must have exactly one matching from_raw. If the foreign developer forgets to call nosqlite_close, memory leaks and the .lock file stays locked forever. If they call nosqlite_close twice on the same pointer, you get a double-free crash.
Welcome back to manual memory management.
The Fun Part: Calling It from Go and Python
After dealing with all these pointer rules and compiler complaints, you compile the crate into a shared library: libnosqlite_ffi.so on Linux, or nosqlite_ffi.dll on Windows. You generate a clean C header file called nosqlite.h.
And then comes the best part: using it from another language.
Here is what the Go code looks like:
package nosqlite
/*
#cgo LDFLAGS: -lnosqlite_ffi
#include <stdlib.h>
#include "nosqlite.h"
*/
import "C"
import (
"errors"
"unsafe"
)
type Database struct {
handle *C.DBHandle
}
func Open(path string) (*Database, error) {
cPath := C.CString(path)
defer C.free(unsafe.Pointer(cPath))
handle := C.nosqlite_open(cPath)
if handle == nil {
return nil, errors.New("failed to open database")
}
return &Database{handle: handle}, nil
}
func (db *Database) Close() {
if db.handle != nil {
C.nosqlite_close(db.handle)
db.handle = nil
}
}Look at how simple that is.
A Go developer who imports this package does not need to know anything about Rust. They do not know about the borrow checker. They do not care about 4096-byte pages or BSON bytes.
They just call nosqlite.Open("test.db") and db.Close().
And in Python, with just four lines of ctypes, you can do the exact same thing:
import ctypes
lib = ctypes.CDLL("./libnosqlite_ffi.so")
lib.nosqlite_open.argtypes = [ctypes.c_char_p]
lib.nosqlite_open.restype = ctypes.c_void_p
handle = lib.nosqlite_open(b"app.db")When you run that Python script, it feels genuinely satisfying.
There is no web server running in the background. There is no heavy Docker container eating 2GB of RAM. There is no 300MB Node runtime just to read a document.
It is your Python script directly running native Rust machine code in microseconds, reading and writing raw disk pages.
The Beauty of Knowing What Is Actually Happening
Writing this FFI layer was easily one of the most frustrating parts of building nosqlite.
When you spend your time writing high-level code, you get spoiled by modern tools. Garbage collectors clean up your mess. Dynamic languages forgive your type mistakes. The compiler holds your hand.
With FFI, all of those safety nets are gone. You have to face the real mechanics of computers: CPU registers, memory addresses, pointer dereferencing, and the fragile boundary between two completely different programming languages.
It gives you a headache. It slows you down.
But there is a very special kind of satisfaction when it finally works. You realize there is no magic in software. There are no wizards behind the screen. There are just memory addresses, CPU registers, and the rules we write to keep them from breaking.
Now nosqlite is no longer just a private Rust project. It is an open, embeddable database engine that can be called from Python, Go, Dart, or any language that can talk to C.
In the next article, we will go back into the core storage engine to look at queries and updates: how we scan page chains to find documents, what happens when an updated document gets bigger than the old one, and why the pain of a linear scan makes B-tree indexes completely necessary.
