FFI in Miri at 8,000 segfaults/sec

I used to work on embedded Linux. That’s how I ended up here: one side project led to another until I needed more from Miri, and somewhere along the way I forgot the original project. None of this has much to do with my work a few years ago.

More than anything, this is a personal drama. Life can’t be all code; there needs to be some action. Someone needs to cry, and what better topic to cry about than program verification?

Miri is an interpreter that executes Rust while tracking enough of the machine state to detect many kinds of undefined behavior. FFI is where execution leaves that model and enters native code Miri cannot see.

The problem with FFI🔗

Miri is powerful, but inherently limited—and slow. It’s meant for testing code; you’re not going to play video games on it. As you’ll see, I made that slowness much worse. “8,000 segfaults per second” will eventually need to become hundreds of thousands before this is broadly useful.

The core of the issue is that code spawned in an FFI call can do almost anything, and while it runs, Miri has no idea what it’s doing. That is a problem for an interpreter whose model depends on tracking pointer provenance and whether individual bytes are initialized. How do you preserve that model while executing random nonsense CPU instructions that were once written in C, C++, Java, or anything else?

My answer is that you will it into existence.

We can’t recover everything a precompiled binary does. The concrete goal I set out to achieve is to link an arbitrary .so, call a function, and have it somehow Just Work™ in terms of translating what it actually did to what Miri can understand. Even an incomplete account can constrain the worst case, which is much better than assuming the foreign code touched anything it could possibly reach.

That means tracing every memory access: its address and size, and whether it was a read, a write, or both.

Faced with this daunting problem, I sought aid from the holy scripture of Google search results and Stack Overflow. I found a random post from 2013 about trapping every memory read and write using SIGSEGV. It can’t be that hard, right?

I wasn’t the first person to have the idea, though perhaps few had tried it in quite this form.

Tracing native accesses🔗

Turning every access into a fault🔗

Miri forks into two processes. The parent becomes a supervisor; the child remains Miri proper and can jump into whatever foreign code it has been given. Right before it does, we use mprotect to mark the Miri-managed pages exposed to that native call as PROT_NONE. Any access to those pages now raises a segfault.

A segfault is only a Unix signal. It is not magic, and execution can continue after one. Signals mean whatever you believe they do, after all. Under ptrace, the supervisor sees the child receive that signal and itself receives a siginfo, including the faulting address.

Great. Amazing. Lovely. Perfect. This all works.

An address is not enough, however. We still need the size of the access, whether it read or wrote, and some way to let the instruction finish. The page is still protected with PROT_NONE; restarted as-is, it would simply fault again.

Decoding and replaying the instruction🔗

Code is data, so the supervisor reads the bytes at the child’s instruction pointer and hands them to a disassembler. The decoder gives us the possible access width and whether the instruction may read, write, or do both.

This is necessarily conservative. In an RW operand, a write may be conditional on a read, for example. Even so, it gives Miri a much narrower bound on what the instruction could have touched. If the instruction could write, the affected bytes might now be initialized; bytes outside that range definitely were not. When native code reads bytes carrying provenance, Miri can conservatively mark that provenance as exposed.

The event type is correspondingly small: an address range, an access kind, and—because the decoder sometimes cannot know—a certainty bit for writes.

enum AccessEvent {
    Read(AccessRange),
    Write(AccessRange, bool), // `true` when the write is certain
}

struct AccessRange {
    addr: usize,
    size: usize,
}

There’s also the matter that getting the information we need from the decoder is highly architecture-specific. The current implementation uses Capstone and supports x86 and x86-64, since these architectures have well-behaved fixed-size accesses on all instructions; other architectures can have register-dependent access widths, such as with ARM scalable vectors.

Once we understand the fault, the supervisor records its conservative access event. ptrace then lets it rewrite registers and memory in the child.

It diverts the instruction pointer into a fake extern "C" function, moves the stack pointer to zeroed scratch memory, and writes the page address into static storage. That function makes the page readable and writable, then raises SIGSTOP instead of returning.

The supervisor restores the interrupted registers and single-steps the original instruction before sending the child through a matching trampoline that restores PROT_NONE. Execution continues, and the event batch returns to Miri when the FFI call ends.

In extremely abridged source form, the control loop looks like this:

new_regs.set_ip(mempr_off as *const () as usize);
ptrace::setregs(pid, new_regs).unwrap();
// … the child unprotects the page, then stops

ptrace::setregs(pid, regs_bak).unwrap();
ptrace::step(pid, None).unwrap();

// … run mempr_on, restore the registers, and continue

This isn’t a calling convention. I don’t know what it is. It is a crime against computers.

Getting allocations back from native code🔗

Tracing accesses only covers pointers that travel from Miri into native code. Usually, pointers tend to come back as well. To Miri, an address returned by an allocator is otherwise just an integer pretending to be a pointer; Miri cannot know if it may safely dereference it on the host hardware.

So let’s trace allocations.

Intercepting libc🔗

Imagine a function that merely wraps malloc. Miri can shim malloc on the Rust side, but in a precompiled library that wrapper just looks like a random function returning an address. We need to recognize the underlying allocation.

We place traps at the relevant libc entry points. And by “place traps”, I do mean overwrite their first instruction with a breakpoint:

ptrace::write(pid, libc::malloc as *mut _, BREAKPT_INSTR.into()).unwrap();
ptrace::write(pid, libc::calloc as *mut _, BREAKPT_INSTR.into()).unwrap();
// …
ptrace::write(pid, libc::free as *mut _, BREAKPT_INSTR.into()).unwrap();

When native execution reaches one, we redirect the instruction pointer into one of our fake functions. The original and replacement entry points use the same extern "C" ABI, so the redirect should be ABI-compatible. I did consider shimming all of libc, but was assured that would be excessive. Whatever you say, Ralf.

Does it work?

No. Of course it doesn’t.

Those tiny trampoline shims do not have a normal Miri context, stack, or runtime environment. More importantly, Miri cannot wait until the FFI call returns to learn about an allocation: the native code may use that memory immediately.

Keeping allocation events in order🔗

We smuggle a pointer to Miri’s interpreter context through a global, then recover it inside the interceptor to process the allocation or deallocation before native execution continues. Sure, that interpreter context type has a lifetime that we just make up in the pointer materialization, but what’s a little unbounded lifetime between friends?

Ordinary access events can be batched until the FFI call returns. Allocation and deallocation break that scheme: one can reuse an address, while the other makes an old address invalid. A pending event may therefore refer to the wrong allocation by the time Miri processes it. Thus we need to flush those events before either boundary, using another IPC channel hidden in a global Mutex.

Surely that was it?

Of course not; libc can call itself.

When libc calls itself🔗

Libc’s internal calls may be inlined or bound directly to private aliases, bypassing the public entry points we patched. Others still reach those breakpoints. If an internal free is redirected while the matching allocation was not, we can end up allocating with one allocator and freeing with the other.

Linux provides the lovely /proc/self/maps, which identifies the object behind each mapping. We use libc’s executable mappings and the saved call-site address to determine whether a call originated inside libc.

Origin is only half the problem; every allocation also needs an owner. A block obtained from real malloc and passed back to Miri must eventually return to libc rather than Miri’s allocator.

This is where I went insane.

Either way, after some headscratching I managed to adapt the above routing code to handle that case as well; and after all of that nonsense, it finally passed CI. The allocation-tracing half still needs cleanup, and its current shape is regrettably split. alloc-tracing-libc-part-1 installs the interception layer and has its review history in PR #4792. The full allocator hand-off lives on alloc-tracing-libc-part-2, which has no PR. The older all-in-one draft PR #4791 remains useful archaeology.

Current status and future work🔗

The two halves are at different stages. Native access tracing has been merged, and people are already using it; allocation tracing remains experimental branch work. The Miri documentation describes the experimental native-library flags. A minimal invocation looks like this:

MIRIFLAGS="-Zmiri-native-lib=/path/to/library.so -Zmiri-native-lib-enable-tracing" \
  cargo +nightly miri test

Yes, the interface could be nicer. It will be nicer. There is a very big to-do list; please add to it by telling me what needs to improve.

Cost and coverage🔗

The prototype measurement behind the title was roughly 8,000 handled faults per second. Treat that as an order-of-magnitude result, not a reproducible benchmark; this article does not preserve the original machine and workload details.

Because every access to a protected Miri-managed page must fault, it is not fast. This is proof that the approach can work for a toy program, not its final form. Possible optimizations include moving some supervision into a small kernel component and replacing the decoder bottleneck.

What “unsound” means here🔗

Also, Miri is now unsound. You’re all very welcome.

Review archaeologyplease don’t

That joke has a less precise scope than one might hope. Miri’s own documentation calls native-library mode unsound because Miri can lose initialization and provenance information for memory shared with native code. The feature is experimental and opt-in. While great care was taken to not make Miri itself literally unsound, one never knows with code like this; it still enables calling arbitrary native binaries from inside Miri itself.

Callbacks and threads🔗

Callbacks are one of the next targets. libffi, which already handles part of the FFI setup, can create ABI-compatible callback entry points. That solves only the first step. Miri must still regain control, schedule the Rust callback, synchronize memory state, and return to native code.

Native threads pose the same kind of coordination problem and might eventually be parked and resumed through Miri’s scheduler.

There is a future in which you pass an arbitrary function pointer into FFI, have it called simultaneously from eighteen threads, and Miri still detects at least a bunch of UB.

That is the point of this story: here is a ridiculous thing, and here is how I willed it into existence despite reality insisting it was a load of crap.

Credits🔗

Ralf and Oli survived many painful review iterations while I learned what I was doing. Strophox created Miri’s previous FFI implementation, and ixi built the yaxpeax features this work needs.

Thanks as well to the Capstone developers and all my dependencies. I began with Iced, which was x86-only and also very nice.