PRs and LLMs

Sat, 27 Jun 2026 10:00:00 +0000

We are hiring at Feldera, so I have been talking to many software engineers. These calls tend to reveal where the “meta of software engineering” is at. Often the signal is that there isn’t a good signal and people are looking for new opportunities for various reasons.

This time it’s different: Senior engineers who were interviewing kept raising the same complaint.

These engineers now review an unholy amount of PRs written by coding agents. They struggle with it for three reasons.

Reviewing now takes up most of the time they used to spend writing code.
They do not perceive AI-generated code as ‘good enough’
They worry about merging large changes into their product in such a short time

I don’t claim to have a general solution for this, but here is what we are starting to do at Feldera:

Reviewing consumes the time I used to spend writing code

One often quoted response is to stop requiring human reviews. That may work for products where correctness issues have a low cost. But I don’t think it’s a good solution for us.

Hence, our approach at feldera is to shift more of the “burden of proof” that a PR is ready to the author.

Our old rule was that each PR must either add or update existing unit tests, and major features had to include integration tests.

I believe this isn’t enough anymore.

Here are some ideas we are experimenting with at the moment:

Set a 99% coverage target for new repositories. This forces agent-written code to include tests, otherwise it won’t get accepted. LLMs can usually write the missing tests once coverage tooling such as llvm-cov identifies the gaps.
Add a rule to CLAUDE/AGENTS.md that new tests are validated to fail by reverting the change to make sure the tests actually capture a real issue.
Each substantial PR must run autonomous end-to-end testing. An agent runs the product end-to-end (started by the PR author). The agent’s job is to validate the new feature/bugfix in different configurations, environments, and failure cases. The test summary should be included in the PR description.
Invest in tests that constrain agent-written code: APIs used by clients must include an executable specification or oracle. Property-based testing runs the same generated inputs through the spec and the implementation. Reviewers should read the spec carefully and skim the implementation. The PR author writes/generates the tests.
Design executable specifications and implementations so they are interchangeable. It should be easy to replace an implementation with its specification, or a specification with the implementation. This makes it easier to isolate bugs in individual components.

Most production software is not structured this way because maintaining executable specifications for every API and component is very labor-intensive. But in a world where writing code is free it turns out to be a great idea.

The engineer does not trust AI-generated code

There are different opinions about how good LLM code is. One engineer with little experience might be amazed by it, an engineer with 15 years of experience might be appalled.

Some of that disagreement is taste. Some of it is measurable. To make reviews less subjective, we try to evaluate generated code along five points:

It works
It uses ‘the right’ data structures and has ‘the right’ algorithmic complexity
It exposes and uses ‘the right’ API surface
The implementation is written using clean code principles (DRY, etc.)
If it matters: It is fast enough

(1) mostly requires someone to state the expected behavior clearly and provide a representative test corpus that gives an LLM enough context. For many applications that gets you quite far. (2) and (3) usually need a careful human review, (4) can often be improved by encoding the right coding conventions in your CLAUDE.md or AGENTS.md.

(5) is where I expect engineering teams like ours to spend most of their time.

Rust's testing ecosystem

We write mostly Rust. And the ecosystem is great to establish (1). If we need a PR to carry this much proof the tooling makes it cheap. We use the following libraries at Feldera excessively:

rstest for fixtures and parameterized cases, so one test body covers a whole table of inputs.
proptest to generate inputs and shrink any failure down to a minimal counterexample.
proptest-state-machine to drive random sequences of operations against a reference model and assert the implementation and the model never disagree.
loom to exhaustively check lock-free code against every interleaving the memory model permits.
shuttle to hunt the same class of concurrency bugs in larger programs, where loom's exhaustive search gets too expensive and randomized exploration wins.

Libraries are a mechanism. More important is that our engineering org thinks about code in terms of specifications, invariants, pre- and post-conditions. None of this was a bad idea before LLMs, just that it's much more important now to handle the increased velocity.

Merging changes faster than the codebase can absorb

A senior engineer knows which areas of the codebase are safe, risky, or outright dangerous to change. This is knowledge a new engineer (or LLM) does not have. It’s learned from too many 3 a.m. pages.

That institutional memory is valuable. At Feldera, this is where senior engineers are expected to push back and slow things down.

That being said, having too much code that falls in the ‘risky’ or ‘outright dangerous’ territory has now become an even bigger liability. Agents are not slowed down by reading or changing very complicated code. But given a module riddled with footguns an agent often performs even worse than a human.

That code usually falls into one of a few categories. It may have grown into a god module that should be split up. It may have accumulated complexity that needs review. Or it may be underspecified, undertested, and below the standard we now need. Luckily, AI also changes the cost of leaving code in that state.

Makes sense?

Does it seem appealing to you to work in such an engineering organization? We are hiring Rust backend and product engineers.

Or do you think it won’t solve any of the “PR-slop” problems? I’m happy to hear about that too.

Context-switching gone wrong

Sat, 13 Mar 2021 12:11:11 +0000

I remember while this bug wasn’t that hard to fix (or find), it still had some “fantastic” potential to it. Because, when it appeared, at first nothing made sense anymore…

When programming, it happens occasionally that you end up with a dead-end: A branch in the code that you determine will never be taken by the CPU with absolute certainty. However, due to limitations in the language or the structure of your algorithm you can’t really avoid to account for it in the program.

An example can be a match statement like this one:

let mut ops = Vec::with_capacity(nop);
for _i in 0..nop {
    let op = orng.gen();
    match op % 2 {
        0 => ops.push(Operation::WriteOperation(OpWr::Pop)),
        1 => ops.push(Operation::WriteOperation(OpWr::Push(orng.gen()))),
        _ => unreachable!("Can only have 0 or 1"),
    }
}

Since we end up doing modulo two on our op variable, the only values we ever expect to see is 0 or 1 in that match statement. However, the match statement in rust needs to be exhaustive and in some cases the rust compiler isn’t able to determine that only two scenarios are valid and so it still wants us to provide a wild-card pattern (using _), which provides an execution path for every number that is not 0 or 1. Because we determine this code-path can never be executed we end up putting an unreachable! statement there, which would just abort the program. That’s perfectly fine because it’s obvious to see that we never go there.

Note aside, the current rust compiler does in-fact eliminate the unreachable branch in release mode (but not in debug mode). Another way to write it is to replace the 1 with a _ in the match statement but this can lead to “hard-to-find bugs” if we change the expression we’re matching on later.

Another (slightly weaker) form of such paths are “trivial” pre-conditions in the form of asserts. For example, here is such a pre-condition in C:

void mutex_obj_hold(kmutex_t *lock)
{
    struct kmutexobj *mo = (struct kmutexobj *)lock;

    KASSERTMSG(mo->mo_magic == MUTEX_OBJ_MAGIC,
        "%s: lock %p: mo->mo_magic (%#x) != MUTEX_OBJ_MAGIC (%#x)",
         __func__, mo, mo->mo_magic, MUTEX_OBJ_MAGIC);

    atomic_inc_uint(&mo->mo_refcnt);
}

Essentially, the KASSERTMSG is saying that in order to call mutex_obj_hold the argument provided must be of type kmutexobj, which is validated in C by checking that the member mo_magic is set to the constant MUTEX_OBJ_MAGIC. This is a way to add some type robustness for a “loosely typed” language such as C. While this pre-condition can certainly be wrong, in practice this is rarely expected to happen. The only time this supposedly can fail is if someone just wrote some locking code and invoked this function with a wrong argument (aka a different type) or forgot to initialize the lock¹.

Branching into the unreachable

As you can probably guess, the manifestation of the bug in this post was that the execution of a program would suddenly end up in such code-paths by taking these “impossible” branches. Which meant for the first example, that it could suddenly fail with

thread 'main' panicked at 'internal error: entered unreachable code: Can only be 0 or 1', src/main.rs:6:5

And, because that wasn’t weird enough, the bug also wasn’t deterministic: Most of the time the program would work “just fine”, but every so often the code would abort due to some random assert, panic, or unreachable! statement. Upon inspection they always turned out to be unlikely or impossible things which now suddenly happened at an alarming rate.

To understand this a bit better, we have to refresh some knowledge about CPU assembly language (we use x86) to see what’s really going on under the hood. It’s important to understand that our match and KASSERT example will eventually be rewritten using just a bunch of if statements by the compiler:

if op % 2 == 0 {
    ops.push(Operation::WriteOperation(OpWr::Pop));
}
else if op % 2 == 1 {
    ops.push(Operation::WriteOperation(OpWr::Push(orng.gen())));
}
else {
    unreachable!("Can only have 0 or 1");
}

Afterwards, the code is further simplified into comparisons (cmp) and branch instructions (jmp, jne etc.) when converted to x86 assembly. To explain, we show the assembly code for the first if statement, if op % 2 == 0:

and     rax, 1
cmp     rax, 0
jne     .LBB126_3

The first instruction does a bitwise and with op (in rax) and the constant 1 (calculates op % 2 by taking the first bit of op and stores this in rax), then it compares the rax to 0 using cmp, and finally if rax contained something not 0, it will jump to another location (e.g., to the label .LBB126_3). However, if it was 0 it will continue with the next instruction after jne.

One thing that’s important to understand here is that there is an implicit dependency between the result of cmp and jne that is not directly visible: Since jne needs to jump (or not) based on the result of the previous cmp instruction, it needs to access its result somehow. On x86, the CPU has a special register called RFLAGS for this purpose. What happens is that the cmp instruction will subtract the second operand (0) from the first operand (rax) and then set flags in RFLAGS based on the result of the subtraction. For example, the Zero Flag (which is the one important for jne), is set to 1 if the subtraction performed by cmp results in zero. This means jne just needs to inspect the Zero Flag in RFLAGS: if it is set it won’t jump because the compared values were equal, but if it remained unset it will jump to the specified location.

Locating the issue

So what went wrong with our code? Why would the CPU suddenly take those odd branches that led nowhere? Assuming our branch logic in the CPU wasn’t broken, surely the bug must be in the software somewhere?

As always there were a couple of options where this bug could be located: For example, since we wrote our own ELF parsing and loading library, if something was messed up in there about how instructions get loaded or relocated it could definitely become a problem. This seemed unlikely though since the bug only affected branches and also wasn’t deterministic.

What really helped was when I eventually figured out that this bug only happened on CPU cores were the OS received interrupts during execution. That limited the scope of the bug to anything that would happen in the interrupt handler. To isolate further, I downsized the interrupt handler by removing all program logic in it (aside from restoring the context). When afterwards the bug was still present, I knew it must be in the context restore logic.

To explain this a bit more: The bread and butter of an operating system is to give an illusion to a program that it is running without any interruption on a CPU. In reality tough, an OS will context-switch many times per second to execute many different programs concurrently. One cause for a context switch can be interrupts: For example from a device, to notify when a packet is ready, or a timer to tell us that it might be time to run some other program now. To hold up the illusion, OS and hardware ideally must capture the exact state of the CPU when the interruption happens and restore back that exact state to resume execution of a program later.

There are two predominant ways to resume execution in user-space from a kernel on modern x86: sysret to return after a program made a syscall, and iretq to return after an interrupt. Both need to restore the CPU context which can include some or all general-purpose registers, the SIMD and floating point state, as well as the aforementioned RFLAGS register. Context-switching usually can be executed only through a series of assembly statements.

Here is a code-snippet that stores the CPU state when an interrupt happens (it misses a register, can you spot it?). A memory region, accessed through %rax, is used to move the CPU register contents to a process-specific save area.

// Interrupt execution starts here...

// Save original %rax temporarily on the stack 
// because we will overwrite it to hold a reference to save area
pushq %rax

// First, get the pointer to the register save area
rdgsbase %rax
movq 0x8(%rax), %rax

// Save CPU general-purpose register context
// We don't save %rax yet since we used it to
// reference the save area location
movq %rbx,  1*8(%rax)
movq %rcx,  2*8(%rax)
movq %rdx,  3*8(%rax)
movq %rsi,  4*8(%rax)
movq %rdi,  5*8(%rax)
movq %rbp,  6*8(%rax)
// We don't save %rsp yet since it is overridden by CPU on irq entry
movq %r8,   8*8(%rax)
movq %r9,   9*8(%rax)
movq %r10, 10*8(%rax)
movq %r11, 11*8(%rax)
movq %r12, 12*8(%rax)
movq %r13, 13*8(%rax)
movq %r14, 14*8(%rax)
movq %r15, 15*8(%rax)

// Save original rax, which we pushed on the stack previously
popq %r15
movq %r15, 0*8(%rax)

// Save %rsp of interrupted process
movq 5*8(%rsp), %r15
movq %r15, 7*8(%rax)

// Save RIP were we were at before we got interrupted
// This is at rsp+16 (put there by the hardware):
movq 2*8(%rsp), %r15
movq %r15, 16*8(%rax)

// Saves the fs register
rdfsbase %r15
movq %r15, 19*8(%rax)

// Save vector registers
fxsave 24*8(%rax)

// Move on to high-level language function
callq handle_generic_exception

And here is the “opposite” code-snippet that restores the CPU context again (from the save area, now in %rdi) and returns to the interrupted program using iretq (this example doesn’t miss anything and is correct):

// Restore fs and gs registers
swapgs
movq 19*8(%rdi), %rsi
wrfsbase %rsi

// Restore SIMD/vector registers
fxrstor 24*8(%rdi)

// Restore general-purpose CPU registers
movq  0*8(%rdi), %rax
movq  1*8(%rdi), %rbx
movq  2*8(%rdi), %rcx
movq  3*8(%rdi), %rdx
movq  4*8(%rdi), %rsi
// %rdi: Restored last (see below) to preserve `save_area`
movq  6*8(%rdi), %rbp
// %rsp: Restored through `iretq` (see below)
movq  8*8(%rdi), %r8
movq  9*8(%rdi), %r9
movq 10*8(%rdi), %r10
movq 11*8(%rdi), %r11
movq 12*8(%rdi), %r12
movq 13*8(%rdi), %r13
movq 14*8(%rdi), %r14
movq 15*8(%rdi), %r15

// Stack segment register
pushq 18*8(%rdi)
// %rsp stack register
pushq 7*8(%rdi)
// RFLAGS register
pushq 17*8(%rdi)
// Code segment register
pushq 19*8(%rdi)

// Instruction pointer
pushq 16*8(%rdi)

// Restore `rdi` register last, since it was used to reach `save_area`
movq 5*8(%rdi), %rdi

// And back we go:
iretq

The bug in this case was that we simply forgot to ever store RFLAGS to the save area in the first snippet. This meant that every time we returned from an interrupt it would use an outdated RFLAGS (the one which remained from the last time the process did a system call).

Going back to our branch example from earlier: Imagine an interrupt happens after the CPU already executed cmp but just before it wanted to start executing jne. Instead of resuming execution at jne with the up-to-date RFLAGS, now we resume with an outdated RFLAGS value that may or may not correspond to what cmp computed one instruction earlier. Without restoring RFLAGS properly, our branching became a seemingly random affair – depending on whether interrupts would happen or not and when :).

Another way would be through memory corruption, but that’s not what was happening with this bug. ↩

When alignment matters

Thu, 11 Mar 2021 02:22:30 +0000

Some bugs are hard to find because when they manifest, they do so in a very misleading way. This one is about fairly unexpected consequences when things aren’t properly aligned. Most of the time alignment problems lead to “just” bad performance. In some other cases, hardware architectures may generate a fault on misaligned accesses (but this is easy to pin-point and fix). However, in our case the problem was a little more subtle.

We were trying to run some more applications on our new research OS. If you haven’t read the previous post: We had a new kernel that we built from scratch and our user-space processes ran with a small libOS that we wrote from scratch too. Basically, we had a lot of code which was fairly new, and bugs had to be expected anywhere.

For example, our user-space runtime – the victim of the bug in this post – contained a custom built memory allocator, user-space scheduler and a system call abstraction for the kernel. In addition, it provided the lowest layer (the hypercalls) that allowed a process to also link with rumpkernel. rumpkernel is a library OS built on NetBSD that gave a lot of POSIX support to a process. Most importantly for this post, it came with an implementation of libpthread. The way it worked was that libpthread provided an implementation of all the pthread APIs, but underneath it would call into our own custom scheduler (through the hypercalls), to create new threads and control which threads would be put in and out of the run queues etc.

Locking gone wrong

Last time we were in the process of porting Redis to our OS. This time I was trying to port a second application, namely memcached. It was a good test because in comparison to Redis, memcached was multi-threaded. So it exercised a lot more code-paths than Redis.

When trying to launch memcached, it became apparent that the process very often ended up stuck somewhere during initialization and didn’t make progress anymore. Unfortunately, it wasn’t really deterministic: Every time it would get stuck in a slightly different function (and loop). And sometimes it even worked: memcached started and waited for incoming connections – great!

In such a scenario, a systems programmer will immediately think of two potential classes of bugs: (1) a memory corruption: for example a stack corruption or a buffer overflow could cause the program to wait in some loop forever. (2) a locking issue: e.g., a deadlock or a race due to improper locking.

After getting stuck a few times, I started to notice that most of the times when execution came to a halt, it was in one of the pthread locking functions. Therefore, I expected a problem with locking. However, I trusted that memcached would use pthread locks correctly and that libpthread (which came from NetBSD) was correct too. What I certainly didn’t have much faith in was my scheduler or the integration of libpthread with it. Memory corruption wasn’t high on my list because most of our code was written in Rust and so we experienced very few memory bugs, if at all.

For libpthread to work with our scheduler, I had to provide a series of low-level APIs. These functions were prefixed with _lwp, which stands for light-weight process. The _lwp APIs can be a fairly complex affair to get right: For example _lwp_park and _lwp_unpark are used to add threads and remove threads from the scheduler run-queue. In addition to _lwp_park having many different parameters, one must also be aware that in multi-threaded code, unpark can potentially arrive before park has even completed… It isn’t surprising that I spent a lot of time trying to find the bug somewhere in my implementation of those APIs. A problem there could easily lead to threads not being woken up or put to sleep incorrectly (which in turn meant someone else might end up waiting on a lock forever). However, as much as I wanted to find the bug there, it just wasn’t there.

Tagged pointers

Through the debug process, I became more familiar with the locking APIs in NetBSD. I especially learned more about how pthreads worked internally than I really wanted to know. But, this proved to be crucial for figuring out what was wrong. What helped getting me on the right path was the following comment in pthread_int.h:

Bits in the owner field of the lock that indicate lock state.  If the
WRITE_LOCKED bit is clear, then the owner field is actually a count of
the number of readers.

What this comment meant to convey is that in case of a reader-writer lock the “owner field” (a 64 bit value) uses a bunch of bits (the first four) to indicate things about the lock’s state (wait, wrlock and wrwant). If wrlock is set to 1, bits 4 to 63 point to an address where the owner struct resides, if wrlock is set to 0 the bits contain the amount of readers instead.

 N                              4        3        2        1        0
 +------------------------------------------------------------------+
 | owner or read count          | <free> | wrlock | wrwant |  wait  |
 +------------------------------------------------------------------+

This is a form of tagged pointers, which encodes a few bits of meta-data along with an address or count. The reason this works is because everything allocated by malloc, such as the owner struct, is expected to have (at least) a 16 byte alignment on our platform (so the first 4 bits of any pointer handed out by malloc are guaranteed to be zero). If one wants to dereference the owner pointer, libpthread just has to make sure to always mask out (set to zero) the lower 4 bits before dereferencing it.

malloc() and free()

As you can probably guess by now, the real culprit ended up being our malloc aligning everything to 8 instead of 16 bytes. This meant that our address to the owner would use the 4th bit, which was reserved for use by libpthread. Even though the bit wasn’t really in use, the pthread code would still clear the bit before a dereference, and thus change the base address for the owner struct slightly – which turned out to be quite confusing for the pthread code.

One last detail about this bug was that our underlying memory allocator was actually doing all the alignment correctly. However, our user-space runtime including the memory allocator was written in Rust. The bug was introduced when I wrote the wrapper code that converted the malloc and free calls (as required by rumpkernel) to Rust’s alloc and dealloc interface.

The dealloc function – Rust’s equivalent of free(ptr) – takes a pointer and a Layout (which encodes size and alignment information for the pointer). With pure Rust code, this is fine since the Rust compiler will insert the correct layout argument on deallocations. Hence, Rust allocators technically don’t need to manually keep track of the size of pointers handed out by alloc (nice!). However to support malloc and free for C code, we need to at be able to find the size of a pointer when it is given back to free (which gets just the pointer as a single argument with no extra size or alignment). Memory allocators written for C often solve this problem by allocating a few bytes more than needed and storing the size of the block in the first few bytes of the newly allocated memory.

Here is our original (buggy) code, which prepends every block with an 8 byte header on malloc to store the size, and then reads the size again during free. As you can see HEADER_SIZE is set to 8 bytes, which means our pointers we hand out are now aligned to 8 bytes (even though we requested a 16 byte alignment from the underlying allocator). The fix is to change HEADER_SIZE and/or ALIGNMENT accordingly.

pub const HEADER_SIZE: usize = 8;
pub const ALIGNMENT: usize = 16;

/// Implementes malloc using the `alloc::alloc` interface.
///
/// We need to add a header to store the size for the
/// `free` and `realloc` implementation.
#[no_mangle]
pub unsafe extern "C" fn malloc(size: c_size_t) -> *mut u8 {
    trace!("malloc {}", size);

    let allocation_size: u64 = (size + HEADER_SIZE) as u64;
    let ptr = alloc::alloc::alloc(Layout::from_size_align_unchecked(
        allocation_size as usize,
        ALIGNMENT,
    ));

    if ptr != ptr::null_mut() {
        *(ptr as *mut u64) = allocation_size;
        ptr.offset(HEADER_SIZE as isize)
    } else {
        ptr::null_mut()
    }
}

/// Implements `free` through the rust `dealloc` interface.
///
/// Recovers the size of the block (needed by dealloc) by reading the
/// pre-pended header first.
#[no_mangle]
pub unsafe extern "C" fn free(ptr: *mut u8) {
    if ptr == ptr::null_mut() {
        return;
    }

    let allocation_size: u64 = *(ptr.offset(-(HEADER_SIZE as isize)) as *mut u64);
    alloc::alloc::dealloc(
        ptr.offset(-(HEADER_SIZE as isize)),
        Layout::from_size_align_unchecked(allocation_size as usize, ALIGNMENT),
    );
}

Afterthoughts

I can think of two things that could’ve been done better here to prevent this bug:

A post conditions that asserts proper alignment for ptr (unfortunately I doubt I was consciously aware of the alignment restriction when I wrote that malloc function anyways)
libpthread would be more robust if it used the high bits¹ in the pointer for tagging rather than the first four (this would prevent alignment issues, but might cause other bugs more easily)

On current x86-64 machines, virtual addresses do not use the full 64 bits available. ↩

76 requests per second?

Sun, 08 Mar 2020 22:27:30 +0000

It was a particularly mischievous bug. At the time, we were writing a new research prototype kernel in our group. Research operating systems are fun because you typically end up writing a lot of code until you get something very basic (like print) running. In fact, it’s often the case that you end up with a system or function call which can’t be fully implemented at the moment (because it would require too much code or other things are still missing for it). Therefore, every so often, you have to cut some corners and do a temporary, partial implementation to make progress on a goal.

Unfortunately, if you cut too many corners, you end up forgetting about them. Then they haunt you as bugs weeks or months later. This post is about such an instance of a bug I had to deal with – approximately a year ago.

The premise

In our prototype OS, we had a kernel that implemented user-space processes, memory management, files, and device pass-through. We also had a user-space runtime for fine-grained scheduling (including locks etc.). However, at the time neither our runtime nor our kernel had full support for POSIX. POSIX is pretty important if you want to run existing applications, which in turn is important if you want to evaluate how well your system is doing. Unfortunately, implementing POSIX is a huge undertaking and our resources were limited.

Fortunately, this problem was solved by a project called rumpkernels – a NetBSD libOS. A libOS is a fancy term for a library that contains an entire operating system (in this case NetBSD). I’m not going into too many details here, but the way it works is that you’ll end up implementing a minimal set of functions (around 50, called hypercalls in rump) which abstract away the underlying machine. In return, you’ll get all the NetBSD code (kernel and user-space) running inside your process: So instead of implementing a TCP/IP stack, disk and network drivers, libpthread and libc, you just have to implement these ~50, arguably pretty basic, functions which primarily deal with low-level memory management, timers, scheduling, and locking. It’s a great trade-off, especially since we already had most of that.

76 Requests?

So there I was, working for a few weeks on writing the implementation for those 50 hypercall functions. It was also around Christmas, so I ended up taking a break at some point. Overall, it probably took about 2-3 months to finish it. Many of the hypercalls were trivial to implement since we already had support for it, but some required significant additional implementation. After the implementation was completed, I figured I would test it by compiling a user-space process, linking the NetBSD code, my hypercall implementation, and redis (a single-threaded key–value store). I definitely had some linking and init issues first, but after some digging, I finally saw a familiar prompt on my terminal:

[INFO] - vibrio::rumprt::crt::tls: tls area size is: data + bss + extra = 48
[INFO] - vibrio::rumprt::prt: rumprun_makelwp 0x37a10008 0x3e019240 0x3e801000--4194304 64 0x37a10198
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.0.6 (ffbeedb9/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1:M 02 Feb 02:02:10.540 # Server started, Redis version 3.0.6
1:M 02 Feb 02:02:10.540 * The server is now ready to accept connections on port 6379

Redis started, great! Eager to see if it worked, I ran the redis-benchmark tool and sent GET requests to the network interface of the VM:

$ redis-benchmark -h 172.31.0.10 -n 1000
====== GET ======
  1000 requests completed in 13.00 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

30.10% <= 2 milliseconds
80.10% <= 3 milliseconds
85.10% <= 4 milliseconds
90.10% <= 5280 milliseconds
93.40% <= 5281 milliseconds
93.90% <= 5282 milliseconds
95.10% <= 6712 milliseconds
96.30% <= 7684 milliseconds
99.20% <= 7685 milliseconds
100.00% <= 7685 milliseconds
76.92 requests per second

76 requests?! That’s pretty terrible for most applications, let alone a key–value store… Just for comparison: a Linux VM with redis can do about 2 million requests per second. So we’re off by a factor of 26’000x, yelp. The issue was also pretty clear from the histogram: 10% of the requests ended up just being really slow (like 5+ seconds slow). What’s going on?

Debugging

With prototype OSes there is usually little support for debugging aside from printing and guessing. And its even harder if you’re debugging performance. However, with 76 requests per second, there ought to be something seriously wrong here so this should be easy? I remember being puzzled for a while though:

At first, I figured surely it must be slow because I am compiling all this code in debug mode without optimizations. So I quickly tried it with compiler optimizations turned on: No change at all, still 76 requests. This was confusing, there should have been at least some impact from the compiler?

My next suspicion was that I must have messed up some of the locking implementation. I instrumented most of them but to no avail. They didn’t take up any significant amount of time during the execution.

I also carefully inspected the scheduler and it’s run queues. Maybe the right threads just weren’t running because of a bug in the scheduling algorithm? I didn’t find anything wrong with it tough, the system was happily running all kinds of different threads at different times.

Interrupts were another potential culprit: Maybe they didn’t arrive or they didn’t work properly. I dismissed the theory when I verified that the interrupt handler did indeed get invoked properly by pinging the system.

Finally, I blamed rump itself (always good to blame others when desperate). But running the original rumpkernel as a unikernel in a VM revealed it does indeed work as intended with millions of GETs per second.

What got me on the right track eventually was the fact that pinging the interface also showed a similar anomaly with latency (although with less total variance).

$ ping 172.31.0.10
PING 172.31.0.10 (172.31.0.10) 56(84) bytes of data.
bytes from 172.31.0.10: icmp_seq=1 ttl=255 time=5.78 ms
bytes from 172.31.0.10: icmp_seq=2 ttl=255 time=5.32 ms
bytes from 172.31.0.10: icmp_seq=3 ttl=255 time=2.50 ms
bytes from 172.31.0.10: icmp_seq=4 ttl=255 time=0.940 ms
bytes from 172.31.0.10: icmp_seq=5 ttl=255 time=8.81 ms

Ping uses ICMP requests, which are handled quite differently from TCP or UDP packets. I figured the issue must be somewhere deep down in the device driver, or device itself.

At one point, I didn’t quickly kill the system and let ping run for a longer time. That’s when I saw some concerning errors in the serial log (and at the same time the ping messages suddenly stopped arriving):

wm0: device timeout (txfree 4032 txsfree 0 txnext 832)
wm0: device timeout (txfree 4032 txsfree 0 txnext 832)
wm0: device timeout (txfree 4032 txsfree 0 txnext 832)

wm0 is our network device (a e1000 PCIe NIC emulated by QEMU here). The device timeout message is due to a watchdog timer expiring in the NetBSD driver code. At this stage, I was still confused. Why would the device not work correctly for me? It definitely works with any other OS. The bug must be somewhere in my implementation, but where?

rumpcomp_pci_dmalloc

I started to piece things together when I finally noticed the following message in the log:

[ERROR] - vibrio::rumprt::dev: rumpcomp_pci_dmalloc size=65536 alignment=4096

rumpcomp_pci_dmalloc is one of the required hypercalls. It’s used by the NetBSD kernel (and its device drivers) to allocate DMA memory which can be read and written to by device drivers and the devices. This is important because the device eventually needs to write incoming packets somewhere into memory or read outgoing packets from memory (using DMA). DMA memory is a bit special because we need to know the underlying physical address of the allocated buffer. This is because a device does not use virtual addresses (like our process), and ends up referring to memory by physical addresses¹.

Looking at our rumpcomp_pci_dmalloc function, nothing is obviously wrong with it:

#[no_mangle]
pub unsafe extern "C" fn rumpcomp_pci_dmalloc(
    size: usize,
    alignment: usize,
    pptr: *mut c_ulong,
    vptr: *mut c_ulong,
) -> c_int {
    let layout = Layout::from_size_align_unchecked(size, alignment);
    if size > 4096 {
        error!("rumpcomp_pci_dmalloc size={} alignment={}", size, alignment);
    }

    let mut p = crate::mem::PAGER.lock();
    let r = (*p).allocate(layout);
    match r {
        Ok((vaddr, paddr)) => {
            *vptr = vaddr.as_u64();
            *pptr = paddr.as_u64();
            0
        }
        Err(_e) => 1,
    }
}

In short, the function takes a size and alignment argument to allocate a buffer that is physically and virtually contiguous. It then expects us to write the corresponding physical and virtual start address of the new buffer into pptr and vptr.

Notice the innocent error message that gets printed if size is >4096? Remember the shortcuts I talked about in the beginning? Well here is one that was taken early on and then forgotten about:

The interesting logic happens in (*p).allocate(layout). Eventually, this function calls into our kernel, which has to allocate memory, map it into our address space and return the virtual (vaddr) and physical (paddr) address to it. When this function was first implemented, our kernel used a buddy allocator to allocate physical memory. The buddy allocator would always return a physically consecutive region of memory, which the kernel mapped as a virtually consecutive region by writing the page-table entries accordingly.

The bug got introduced when sometime in the middle of implementing the hypercalls, the underlying kernel memory allocator changed: Rather than using a buddy allocator (which is fairly slow), an optimization got added that held a bunch of pages as 4 KiB frames on a stack to allocate from them quickly. However, the problem now was that these frames were no longer necessarily physically consecutive. For buffers larger than 4 KiB the kernel would happily pop off frames from the stack, add them together in virtual space, and then just return the physical address of the first page that got mapped as paddr.

This meant that if the size of the buffer was exactly 4 KiB, everything was virtually and physically consecutive. For larger memory areas like the one allocated by the NIC driver (64 KiB), the returned paddr would only match for the first 4 KiB of the buffer. Afterwards all bets were off². The driver would then take this large buffer and slice it up into smaller packet buffers. The physical address of those individual buffers was calculated by taking the original paddr and adding a offset to it. What this meant for the system was that only a small percentage of the packet buffers were actually working. For the rest, the device would write to them using some physical address but it didn’t match with the corresponding virtual address in the driver code.

What was surprising here is that even though only a fraction of the packet buffers were working (e.g., were correctly visible in both device and driver), the system still “worked”, thanks to the all robustness (retransmission, error checking etc.) built into TCP/IP. It just had really terrible performance.

What went wrong?

The contract between the user-space allocator and the kernel allocator clearly broke when the kernel allocator got optimized. So, when I initially changed the allocation scheme in the kernel, I added the error message in rumpcomp_pci_dmalloc to indicate that this should be fixed later for buffers larger than 4 KiB (cutting a corner).

However, fixing it wasn’t a top priority at the time: we wanted to bring up the system without drivers anyways first. So instead of using an assert, logging an error was a better way to make progress on other parts of the system. Unfortunately, after a while the error message just got forgotten, only to be rediscovered later in a pile of other log messages.

Once I fixed the bug, redis jumped from 76 to 24k GET requests per second. That was a quick 325x improvement! It took getting rid of 2-3 other bugs to go past a million, but those are a story for another time.

For simplicity, we ignore the IOMMU here. ↩
Given how our stack based allocator worked it would, with high chance, often have some other frames in there that would align but there were no guarantees for that. ↩