Making it easy to write a 9P server
I want to make it as easy to write a server for the 9P network file protocol as it is to write an HTTP server using frameworks like Flask or Express.
Why 9P? Because Linux has native support for it, as does a variety of software like QEMU, v86, ZeroFS.
Ostensibly 9P also has a wide variety of programming language implementations, though a closer inspection will show that the support isn't as extensive as you'd expect.
Encouraged by what seemed at the time like a wide range of support for 9P, Progrium and I started down the path of making it simpler to write a 9P server sometime around 2024. We expected to get a basic proof of concept done in a weekend, or maybe a week or two?
My initial goal was to write a calculator file systems, something that would work like this:
$ ls add sub mul div result $ echo 2 > add $ echo 3 > add $ cat result 5
Well, as it turned out, it would take me two years before I could easily write a dynamic file system like the one I sketched above. With that hindsight and hundreds of hours spent on this project, I can confidently say that I underestimated the level of effort that it would take.
One of the major reasons for this is that while the 9P protocol itself is simple, the details of writing a fully functional filesystem is anything but.
Even a small filesystem has a lot of edge cases that need to be considered, for example:
- Do you pre-allocate large files?
- Does the modification time of a directory get updated when a file inside of that directory is modified?
- Is file truncation supported? What about at an offset?
The list goes on, but that is a pretty representative example of what you need to take into account when writing a 9P server. Part of why I dramatically underestimated the level of effort for writing a 9P server is that I assumed that writing one would be roughly on the order of writing a HTTP server, something that I've done many times before. Dealing with all of these edge cases really helped me understand how simple and elegant the core of HTTP is!
So, how did I handle that complexity and finally get something working? In a word: Testing.
Before I get into the testing approach that I ended up using, I'd like to spend some time talking about my motivations and how they informed what I built.
The goal of this project was to make it as easy to write a 9P server with dynamic files as it is to write a HTTP server with dynamic resources. But why focus on files when HTTP is everywhere?
Simply put, it's because files are the most widely supported interface in computing.
The first hypothesis that I'm testing, which I'm fairly certain about, is that reading and writing files is the computer interaction metaphor that is the easiest to understand. Of all of the methods for a programmer to interact with data, using files is the method that has the largest adoption. I say this not as a ivory tower philosopher, but as someone who has spent the majority of their career on teaching programmers how to use HTTP based protocols.
The other hypothesis is that if we make it easy to write file servers, we can extend the useful life of existing and reliable software and also create new ways for software systems to interact.
Some hypothetical examples:
- Editing the sprites for a running Game Boy game using MS Paint.
- Allowing Bash scripts to manipulate the DOM of a webpage.
- Controlling lights by writing "on" or "off" to a file.
For as interesting as these hypothesis are, the question that I asked myself at this point, which you might be asking yourself too, is "why hadn't this done before?" and in this case, I'm pretty certain that the answer is: "because filesystems are very annoying to implement"
To explain how annoying it was, I'll cover what my original plan was:
- Write a 9P server that works
- Create a Flask-like API around the working 9P implementation.
- Write a testing tool that interactively teaches a programmer how to implement the 9P protocol.
The thinking was to get something working so that I understood how it worked, then build out a test suite which would start with the easiest possible test to pass and then gradually build from there. The idea for the test suite came from the "okta-scim-beta" project that I worked on at Okta in 2016 which taught developers how to implement the SCIM protocol.
Without getting into the details, that plan didn't work. I kept getting stuck trying to implement a 9P server and got pretty discouraged until I ran across the ZeroFS project which makes heavy use of 9P and, critically, makes heavy use of filesystem tests!
Learning that projects like pjdfstest and xfstest existed was the breakthrough that I needed.
I used existing filesytem tests to create a tool that I call "Virgil" which interactively teaches a programmer how to implement 9P.
I'm happy to report that Virgil works, and works very well. I've used it to create 9P implementations in every programming language that I could think of (Forth, COBOL, Fortran) for a wide range of systems including browsers, the Nintendo 3DS, and the Playdate.
However, I have a confession to make: I made extensive use of an LLM (Claude in particular) to write Virgil. I'm not proud of this and the code it created isn't something that I'd want to put my name on.
So rather than release more slop into the world, what I am doing instead is re-writing Virgil with my human brain. Partly as a form of penance, partly to really understand how it works, but mostly because I want to spend time deeply considering the 9P specific questions which don't have easy answers, things like:
- Should a 9P server hint that a file is "dynamic" and if so, how?
- What is the right default block size to use for storing files?
- How should the inode number (the "path" component of a 9P QID) of a file be selected?
- What user should be the owner of "/" by default?
I also want to write about how using LLMs to write Virgil has been an interesting counterweight to a lot of the hype around LLMs because, to date, I have yet to see an LLM that can write a functional 9P server in "one shot". These experiences have also given me a pretty strong conviction that testing is an even more important part of software development than it has ever been before.