Virgil today
Using a tool called Virgil that I describe in the posts "Making it easy to write a 9P server" and "Virgil foundations" I had LLMs write a wide range of 9P servers.
While the code that was written works, it's not in a state where I'd feel comfortable releasing it. Instead I will be paying for my LLM sins by rewriting Virgil and at least some of the servers.
Before I get started with that however, I want to describe what I've gotten working so far, as a way of setting down a sign post to show what I've accomplished so far, even if it's largely slop.
What I have working now
Using Virgil and LLMs, the code that I've generated so far falls into these categories:
- 9P libraries: Libraries that allow you to write a dynamic fileserver. By default, each library supports serving an in-memory file system.
- 9P servers: Broadly, code that runs on hardware and exposes that hardware as a mountable 9P file server.
- 9P infrastructure: Code that speaks 9P, but is intended to support the use and development of 9P servers.
9P libraries
This is where I got started. The idea was to build a "Flask for 9P". What this meant in practice was starting with language specific implementation of an in-memory file system, with the ability to define functions that handle specific paths. A full file system implementation isn't strictly necessary if all that you're exposing is a single file. For more than a few files however, it becomes very important.
The reason for implementing an in-memory file system was originally because I needed something to pass the automated tests. Over time however, it became clear that there are a lot of edge cases in how a file is expected to behave that it made sense to default to an in-memory filesystem and make the dynamic files the edge case instead of vice versa.
In any event, an unexpected outcome of having an LLM pass the Virgil tests using a lot of different languages is that new language implementations would sometimes expose subtle bugs that I'm not sure would have been obvious otherwise.
Here is what I had Claude generate, whith comments as appropriate:
9P libraries that support 9P2000 or "styx"
- Clojure
Here is the prompt that I used:
goal write 9P2000 / Styx server in Clojure. Start by running "make 9ptest" in ./server. Then create a directory in ./server named clj-styx. Implement the server using the stdio:/ interface in ./server/bin/styx-test (built with "make 9ptesst). Read the help text of styx-text carefully before getting started. Pass only one styx-test at a time and then move on. Write idomatic code that an expert developer would write, do not take shortcuts. Write a failing test that matches the error in styx-test before passing both tests. styx-test is the final authority on how to implement styx
Something that I noticed when having LLMs write lisp-like languages is that they'd spend a lot of time matching parenthesis.
- C++
- Dafney
I was inspired by an article on using LLMs to write Dafney so I had Claude generate 9P servers in both C++ and Java. LLM written formal verification didn't expose as many bugs as I expected, but the process did find that I wasn't testing that in-memory 9P servers were freeing up unused blocks.
- Erlang
Here is the prompt that I used:
goal write 9P2000 / Styx server in Erlang. Start by running "make 9ptest" in ./server. Then create a directory in ./server named erlang-styx. Implement the server using the stdio:/ interface in ./server/bin/styx-test (built with "make 9ptesst). Read the help text of styx-text carefully before getting started. Pass only one styx-test at a time and then move on. Write idomatic code that an expert developer would write, do not take shortcuts. Write a failing test that matches the error in styx-test before passing both tests. styx-test is the final authority on how to implement styx
- Java
- JavaScript
- Rust
9P libraries that support 9P2000.L
- C
- Forth
- Fortran
This one was surprising! I guess that by default Fortran pre-allocates all of the variables used, so the initial binary that Claude created was enormous. This implementation also turned up some weird bugs. I don't remember what they were, just that I was glad to have found them.
- J
J is a purposely terse language. I remember it going a lot faster than other implementations and my hypothesis is because more of the code can fit into the context window.
- Java
- JavaScript
- Perl
- Python
- Rust
9P servers
I started off this project wanting to make it easy to write 9P file servers. Once I got that ability, the thing that I started working on next was having these servers run on actual hardware. What I didn't realize at the time is that I was poorly re-implementing what is known as a "drawterm" in the Plan 9 world.
Part of why I think I didn't make the connection initially is because
drawterms are weird. Not quite ssh, not quite sftp, not quite
VNC or RDP, they are an interesting intersection of a bunch of
those concepts.
To explain what a drawterm is, I think it helps to start with how those tools work, so we can contrast them with what a drawterm does.
sshconnects to a remote machine and allows you to run commands on the remote machine.sftpallows you to copy files to and from a remote machine.VNCconnects to a remote machine. Gives you a graphical view of the remote desktop by sending over the raw pixels on the remote machine. Allows you to control the remote machine's keyboard and mouse.RDPlikeVNCexcept in addition to sending raw pixels, it can also send window draw commands, which can be more bandwidth efficient.
By contrast, what a "drawterm" does is this:
- Implements fileservers for the client's keyboard, mouse, clipboard, local filesystem, and other capabilities as files.
- Implements the
/dev/drawfilesystem, which is the same filesystem that a Plan 9 machine uses to draw on a directly attached monitor. - Connects to the remote machine using TLS, and then mounts the filesystems above on the remote machine.
- Launches a shell (e.g.
rc) or other program (like theriowindowing system) on the remote machine, binding in the filesystems mentioned above.
This all works because:
- Everything in Plan 9 is a file.
- Plan 9 allows you to change what filesystems are bound to a specific process.
So, instead of having to trick the remote system into thinking that
your client is local (like ssh, sftp, etc have to do) you can
change what filesystems a process sees and things will "just work"
because it's using the same "API" that it always uses.
What this means in practice is that once a drawterm is written for a specific hardware platform, that hardware can connect to a remote Plan 9 system (a "cpu" server in Plan 9 terms) without needing to make any changes on the Plan 9 side. Furthermore, you don't need to learn or implement any new APIs or protocols to do remote computing, you can use filesystem mounts instead.
Here are the systems or languages where I had a drawterm implemented:
Go
Java
JavaScript
This one is fun. It can run in a web browser!
Rust
DOS
I've gotten this to work in emulation, but have yet to get it working on real hardware. I will eventually though!
Flutter
Python
Nintendo 3DS
This is where things started getting cool. How do you expose two screens to a remote system? I don't know yet!
Playdate
Probably the most fun of them all. I was delighted to learn that Plan 9 has native support for 1 bit displays. I was honestly surprised that it worked!
9P infrastructure
In the process of experimenting with all of the clients and servers described above, I found the need for two programs that don't fit nicely into any of the categories above
9P Relay
I originally wrote this so that Progrium could connect remotely to my Nintendo 3DS which was behind NAT and a firewall. It started off as a 9P specific "ngrok" type service, allowing two systems to communicate via outbound connections. My server running behind NAT would connect to a public "9P Relay" as a client and then start listening as a server. Another client could connect to the same relay and use it as a server, without having to care or know that the server is actually behind NAT.
I later had a feature added where the relay could also proxy a 9P2000.L connection to a 9P2000 server, some of the 9P2000.L messages can be passed along unchanged, others had to be emulated. But it worked!
Plan 9 kernel shim
This was the most surprising of these experiments. This is a Plan 9 kernel emulator that translates Plan 9 system calls into Linux system calls. It allows for unmodified Plan 9 binaries to run under Linux.
I had this written because I wanted an easy way to connect to a real Plan 9 server that was also available over a WebSocket.
Closing
I got a lot of 9P related clients, servers, and infrastructure working with Claude. I'm not proud of it so I want to rewrite a lot of it, but I wanted to write about whats been done so far before I forget more about what has been done so far.