Next, let’s look at the differences between the virtual memory systems of NetBSD, OpenBSD, Linux, and OpenIndiana.
First, let’s give just a tiny summary of where each of these OSes come from. We covered the history of BSD in the article on BSD Virtual Memory, so I’ll skip those. If we go back to the Wiki entry on the History of Unix, we can see that SunOS derived originally from BSD 3.0 to 4.1, then evolved alongside BSD until about the same time as BSD4.4. At which point, Sun built a new OS called Solaris based on Unix System V. Finally, OpenSolaris split from Solaris 10. From the Wiki entry on OpenIndiana, it seems to have come into existence when Sun was being acquired by Oracle.
As for Linux, the kernel was created in 1991 during the period immediately before 386BSD was first released. At this time, people were becoming aware that the 386 was powerful enough to support some flavor of Unix, but nothing had yet been written. It dovetailed perfectly with the GNU effort begun in 1983 to make a wholly free flavor of Unix.
With the history out of the way, let’s get into the differences. They are the following:
- How the kernel deals with being out of memory
- Copy-on-write mechanisms
- How the kernel accesses user memory
- Dead queue (OpenBSD versus NetBSD)
How the kernel deals with being out of memory
When OpenBSD’s kernel runs out of memory, it simply panics. I think the same is true of OpenIndiana.
Linux infamously has an Out-Of-Memory (OOM) killer. This is a heuristic that runs when the kernel runs out of memory, and decides on a process to kill in order to free up memory.
In UVM, copy-on-write simply creates a read-only reference to a backing
uvm_object. When written to, UVM creates an anonymous map, and copies the
uvm_object’s data into the anonymous map. (See page 51 of Cranor’s dissertation.)
This simplifies the original VM system, which used chains of copy/shadow objects which could get arbitrarily long.
How the kernel accesses user memory
On Linux, the kernel maps all of physical memory into the kernel’s address space. This makes it possible to copy data from a user process into the kernel’s memory without the overhead of first mapping the process’ physical memory into the kernel’s address space. There’s a great article by IBM on how this works in linux.
To figure out how this works on NetBSD, I found an interesting article on linux compatibility on NetBSD, which led me to the code for the POSIX
read() system call in
So that just calls
dofileread() in the same file (
Actually, I think that the
SCARGS() macro uses something like linux’s
copy_from_user() function, whose equivalent is
copyin() in NetBSD.
Chuck Silvers’ UBC paper states the intention to implement
copyout() using UVM page loans, but this doesn’t seem to be implemented.
It turns out, the implementation for
copyin() is hardware-dependent since it is related to how pmap actually stores the memory. The x86-64 implementation is in
As unsatisfying as it is, I’m not really sure what this does. It seems to copy byte-by-byte from user space to kernel space. Does this mean that all of physical memory exists in the kernel’s virtual address space, since assembly instructions act on virtual addresses?
I’ll go through line by line and add comments where necessary to try and explain what the assembly does. The most helpful documentation I could find for gas or AT&T syntax x86-64 code was Oracle’s x86 assembly language reference manual, which documents the solaris assembler, which is very similar to gas. Another useful resource was the x86 assembly wikibook on shift and rotate and the x86 instruction set reference on MOVS.
It certainly copies bytes around, using
rep movsq, but it’s still not clear how the kernel indexes into user space without mapping all of physical memory into its address space.
Dead queue (OpenBSD versus NetBSD)
I’ve mostly been studying UVM since I began to study virtual memory. When I began, I assumed that OpenBSD was mostly still running a VM system that was based on Cranor’s UVM. However, it seems that OpenBSD’s VM system was substantially rewritten in 2011 by Ariane van der Steldt.
One of the changes Ariane made was to add a “dead entry queue,” and as far as I can tell there is no official documentation about what a dead entry is. We can see that the dead entry queue was added in revision 1.45 in 2011 by ariane. This was part of Ariane’s VM rewrite, so it isn’t clear what the dead entry queue in particular was meant to do.
Since OpenBSD seems to differ significantly from Cranor’s design, I’m going to switch to studying NetBSD instead.