blog.pr4tt.com

goasm - a Go assembly generator

Thu, 02 Feb 2017 00:00:00 +0000

Recently, I’ve been learning Go. One thing I’ve learned is that the Go compiler generates its own assembly language. There’s a great talk by Rob Pike about it. While I was learning, I began to wish there was a tool similar to the Compiler Explorer (more commonly referred to as Godbolt), but for Go assembly. So I built goasm, a Go assembly generator.

Details

The design is straightforward. It prints the Go code to a temporary file, and then compiles it (sending the output object file to /dev/null). One lesson I learned was that if you send an XMLHTTPRequest using JavaScript, and your form data is encoded as a FormData object, this is received by the back-end as multi-part form data. This threw me off at first, because simply submitting the form doesn’t do this.

I deployed the app on Heroku, since they support Go apps. Normally, Go is only available on your Heroku dyno while building your app, but I used jpillora’s custom heroku Go buildpack, because it keeps Go in the run environment. Jpillora built this in order to build the pretty neat cloud-gox tool.

Future

I would like to implement setting arbitrary GOOS and GOARCH variables. It would be very easy, except that Go doesn’t yet support specifying an environment variable for a single command.

Right now, the output is taken raw from the output of the Go compiler. I’d like to parse the output for just the assembly code, and make it look much more like Godbolt.

lc2.js - an LC-2 Simulator in JS

Mon, 23 Jan 2017 00:00:00 +0000

The book “Introduction to Computing Systems: from Bits & Gates to C & Beyond” by Patt & Patel introduces in some detail a computer called LC-2. This computer never physically existed, but the publisher’s website includes Windows and Unix versions of a simulator for the computer. I couldn’t get the Unix version to run on my system, and I didn’t want to dual-boot just for that, so I decided to write a JavaScript simulator. And so I built lc2.js.

About the LC-2

The LC-2 is a 16-bit CPU with 8 general-purpose registers (r0 to r7), 2 special purpose registers: PC, which stores the memory address of the next instruction to be loaded; and IR, which stores the most recently loaded instruction. Additionally, the LC-2’s memory management unit (MMU) has 2 additional registers: MAR, which stores an address in memory; and MDR, which stores the data at that address in memory. These registers are used to read and write to memory.

Technology

The simulated MMU in lc2.js uses typed arrays of unsigned 16-bit integers to model the memory. In order to minimize wasted memory, each page (9 bit index, 20 512 memory addresses) are only initialized when they are first read/written. For any reasonably program, lc2.js will only initialize 3 pages of memory. The first and last pages contain OS code, and user code exists in the page starting with address 0x3000 by convention. This means that lc2.js normally only uses 3kb of browser memory, rather than the full 131 kb (2¹⁶ = 65536 memory locations, each of which is 2 bytes).

I wrote lc2.js in a test-driven way. I began by specifying a series of unit tests in QUnit, and then worked backwards from there to make the tests pass with minimal code changes. It worked well enough, but there was at least one instance where I wrote a wrong test, and only caught it when I was trying to run a real program.

Project Future

The book that defines LC-2 has a gap. In one chapter, we are using LC-2 assembly, and in the next we are using C. I would like to use lc2.js to bridge that gap. I think it would be interesting to build a Forth on top of lc2.js, and use that to bootstrap a simple Lisp dialect.

Virtual Memory Differences

Wed, 24 Feb 2016 00:00:00 +0000

Previously, we’ve been looking at how POSIX specifies memory management, an academic design for virtual memory and BSD virtual memory in particular:

Next, let’s look at the differences between the virtual memory systems of NetBSD, OpenBSD, Linux, and OpenIndiana.

History

First, let’s give just a tiny summary of where each of these OSes come from. We covered the history of BSD in the article on BSD Virtual Memory, so I’ll skip those. If we go back to the Wiki entry on the History of Unix, we can see that SunOS derived originally from BSD 3.0 to 4.1, then evolved alongside BSD until about the same time as BSD4.4. At which point, Sun built a new OS called Solaris based on Unix System V. Finally, OpenSolaris split from Solaris 10. From the Wiki entry on OpenIndiana, it seems to have come into existence when Sun was being acquired by Oracle.

As for Linux, the kernel was created in 1991 during the period immediately before 386BSD was first released. At this time, people were becoming aware that the 386 was powerful enough to support some flavor of Unix, but nothing had yet been written. It dovetailed perfectly with the GNU effort begun in 1983 to make a wholly free flavor of Unix.

Differences

With the history out of the way, let’s get into the differences. They are the following:

How the kernel deals with being out of memory
Copy-on-write mechanisms
How the kernel accesses user memory
Dead queue (OpenBSD versus NetBSD)

How the kernel deals with being out of memory

When OpenBSD’s kernel runs out of memory, it simply panics. I think the same is true of OpenIndiana.

Linux infamously has an Out-Of-Memory (OOM) killer. This is a heuristic that runs when the kernel runs out of memory, and decides on a process to kill in order to free up memory.

Copy-on-write mechanisms

In UVM, copy-on-write simply creates a read-only reference to a backing uvm_object. When written to, UVM creates an anonymous map, and copies the uvm_object’s data into the anonymous map. (See page 51 of Cranor’s dissertation.)

This simplifies the original VM system, which used chains of copy/shadow objects which could get arbitrarily long.

How the kernel accesses user memory

On Linux, the kernel maps all of physical memory into the kernel’s address space. This makes it possible to copy data from a user process into the kernel’s memory without the overhead of first mapping the process’ physical memory into the kernel’s address space. There’s a great article by IBM on how this works in linux.

To figure out how this works on NetBSD, I found an interesting article on linux compatibility on NetBSD, which led me to the code for the POSIX read() system call in src/sys/kern/sys_generic.c:

/*
 * Read system call.
 */
/* ARGSUSED */
int
sys_read(struct lwp *l, const struct sys_read_args *uap, register_t *retval)
{
	/* {
		syscallarg(int)		fd;
		syscallarg(void *)	buf;
		syscallarg(size_t)	nbyte;
	} */
	file_t *fp;
	int fd;

	fd = SCARG(uap, fd);

	if ((fp = fd_getfile(fd)) == NULL)
		return (EBADF);

	if ((fp->f_flag & FREAD) == 0) {
		fd_putfile(fd);
		return (EBADF);
	}

	/* dofileread() will unuse the descriptor for us */
	return (dofileread(fd, fp, SCARG(uap, buf), SCARG(uap, nbyte),
	    &fp->f_offset, FOF_UPDATE_OFFSET, retval));
}

So that just calls dofileread() in the same file (src/sys/kern/sys_generic.c):

int
dofileread(int fd, struct file *fp, void *buf, size_t nbyte,
	off_t *offset, int flags, register_t *retval)
{
	struct iovec aiov;
	struct uio auio;
	size_t cnt;
	int error;
	lwp_t *l;

	l = curlwp;

	aiov.iov_base = (void *)buf;
	aiov.iov_len = nbyte;
	auio.uio_iov = &aiov;
	auio.uio_iovcnt = 1;
	auio.uio_resid = nbyte;
	auio.uio_rw = UIO_READ;
	auio.uio_vmspace = l->l_proc->p_vmspace;

	/*
	 * Reads return ssize_t because -1 is returned on error.  Therefore
	 * we must restrict the length to SSIZE_MAX to avoid garbage return
	 * values.
	 */
	if (auio.uio_resid > SSIZE_MAX) {
		error = EINVAL;
		goto out;
	}

	cnt = auio.uio_resid;
	error = (*fp->f_ops->fo_read)(fp, offset, &auio, fp->f_cred, flags);
	if (error)
		if (auio.uio_resid != cnt && (error == ERESTART ||
		    error == EINTR || error == EWOULDBLOCK))
			error = 0;
	cnt -= auio.uio_resid;
	ktrgenio(fd, UIO_READ, buf, cnt, error);
	*retval = cnt;
 out:
	fd_putfile(fd);
	return (error);
}

Actually, I think that the SCARGS() macro uses something like linux’s copy_from_user() function, whose equivalent is copyin() in NetBSD.

Chuck Silvers’ UBC paper states the intention to implement copyin()/copyout() using UVM page loans, but this doesn’t seem to be implemented.

It turns out, the implementation for copyin() is hardware-dependent since it is related to how pmap actually stores the memory. The x86-64 implementation is in src/sys/arch/amd64/amd64/copy.S:

ENTRY(copyin)
	DEFERRED_SWITCH_CHECK

	xchgq	%rdi,%rsi
	movq	%rdx,%rax

	addq	%rsi,%rdx		/* check source address not wrapped */
	jc	_C_LABEL(copy_efault)
	movq	$VM_MAXUSER_ADDRESS,%r8
	cmpq	%r8,%rdx
	ja	_C_LABEL(copy_efault)	/* j if end in kernel space */

.Lcopyin_start:
3:	/* bcopy(%rsi, %rdi, %rax); */
	movq	%rax,%rcx
	shrq	$3,%rcx
	rep
	movsq
	movb	%al,%cl
	andb	$7,%cl
	rep
	movsb
.Lcopyin_end:
	xorl	%eax,%eax
	ret
	DEFERRED_SWITCH_CALL

As unsatisfying as it is, I’m not really sure what this does. It seems to copy byte-by-byte from user space to kernel space. Does this mean that all of physical memory exists in the kernel’s virtual address space, since assembly instructions act on virtual addresses?

I’ll go through line by line and add comments where necessary to try and explain what the assembly does. The most helpful documentation I could find for gas or AT&T syntax x86-64 code was Oracle’s x86 assembly language reference manual, which documents the solaris assembler, which is very similar to gas. Another useful resource was the x86 assembly wikibook on shift and rotate and the x86 instruction set reference on MOVS.

ENTRY(copyin)
	DEFERRED_SWITCH_CHECK

	/* the 'q' suffix after an instruction indicates that the instruction
	 * works on a "quadword" or 64- bits
	 */

	xchgq	%rdi,%rsi		/* Exchange rdi, rsi */
	movq	%rdx,%rax		/* Move rdx into rax */

	addq	%rsi,%rdx		/* add rsi to rdx */
	jc	_C_LABEL(copy_efault)	/* jump if carry bit set after prev. */
					/* instruction */
	movq	$VM_MAXUSER_ADDRESS,%r8	/* move the max user address into r8 */
	cmpq	%r8,%rdx			/* compare r8 and rdx */
	ja	_C_LABEL(copy_efault)	/* jump if r8 is above rdx */

.Lcopyin_start:
3:	/* bcopy(%rsi, %rdi, %rax); */
	movq	%rax,%rcx	/* move rax to rcx */
	shrq	$3,%rcx		/* shift rcx right by 3 bits */
	rep			/* repeat the following until rcx is 0 ... */
	movsq			/* move quadword from rsi to rdi */
	movb	%al,%cl		/* move byte from al to cl */
	andb	$7,%cl		/* logical AND bytes 7 and cl */
	rep			/* repeat the following until rcx is 0 ... */
	movsb			/* move byte from rsi to rdi */
.Lcopyin_end:
	xorl	%eax,%eax	/* logical XOR long (32 bit) eax with itself */
				/* effectively, zero out eax */
	ret
	DEFERRED_SWITCH_CALL

It certainly copies bytes around, using rep movsq, but it’s still not clear how the kernel indexes into user space without mapping all of physical memory into its address space.

Dead queue (OpenBSD versus NetBSD)

I’ve mostly been studying UVM since I began to study virtual memory. When I began, I assumed that OpenBSD was mostly still running a VM system that was based on Cranor’s UVM. However, it seems that OpenBSD’s VM system was substantially rewritten in 2011 by Ariane van der Steldt.

Ariane posted a diff in OpenBSD’s tech mailing list detailing a vmmap rewrite, and also made a presentation about it.

One of the changes Ariane made was to add a “dead entry queue,” and as far as I can tell there is no official documentation about what a dead entry is. We can see that the dead entry queue was added in revision 1.45 in 2011 by ariane. This was part of Ariane’s VM rewrite, so it isn’t clear what the dead entry queue in particular was meant to do.

Since OpenBSD seems to differ significantly from Cranor’s design, I’m going to switch to studying NetBSD instead.

OpenBSD Virtual Memory

Tue, 23 Feb 2016 00:00:00 +0000

Today, we answer the following questions about OpenBSD’s virtual memory system:

How are contiguous regions of memory managed?
How is memory freed?
- What happens when the kernel runs out of memory?
Is there any special code for Non-Uniform Memory Access (NUMA) architectures?
How is physical memory managed?
- Does it assume more virtual than physical memory?
What are its data structures which store information about memory?

Along the way, I’ll show the relevant parts of the OpenBSD source and link to their location in their web CVS, which should serve both as a map and let anyone interested easily find out more. But first, we’ll look at UVM’s design at a high-level.

UVM Design Overview

As we learned in the article on BSD’s virtual memory system, Charles Cranor wrote his PhD dissertation on his massive rewrite of NetBSD’s VM system, which he called UVM. This rewrite was meant to address the following issues in 4.4BSD’s VM, which was based on Mach’s VM:

shadow/copy object chaining
one page at a time I/O operations
poor integration with the kernel

When a process is forked, its existing address space is copied to the new process. Since actually copying the entire address space would be a tremendous overhead on every call to fork, and since a common pattern in unix is to immediately fork and kill the original process to start a process in the background, POSIX systems typically don’t actually copy the address space. In 4.4BSD, a fork creates a “copy” object between the vm_map_entry and the vnode. This allows the kernel to fault on any writes to this memory, and only then copy the data to the writing process’ address space. This mechanism is called copy-on-write. (See section 5.5, page 146 of The Design and Implementation of the 4.4BSD Operating System.)

Similarly, when a process has a private mapping from memory to a file, this means that changes made by that process are not reflected in the file but are visible to that process. To support this, 4.4BSD creates a “shadow” object between the vm_map_entry and the file object, which stores the private changes made by the process to the file. (See section 5.5, page 142 of The Design and Implementation of the 4.4BSD Operating System.)

These shadow/copy object chains can get quite long, which slows memory search times and can contain inaccessible redundant copies of the same page of data. If left unchecked, this can fill swap space with redundant data and cause the system to deadlock, these are called swap memory leak deadlocks. 4.4BSD addresses this problem with a collapse operation which can bypass or discards any redundant pages that are still accessible, but this process can’t do anything to redundant pages that have become inaccessible.

UVM simplifies the copy-on-write mechanisms by replacing shadow/copy objects with a 2-level scheme based on page reference counters. UVM also uses these reference counters to eliminate swap memory leaks.

Second on the list of issues meant to be addressed by UVM is the fact that I/O operations in the BSD VM are performed one page at a time, which slows paging response time. UVM addresses this by allowing such operations on multi-page clusters.

Finally, Cranor claims that BSD’s VM is poorly integrated with the kernel, and gives the example that unreferenced memory-mapped file objects are cached both at the I/O system (vnode) layer, and redundantly at the VM layer as well.

Locking

Section 3.3.2 of Cranor’s dissertation describes data structure locking, noting that OS data structures can be protected either by one big lock or by many small “fine-grained” locks. It goes on to say that when Mach’s VM system was ported to BSD, they ignored its fine-grained locking support. One of UVM’s goals is to restore this fine-grained locking support.

Contiguous Memory Management

What data structure is used to represent a contiguous region of memory? Well, the uvm_object struct is used to provide a reference to a backing store (such as physical memory), and can hold references to a contiguous sequence of pages. But there’s no mechanism to allocate contiguous regions of physical memory aside from the direct-memory-access (DMA) interface meant for driver developers.

Freeing Memory

What actually happens when we call munmap()? Well, rather than go through yet another trail of indirection citing the code at every step, I’ll simply look through the code and enumerate the calls here:

sys_munmap() system call in src/sys/uvm/uvm_mmap.c calls uvm_unmap_remove(map, addr, addr + size, &dead_entries, FALSE, TRUE)
uvm_unmap_remove(struct vm_map *map, vaddr_t start, vaddr_t end, struct uvm_map_deadq *dead, boolean_t remove_holes, boolean_t markfree) in src/sys/uvm/uvm_map.c calls uvm_mapent_mkfree(map, entry, &prev_hint, dead, markfree)
uvm_mapent_mkfree(struct vm_map *map, struct vm_map_entry *entry, struct vm_map_entry **prev_ptr, struct uvm_map_deadq *dead, boolean_t markfree) in the same file, which mostly just seems to put the entry into a queue of dead entries.

What happens when the kernel runs out of memory?

When a user process runs out of memory, a call to mmap simply returns the ENOMEM error, as we can see in src/sys/uvm/uvm_mmap.c:

/*
 * sys_mmap: mmap system call.
 *
 * => file offset and address may not be page aligned
 *    - if MAP_FIXED, offset and address must have remainder mod PAGE_SIZE
 *    - if address isn't page aligned the mapping starts at trunc_page(addr)
 *      and the return value is adjusted up by the page offset.
 */
int
sys_mmap(struct proc *p, void *v, register_t *retval)
{
	/* ... */
	/* check for file mappings (i.e. not anonymous) and verify file. */
	if ((flags & MAP_ANON) == 0) {
	/* ... */
	} else {		/* MAP_ANON case */
		if ((flags & MAP_ANON) != 0 ||
		    ((flags & MAP_PRIVATE) != 0 && (prot & PROT_WRITE) != 0)) {
			if (size >
			    (p->p_rlimit[RLIMIT_DATA].rlim_cur - ptoa(p->p_vmspace->vm_dused))) {
				return ENOMEM;
			}
		}
	/* ... */
	}
	/* ... */
}

But when the kernel runs out of memory, we see a panic as in src/sys/uvm/uvm_map.c:

/*
 * uvm_mapent_alloc: allocate a map entry
 */
struct vm_map_entry *
uvm_mapent_alloc(struct vm_map *map, int flags)
{
	struct vm_map_entry *me, *ne;
	int pool_flags;
	int i;

	pool_flags = PR_WAITOK;
	if (flags & UVM_FLAG_TRYLOCK)
		pool_flags = PR_NOWAIT;

	if (map->flags & VM_MAP_INTRSAFE || cold) {
		mtx_enter(&uvm_kmapent_mtx);
		me = uvm.kentry_free;
		if (me == NULL) {
			ne = km_alloc(PAGE_SIZE, &kv_page, &kp_dirty,
			    &kd_nowait);
			if (ne == NULL)
				panic("uvm_mapent_alloc: cannot allocate map "
				    "entry");
			/* ... */
		}
		/* ... */
	}
	/* ... */
}

NUMA

Non-Uniform Memory Access (NUMA) is when each processor has some section of fast memory which is considered “local,” and the rest is much slower and considered “remote.” Section 5.2.16 of the ACPI specification defines the System Resource Affinity Table (SRAT) which associates processors and memory to proximity domains, essentially what memory is local to which processor. Section 5.2.17 of the same spec defines the System Locality Distance Information Table (SLIT) which provides information about the memory latency for each proximity domain.

I learned on hubertf’s blog that Christoph Egger wrote an ACPI SLIT parser and an ACPI SRAT parser for NetBSD. Indeed, the src/sys/dev/acpi directory in NetBSD’s CVS contains acpi_slit.h, acpi_slit.c, acpi_srat.h, and acpi_srat.c. No similar files exist in OpenBSD, which suggests that they haven’t been ported.

There seems to have been some work done on adding NUMA support as well as the concept of memory and CPU affinity to OpenBSD, but nothing since 2009. A search within OpenBSD’s web CVS reveals no mention of numa, which suggests that Ariane’s patch was never accepted.

What about other BSDs?

The paper describing DragonFly BSD’s design goals as a fork of FreeBSD states that it is designed with a NUMA-centric view, by explicitly partitioning the workload among multiple processors. There was also a Google Summer of Code project in 2010 to make DragonFly BSD NUMA-aware.

Physical Memory Management

The BSD VM system is split into two layers: machine dependent and independent. UVM’s machine-dependent layer is essentially the same as BSD’s, which is called the pmap layer. pmap handles adding to, removing from, or querying for virtual or physical mappings within the processor’s memory management unit (MMU).

The pmap layer is a deliberately thin abstraction over the MMU, in order that the VM code built on top can reuse as much code as possible across every supported platform. This also lets us hide shortcomings of the MMU, such as painfully small page sizes on the VAX, by dealing with such things at the pmap layer. Unfortunately, it has a symmetric shortcoming of its own, in that it abstracts away the nicer features of any MMU in order to provide a simple interface supported by all architectures. Also, this means that memory information is stored both in the page tables manipulated by the pmap layer, and in the higher-level UVM structures.

The pmap structure is in src/sys/arch/i386/include/pmap.h:

/*
 * The pmap structure
 *
 * Note that the pm_obj contains the reference count,
 * page list, and number of PTPs within the pmap.
 */

struct pmap {
	uint64_t pm_pdidx[4];		/* PDIEs for PAE mode */

	struct mutex pm_mtx;
	struct mutex pm_apte_mtx;

	paddr_t pm_pdirpa;		/* PA of PD (read-only after create) */
	vaddr_t pm_pdir;		/* VA of PD (lck by object lock) */
	int	pm_pdirsize;		/* PD size (4k vs 16k on PAE) */
	struct uvm_object pm_obj;	/* object (lck by object lock) */
	LIST_ENTRY(pmap) pm_list;	/* list (lck by pm_list lock) */
	struct vm_page *pm_ptphint;	/* pointer to a PTP in our pmap */
	struct pmap_statistics pm_stats;  /* pmap stats (lck by object lock) */

	vaddr_t pm_hiexec;		/* highest executable mapping */
	int pm_flags;			/* see below */

	struct segment_descriptor pm_codeseg;	/* cs descriptor for process */
	union descriptor *pm_ldt;	/* user-set LDT */
	int pm_ldt_len;			/* number of LDT entries */
	int pm_ldt_sel;			/* LDT selector */
};

Do they assume more virtual than physical addresses?

Although this used to be a reasonable assumption on x86, the x86-64 specification has a bigger physical address space (52 bits) than virtual address space (48 bits).

On Linux, the kernel allocates every page of physical memory into the kernel’s virtual address space. This makes the fundamental assumption that there will always be more virtual than physical memory.

I can’t find any definitive proof, but I will show how UVM initializes memory. First, let’s see how the kernel is initialized in src/sys/kern/init_main.c:

/*
 * System startup; initialize the world, create process 0, mount root
 * filesystem, and fork to create init and pagedaemon.  Most of the
 * hard work is done in the lower-level initialization routines including
 * startup(), which does memory initialization and autoconfiguration.
 */
/* XXX return int, so gcc -Werror won't complain */
int
main(void *framep)
{
	struct proc *p;
	struct process *pr;
	struct pdevinit *pdev;
	quad_t lim;
	int s, i;
	extern struct pdevinit pdevinit[];
	extern void disk_init(void);

	/*
	 * Initialize the current process pointer (curproc) before
	 * any possible traps/probes to simplify trap processing.
	 */
	curproc = p = &proc0;
	p->p_cpu = curcpu();

	/*
	 * Initialize timeouts.
	 */
	timeout_startup();

	/*
	 * Attempt to find console and initialize
	 * in case of early panic or other messages.
	 */
	config_init();		/* init autoconfiguration data structures */
	consinit();

	printf("%s\n", copyright);

	KERNEL_LOCK_INIT();
	SCHED_LOCK_INIT();

	uvm_init();
	/* ... */
}

And we can find the code for uvm_init in src/sys/uvm/uvm_init.c:

/*
 * uvm_init: init the VM system.   called from kern/init_main.c.
 */
void
uvm_init(void)
{
	vaddr_t kvm_start, kvm_end;

	/* step 0: ensure that the hardware set the page size */
	if (uvmexp.pagesize == 0) {
		panic("uvm_init: page size not set");
	}

	/* step 1: set up stats. */
	averunnable.fscale = FSCALE;

	/*
	 * step 2: init the page sub-system.  this includes allocating the
	 * vm_page structures, and setting up all the page queues (and
	 * locks).  available memory will be put in the "free" queue.
	 * kvm_start and kvm_end will be set to the area of kernel virtual
	 * memory which is available for general use.
	 */
	uvm_page_init(&kvm_start, &kvm_end);

	/*
	 * step 3: init the map sub-system.  allocates the static pool of
	 * vm_map_entry structures that are used for "special" kernel maps
	 * (e.g. kernel_map, kmem_map, etc...).
	 */
	uvm_map_init();

	/*
	 * step 4: setup the kernel's virtual memory data structures.  this
	 * includes setting up the kernel_map/kernel_object and the kmem_map/
	 * kmem_object.
	 */

	uvm_km_init(vm_min_kernel_address, kvm_start, kvm_end);

	/*
	 * step 4.5: init (tune) the fault recovery code.
	 */
	uvmfault_init();

	/*
	 * step 5: init the pmap module.   the pmap module is free to allocate
	 * memory for its private use (e.g. pvlists).
	 */
	pmap_init();

	/*
	 * step 6: init the kernel memory allocator.   after this call the
	 * kernel memory allocator (malloc) can be used.
	 */
	kmeminit();

	/*
	 * step 6.5: init the dma allocator, which is backed by pools.
	 */
	dma_alloc_init();

	/*
	 * step 7: init all pagers and the pager_map.
	 */
	uvm_pager_init();

	/*
	 * step 8: init anonymous memory system
	 */
	amap_init();

	/*
	 * step 9: init uvm_km_page allocator memory.
	 */
	uvm_km_page_init();

	/*
	 * the VM system is now up!  now that malloc is up we can
	 * enable paging of kernel objects.
	 */
	uao_create(VM_KERNEL_SPACE_SIZE, UAO_FLAG_KERNSWAP);

	/*
	 * reserve some unmapped space for malloc/pool use after free usage
	 */
#ifdef DEADBEEF0
	kvm_start = trunc_page(DEADBEEF0) - PAGE_SIZE;
	if (uvm_map(kernel_map, &kvm_start, 3 * PAGE_SIZE,
	    NULL, UVM_UNKNOWN_OFFSET, 0, UVM_MAPFLAG(PROT_NONE,
	    PROT_NONE, MAP_INHERIT_NONE, MADV_RANDOM, UVM_FLAG_FIXED)))
		panic("uvm_init: cannot reserve dead beef @0x%x", DEADBEEF0);
#endif
#ifdef DEADBEEF1
	kvm_start = trunc_page(DEADBEEF1) - PAGE_SIZE;
	if (uvm_map(kernel_map, &kvm_start, 3 * PAGE_SIZE,
	    NULL, UVM_UNKNOWN_OFFSET, 0, UVM_MAPFLAG(PROT_NONE,
	    PROT_NONE, MAP_INHERIT_NONE, MADV_RANDOM, UVM_FLAG_FIXED)))
		panic("uvm_init: cannot reserve dead beef @0x%x", DEADBEEF1);
#endif
	/*
	 * init anonymous memory systems
	 */
	uvm_anon_init();

#ifndef SMALL_KERNEL
	/*
	 * Switch kernel and kmem_map over to a best-fit allocator,
	 * instead of walking the tree.
	 */
	uvm_map_set_uaddr(kernel_map, &kernel_map->uaddr_any[3],
	    uaddr_bestfit_create(vm_map_min(kernel_map),
	    vm_map_max(kernel_map)));
	uvm_map_set_uaddr(kmem_map, &kmem_map->uaddr_any[3],
	    uaddr_bestfit_create(vm_map_min(kmem_map),
	    vm_map_max(kmem_map)));
#endif /* !SMALL_KERNEL */
}

As far as I can tell, none of these _init() functions does anything like map all of physical memory into the kernel’s virtual address space.

Data Structures

For the upper level, we have the following from Cranor’s Usenix paper about UVM:

We’ll just go through and look at each of the struct definitions for now.

The vmspace struct is defined in src/sys/uvm/uvm_extern.h:

/*
 * Shareable process virtual address space.
 * May eventually be merged with vm_map.
 * Several fields are temporary (text, data stuff).
 */
struct vmspace {
	struct	vm_map vm_map;	/* VM address map */
	int	vm_refcnt;	/* number of references */
	caddr_t	vm_shm;		/* SYS5 shared memory private data XXX */
/* we copy from vm_startcopy to the end of the structure on fork */
#define vm_startcopy vm_rssize
	segsz_t vm_rssize; 	/* current resident set size in pages */
	segsz_t vm_swrss;	/* resident set size before last swap */
	segsz_t vm_tsize;	/* text size (pages) XXX */
	segsz_t vm_dsize;	/* data size (pages) XXX */
	segsz_t vm_dused;	/* data segment length (pages) XXX */
	segsz_t vm_ssize;	/* stack size (pages) */
	caddr_t	vm_taddr;	/* user virtual address of text XXX */
	caddr_t	vm_daddr;	/* user virtual address of data XXX */
	caddr_t vm_maxsaddr;	/* user VA at max stack growth */
	caddr_t vm_minsaddr;	/* user VA at top of stack */
};

The vm_map struct is defined in src/sys/uvm/uvm_map.h:

/*
 *	A Map is a rbtree of map entries, kept sorted by address.
 *	In addition, free space entries are also kept in a rbtree,
 *	indexed by free size.
 *
 *
 *
 *	LOCKING PROTOCOL NOTES:
 *	-----------------------
 *
 *	VM map locking is a little complicated.  There are both shared
 *	and exclusive locks on maps.  However, it is sometimes required
 *	to downgrade an exclusive lock to a shared lock, and upgrade to
 *	an exclusive lock again (to perform error recovery).  However,
 *	another thread *must not* queue itself to receive an exclusive
 *	lock while before we upgrade back to exclusive, otherwise the
 *	error recovery becomes extremely difficult, if not impossible.
 *
 *	In order to prevent this scenario, we introduce the notion of
 *	a `busy' map.  A `busy' map is read-locked, but other threads
 *	attempting to write-lock wait for this flag to clear before
 *	entering the lock manager.  A map may only be marked busy
 *	when the map is write-locked (and then the map must be downgraded
 *	to read-locked), and may only be marked unbusy by the thread
 *	which marked it busy (holding *either* a read-lock or a
 *	write-lock, the latter being gained by an upgrade).
 *
 *	Access to the map `flags' member is controlled by the `flags_lock'
 *	simple lock.  Note that some flags are static (set once at map
 *	creation time, and never changed), and thus require no locking
 *	to check those flags.  All flags which are r/w must be set or
 *	cleared while the `flags_lock' is asserted.  Additional locking
 *	requirements are:
 *
 *		VM_MAP_PAGEABLE		r/o static flag; no locking required
 *
 *		VM_MAP_INTRSAFE		r/o static flag; no locking required
 *
 *		VM_MAP_WIREFUTURE	r/w; may only be set or cleared when
 *					map is write-locked.  may be tested
 *					without asserting `flags_lock'.
 *
 *		VM_MAP_BUSY		r/w; may only be set when map is
 *					write-locked, may only be cleared by
 *					thread which set it, map read-locked
 *					or write-locked.  must be tested
 *					while `flags_lock' is asserted.
 *
 *		VM_MAP_WANTLOCK		r/w; may only be set when the map
 *					is busy, and thread is attempting
 *					to write-lock.  must be tested
 *					while `flags_lock' is asserted.
 *
 *		VM_MAP_GUARDPAGES	r/o; must be specified at map
 *					initialization time.
 *					If set, guards will appear between
 *					automatic allocations.
 *					No locking required.
 *
 *		VM_MAP_ISVMSPACE	r/o; set by uvmspace_alloc.
 *					Signifies that this map is a vmspace.
 *					(The implementation treats all maps
 *					without this bit as kernel maps.)
 *					No locking required.
 *
 *
 * All automatic allocations (uvm_map without MAP_FIXED) will allocate
 * from vm_map.free.
 * If that allocation fails:
 * - vmspace maps will spill over into vm_map.bfree,
 * - all other maps will call uvm_map_kmem_grow() to increase the arena.
 * 
 * vmspace maps have their data, brk() and stack arenas automatically
 * updated when uvm_map() is invoked without MAP_FIXED.
 * The spill over arena (vm_map.bfree) will contain the space in the brk()
 * and stack ranges.
 * Kernel maps never have a bfree arena and this tree will always be empty.
 *
 *
 * read_locks and write_locks are used in lock debugging code.
 */
struct vm_map {
	struct pmap *		pmap;		/* Physical map */
	struct rwlock		lock;		/* Lock for map data */
	struct mutex		mtx;

	struct uvm_map_addr	addr;		/* Entry tree, by addr */

	vsize_t			size;		/* virtual size */
	int			ref_count;	/* Reference count */
	int			flags;		/* flags */
	struct mutex		flags_lock;	/* flags lock */
	unsigned int		timestamp;	/* Version number */

	vaddr_t			min_offset;	/* First address in map. */
	vaddr_t			max_offset;	/* Last address in map. */

	/*
	 * Allocation overflow regions.
	 */
	vaddr_t			b_start;	/* Start for brk() alloc. */
	vaddr_t			b_end;		/* End for brk() alloc. */
	vaddr_t			s_start;	/* Start for stack alloc. */
	vaddr_t			s_end;		/* End for stack alloc. */

	/*
	 * Special address selectors.
	 *
	 * The uaddr_exe mapping is used if:
	 * - protX is selected
	 * - the pointer is not NULL
	 *
	 * If uaddr_exe is not used, the other mappings are checked in
	 * order of appearance.
	 * If a hint is given, the selection will only be used if the hint
	 * falls in the range described by the mapping.
	 *
	 * The states are pointers because:
	 * - they may not all be in use
	 * - the struct size for different schemes is variable
	 *
	 * The uaddr_brk_stack selector will select addresses that are in
	 * the brk/stack area of the map.
	 */
	struct uvm_addr_state	*uaddr_exe;	/* Executable selector. */
	struct uvm_addr_state	*uaddr_any[4];	/* More selectors. */
	struct uvm_addr_state	*uaddr_brk_stack; /* Brk/stack selector. */
};

The reference to uvm_map_addr is actually a reference to a red-black tree defined earlier:

RB_HEAD(uvm_map_addr, vm_map_entry);

The RB_HEAD call is a standard part of NetBSD/OpenBSD, and we can look at the relevant man page:

RED-BLACK TREES
     A red-black tree is a binary search tree with the node color as an extra
     attribute.  It fulfills a set of conditions:
           1.   every search path from the root to a leaf consists of the same
                number of black nodes,
           2.   each red node (except for the root) has a black parent,
           3.   each leaf node is black.

     Every operation on a red-black tree is bounded as O(lg n).  The maximum
     height of a red-black tree is 2lg (n+1).

     A red-black tree is headed by a structure defined by the RB_HEAD() macro.
     A RB_HEAD structure is declared as follows:

           RB_HEAD(HEADNAME, TYPE) head;

     where HEADNAME is the name of the structure to be defined, and struct
     TYPE is the type of the elements to be inserted into the tree.

So the type of elements stored in the uvm_map_addr red-black tree is vm_map_entry which is defined earlier in the same file as vm_map:

/*
 * Address map entries consist of start and end addresses,
 * a VM object (or sharing map) and offset into that object,
 * and user-exported inheritance and protection information.
 * Also included is control information for virtual copy operations.
 */
struct vm_map_entry {
	union {
		RB_ENTRY(vm_map_entry)	addr_entry; /* address tree */
	} daddrs;

	union {
		RB_ENTRY(vm_map_entry)	rbtree;	/* Link freespace tree. */
		TAILQ_ENTRY(vm_map_entry) tailq;/* Link freespace queue. */
		TAILQ_ENTRY(vm_map_entry) deadq;/* dead entry queue */
	} dfree;

#define uvm_map_entry_start_copy start
	vaddr_t			start;		/* start address */
	vaddr_t			end;		/* end address */

	vsize_t			guard;		/* bytes in guard */
	vsize_t			fspace;		/* free space */

	union vm_map_object	object;		/* object I point to */
	voff_t			offset;		/* offset into object */
	struct vm_aref		aref;		/* anonymous overlay */

	int			etype;		/* entry type */

	vm_prot_t		protection;	/* protection code */
	vm_prot_t		max_protection;	/* maximum protection */
	vm_inherit_t		inheritance;	/* inheritance */

	int			wired_count;	/* can be paged if == 0 */
	int			advice;		/* madvise advice */
#define uvm_map_entry_stop_copy flags
	u_int8_t		flags;		/* flags */

#define UVM_MAP_STATIC		0x01		/* static map entry */
#define UVM_MAP_KMEM		0x02		/* from kmem entry pool */

	vsize_t			fspace_augment;	/* max(fspace) in subtree */
};

Notice also that a vm_map_entry has makes a couple of TAILQ_ENTRY calls, which create another standard data structure, this time a tail queue:

...
     TAILQ_ENTRY(TYPE);

     TAILQ_HEAD(HEADNAME, TYPE);
...
DESCRIPTION    [Toc]    [Back]

     These macros define and operate on five types of data structures: singlylinked
  lists, simple queues, lists, tail queues, and circular queues.
     All five structures support the following functionality:

           1.   Insertion of a new entry at the head of the list.
           2.   Insertion of a new entry after any element in the
list.
           3.   Removal of an entry from the head of the list.
           4.   Forward traversal through the list.
...
     Tail queues add the following functionality:

           1.   Entries can be added at the end of a list.
           2.   They may be traversed backwards, at a cost.

     However:

           1.   All list insertions and removals must specify the
head of the
                list.
           2.   Each head entry requires two pointers rather than
one.
           3.   Code size is about 15% greater and operations run
about 20%
                slower than singly-linked lists.

Exactly what the “dead” entry queue does, I’m not sure. If we look at the same struct in NetBSD:

/*
 * Address map entries consist of start and end addresses,
 * a VM object (or sharing map) and offset into that object,
 * and user-exported inheritance and protection information.
 * Also included is control information for virtual copy operations.
 */
struct vm_map_entry {
	struct rb_node		rb_node;	/* tree information */
	vsize_t			gap;		/* free space after */
	vsize_t			maxgap;		/* space in subtree */
	struct vm_map_entry	*prev;		/* previous entry */
	struct vm_map_entry	*next;		/* next entry */
	vaddr_t			start;		/* start address */
	vaddr_t			end;		/* end address */
	union {
		struct uvm_object *uvm_obj;	/* uvm object */
		struct vm_map	*sub_map;	/* belongs to another map */
	} object;				/* object I point to */
	voff_t			offset;		/* offset into object */
	int			etype;		/* entry type */
	vm_prot_t		protection;	/* protection code */
	vm_prot_t		max_protection;	/* maximum protection */
	vm_inherit_t		inheritance;	/* inheritance */
	int			wired_count;	/* can be paged if == 0 */
	struct vm_aref		aref;		/* anonymous overlay */
	int			advice;		/* madvise advice */
	uint32_t		map_attrib;	/* uvm-external map attributes */
#define uvm_map_entry_stop_copy flags
	u_int8_t		flags;		/* flags */

#define	UVM_MAP_KERNEL		0x01		/* kernel map entry */
#define	UVM_MAP_STATIC		0x04		/* special static entries */
#define	UVM_MAP_NOMERGE		0x08		/* this entry is not mergable */

};

We can see that it doesn’t have a dead entry queue at all. :/

The vm_map_object union is defined earlier in the same file:

/*
 * Objects which live in maps may be either VM objects, or another map
 * (called a "sharing map") which denotes read-write sharing with other maps.
 *
 * XXXCDC: private pager data goes here now
 */

union vm_map_object {
	struct uvm_object	*uvm_obj;	/* UVM OBJECT */
	struct vm_map		*sub_map;	/* belongs to another map */
};

The uvm_object struct is defined in src/sys/uvm/uvm_object.h

/*
 * uvm_object: all that is left of mach objects.
 */

struct uvm_object {
	struct uvm_pagerops		*pgops;		/* pager ops */
	RB_HEAD(uvm_objtree, vm_page)	 memt;		/* pages in object */
	int				 uo_npages;	/* # of pages in memt */
	int				 uo_refs;	/* reference count */
};

From the definition of uvm_object we can see that it also contains a red-black tree of vm_page objects, which are defined in src/sys/uvm/uvm_page.h:

/*
 *	Management of resident (logical) pages.
 *
 *	A small structure is kept for each resident
 *	page, indexed by page number.  Each structure
 *	contains a list used for manipulating pages, and
 *	a tree structure for in object/offset lookups
 *
 *	In addition, the structure contains the object
 *	and offset to which this page belongs (for pageout),
 *	and sundry status bits.
 *
 *	Fields in this structure are possibly locked by the lock on the page
 *	queues (P).
 */

TAILQ_HEAD(pglist, vm_page);

struct vm_page {
	TAILQ_ENTRY(vm_page)	pageq;		/* queue info for FIFO
						 * queue or free list (P) */
	RB_ENTRY(vm_page)	objt;		/* object tree */

	struct vm_anon		*uanon;		/* anon (P) */
	struct uvm_object	*uobject;	/* object (P) */
	voff_t			offset;		/* offset into object (P) */

	u_int			pg_flags;	/* object flags [P] */

	u_int			pg_version;	/* version count */
	u_int			wire_count;	/* wired down map refs [P] */

	paddr_t			phys_addr;	/* physical address of page */
	psize_t			fpgsz;		/* free page range size */

	struct vm_page_md	mdpage;		/* pmap-specific data */

#if defined(UVM_PAGE_TRKOWN)
	/* debugging fields to track page ownership */
	pid_t			owner;		/* proc that set PG_BUSY */
	char			*owner_tag;	/* why it was set busy */
#endif
};

The lowest level in the above diagram shows a “pager” object, which corresponds to the uvm_pagerops struct, which is defined in src/sys/uvm/uvm_pager.h:

struct uvm_pagerops {
						/* init pager */
	void			(*pgo_init)(void);
						/* add reference to obj */
	void			(*pgo_reference)(struct uvm_object *);
						/* drop reference to obj */
	void			(*pgo_detach)(struct uvm_object *);
						/* special nonstd fault fn */
	int			(*pgo_fault)(struct uvm_faultinfo *, vaddr_t,
				 vm_page_t *, int, int, vm_fault_t,
				 vm_prot_t, int);
						/* flush pages out of obj */
	boolean_t		(*pgo_flush)(struct uvm_object *, voff_t,
				 voff_t, int);
						/* get/read page */
	int			(*pgo_get)(struct uvm_object *, voff_t,
				 vm_page_t *, int *, int, vm_prot_t, int, int);
						/* put/write page */
	int			(*pgo_put)(struct uvm_object *, vm_page_t *,
				 int, boolean_t);
						/* return range of cluster */
	void			(*pgo_cluster)(struct uvm_object *, voff_t,
				 voff_t *, voff_t *);
						/* make "put" cluster */
	struct vm_page **	(*pgo_mk_pcluster)(struct uvm_object *,
				 struct vm_page **, int *, struct vm_page *,
				 int, voff_t, voff_t);
};

As we can see, this effectively implements a kind of multiple dispatch in which the underlying “pager” implements these functions and stores pointers to them in a uvm_pagerops struct.

Conclusion

TODO

Aside: x86 MMU

The x86 implementation of pmap, located in src/sys/arch/i386/i386/pmap.c contains some nice documentation for the x86 MMU:

/*
 * this file contains the code for the "pmap module."   the module's
 * job is to manage the hardware's virtual to physical address mappings.
 * note that there are two levels of mapping in the VM system:
 *
 *  [1] the upper layer of the VM system uses vm_map's and vm_map_entry's
 *      to map ranges of virtual address space to objects/files.  for
 *      example, the vm_map may say: "map VA 0x1000 to 0x22000 read-only
 *      to the file /bin/ls starting at offset zero."   note that
 *      the upper layer mapping is not concerned with how individual
 *      vm_pages are mapped.
 *
 *  [2] the lower layer of the VM system (the pmap) maintains the mappings
 *      from virtual addresses.   it is concerned with which vm_page is
 *      mapped where.   for example, when you run /bin/ls and start
 *      at page 0x1000 the fault routine may lookup the correct page
 *      of the /bin/ls file and then ask the pmap layer to establish
 *      a mapping for it.
 *
 * note that information in the lower layer of the VM system can be
 * thrown away since it can easily be reconstructed from the info
 * in the upper layer.
 *
 * data structures we use include:
 *
 *  - struct pmap: describes the address space of one thread
 *  - struct pv_entry: describes one <PMAP,VA> mapping of a PA
 *  - struct pv_head: there is one pv_head per managed page of
 *	physical memory.   the pv_head points to a list of pv_entry
 *	structures which describe all the <PMAP,VA> pairs that this
 *      page is mapped in.    this is critical for page based operations
 *      such as pmap_page_protect() [change protection on _all_ mappings
 *      of a page]
 */
/*
 * i386 MMU hardware structure:
 *
 * the i386 MMU is a two-level MMU which maps 4GB of virtual memory.
 * the pagesize is 4K (4096 [0x1000] bytes), although newer pentium
 * processors can support a 4MB pagesize as well.
 *
 * the first level table (segment table?) is called a "page directory"
 * and it contains 1024 page directory entries (PDEs).   each PDE is
 * 4 bytes (an int), so a PD fits in a single 4K page.   this page is
 * the page directory page (PDP).  each PDE in a PDP maps 4MB of space
 * (1024 * 4MB = 4GB).   a PDE contains the physical address of the
 * second level table: the page table.   or, if 4MB pages are being used,
 * then the PDE contains the PA of the 4MB page being mapped.
 *
 * a page table consists of 1024 page table entries (PTEs).  each PTE is
 * 4 bytes (an int), so a page table also fits in a single 4K page.  a
 * 4K page being used as a page table is called a page table page (PTP).
 * each PTE in a PTP maps one 4K page (1024 * 4K = 4MB).   a PTE contains
 * the physical address of the page it maps and some flag bits (described
 * below).
 *
 * the processor has a special register, "cr3", which points to the
 * the PDP which is currently controlling the mappings of the virtual
 * address space.
 *
 * the following picture shows the translation process for a 4K page:
 *
 * %cr3 register [PA of PDP]
 *      |
 *      |
 *      |   bits <31-22> of VA         bits <21-12> of VA   bits <11-0>
 *      |   index the PDP (0 - 1023)   index the PTP        are the page offset
 *      |         |                           |                  |
 *      |         v                           |                  |
 *      +--->+----------+                     |                  |
 *           | PD Page  |   PA of             v                  |
 *           |          |---PTP-------->+------------+           |
 *           | 1024 PDE |               | page table |--PTE--+   |
 *           | entries  |               | (aka PTP)  |       |   |
 *           +----------+               | 1024 PTE   |       |   |
 *                                      | entries    |       |   |
 *                                      +------------+       |   |
 *                                                           |   |
 *                                                bits <31-12>   bits <11-0>
 *                                                p h y s i c a l  a d d r
 *
 * the i386 caches PTEs in a TLB.   it is important to flush out old
 * TLB mappings when making a change to a mapping.   writing to the
 * %cr3 will flush the entire TLB.    newer processors also have an
 * instruction that will invalidate the mapping of a single page (which
 * is useful if you are changing a single mapping because it preserves
 * all the cached TLB entries).
 *
 * as shows, bits 31-12 of the PTE contain PA of the page being mapped.
 * the rest of the PTE is defined as follows:
 *   bit#	name	use
 *   11		n/a	available for OS use, hardware ignores it
 *   10		n/a	available for OS use, hardware ignores it
 *   9		n/a	available for OS use, hardware ignores it
 *   8		G	global bit (see discussion below)
 *   7		PS	page size [for PDEs] (0=4k, 1=4M <if supported>)
 *   6		D	dirty (modified) page
 *   5		A	accessed (referenced) page
 *   4		PCD	cache disable
 *   3		PWT	prevent write through (cache)
 *   2		U/S	user/supervisor bit (0=supervisor only, 1=both u&s)
 *   1		R/W	read/write bit (0=read only, 1=read-write)
 *   0		P	present (valid)
 *
 * notes:
 *  - on the i386 the R/W bit is ignored if processor is in supervisor
 *    state (bug!)
 *  - PS is only supported on newer processors
 *  - PTEs with the G bit are global in the sense that they are not
 *    flushed from the TLB when %cr3 is written (to flush, use the
 *    "flush single page" instruction).   this is only supported on
 *    newer processors.    this bit can be used to keep the kernel's
 *    TLB entries around while context switching.   since the kernel
 *    is mapped into all processes at the same place it does not make
 *    sense to flush these entries when switching from one process'
 *    pmap to another.
 */
/*
 * A pmap describes a process' 4GB virtual address space.  This
 * virtual address space can be broken up into 1024 4MB regions which
 * are described by PDEs in the PDP.  The PDEs are defined as follows:
 *
 * Ranges are inclusive -> exclusive, just like vm_map_entry start/end.
 * The following assumes that KERNBASE is 0xd0000000.
 *
 * PDE#s	VA range		Usage
 * 0->831	0x0 -> 0xcfc00000	user address space, note that the
 *					max user address is 0xcfbfe000
 *					the final two pages in the last 4MB
 *					used to be reserved for the UAREA
 *					but now are no longer used.
 * 831		0xcfc00000->		recursive mapping of PDP (used for
 *			0xd0000000	linear mapping of PTPs).
 * 832->1023	0xd0000000->		kernel address space (constant
 *			0xffc00000	across all pmaps/processes).
 * 1023		0xffc00000->		"alternate" recursive PDP mapping
 *			<end>		(for other pmaps).
 *
 *
 * Note: A recursive PDP mapping provides a way to map all the PTEs for
 * a 4GB address space into a linear chunk of virtual memory.  In other
 * words, the PTE for page 0 is the first int mapped into the 4MB recursive
 * area.  The PTE for page 1 is the second int.  The very last int in the
 * 4MB range is the PTE that maps VA 0xffffe000 (the last page in a 4GB
 * address).
 *
 * All pmaps' PDs must have the same values in slots 832->1023 so that
 * the kernel is always mapped in every process.  These values are loaded
 * into the PD at pmap creation time.
 *
 * At any one time only one pmap can be active on a processor.  This is
 * the pmap whose PDP is pointed to by processor register %cr3.  This pmap
 * will have all its PTEs mapped into memory at the recursive mapping
 * point (slot #831 as show above).  When the pmap code wants to find the
 * PTE for a virtual address, all it has to do is the following:
 *
 * Address of PTE = (831 * 4MB) + (VA / PAGE_SIZE) * sizeof(pt_entry_t)
 *                = 0xcfc00000 + (VA / 4096) * 4
 *
 * What happens if the pmap layer is asked to perform an operation
 * on a pmap that is not the one which is currently active?  In that
 * case we take the PA of the PDP of the non-active pmap and put it in
 * slot 1023 of the active pmap.  This causes the non-active pmap's
 * PTEs to get mapped in the final 4MB of the 4GB address space
 * (e.g. starting at 0xffc00000).
 *
 * The following figure shows the effects of the recursive PDP mapping:
 *
 *   PDP (%cr3)
 *   +----+
 *   |   0| -> PTP#0 that maps VA 0x0 -> 0x400000
 *   |    |
 *   |    |
 *   | 831| -> points back to PDP (%cr3) mapping VA 0xcfc00000 -> 0xd0000000
 *   | 832| -> first kernel PTP (maps 0xd0000000 -> 0xe0400000)
 *   |    |
 *   |1023| -> points to alternate pmap's PDP (maps 0xffc00000 -> end)
 *   +----+
 *
 * Note that the PDE#831 VA (0xcfc00000) is defined as "PTE_BASE".
 * Note that the PDE#1023 VA (0xffc00000) is defined as "APTE_BASE".
 *
 * Starting at VA 0xcfc00000 the current active PDP (%cr3) acts as a
 * PTP:
 *
 * PTP#831 == PDP(%cr3) => maps VA 0xcfc00000 -> 0xd0000000
 *   +----+
 *   |   0| -> maps the contents of PTP#0 at VA 0xcfc00000->0xcfc01000
 *   |    |
 *   |    |
 *   | 831| -> maps the contents of PTP#831 (the PDP) at VA 0xcff3f000
 *   | 832| -> maps the contents of first kernel PTP
 *   |    |
 *   |1023|
 *   +----+
 *
 * Note that mapping of the PDP at PTP#831's VA (0xcff3f000) is
 * defined as "PDP_BASE".... within that mapping there are two
 * defines:
 *   "PDP_PDE" (0xcff3fcfc) is the VA of the PDE in the PDP
 *      which points back to itself.
 *   "APDP_PDE" (0xcff3fffc) is the VA of the PDE in the PDP which
 *      establishes the recursive mapping of the alternate pmap.
 *      To set the alternate PDP, one just has to put the correct
 *	PA info in *APDP_PDE.
 *
 * Note that in the APTE_BASE space, the APDP appears at VA
 * "APDP_BASE" (0xfffff000).
 */

Attribution

The diagram for UVM’s data structures comes from Charles Craynor’s paper at Usenix in 1999, of which there is also a PDF version.

RadixVM

Sun, 07 Feb 2016 00:00:00 +0000

So far this blog has investigated the virtual memory interface in POSIX, and dived into some of the details of how it’s implemented in BSD. Today, we’ll look at the state of the art in academic virtual memory design, in particular a design called RadixVM by Austin T. Clements, M. Frans Kaashoek, Nickolai Zeldovich at MIT’s CSAIL.

Background

Recall that virtual memory allows us to pretend that each process has exclusive access to a huge amount of (virtual) memory. Each fixed-size page of this virtual address space is usually stored either resident in an equal-sized frame of physical memory, or swapped out to disk. A process’ operations on these virtual addresses are passed through a translation lookaside buffer (TLB) that caches the actual physical memory locations for each virtual page. In the case where an entry for a page is not found in the TLB, a page table is consulted. If the page is not presently loaded in physical memory (referred to as a page fault), it is loaded before continuing. Finally, the TLB is updated with the address and the process continues operating in the virtual address space.

These days, memory allocation such as a call to malloc is often handled by mapping an anonymous region in memory using an underlying call to mmap. One of the most well-known malloc implementations, jemalloc, explicitly states in its documentation that it prefers to call mmap over sbrk. When such regions are unmapped, it’s important to immediately flush their entries in the TLB otherwise a dangling pointer referring to that allocation may end up modifying a region of memory which was already reallocated. This process is called a TLB shootdown. When multiple threads of the same process are spread across multiple cores, and one thread unmaps the region, any core running threads from this process needs to have its TLB flushed. This is called remote TLB shootdown.

Problem

Unfortunately, most operating systems serialize calls to mmap and munmap (single lock per shared address space). mmap and munmap should be perfectly parallelizable since they should be operating on different parts of the address space.

Operating systems typically use balanced trees to keep track of mapped memory regions within a process’ address space. Linux uses red/black trees, FreeBSD uses splay trees, Solairs and Windows use AVL trees. Since these data structures require rebalancing on insert and delete (to maintain O(log n) height), they use a single lock to serialize changes to the entire data structure.

RadixVM

RadixVM attempts to solve this problem. RadixVM has 3 novel parts: its radix-tree-based data structure for tracking mapped memory, its method of avoiding remote TLB shootdowns, and its memory-efficient distributed reference counting scheme.

Refcache: Reference Counting

Reference counting is a common form of memory management in which memory is automatically freed when the number of references to a memory region drops to zero.

The RadixVM paper introduces a memory-efficient distributed reference counting scheme, called Refcache. Roughly speaking, refcache counts references to a memory location across possibly many cores. Refcache divides time into fixed-length epochs (the canonical implementation uses an epoch of length 10ms), and only frees unreferenced memory after its reference count has dropped to zero and remains at zero for an entire epoch.

Data Structure

Radix tree is a type of data structure better known as a prefix tree, or a trie. This data structure is a tree for storing strings of characters from a set called the alphabet. A string is represented as a leaf in the tree by its root to leaf path, each edge of which is labeled with a sequence of characters. At each node along this path, the concatenation of the edges of the root to node path form a prefix shared by all children of this node.

In the context of operating systems, we normally use a variant of a radix tree which is built on bit strings has fixed depth. That is, each outgoing edge from a level has the same number of bits. This is very commonly used to represent a hierarchical page table, indexed by a 32 or 64-bit word, whose first few bits are the index into the highest level page table, the next few bits are the index into the next-highest level page table, etc.

RadixVM uses a data structure based on a radix tree, very similar to a typical page table, in which each level is indexed by 9 (or fewer) bits. Unlike a typical page table, RadixVM’s data structure stores a separate copy of the mapping metadata in the radix tree for each page in the mapped range.

Avoiding Remote TLB Shootdowns

Each core has its own TLB, and the x86 architecture doesn’t inform the kernel about its contents. Therefore, most operating systems perform the simplest possible kind of TLB shootdown when memory is unmapped, which is to shootdown the TLB on every core. To avoid remote TLB shootdowns, RadixVM uses a scheme that tracks which cores have what mappings, and only performs remote TLB shootdowns when necessary.

Implementation

The authors learned in a previous paper about a lock-free VM implementation called BonsaiVM, that big operating systems like Linux have a high level of coupling between the VM system and other parts of the operating system. This means that implementing a novel VM design can be a monumental task. So instead, the authors chose to implement radixvm on xv6. xv6 is a recently rewritten variant of Unix v6 ported to modern hardware for academic use at MIT.

Performance

The authors were able to compare the behavior of Metis, a single-server multithreaded MapReduce library running on RadixVM, Bonsai, and regular linux. The Metis workload stresses concurrent mmaps and pagefaults, but not munmaps. In their experiments, RadixVM and Bonsai (only with 8MB block sizes) scale very well. Linux/64kb seems to peak at 10 cores before dropping off. Linux/8MB and Bonsai/64kb drop off after 20 cores.

Since most large, complex applications have been built around the traditional limitations of mmap and munmap, they don’t allow for good measurements of improvements to mmap/munmap. Thus, the authors were forced to rely on microbenchmarks to measure performance in other workloads. According to these microbenchmarks on the workloads they tested, only RadixVM really scales.

Do we really need all 3 parts?

One final interesting thing to note about RadixVM is that the authors consider very strongly whether all 3 parts of their design are necessary. Of course, they determine that yes, all parts of their design are necessary. But it is still good that they spent the time to explain why.

Attribution

Patricia trie diagram by Saffles (Microsoft Visio) CC BY-SA 3.0, via Wikimedia Commons.

Page table diagram by RokerHRO (eigene Arbeit / own work using Xfig.) CC BY-SA 3.0, via Wikimedia Commons.

BSD Virtual Memory

Tue, 02 Feb 2016 00:00:00 +0000

After the article on POSIX memory management, we have a good grasp of the high-level memory allocation functions specified in POSIX.1. We learned how malloc is a C library function which can be implemented using mmap. Now it’s time to dive into the details about how mmap is implemented.

Under the hood, mmap touches a fundamental concept in operating systems. Specifically, it must interface with the kernel’s implementation of virtual memory. Virtual memory is the way in which we give each process the illusion that it is the only running process and it has full access to the system’s memory.

There are 3 main ways to virtualize memory: segmentation, paging, and swapping. Segmentation builds on the concept we touched on in the previous article, that each process is split into several “segments,” such as heap, stack and text. These segments can be placed independently in physical memory and with hardware support their virtual addresses could be translated to physical ones. However, keeping these segments contiguous in memory can be a lot of overhead. For that reason, paging divides virtual memory into fixed-size pages, which are placed independently in fixed-size frames in physical memory. Finally, swapping moves allocated but inactive parts of memory to disk to free up space for active processes.

POSIX.1 doesn’t specify implementation details, so we have to pick a particular operating system to investigate. For no particularly good reason, we’re going with BSD, and we’ll be looking at NetBSD and FreeBSD in particular. We’ll be looking at those two BSDs in particular for historical reasons, so it’s worth taking a brief aside to go over that history.

BSD History

UNIX was started at Bell Labs in 1969. In 1974, the University of California at Berkeley received a copy of UNIX which they ran on a PDP-11. In 1975, they installed UNIX v6. By 1977, they had written enough software in which other Universities were interested that they started bundling it together as 1BSD.

In 1978, a VAX computer was installed at Berkeley, and though the original UNIX for the PDP-11 supported paging and swapping, the UNIX version ported to this machine, UNIX/32V, was released without paging. So 2BSD included a largely rewritten kernel for the VAX which included support for paging.

In 1985, a group at Carnegie Mellon University began work on a research kernel called Mach that would replace the BSD kernel. Importantly, Mach was designed to be platform-independent (as opposed to all previous versions of BSD or UNIX). This decision likely led to the good kernel design that inspired the design of several future BSD variants.

In 1988, the decision was finally made to move BSD away from VAX and cleanly separate its machine-dependent and machine-independent code. 4.3BSD-Tahoe was the interrim BSD release for the Power 6/32 platform which put this into practice, though the port for that platform was very quickly abandoned.

Also in 1988, the Single UNIX Specification officially became the first version of POSIX, IEEE 1003.1-1988.

By 1989, much of BSD relied on proprietary UNIX code which relied on having an AT&T software license, which was becoming increasingly expensive. Thus began an effort to rewrite all the proprietary dependencies which became BSD Net/1.

In 1990, 4.3BSD-Reno, another interrim version, was released which pushed towards POSIX compliance. By 1991, most of the AT&T code had been replaced, and Net/2 was released. Net/2 used the VM system from Mach 2.5.

In 1992, 4.3BSD-Reno and Net/2 were used as the basis of a port to the Intel 386 processor, which was called 386BSD. Notably, this finally replaced all the proprietary code that was still left in Net/2. Also in 1992, BSD/386, a commercial port to the 386, was also released, but even before its release AT&T filed a lawsuit against them for breach of their copyright.

In 1993, 386BSD was forked into FreeBSD and NetBSD. By 1994, 386BSD was abandoned. Also, the AT&T lawsuit was resolved, and 4.4BSD-Lite was released, containing no AT&T source code. This release incorporated many of Mach’s design decisions and were largely adopted by both FreeBSD and NetBSD over the following years. In 1996, NetBSD was forked into OpenBSD.

In 1998, Charles Cranor wrote a dissertation on the design of UVM, a virtual memory system which was to replace Mach’s in NetBSD. Much of this was also ported to OpenBSD.

In 2004, version 4.8 of FreeBSD was forked into DragonFly BSD.

Virtual Memory Systems

So finally, we can talk about virtual memory systems. We can summarize the above by saying that there are 2 main BSD variants today: FreeBSD and NetBSD. Each of which has its own sub-variant: DragonFly BSD and OpenBSD, respectively. FreeBSD and DragonFly BSD have virtual memory systems based on Mach (or 4.4BSD-Lite if you prefer), and NetBSD and OpenBSD have virtual memory systems based on UVM.

FreeBSD (Mach)

Data Structures

The FreeBSD kernel stores address space information about each process in a vmspace structure, which encapsulates both machine-dependent and machine-independent information. We describe here only the machine-independent data structures. Each vmspace structure contains a vm_map structure, which points to an ordered linked-list of vm_map_entry structure objects.

Each vm_map_entry describes a contiguous region of virtual memory and contains a pointer to a next entry, and to a chain of vm_object structures. Each contiguous region of virtual memory pointed to by a vm_map_entry has the same attributes (such as protection), so these are also stored in this structure. Between a vm_map_entry and its chain of vm_objects, there are zero or more “shadow” vm_objects which track changes to the vm_object, each of which stores a pointer to a vm_page containing the changes and another pointer to the unmodified vm_object.

A vm_object structure contains a pointer to another linked list of vm_page structures which represent the physical memory cache of the vm_object. The vm_page also keeps a radix tree of these vm_page structures (keyed by its logical offset from the start of the vm_object). This radix tree makes searching for vm_page structures much quicker.

DragonFly BSD Changes

A notable difference between DragonFly BSD and FreeBSD is that the former also stores the vm_map_entry structures in a tree.

NetBSD and OpenBSD (UVM)

UVM was based on 4.4BSD which itself was based on Mach, and so it initially bares some resemblance to the virtual memory system in FreeBSD. However, UVM was designed to support memory sharing using three mechanisms: page loanout, page transfer, and map entry passing.

Page loanout is when a process loans its memory to another process. This is useful particularly in networking, in which data can be sent to the kernel’s network stack simply by loaning the appropriate pages. This avoids the need for costly copy operations.

Page transfer is similar to loanout, except the pages remain in the possession of the receiving process.

Map entry passing, rather than transfering the pages of virtual memory, copies the higher-level map objects in order to export a large range of memory to another process. This saves nothing when copying a single page, but scales very well since a map object can point to very many pages in memory, and only the map structure itself need be copied.

Data Structures

UVM is designed with vmspace, vm_map, vm_object, and vm_page objects which function similarly to those in FreeBSD. We will describe the differences.

In addition to what was described above, each vm_object has a vm_pager object which describes how the backing store can be accessed. Essentially, this is a pointer to a list of functions which fetch and store pages between the memory pointed to by the vm_page object and whatever backing store (such as a disk) underlies the vm_object.

Implementation

It’s interesting to look at the implementation to see all the gory details. We can find the definition of the mmap system call for OpenBSD in their CVS repo in src/sys/uvm/uvm_mmap.c. In fact, there are too many gory details here, so I will show a version here with lots of extra stuff cut out. We only really care about the case where extra anonymous memory is allocated.

/*
 * sys_mmap: mmap system call.
 *
 * => file offset and address may not be page aligned
 *    - if MAP_FIXED, offset and address must have remainder mod PAGE_SIZE
 *    - if address isn't page aligned the mapping starts at trunc_page(addr)
 *      and the return value is adjusted up by the page offset.
 */
int
sys_mmap(struct proc *p, void *v, register_t *retval)
{
	/* ... */

	/* first, extract syscall args from the uap. */
	/* ... */

	/* validate the flags */
	/* ... */

	error = pledge_protexec(p, prot);
	if (error)
		return (error);

	/* align file position and save offset.  adjust size. */
	ALIGN_ADDR(pos, size, pageoff);

	/* now check (MAP_FIXED) or get (!MAP_FIXED) the "addr" */
	if (flags & MAP_FIXED) {
		/* ... */

	}

	/* check for file mappings (i.e. not anonymous) and verify file. */
	if ((flags & MAP_ANON) == 0) {
	/* ... */
	} else {		/* MAP_ANON case */
		/*
		 * XXX What do we do about (MAP_SHARED|MAP_PRIVATE) == 0?
		 */
		if (fd != -1)
			return EINVAL;

is_anon:	/* label for SunOS style /dev/zero */

		if ((flags & MAP_ANON) != 0 ||
		    ((flags & MAP_PRIVATE) != 0 && (prot & PROT_WRITE) != 0)) {
			if (size >
			    (p->p_rlimit[RLIMIT_DATA].rlim_cur - ptoa(p->p_vmspace->vm_dused))) {
				return ENOMEM;
			}
		}
		maxprot = PROT_MASK;
		error = uvm_mmapanon(&p->p_vmspace->vm_map, &addr, size, prot, maxprot,
		    flags, p->p_rlimit[RLIMIT_MEMLOCK].rlim_cur, p);
	}

	if (error == 0)
		/* remember to add offset */
		*retval = (register_t)(addr + pageoff);

	/* ... */
	return (error);
}

And uvm_mmapanon is defined in the same file:

/*
 * uvm_mmapanon: internal version of mmap for anons
 *
 * - used by sys_mmap
 */
int
uvm_mmapanon(vm_map_t map, vaddr_t *addr, vsize_t size, vm_prot_t prot,
    vm_prot_t maxprot, int flags, vsize_t locklimit, struct proc *p)
{
	int error;
	int advice = MADV_NORMAL;
	unsigned int uvmflag = 0;
	vsize_t align = 0;	/* userland page size */

	/*
	 * for non-fixed mappings, round off the suggested address.
	 * for fixed mappings, check alignment and zap old mappings.
	 */
	if ((flags & MAP_FIXED) == 0) {
		*addr = round_page(*addr);	/* round */
	} else {
		if (*addr & PAGE_MASK)
			return(EINVAL);

		uvmflag |= UVM_FLAG_FIXED;
		if ((flags & __MAP_NOREPLACE) == 0)
			uvmflag |= UVM_FLAG_UNMAP;
	}

	if ((flags & MAP_FIXED) == 0 && size >= __LDPGSZ)
		align = __LDPGSZ;
	if ((flags & MAP_SHARED) == 0)
		/* XXX: defer amap create */
		uvmflag |= UVM_FLAG_COPYONW;
	else
		/* shared: create amap now */
		uvmflag |= UVM_FLAG_OVERLAY;

	/* set up mapping flags */
	uvmflag = UVM_MAPFLAG(prot, maxprot,
	    (flags & MAP_SHARED) ? MAP_INHERIT_SHARE : MAP_INHERIT_COPY,
	    advice, uvmflag);

	error = uvm_mapanon(map, addr, size, align, uvmflag);

	if (error == 0)
		error = uvm_mmaplock(map, addr, size, prot, locklimit);
	return error;
}

The function prototype for uvm_mapanon is in src/sys/uvm/uvm_extern.h but its definition is in src/sys/uvm/uvm_map.c:

/*
 * uvm_mapanon: establish a valid mapping in map for an anon
 *
 * => *addr and sz must be a multiple of PAGE_SIZE.
 * => *addr is ignored, except if flags contains UVM_FLAG_FIXED.
 * => map must be unlocked.
 *
 * => align: align vaddr, must be a power-of-2.
 *    Align is only a hint and will be ignored if the alignment fails.
 */
int
uvm_mapanon(struct vm_map *map, vaddr_t *addr, vsize_t sz,
    vsize_t align, unsigned int flags)
{
	/* ... */

	/*
	 * Before grabbing the lock, allocate a map entry for later
	 * use to ensure we don't wait for memory while holding the
	 * vm_map_lock.
	 */
	new = uvm_mapent_alloc(map, flags);
	if (new == NULL)
		return(ENOMEM);
	
	/* ... */

	/*
	 * Create new entry.
	 * first and last may be invalidated after this call.
	 */
	entry = uvm_map_mkentry(map, first, last, *addr, sz, flags, &dead,
	    new);
	if (entry == NULL) {
		error = ENOMEM;
		goto unlock;
	}
	new = NULL;
	KDASSERT(entry->start == *addr && entry->end == *addr + sz);
	entry->object.uvm_obj = NULL;
	entry->offset = 0;
	entry->protection = prot;
	entry->max_protection = maxprot;
	entry->inheritance = inherit;
	entry->wired_count = 0;
	entry->advice = advice;
	if (flags & UVM_FLAG_NOFAULT)
		entry->etype |= UVM_ET_NOFAULT;
	if (flags & UVM_FLAG_COPYONW) {
		entry->etype |= UVM_ET_COPYONWRITE;
		if ((flags & UVM_FLAG_OVERLAY) == 0)
			entry->etype |= UVM_ET_NEEDSCOPY;
	}
	if (flags & UVM_FLAG_OVERLAY) {
		KERNEL_LOCK();
		entry->aref.ar_pageoff = 0;
		entry->aref.ar_amap = amap_alloc(sz,
		    ptoa(flags & UVM_FLAG_AMAPPAD ? UVM_AMAP_CHUNK : 0),
		    M_WAITOK);
		KERNEL_UNLOCK();
	}

	/* Update map and process statistics. */
	map->size += sz;
	((struct vmspace *)map)->vm_dused += uvmspace_dused(map, *addr, *addr + sz);

unlock:
	vm_map_unlock(map);

	/*
	 * Remove dead entries.
	 *
	 * Dead entries may be the result of merging.
	 * uvm_map_mkentry may also create dead entries, when it attempts to
	 * destroy free-space entries.
	 */
	uvm_unmap_detach(&dead, 0);
out:
	if (new)
		uvm_mapent_free(new);
	return error;
}

So that largely defers to uvm_map_mkentry, which is again defined in the same file.

/*
 * Create and insert new entry.
 *
 * Returned entry contains new addresses and is inserted properly in the tree.
 * first and last are (probably) no longer valid.
 */
struct vm_map_entry*
uvm_map_mkentry(struct vm_map *map, struct vm_map_entry *first,
    struct vm_map_entry *last, vaddr_t addr, vsize_t sz, int flags,
    struct uvm_map_deadq *dead, struct vm_map_entry *new)
{
	/* ... */

	/* Initialize new entry. */
	if (new == NULL)
		entry = uvm_mapent_alloc(map, flags);
	else
		entry = new;
	if (entry == NULL)
		return NULL;
	/* ... */
	return entry;
}

Again, uvm_mapent_alloc is in the same file.

/*
 * uvm_mapent_alloc: allocate a map entry
 */
struct vm_map_entry *
uvm_mapent_alloc(struct vm_map *map, int flags)
{
	struct vm_map_entry *me, *ne;
	int pool_flags;
	int i;

	pool_flags = PR_WAITOK;
	if (flags & UVM_FLAG_TRYLOCK)
		pool_flags = PR_NOWAIT;

	if (map->flags & VM_MAP_INTRSAFE || cold) {
		mtx_enter(&uvm_kmapent_mtx);
		me = uvm.kentry_free;
		if (me == NULL) {
			ne = km_alloc(PAGE_SIZE, &kv_page, &kp_dirty,
			    &kd_nowait);
			if (ne == NULL)
				panic("uvm_mapent_alloc: cannot allocate map "
				    "entry");
			for (i = 0;
			    i < PAGE_SIZE / sizeof(struct vm_map_entry) - 1;
			    i++)
				RB_LEFT(&ne[i], daddrs.addr_entry) = &ne[i + 1];
			RB_LEFT(&ne[i], daddrs.addr_entry) = NULL;
			me = ne;
			if (ratecheck(&uvm_kmapent_last_warn_time,
			    &uvm_kmapent_warn_rate))
				printf("uvm_mapent_alloc: out of static "
				    "map entries\n");
		}
		uvm.kentry_free = RB_LEFT(me, daddrs.addr_entry);
		uvmexp.kmapent++;
		mtx_leave(&uvm_kmapent_mtx);
		me->flags = UVM_MAP_STATIC;
	} else if (map == kernel_map) {
		splassert(IPL_NONE);
		me = pool_get(&uvm_map_entry_kmem_pool, pool_flags);
		if (me == NULL)
			goto out;
		me->flags = UVM_MAP_KMEM;
	} else {
		splassert(IPL_NONE);
		me = pool_get(&uvm_map_entry_pool, pool_flags);
		if (me == NULL)
			goto out;
		me->flags = 0;
	}

	if (me != NULL) {
		RB_LEFT(me, daddrs.addr_entry) =
		    RB_RIGHT(me, daddrs.addr_entry) =
		    RB_PARENT(me, daddrs.addr_entry) = UVMMAP_DEADBEEF;
	}

out:
	return(me);
}

Finally, we’re starting to get to the good stuff. This calls km_alloc, which is defined in src/sys/uvm/uvm_km.c:

void *
km_alloc(size_t sz, const struct kmem_va_mode *kv,
    const struct kmem_pa_mode *kp, const struct kmem_dyn_mode *kd)
{
	struct vm_map *map;
	struct vm_page *pg;
	struct pglist pgl;
	int mapflags = 0;
	vm_prot_t prot;
	paddr_t pla_align;
	int pla_flags;
	int pla_maxseg;
	vaddr_t va, sva;

	KASSERT(sz == round_page(sz));

	TAILQ_INIT(&pgl);

	if (kp->kp_nomem || kp->kp_pageable)
		goto alloc_va;

	pla_flags = kd->kd_waitok ? UVM_PLA_WAITOK : UVM_PLA_NOWAIT;
	pla_flags |= UVM_PLA_TRYCONTIG;
	if (kp->kp_zero)
		pla_flags |= UVM_PLA_ZERO;

	pla_align = kp->kp_align;
#ifdef __HAVE_PMAP_DIRECT
	if (pla_align < kv->kv_align)
		pla_align = kv->kv_align;
#endif
	pla_maxseg = kp->kp_maxseg;
	if (pla_maxseg == 0)
		pla_maxseg = sz / PAGE_SIZE;

	if (uvm_pglistalloc(sz, kp->kp_constraint->ucr_low,
	    kp->kp_constraint->ucr_high, pla_align, kp->kp_boundary,
	    &pgl, pla_maxseg, pla_flags)) {	
		return (NULL);
	}

#ifdef __HAVE_PMAP_DIRECT
	/*
	 * Only use direct mappings for single page or single segment
	 * allocations.
	 */
	if (kv->kv_singlepage || kp->kp_maxseg == 1) {
		TAILQ_FOREACH(pg, &pgl, pageq) {
			va = pmap_map_direct(pg);
			if (pg == TAILQ_FIRST(&pgl))
				sva = va;
		}
		return ((void *)sva);
	}
#endif
alloc_va:
	prot = PROT_READ | PROT_WRITE;

	if (kp->kp_pageable) {
		KASSERT(kp->kp_object);
		KASSERT(!kv->kv_singlepage);
	} else {
		KASSERT(kp->kp_object == NULL);
	}

	if (kv->kv_singlepage) {
		KASSERT(sz == PAGE_SIZE);
#ifdef __HAVE_PMAP_DIRECT
		panic("km_alloc: DIRECT single page");
#else
		mtx_enter(&uvm_km_pages.mtx);
		while (uvm_km_pages.free == 0) {
			if (kd->kd_waitok == 0) {
				mtx_leave(&uvm_km_pages.mtx);
				uvm_pglistfree(&pgl);
				return NULL;
			}
			msleep(&uvm_km_pages.free, &uvm_km_pages.mtx, PVM,
			    "getpage", 0);
		}
		va = uvm_km_pages.page[--uvm_km_pages.free];
		if (uvm_km_pages.free < uvm_km_pages.lowat &&
		    curproc != uvm_km_pages.km_proc) {
			if (kd->kd_slowdown)
				*kd->kd_slowdown = 1;
			wakeup(&uvm_km_pages.km_proc);
		}
		mtx_leave(&uvm_km_pages.mtx);
#endif
	} else {
		struct uvm_object *uobj = NULL;

		if (kd->kd_trylock)
			mapflags |= UVM_KMF_TRYLOCK;

		if (kp->kp_object)
			uobj = *kp->kp_object;
try_map:
		map = *kv->kv_map;
		va = vm_map_min(map);
		if (uvm_map(map, &va, sz, uobj, kd->kd_prefer,
		    kv->kv_align, UVM_MAPFLAG(prot, prot, MAP_INHERIT_NONE,
		    MADV_RANDOM, mapflags))) {
			if (kv->kv_wait && kd->kd_waitok) {
				tsleep(map, PVM, "km_allocva", 0);
				goto try_map;
			}
			uvm_pglistfree(&pgl);
			return (NULL);
		}
	}
	sva = va;
	TAILQ_FOREACH(pg, &pgl, pageq) {
		if (kp->kp_pageable)
			pmap_enter(pmap_kernel(), va, VM_PAGE_TO_PHYS(pg),
			    prot, prot | PMAP_WIRED);
		else
			pmap_kenter_pa(va, VM_PAGE_TO_PHYS(pg), prot);
		va += PAGE_SIZE;
	}
	pmap_update(pmap_kernel());
	return ((void *)sva);
}

Whew. That’s a lot to unravel, so I’ll leave it here for now.

References

For the section on FreeBSD, I referred to “The Design and Implementation of FreeBSD.” For the section on NetBSD and OpenBSD, I referred to Charles Cranor’s dissertation “Design and Implementation of the UVM Virtual Memory System.”

Attribution

The diagram for UVM’s data structures comes from Charles Craynor’s paper at Usenix in 1999, of which there is also a PDF version.

POSIX Memory Management

Mon, 01 Feb 2016 00:00:00 +0000

As an application programmer using a unix-flavored operating system, one’s view of memory management is usually limited to malloc and its various alternative forms. However, malloc is part of the C standard library and while it is specified as part of POSIX.1, the specification defers to the ISO C standard. In fact, since malloc is part of the C standard library it is not a part of the kernel and can’t allocate memory directly. For that, we need to use kernel functions.

POSIX.1-1998 specifies both sbrk and mmap for allocating memory. But sbrk was allegedly removed in POSIX.1-2001, and certainly we can see it no longer exists in the version of POSIX.1-2004 still available online. To understand what they do and why sbrk was probably removed, we need to understand how a program’s code and data are laid out in memory.

Memory Layout

Modern operating systems implement virtual memory, which gives each application the illusion that each has access to all (or most) of the machine’s physical memory. In reality, the kernel maps virtual memory locations to physical ones and, with help from the CPU, translates virtual to physical addresses on the fly. Since we ostensibly have the entire address space in which to arrange our code and data, this gives us a lot of flexibility.

The diagram on the right shows a pretty typical way to arrange the code and data for a program. Starting at the bottom (which has the lowest memory address), we have the text region. This is where a prorgam’s code would be stored. Above that is the BSS (Block Started by Symbol) region and above that is the initialized data region, which are where static variables (uninitialized and initialized, respectively) are stored. Above that, we have the heap, free memory, and the stack.

Any standard C course will explain the difference between the stack and the heap. The stack contains variables like function arguments and variables defined locally within a function. These variables are automatically pushed onto the stack when a function is called, and are popped off the stack when the function returns a value. In order to keep a variable around for longer than the life of a function call, we need to allocate on the heap.

Keep in mind that this memory layout is only the typical case. An OS that implements the POSIX.1 specification may not lay out memory in the same way.

`sbrk`

The brk and sbrk kernel functions move the boundary between the heap and free memory, effectively increasing the size of the heap and allocating more memory for a program. In particular, brk takes a memory address and sets that as the new “break” (the byte past the last heap-allocated memory), whereas sbrk more usefully takes a number of bytes and increases the size of the heap by that many bytes. We can effectively decrease the size of the heap by giving sbrk a negative value.

We could get away with only using this function if we treated the heap as another stack. In other words, if we always deallocated in the reverse order as we allocated memory, then we could easily just use sbrk.

However, it’s much more useful as an application programmer to be able to allocate and deallocate from the heap in any order. For this reason, when malloc is implemented using sbrk, it usually allocates more space than it needs and uses something like a linked-list to keep track of unused memory locations.

A naive implementation of malloc might assume that no programmer would both use malloc and call sbrk directly, so the specification warns that the behavior of sbrk is unspecified if an application also uses any other functions (such as malloc or mmap).

Notice that sbrk makes this strong assumption that the heap is a contiguous area of memory with a single boundary which can grow or shrink to allocate more or less memory to a process. This, and that sbrk doesn’t play well with malloc, are probably why it was removed from the POSIX.1 standard. That said, most unix-flavored OSes typically still implement it. Thankfully, we have a nice alternative: mmap.

ASIDE: For more on implementing malloc using sbrk, there’s a great code review cleaning up the malloc implementation from K&R which uses a function morecore which seems to be analagous to sbrk. There’s also a high-level view of how malloc works, as well as a tutorial on implementing a simple version of malloc based on a tutorial on implmeneting a complex version of malloc.

`mmap`

mmap is specified in the latest version of the POSIX.1 specification. Its brief description is “The mmap() function shall establish a mapping between an address space of a process and a memory object.” Immediately, we can see that this doesn’t make any assumptions about how a program is laid out in memory. Although at first glance, it may seem that this function is more about mapping something like I/O into the address space of a process.

The function prototype for mmap is:

void *mmap(void *addr, size_t len, int prot, int flags, int fildes, off_t off);

The specification also goes on to state that if addr is 0, and FLAG_FIXED is not set, then the implementation is free to find unused memory into which to map. Also, most operating systems have an additional flag MAP_ANON which effectively just allocates the desired memory in the same way that malloc would, without making any assumptions about memory layout. To illustrate this, we can implement a very rudimentary version of malloc using only mmap (and munmap to implement free).

#include <sys/mman.h>
#include <stdio.h>

void* mymalloc(size_t len) {
  void* addr = mmap(0,                      // addr
                    len + sizeof(size_t),   // len
                    PROT_READ | PROT_WRITE, // prot
                    MAP_ANON | MAP_PRIVATE, // flags
                    -1,                     // filedes
                    0);                     // off
  *(size_t*)addr = len;
  return addr + sizeof(size_t);
}

int myfree(void* addr) {
  return munmap(addr - sizeof(size_t),      // addr
                (size_t) addr);             // len
}

int main(int argc, char* argv[]) {
  puts("Allocating first integer...\n");
  int* heap_integer_1 = mymalloc(sizeof(int));
  puts("Allocating char...\n");
  char* heap_char_1 = mymalloc(sizeof(char));
  puts("Allocating second integer...\n");
  int* heap_integer_2 = mymalloc(sizeof(int));
  puts("Writing to char...\n");
  *heap_char_1 = 'o';
  puts("Writing to first integer...\n");
  *heap_integer_1 = 1111;
  puts("Writing to second integer...\n");
  *heap_integer_2 = 2222;
  puts("Reading from allocated integers...");
  printf("First allocated integer:  %d\n", *heap_integer_1);
  printf("Heap allocated char:      %c\n", *heap_char_1);
  printf("Second allocated integer: %d\n", *heap_integer_2);
  puts("Deallocating second integer...");
  myfree(heap_integer_2);
  puts("Deallocating char...");
  myfree(heap_char_1);
  puts("Deallocating first integer...");
  myfree(heap_integer_1);
  return 0;
}

You can run this code online at codepad.

Attribution

The memory arrangement diagram is by Majenko (Own work) CC BY-SA 4.0, via Wikimedia Commons. It appears on the Wikipedia page for the data segment.

Possession, Ownership, and Copyright

Sun, 12 May 2013 00:00:00 +0000

Possession is the natural state of actually physically having some item. An item can only either be in your possession or not. Maybe there’s some ambiguity if several people are holding the same item, but we can say that everyone holds the item possesses it. The idea of possession comes naturally from being able to carry and use things.

From the state of possession comes the natural rights to take and leave. If an item exists, taking it is the action of moving from a state of not possessing that item to the opposite. Similarly, leaving an item is the opposite action.

With that understood, let’s consider the concept of ownership. If an item is not owned by anyone else (or if the owner permits), I can claim ownership of that item. Ownership supersedes the right of others to take that item without my granting them license to do so.

Items that are owned are said to be property. Such property can be privately owned by a single individual, collectively owned by several individuals, or publicly owned by some public entity like a government.

Ownership is an old concept. Plato argued that collective ownership was necessary to promote common pursuit of the common interest, while Aristotle said “when everyone has a distinct interest, men will not complain of one another, and they will make more progress, because every one will be attending to his own business.”

Interestingly, Aristotle defined freedom as the extent to which one owns one’s self. In other words, your ability to control yourself determines your level of freedom. If you have absolute self control, you are absolutely free. Otherwise you are to some extent a natural slave to your desires.

Hume argued that there is no natural “mine” or “thine”, and that ownership must be understood as the creation of a sovereign state or the artificial product of a convention “enter’d into by all the members of the society to bestow stability on the possession of…external goods, and leave every one in the peaceable enjoyment of what he may acquire by his fortune and industry.”

Locke argued that ownership arises naturally since we naturally own ourselves and from this follows ownership of other items by our own labour, but this seems to confuse the concepts of possession and ownership. We naturally possess ourselves, but do we naturally own ourselves? As defined above, ownership is a protection from the rights of others to take items that are not in our possession. How can we cease to possess ourselves that ownership might prevent others from exercising their right to take? The idea is nonsensical.

While not arising naturally, ownership of any kind is useful. It would be impractical to require that in order to assure your continued ability to possess an item, you must keep others from possessing it at all times. A collective agreement that you own an item means you can spend your time productively instead.

Intellectual property dates back at least to the Ancient Greeks, but it is clear that it isn’t the same as regular property since as with self-ownership, it isn’t obvious that we can ever cease to possess an idea. If I tell someone an idea of mine, we both possess the idea. There is no scarcity of ideas.

The protection of intellectual property by ancient societies was not codified in law, but by franchises or royal favours. Bugbee makes the distinction that these cases were works that were in the public domain being removed from the publi by some authority, as opposed to a creator depriving the public of something that did not previously exist.

One of the first codified protections was by the Republic of Florence in 1421, wherein an architect was granted certain rights to an his invention of a barge with hoisting gear for carrying marble. This precedent was useful at least in that it incentivized creation of new works. This became known as the Florentine patent statute. Specifically, the statute protected the architects right to build the barge, not the idea itself.

Literary works were unprotected until the Gutenberg printing press was invented in 1450, and even then few copyrights were granted.

The English Statute of Monopolies (1624) and the Statute of Anne (1710) extended this idea by ending the practice of patenting unoriginal ideas and works which were already in the public domain. The Statute of Anne is considered to be the first codification of copyrights, in which authors were granted the control of the copying of their work for a fixed period with the explicit intent to incentivize writing. The statute specifically mentions the common practice of copying and reprinting works by printers.

The Statute of Anne shifted the protection from the ability to create an object based on an original idea to the protection of the idea itself. The protection was from physical books being created from the same intellectual material, but the underlying notion was that the creators deserved to be rewarded for their creation. International copyright law has since been expanded by the Berne convention and the TRIPS agreement, but the same notion applies.

One fundamental question that must be asked is to what extent do creators have the right to be rewarded? Unquestionably, the creation of novel ideas benefits us all, but each protection we give creators is a right that we are taking from users.

For example, the recently passed Canadian Copyright Modernization Act Bill C-11 extends many user rights to copyrighted work, but all of these rights are trumped by a digital lock on the work. Additionally, it is now illegal to break digital locks for any purpose, and it is illegal to distribute tools which might break digital locks. Is it really within the purview of protecting creators’ rights to be rewarded to remove the right to distribute creations not in any way related to the creator’s work?

In answering the question of rewarding creators, we must also answer the question: how much does copying of a work hurt a creator? Cory Doctrow argues that excessive copyright hurts creators. One of Doctrow’s main points is that digital locks send the message that if you buy a work then it may be taken away at any time, but if you steal it you can keep it forever. In this sense, Doctrow is saying that digital locks prevent ownership of a copy of the work.

Doctrow’s eponymous law states “anytime someone puts a lock on something you own, against your wishes, and doesn’t give you the key, they’re not doing it for your benefit.”

Readability

Sun, 07 Apr 2013 00:00:00 +0000

If you’re reading this in a maximized browser on a wide-screen monitor, you may wonder why the margins are so big. The answer isn’t that this page is optimized for small displays. In fact, I asked the same question of LaTeX margins and I found an excellent answer on the LaTeX stackexchange that points out that the problem isn’t large margins, but large paper with respect to optimal line length.

The Elements of Typographic Style Applied to the Web states that line lengths are optimally between 45 and 75 characters including spaces. Emil Ruder suggests 50-60 characters in his book, Typographie. Emil goes on to say that readers are energized when their eyes move from the end of one line to the beginning of the next, as long as it doesn’t happen too frequently, and that this energy diminishes over the length of the line.

The Web Style Guide claims that research that reading slows as line lengths exceed the maximum width of about 12 words. With a space between each word and assuming 4 letters per word on average, this converts to about 12*4+11=59 characters per line.

You can use this bookmarklet to check the number of characters on a line in your browser. As of this writing, line lengths on this blog tend to be around 71 characters.

I would also like to take this opportunity to recommend Readability which converts any web page to a more readable version, and whose novel theme inspired the design of this blog.