os lab part 2 blog post

data/blog/2023-12-04-fall-2023-os-part-2-lab.md

@@ -0,0 +1,62 @@
+---
+title: Secure OS Lab Part 2
+authors: [Jason An]
+category: Projects
+tags: [fall-2023, cyber-lab]
+description: Creating a secure OS
+---
+
+# Cyber OS Blogpost
+## Overview
+We started this quarter with basically just a bootloader, terminal driver, basic libc library, and an empty kernel. From this starting point, we had to start building out all of the components of the kernel that allow an operating system to properly function. Specific details are in the sections below.
+
+## Virtual Memory/Paging
+In x86 and most CPU architectures, there's a distinction between physical and virtual memory. Physical memory addresses directly and linearly into RAM, and has access to the entire memory space. When the OS first boots, this is the default addressing mode. This is unideal for many reasons. Every process would be given access to the entire memory space, which lets them read and modify the memory of other processes, or worse, the kernel. It's also hard to partition memory properly between processes, as each process would somehow have to know what regions of memory are safe to write into without any collisions.
+
+This is the purpose of virtual memory: by utilizing a technique called "paging", every process can be given its own virtual memory space, which non-linearly maps onto physical memory in a manner that can be controlled by the kernel, along with additional protections, like non-executable memory or kernel-only memory.
+
+A page of memory in x86 is 4096 (0x1000) aligned bytes, and is the minimum unit of memory for address translation. A page table describes how these pages in virtual memory map to pages in physical memory, if they even map anywhere at all.
+
+Concretely, how paging works in x86 is that the kernel writes a top-level page table to physical memory then moves its physical address into the cr3 control register, which enables the memory management unit (MMU) that manages paging. Another layer of optimization is the translation lookaside buffer (TLB) which caches page table entries, which means that the TLB has to be flushed every time a page table entry is changed. The reason the page table is referred to as "top-level" is because x86 actually has hierarchical paging with multiple levels of page tables. In 32-bit x86, there are 2 levels, and in 64-bit x86, there are 4 levels with an optional 5th with CPU extensions. This is because a flat page table containing everything would take up too much memory by itself, so each level of page table only encodes a portion of the bits of the virtual address.
+
+We wrote a simple freelist page frame allocator exposed as a `palloc` function, which is responsible for taking in a virtual address, then finding a free page in physical memory and creating the respective entry in the page table to map the virtual address there. There's also a `pfree` function that takes in a virtual address and deletes the page table entry for it. This is where the freelist comes in: in order to reuse freed physical pages, a singly linked list of freed pages is maintained in the freed pages themselves, so it uses no additional space. To free a page, the freelist head is stored in the page, then the freelist head is set to the freed page. To allocate a page, if the freelist head is not NULL, then the freelist head will be the allocated page, and the head will be advanced to the next element in the list. If the freelist head is NULL, that means there are no reusable pages, so a new page is chosen at the end of the allocated pages.
+
+## Heap Allocator
+The heap is a pool of memory conceptually divided into fixed-size regions called chunks, providing a method for dynamic allocation. The OS can request arbitrarily sized chunks for use at run-time through the `kalloc` function and free these chunks for reuse through the `kfree` function. The heap allocator is responsible for appropriately reusing and consolidating freed chunks to minimize fragmentation.
+
+Our implementation stores metadata including the size and in-use status of chunks immediately before the chunk data in memory. Freed chunks are stored in a doubly linked-list and repurpose the chunk data to store free-list pointers. Upon freeing a chunk, the in-use status of immediately adjacent chunks in memory (forwards and backwards) are checked to perform consolidation such that adjacent free chunks are merged into a single larger free chunk. An iteration through the free-list upon subsequent allocations utilizes a first-fit remaining strategy for speed.
+
+The heap itself is implemented as a contiguous region of memory. The wilderness chunk represents the last page of memory that has been allocated for the heap using the kernel's `palloc` function, and can be dynamically grown to satisfy allocation requests that are not satisfied by the free-list.
+
+An improvement to our heap allocator could utilize fixed-size bins as a caching mechanism for freed chunks, serving as both a space and time efficiency improvement, as a best-fit strategy would be implicitly utilized in constant time.
+
+## Interrupts + IDT
+Interrupts allow for processes and hardware devices to alert the kernel of various events occurring. For example, every time you're in your browser and you move your mouse, your operating system needs to be alerted that signals are being sent from your mouse. After the operating system is alerted, the input is then received and handled, usually by updating the position of the cursor and changing the location it is displayed on the screen.
+
+In order to enable this functionality, we first had to create an Interrupt Descriptor Table, or IDT, which is essentially a list of pointers that tell the OS where to jump to upon receiving an interrupt signal. These "pointers" are not actually pointers denoted by `*variable_name` like you would see in C source code but a specially formatted struct which we call `idt_entry_t` which contain the address of the function to jump to as well as segment selectors for kernel code in our Global Descriptor Table. In addition to creating the structs for the IDT, the functions that the IDT point to also need some modifications from the default compiled GCC functions. Our Interrupt Service Routines, which are the first 32 functions loaded into our IDT, are reserved by the Intel x86 CPU architecture to allow for the OS kernel to be informed of certain CPU events like page faults, segmentation faults, divide by 0 errors, etc. Additionally, 16 Interrupt Requests, which are raised by hardware devices like the keyboard and timer, also require special function procedures to properly initialize the stack frames. For this, we created function stubs in ASM and used `extern` to access them within our C source code.
+
+After creating these function stubs and linking them to our C kernel code, we also needed to created custom structs to access the parameter data getting pushed to the stack frame by our previously defined interrupt routine stubs, which we named `registers_t` and contains all the values of the registers pushed by the interrupt routine stubs as well as the interrupt numbers and error codes.
+
+Then, after all of this setup was done, we enabled the PIC, which is a piece of hardware that controls our hardware interrupts, to remap the PIC pins to interrupt numbers and initialize some additional settings on the PIC. To tell the CPU the location of the IDT, we used the `lidt` instruction with the address of the IDT and then ran the `sti` instruction to enable interrupts.
+
+## Keyboard Driver
+In order to receive text input from the user, the OS needs some kind of way to interpret electrical signals from the keyboard into valid ASCII. To do this, we need a keyboard driver to translate keyboard events like `Q_PRESSED` and `Q_RELEASED` into the correct characters that we want the kernel to receive.
+
+After setting up the IDT, creating a keyboard driver is quite trivial. All we need is a function in the correct format, specifically a function of type `typedef void (*isr_t)(registers_t *);`, which just means that we have a function that returns `void` and accepts a single parameter of type `registers_t*` and we define a type called `isr_t` that is a function pointer that follows these constraints.
+
+To actually get the signal from the keyboard, we have to read the CPU pin/port labeled 0x60, which is accomplished by `uint8_t scan_code = inb(0x60);`. Then, to handle the input from the keyboard, all we have to do is map the scan codes from the keyboard to ASCII characters and we're done with our driver! What this looks like in C code is basically just a long switch-case statement for and printing the ASCII character on the screen.
+
+## Serial Driver
+Given our bare bones infrastructure, debugging is quite a challenge.
+
+Serial is a method of transmitting data one bit at a time.
+
+While writing a serial driver might seem a less than critical task, this is not the case for one particular reason: debugging. At the beginning of the quarter, debugging was a serious hassle. Simple print statement debugging generally requires some sort of terminal. However, our entire application is a single terminal and it's common for it to break completely when there is a bug. Additionally, the data we can display is limited to the size of the terminal, and the limited set of ASCII characters we support.
+
+To make debugging significantly easier we can instead use serial to write any number of arbitrary bytes to some application outside the operating system (eg. terminal, file, gdb, etc.).
+
+We used this to create unit tests that write error messages and IDs or to print the contents of each registry with the intent to create an output pipeline to the serial driver.
+
+With this, we were able to create a workaround for our debugging process by directly writing data to the serial port, rather than to the emulation terminal. This resulted in making our quality of life easier while continuing our development of the kernel.
+
+Additionally, we also took a look at implementing the ability to read from serial, which would allow us to read bit data from the serial port stream. Due to time limit constraints, we did not implement this entirely, and we made the decision to focus on other critical aspects of the project instead.