4 (Kernel Cleanup)
primitives - The core of the VSTa kernel lacks any reusable primitives. There are no basic linked list primitives, lock primitives, hashing primitives, bit operation primitives, ect.. This results in mass amounts of duplicated code that is difficult to maintain. Correcting bugs in something like linked list handling requires huge changes to the entire code base. We desperately need valid primitives for a large variety of basic data types to make refactoring the kernel less painful.
x86 ICU rewrite - The current 8259 interrupt handler code in VSTa is not interrupt safe, no that was not a joke. This results in a rather large number of odd issues. Further more, spurious interrupts on interrupts thought to be disabled will result in a kernel assertion unless the interrupt is on IRQ7. This is particularly painful as it is possible for an interrupt to be reenabled w/out permission of the ICU code.
scheduler - The current scheduler is not only incredibly huge for what it does, it lacks some fudemental functionality that is greatly desired. Like proper job control. Another area of concern is that the scheduler expects the caller to hold semaphore locks before the scheduler is entered, which makes the code that much more convoluted. There are a large variety of better algorithms to choose from for process scheduling. Perhaps the most attractive is the 0(1) scheduler currently in use in the Linux 2.6 kernel. This schedueling algorithm is used in a number of real-time embedded systems (Ed Skinner at MontaVista software teaches about this schedueler in his real-time operating systems class). This algorithm works by using a bitmap where each bit corresponds to an element in an priority array. Each element in the array contains a link list of all runnable tasks for that priority, the head of the list is the first runnable task at that priority. What happens is that in effect your operation for finding the next runnable task looks like: task = priority![FindFirstSet(bitmap)] The first task on the linked list is returned and assigned. From there you just need to tear down the old task and fire up the new one. The work of adding a process back to one of the link lists simply becomes appending the task to the end of the link list for that priority when it becomes runnable and setting the appropriate bit high in the bitmap. If the task is preempted, then you simply round-robin it to the end of the list, or the end of the next lower priority list if you are going to consider process starvation. We don't need to track sleeping processes because they will be given to us when they are woken back up via either a timer, and interrupt, or an ipc message. Fixing this up is dependant on the primitives cleanup.
semaphores - Semaphores as they currently exist can be fairly racey on SMP. Part of the issue is that the semaphores themselves do not handle the task of saving the jump address for a slept process but instead hand that off to the scheduler who deals with it via setjmp/longjmp, which ends up painfully overloading the scheduler itself besides being ugly. The semaphores really need to be replaced with simpler priority driven ones, likely using the same findfirstset bit trickery as proposed for the scheduler.
spinlocks - Old VSTa spinlocks are hopelessly useless for SMP work. They do not spin, and they barely lock. Replace with something that does.
memory management - While the current memory management subsystem does in fact do its job as expected, it is likely the single most convoluted, unorganized, and painfully poorly written piece of code in the entire kernel. Further more, it is most of the kernel. This needs to be flat out replaced, but in doing so we will impact the IPC subsystem as it is written with a bit to much knowledge about how the memory manager functions.
msg passing - Besides being ridden with untold number of bugs and deadlock conditions, the current message passing layer lacks the functionality for doing proper asyncronous I/O operations. It also forces daemons to behave in certain ways that may not be desirable. Once such example is that msgsend forces the client to go into an I/O wait state. What would be nice is if we replaced msgsend w/ a new msgcall that forced the client into an I/O wait state there by implying that the communication path being set up requires the client block. If the client does not have to block for the message, it could use a modified msgsend that simply attaches the msg to the servers queue and control is returned back to the client if it still has allowed time on the CPU. An example of where this would be particularly useful is in the case of journaled file systems where a filesystem would msgcall for the journal writes, and once those are done could msgsend the filesystem updates. Once the replies come back for the updates it could msg_call to clear the journal entries.