I have tutored a great number of students in undergraduate operating systems. Personally, I enjoy the topic. My dissertation is in the field of distributed systems. Distributed systems is an academic offspring of operating systems research where partial failures are expected and allowed. The operating systems course is almost universally taught with a laboratory component. I have not (yet) found the perfect operating systems course. These are my observations on the various approaches that I have seen.
Production Operating System
Linux and Android are the most popular options for courses that teach real world open source operating systems. This style of teaching generally uses a virtual machine (such as VirtualBox) or an emulator (such as QEMU). The benefits of this approach are that students have exposure to real production operating systems source code. The drawbacks of this approach are that students have exposure to real production operating systems source code. Linux and Android are large complex projects. They have non-trivial build systems. They are implemented in C. For many students this is their first exposure to programming in C. This may be the student’s only exposure to C in their undergraduate curriculum. It is a challenge to teach the textbook learning objectives of the operating system course and teach the practical C programming skills necessary for Linux and Android. And teach both kinds of topics at the same time in the same course. A simple build process goes a long way. I prefer running QEMU to VirtualBox. The implement-compile-test feedback cycle is faster on QEMU. Emphasize and discourage common mistakes. For example, when running a virtual machine with an ext2 filesystem with a high probability of filesystem corruption when the OS crashes, always make a snapshot of the operating system before booting new changes. In these real-world build environments, it becomes possible to lose hours of work due to a simple workflow error. These kinds of traps are discouraging to students who are just trying to learn the material.
Research/Educational Operating System
xv6 is the most popular option for small teaching operating systems in the UNIX family that are implemented in C. QEMU is the most common execution environment. These operating systems are on the order of 10,000 lines of code. Compare that to the Linux kernel which is around 10,000,000 lines of code. The implement-compile-test feedback cycle is faster than on a production operating system. The build environment is typically disposable; the state of the virtual disk is not saved on shutdown. As an exposure to the C programming language the development workflow is dramatically simplified. Using an educational operating system has become my preferred method for operating systems labs. It is realistic enough for students to learn the concepts of operating systems and not too realistic to bog students down in implementation details. There are a few drawbacks. The x86 version of xv6 is no longer being maintained. New development has continued on the RISC-V version while schools continue to use the x86 code.
Managed Runtime Environment