operating system structures
(the unholy trinity)

(section 1.4)

Monolithic

e.g. Windows 3.1, UNIX

Like a big, black, shiny box. External interface, but no externally-visible structure. Calls to OS typically through trap instruction.

advantages: fine for few API calls, simple, easy to implement & write (OS single linked program), efficient to implement.
disadvantages: difficult to modify (must recompile kernel), unwieldy if there's lots of functionality, all OS code must be trusted

Layered

e.g. MULTICS

Like an onion-skinned monolithic system. Innermost layer provides essential OS services (like process-scheduling & interrupt-handling); each additional outer layer provides a higher-level VM to the layer above; each additional layer is less 'trusted' (trust the layer below, but mistrust the layer above). Typically uses hardware assistance to define & enforce layer boundaries.

advantages: more modular than monolithic, possibly more robust
disadvantages: calls between multiple layers may incur overheads, inflexible in terms of inserting additional layers and adding/removing features (like device drivers) to/from a layer. Layer structure is fixed. Need special hardware to enforce layer boundries.

Client-server (microkernel)

e.g. Plan 9, Mach

More commonly known nowadays as a 'microkernel' architecture.

Single, small kernel provides only fundamental services, while federation of reasonably independent processes provide all remaining OS services. For example, in operating systems based on the Mach kernel, even memory management is handled by non-kernel code. Most of the OS runs in 'user mode' with no special privileges (though certain services may require special privileges in order to get their work done).

advantages: easier to add and change modules. Modules are easier to develop, and robust (because they may be protected from each other). Kernel can be very small and efficient. Modules need not be on same machine. Can accommodate much larger APIs (because it's broken up into modules).
disadvantages: more sophisticated; probably needs more modern computers. OS structure visible to programs (good or bad); requires a more sophisticated security model to allow modules to provide services without stomping all over each other.

OS designers strive to reduce the size of the kernel, and what is in the kernel. It needs to provide only the most essential OS services: a framework for the other OS services to do their job and to communicate with each other.

what needs to be in a (micro)kernel?

process dispatcher/scheduler
interrupt handler
(perhaps low-level device-handling)
process communication

Reality check

In reality, nobody builds monolithic OSs any more. Layered systems offer too much complexity for too little gain. Most modern OSs have (or claim to have) a modular, object-oriented microkernel architecture. This structure has all the good points of the older two approaches: kernel is monolithic---simple, small, efficient---and 'layers' emerge from the way that higher-level modules use lower-level services.