Mac OS X Internals: A Systems Approach

Books Details :

Author : Amit Singh
Hardcover: 1680 pages
Publisher: Addison-Wesley Professional (June 19, 2006)
Language: English
ISBN: 0321278542
Product Dimensions: 9.4 x 7.7 x 2.6 inches




Books Description :

From the Back Cover

Mac OS X was released in March 2001, but many components, such as Mach and BSD, are considerably older. Understanding the design, implementation, and workings of Mac OS X requires examination of several technologies that differ in their age, origins, philosophies, and roles.

Mac OS X Internals: A Systems Approach is the first book that dissects the internals of the system, presenting a detailed picture that grows incrementally as you read. For example, you will learn the roles of the firmware, the bootloader, the Mach and BSD kernel components (including the process, virtual memory, IPC, and file system layers), the object-oriented I/O Kit driver framework, user libraries, and other core pieces of software. You will learn how these pieces connect and work internally, where they originated, and how they evolved. The book also covers several key areas of the Intel-based Macintosh computers.

A solid understanding of system internals is immensely useful in design, development, and debugging for programmers of various skill levels. System programmers can use the book as a reference and to construct a better picture of how the core system works. Application programmers can gain a deeper understanding of how their applications interact with the system. System administrators and power users can use the book to harness the power of the rich environment offered by Mac OS X. Finally, members of the Windows, Linux, BSD, and other Unix communities will find the book valuable in comparing and contrasting Mac OS X with their respective systems.

Mac OS X Internals focuses on the technical aspects of OS X and is so full of extremely useful information and programming examples that it will definitely become a mandatory tool for every Mac OS X programmer.

About the Author

Amit Singh is an operating systems researcher currently working at Google. Amit was formerly with the IBM Almaden Research Center. Previously, he worked for a Silicon Valley startup that did cutting-edge work in the virtualization of operating systems. Amit also was a member of technical staff in the Information Sciences Research Center at Bell Laboratories, where he worked on operating systems and networking. He created and maintains http://www.osxbook.com and http://www.kernelthread.com.

Excerpt. © Reprinted by permission. All rights reserved.

Although Mac OS X is a relatively new operating system, its lineage is rather colorful, and the history of most of its components is much older. Mac OS X, and Apple in general, has attracted a lot of attention in recent years. Given Apple's "cult-like" status, and the unique cultural and technical composition of Mac OS X, it is not surprising that people with vastly different backgrounds and interests—both users and non-users of Mac OS X—are curious about the system.

After working on, programming on, and extending several operating systems over the years, I was introduced to Mac OS X on April 1, 2003. I was soon curious about the structure of the system. Despite the availability of several good books on Mac OS X, I was dismayed to find that I could not learn the details of Mac OS X internals from a book—no such book existed. There are books describing how to perform various tasks on Mac OS X; how to configure, customize, and tweak the system; and how certain user-visible aspects differ between Mac OS X and Windows. There are also books covering specific Mac OS X programming topics, such as the Cocoa and Carbon APIs. Other books make life easier for a Unix user migrating to Mac OS X—such books typically discuss the command-line interface to the operating system. Although these books play important roles in making Mac OS X accessible, the core architecture and implementation of Mac OS X and its components have remained mysteries. To make matters worse, besides the lack of information, it is common to find misinformation on the composition of Mac OS X. Consequently, the system is often misunderstood because of long-standing myths and stereotypes, or is perceived as a black box.

The purpose of this book is to deconstruct and demystify Mac OS X by describing it from a system-design perspective. It takes an implementation-oriented approach to understanding the system. Consider the example of interprocess communication (IPC). Mac OS X has numerous user-level and kernel-level IPC mechanisms, some of which are well known and documented. Rather than just showing how to use these mechanisms, the book explains the design and implementation of the most fundamental mechanism (Mach IPC), and then discusses how the other mechanisms are layered on top of one another. My goal is not to teach you how to do something specific—my goal is to provide you with enough knowledge and examples so that after reading the book, depending on your interests and background, you can build upon your newly gained knowledge and do what you choose.

Along with text, the book uses detailed illustrations, function call graphs, annotated code excerpts, and programming examples to present a detailed examination of Mac OS X. To keep the subject matter interesting and accessible—even to the casual reader—the book contains relevant trivia, digressions, and other tidbits.

Who This Book Is For

I hope the book will be useful to anyone curious about the composition and working of Mac OS X.

Application programmers can gain a deeper understanding of how their applications interact with the system. System programmers can use the book as a reference and to construct a better picture of how the core system works. In my experience as a programmer, a solid understanding of system internals is immensely useful in design, development, and debugging. For example, you know what the system is capable of, what is feasible and what is not, what the "best" option is in a given situation, and what the plausible reasons are for certain program behavior. This book's primary goal is to build a strong foundation for anyone who programs on Mac OS X.

Mac OS X users can read the book to better understand how the system is designed and how it comes together. System administrators and technical support staff will also find value in the book.

Besides those who use Mac OS X, the intended audience includes members of other technical communities, such as the BSD, Linux, and Windows communities. Given that many internal aspects of Mac OS X are radically different from these systems (for example, how the Mach kernel is used), the book will help such readers broaden their knowledge, and will assist them in comparing and contrasting Mac OS X with other operating systems.

The book will also be useful in an advanced operating systems course, particularly if you wish to do a case study on Mac OS X. The book is not suitable, however, as an introductory text. Most of the material is not presented at an introductory level, although I introduce many of the advanced topics with at least some background information.

The Book's Structure

Modern operating systems have become so large and complex that it is impossible to reasonably describe an entire system in a book. This book is somewhat ambitious in that it attempts to cover Mac OS X in substantial breadth and depth. The most important contributors to the book's depth are the carefully selected programming examples. The book is organized into twelve chapters. Although much of the book's content is rather technical, each chapter has sections that should be accessible to non-programmers.

Chapter 1, "Origins of Mac OS X," describes the technical history of Mac OS X and the systems it derives from. An unabridged version of Chapter 1, which covers all of Apple's past and present operating systems, is available on this book's accompanying web site.

Chapter 2, "An Overview of Mac OS X," is an eclectic tour of Mac OS X and its important features. It contains brief overviews of various layers that constitute the system.

Chapter 3, "Inside an Apple," describes the PowerPC architecture, using the PowerPC 970 ("G5") processor line as a specific example. It also discusses the PowerPC assembly language and calling conventions.

Chapter 4, "The Firmware and the Bootloader," describes both Open Firmware and the Extensible Firmware Interface (EFI), along with their respective boot-loaders. It discusses the roles the firmware and the bootloader play in the system's operation, usage scenarios, and events that occur during early bootstrapping.

Chapter 5, "Kernel and User-Level Startup," describes the sequence of events—including initializations of kernel subsystems—from where the kernel starts executing to the point where the first user-space program (launchd) is run by the kernel. The discussion includes launchd's function and implementation.

Chapter 6, "The xnu Kernel," describes the core kernel architecture of Mac OS X. The discussion includes system call families and their implementation, low-level tracing and debugging mechanisms, and special features such as the virtual machine monitor in the PowerPC version of the kernel.

Chapter 7, "Processes," describes abstractions such as tasks, threads, and processes, the various forms in which they exist in Mac OS X subsystems, and processor scheduling. The discussion includes using various kernel-level and user-level interfaces for manipulating the aforementioned abstractions.

Chapter 8, "Memory," describes the Mac OS X memory subsystem's architecture, including discussions of the Mach virtual memory architecture, paging, the unified buffer cache, the working-set detection mechanism, kernel-level and user-level memory allocators, and support for 64-bit addressing.

Chapter 9, "Interprocess Communication," describes various IPC and synchronization mechanisms available in Mac OS X. In particular, it discusses the implementation and usage of Mach IPC.

Chapter 10, "Extending the Kernel," describes the I/O Kit, the object-oriented driver subsystem in Mac OS X.

Chapter 11, "File Systems," describes the overall file system layer in Mac OS X, including brief discussions of each file system type. The discussion also covers partitioning schemes, disk management, and the Spotlight search technology.

Chapter 12, "The HFS Plus File System," describes the internals of the HFS Plus file system. The discussion is aided by the use of a custom file system debugger written for this chapter.

Appendix A, "Mac OS X on x86-Based Macintosh Computers," highlights the key differences between the x86-based and PowerPC-based versions of Mac OS X. Besides this appendix, the book covers the details of several key x86-specific topics, such as EFI, GUID-based partitioning, and Universal Binaries. Most of Mac OS X is architecture-independent, and consequently, the majority of the book is architecture-independent.

Given the book's length, I chose to exclude several topics that are well covered in other texts. The TCP/IP stack is an example—there is no "networking" chapter in the book since the Mac OS X TCP/IP stack is largely a derivative of the FreeBSD stack, which is already well documented. In general, information that is generic across Unix variants and can be found in standard texts is not included in this book.

How to Read This Book

Since the first two chapters provide the background and overall picture of Mac OS X, respectively, I recommend that you read these chapters first. The subsequent chapters are best read sequentially, although, depending on your interests and familiarity with the topics, you can skip certain sections (and perhaps even chapters) and still gain value from the book.

It will help if you have some familiarity with operating system concepts and have used a Unix operating system.

Given that the book has a large number of C programs and program excerpts, you should have some programming experience, and in particular, knowledge of the C programming language. I sometimes use code not only to demonstrate the working of a concept, but also to describe the concept. I realize that it is usually considered "difficult" to "read" code, and authors often expect that many readers will simply skip the code. My belief is that reading the code (as opposed to only running it) in this book will be particularly helpful to programmers.

Spotlight Reviews :

Reviewer: S. Gylfason "snorrigy" (Iceland)
I had the opportunity to review the draft manuscript of Amit Singh's Mac OS X Internals book. With so many different types of operating systems books out there, let me try to place the book to give a better idea what to expect. There are general introduction books that normally introduce the operating system to the reader, without explaining what is actually going on. We have concept books (I put "The Design and Implementation of the FreeBSD Operating System" in this category) which are usually a good introduction to a new system. I used to be a great fan of this type of books, and I still enjoy reading them, but I don't anticipate much new from them. The fact is that operating systems today differ not that much in concepts and abstractions but more in their implementation. Then we have the kernel programming books that either cover the kernel programming in general, like Linux Kernel Internals, or focus on specific parts of the kernel, like Linux Device Drivers.

The Mac OS X Internals book falls into a category that I call OS Internals books category. Books in this category (like the popular Inside Windows books) are similar to the concept books in the sense that they are not focusing on solving some predefined problems, but rather share knowledge. They differ from the concept books in that they approach the concepts from implementation point of view. In recent years I have become a great fan of this type of books. Books in this category are both very enjoyable for anyone interested in the OS but also very useful for application developers.

Actually Amit's book does start off as more of a concept book, and in the first part of the book he gives a great overview of the Mac OS X system, which should be an interesting read even for people not using Mac, but with general interest in operating systems. I found it particularly interesting to read because OS X is so different from other operating systems I know and love, like Linux and Solaris. We've all heard how Mac OS X is built on top of the Mach kernel, uses large parts from BSD, supports backward compatibility via the Carbon API, etc. Not until I had read the first part of this book I fully understood how all the pieces fit together.

In my opinion the first two chapters are rich enough in content and interesting and fun to read for me to recommend this book to anyone interested in operating systems, regardless if they will ever do any programming on a Mac.

The remaining parts of the book cover OS X in a logical order, from the bottom up. Since the book is more focused on educating the reader of how things work rather that trying to teach how to do some particular task (like a network programming book would do) it is important to realize that the book is covering a lot of content, which may not all be of interest to you. If you are more interested in some particular area it is probably wise to jump directly to the appropriate chapter.

The book covers a lot, including the xnu kernel, the boot process, and the role of the firmware. It covers typical OS topics, like processes, virtual memory management, IPC, file systems, and the I/O Kit. What made these chapters especially interesting was to see exactly what part the Mach kernel plays here.

While reading the book I have to say that I have become a great fan of Amit's style. He manages to find a good balance between breadth and depth in his coverage, while keeping the text interesting. Each chapter covers each topic to a reasonable level. He achieves the depth by carefully selecting parts where he drills in quite extensively. Here he uses examples to clarify things. His examples are almost all excellent, both usually short, and to the point. The examples both lend the reader a first hand experience with some low level concepts, but also work as a starting point for the reader to actually try out things for themselves. Amazingly the examples usually don't require anything special beyond the normal dev setup. Meaning, you don't have to do any kernel programming to run most of the examples, which is great for application programmers like me.

After reading this book, I can recommend this book to anybody interested in operating systems and to all developers for the OS X system. OS enthusiasts will get a great overview of the Mac OS X system which will allow them to compare OS X to their own system, be it Linux, Windows, or Solaris. OS X application developers will probably gain the most from this book. This is not a kernel programming book but a book that builds up a strong base for application programmers. Whereas the book is not directly covering any specific class of API, Cocoa, Carbon or POSIX, it builds up a very strong base. For example, after reading the IPC chapter, things like Cocoa notification, Cocoa tasks and threads, remote objects, all become very clear. It will also make it much easier for anyone familiar with for instance Carbon or POSIX to move to Cocoa, because you will understand the common part of the API you know and the new one. Finally, for anyone doing anything advanced in Mac OS X, this book is a must have.

Reviewer: M. D. Silverman "bookworm" (Lawrence, KS USA)
"MacOS X Internals" is the first book introduced since the advent of OS X that focuses on the low-level details of the MacOS in a way that will give technical people, ranging from geeky end-users to hardware-level programmers, a thorough and fascinating tour of the MacOS. This book is not for "average" end users; it is not an introduction to how to use a Mac. Nor is this book for people looking to get started programming on the Mac; there are plenty of books designed for new Mac developers. What "MacOS X Internals" tries to be is something unique: a guide to the MacOS from the bottom up, and it hits this mark very well.

"MacOS X Internals" can be roughly divided into three parts: a technical history of the MacOS, a review of the MacOS firmware and booting process, and a technical tour of various parts of the OS itself (such as interprocess communication and the file system, to pick two at random). The first section will appeal to any technically-included user of the MacOS, the latter two are aimed more at programmers, although there is plenty there to increase the knowledge of anyone with a strong technical bend, whether they have coded before or not. The MacOS history is a detailed trip down memory lane from the very first post-Next developer builds of MacOS X right up to Tiger and beyond. The features introduced in each OS are described in detail with emphasis on technical and "under the hood" changes. The author is clearly an enthusiast as well, as he frequently sprinkles entertaining bits of trivia in with the meat of the main narrative.

Moving on from the historical sections to the present, author Amit Singh gives a blow-by-blow description of the process of the Mac booting. If you have ever wondered exactly what the Mac is doing as the power flows and the grey apple logo appears, this is for you. Everything from the sequence of initial power-on tests, to the bootloaders and Open Firmware onto the higher level unix boot process are covered in detail. Of special note is the extensive coverage of Open Firmware, which is much more then a merely the "BIOS" of a PowerPC Mac. Singh describes many activities that a system programmer can do in Open Firmware, including making a draggable windowing environment and even programming the famous "Towers of Hanoi." Unfortunately, this book was mostly finished before the Intel Macs arrived on the scene, and while there is a section on the EFI (extensible firmware interface) that is the "BIOS" of all Intel Macs, it is not nearly as deep as the Open Firmware section (updated information on EFI and other topics can be found on the author's web site at http://osxbook.com/).

Roughly following the guide to the MacOS booting, Singh delves into the meat of OS X's Unix underpinnings, describing how the MacOS really runs "under the hood." This section covers all the expected topics, including the kernel, interprocess communication, and memory is very thorough fashion, with tons of code snippets all clearly explained. This will be especially useful for readers who are familiar with how other Unixes work behind the scenes, as Singh frequently and helpfully points of the areas where MacOS differs from other flavors of Unix. There is a great section on the various file systems that OS X supports, which is notable because other then a simple list, I have never seen this information in a book with such detail. The section of the primary OS X filesystem, HFS Plus, is very useful for both developers and power users alike.

"Mac OS X Internals" clocks in at 1641 pages in the end, and is over two inches thick. I was thoroughly impressed by this book. The author's technical knowledge, as well as his enthusiasm and skill at presenting these topics is excellent. It's hard to find anything negative to say about this book other then the the above-mentioned need to have more information on EFI (which isn't the author's fault, it was a matter of timing). I do wish the book came with an electronic version, as this would have made searching and copying the code snippets much easier. There is a coupon included for a time-limited trial of online access to the text, but I would prefer that a PDF be included on a CD with the book, even if this required an increase in the price (you can buy an electronic version on Amazon, but this is separate from the print version). These minor complaints aside, I cannot recommend "MacOS X Internals" enough. It is the finest technical book on MacOS X yet published, and puts Apple's own technical documentation to shame in comparison. If you are a highly technical end user of developer, this book belongs on your shelf.