Xserve Notes

Moving to a Mini Server

September 25, 2005:

I recently got a new Mac Mini (1.42GHz, 1GB RAM, 80GB disk) which I hope to use as a small, quiet, but very capable replacement for my aging blue and white PowerMac G3 (350Mhz, 256MB RAM, 4GB disk) which is running Mac OS X 10.3.4 Panther Server. This trusty Web server has been used by my family and also serves several other Web sites of mine with various boutique domains.

It has run pretty flawlessly for the past couple of years, during which time I’ve also added a couple of blogs, Movable Type and, more recently, PixelPost. I used to run a mail server, but it succumbed to massive waves of spam a couple of years ago and I since moved all of my mail accounts to Digital Forest (http://www.digitalforest.net) which enables me to leverage the excellent spam service (Postini) to which they subscribe. Well worth the money as this has been very reliable and my family no longer hounds me about email problems. On the rare occasion when Digital Forest goes down, maybe 2 or 3 times a year, it’s usually back on line pretty quickly and everyone is happy. If not, at least I can foist the blame off on them.

The blue and white PowerMac, which came out in 1999, was the first Power Mac that came in the cool new plastic case with built-in handles, taking a cue from the fabulous colored iMacs of the day. More about that model can be found on this Apple history page. 6 years later, the Mac mini is least 5 times the speed, with 4 times the memory and 20 times the disk space. My mini also has bluetooth, WiFi and the ability to burn CDs & DVDs all in a package that's about a tenth the size and probably about half of the original cost of the blue & white G3 in 1999. The wonders of computing in the 21st century!

My plan was as follows:
1. Set up Mini with Tiger Server
2. Migrate Web data, accounts, MySQL data
3. Configure various files for Movable Type and PixelPost
4. Get it all working
5. Switch the servers (get my router to point to the new server for all Web services)

I did a clean install of Mac OS X 10.4 Tiger Server, upgraded it to the current 10.4.2 release and all was fine. But I suspected that the real fun was just about to begin!

Software RAID Partitioning Best Practices

All recent Macs support software RAID in the BIOS, including the Xserve. However, it can be difficult to set up, and there is very little advice on best practices for how best to use it.

The key issue is that unlike many other operating systems that support software-based RAID, you can only RAID entire drives, not individual partitions.

Thus on the Xserve G5, there appear to be two choices:

• Build a mirrored RAID 1 out of the first two drives, install the OS on it, and use the third drive as a backup drive that is easily removable, or as a hot spare. This solution appears to offer the most reliable boot, as if the first drive fails, the second drive is still bootable. However, the system files and data share the same partition, making backup and restore more difficult.
• Install the OS on a non-RAID drive, and use the second and third drives for data using mirrored RAID 1. The advantage of this is that you can partition the first drive into several smaller volumes, making it easier to restore the system. The disadvantage is that if the boot drive fails, you can't boot until you recreate it. Another disadvantage is that the Mac OS X Server 10.4 installer doesn't easily allow you to separate system files from data.

I've tried both approaches, but I'm now leaning toward second. My first drive has 4 partitions -- 3 identical 30GB partitions, the remainder a spare volume. The first 30GB partition I boot from and is my working OS. The second partition is a clone of the first, in which I use Carbon Copy Cloner on a regular basis to keep up with the boot drive. The third is for experimenting, but for now I keep a copy of the OS X Server in stand-alone mode before I made any changes or added any services.

One particular advantage of this scheme is that I can use Carbon Copy Cloner to update the clone on the second partition, and then use Disk Utility to image it without rebooting the system (you can't image a boot volume with Disk Utility). This image I then store on an old iPod. This means that if anything goes wrong, I can quickly restore the disk image into a replacement boot drive without touching the RAIDed data drive.

There is also an option in Carbon Copy Cloner to support the creation of something called ASR  (Apple Software Restore) disk image which can be burned onto a DVD-RW which sounds like it would make reinstalling even easier.

The challenge of using this scheme is separating out the data from the boot volume, which I will leave for future blog posts.

VNC on OS X Server

I use VNC (Virtual Network Computing), a cross-platform, open source protocol for allowing remote control between different types of computer, to remotely administer the GUI on a number of different servers. Thus one of the first things that I wanted to get working after my initial install of OS X Server was some form of VNC so that I could install the server in my rack in the basement, but still run it from upstairs.

Apple does support VNC , in both OS X 10.4 (Panther) Client as well as OS X 10.4 Server. To turn it on, go to "System Preferences" and in the "Sharing" panel click on "Apple Remote Desktop". Then click on the "Access Privileges" button and turn on "VNC viewers may control screen with password" and enter a password. When you click "OK" you can then click on the check box next to "Apple Remote Desktop" to turn on the service.

Now you can connect to the OS X  GUI via any VNC client and the password. I like free Chicken of the VNC on the Mac, and Ultra VNC Viewer on the PC, but some people swear by RealVNC which has versions for many platforms.

My personal experience with the default VNC server in OS X is that it is a little flaky, probably because the service appears to be oriented toward allowing the Apple Remote Desktop administrative server software to update client computers, rather then focused on the VNC functionality.

Thus I instead use OSXvnc, an free VNC server. You need to make sure that Apple Remote Desktop is turned off in the Sharing panel, but once installed in your applications or utilities folder it appears to offer a much more reliable VNC experience.

One particular thing that is interesting about the most recent version of OSXvnc is that now it offers a capability to have different people using multiple desktops at the same time -- normally VNC only gives you access to the primary monitor that must be shared by all ciients. But taking advantage of the Fast User Switching capability of OS X, now each user can use a different virtual desktop.

Another useful trick that you can do with  VNC is to allow multiple machines with monitors that are side-by-side use the same mouse and keyboard, even if they are not on the same platform. If you have multiple monitors, this eliminates the requirement of KVM switch.  I've not used any these tools yet, but osx2x looks very Macintosh centric.

There are some other good links at the excellent Tao of Mac blog under the VNC tag.

Headless Video Support

Inspite of my problems installing OS X on a the Village Tronic MPDD Pro video card, the card does offer one feature that I like: optional headless support.

You might wonder why this might be important feature to have on a video card for a server? I've run into this problem before -- you attach a large monitor to a server and set the resolution high. You then put the server in the rack, and maybe a year later you need to put a monitor on it. However, this time, the monitor is the one in your machine room and it doesn't support the high resolution of the first monitor, so you can't get your monitor to work. This can be a real pain to solve.

What this optional patch does detect if there is no monitor attached and then lower the resolution to broadly supported size. The next time you attach a monitor, it will use this lower size.

OS X First User

A note for "Best Practices":

During a fresh install, the OS X installer will ask for the name of the first user and will give this user a different set of default items in the Dock. As you typically will be making a disk image of this install for future recover, use "Administrator|admin" and a long-term memorable group root password for this first user, rather then your own name and private secure password, as it may not be yourself that reinstalls the server.

Installing OS X 10.4 Panther with a Village Tronic MPDD Pro Video Card

I just purchased two used Macintosh Xserve G5s, however, they were configured very unusually. Rather then an Apple-branded video adapter they had a third party video adapter, the Village Tronic MPDD Pro.

The first Xserve doesn't have a RAID card, so attempted to install it first. When I tried installing OS X 10.4 Panther Server from DVD-ROM, it would start up OK, however, as soon as the bios was complete and the OS was started, the screen would go blank. My best guess is that there are no drivers for this card by default in the OS X installer DVD.

My first attempt was to boot the Xserve with FireWire Target Disk mode, and install using my Powerbook. I was able to use Disk Utility to mirrored software RAID drives 1 & 2, however, the installer would not let me install the RAID volume in FireWire Target Disk mode. So I installed the server on the spare drive 3. This worked up to very basic level, but even once installed, once OS X started, the video display would still die. Thus the drivers for this video card are not only not on the installer, they are not included with the system. So I couldn't finish the installation without a display.

As you can do a fresh install of OS X on headless Xserve with no video card at all, my next attempt was to use that technique. I got that working after a lot of pain searching for the correct serial number, which is used as the default password. This time I was able to install OS X on the mirrored software RAID volume. However, once you install, how do you install a video driver? I've never figured out how to remotely turn on Apple Remote Desktop and to configure it to turn on the "VNC viewers may control screen with password" option. So without the GUI, how do I install that driver?

So next I booted my Powerbook on the freshly installed OS X Server using FireWire Target Disk mode. This gave me the GUI, and was able to install the video drivers. I had to reboot twice to get them to work, but now my video card was working and I could use my regular monitor and keyboard.

I am mystified how the company that I bought the used Xserves from installed their OS X Server. It was clear to me from their setup of their machines that they had little experience with OS X Server, and I doubt that they installed using my technique. So either it came from CDW with the video driver already installed, or they figured out some other technique.

Now that I've got the first Xserve running, next I've got to try the same technique on the next Xserve, which has a hardware RAID card. Apparently the installer doesn't understand the RAID card, so you have to ssh into the installer and use the CLI (command-line interface) to configure the RAID. I'm not looking forward to it.

Mysteries in Anandtech's "No More Mysteries"

Anandtech published this summer two articles very critical of OS X Server's performance as compared to Linux:

• No more mysteries: Apple's G5 versus x86, Mac OS X versus Linux

  The server performance of the Apple platform is, however, catastrophic. When we asked Apple for a reaction, they told us that some database vendors, Sybase and Oracle, have found a way around the threading problems. We'll try Sybase later, but frankly, we are very sceptical. The whole "multi-threaded Mach microkernel trapped inside a monolithic FreeBSD cocoon with several threading wrappers and coarse-grained threading access to the kernel", with a "backwards compatibility" millstone around its neck sounds like a bad fusion recipe for performance.

  Workstation apps will hardly mind, but the performance of server applications depends greatly on the threading, signalling and locking engine. I am no operating system expert, but with the data that we have today, I think that a PowerPC optimised Linux such as Yellow Dog is a better idea for the Xserve than Mac OS X server.

• No More Apple Mysteries, Part Two

  It is clear that if you plan to run MySQL on Apple hardware, it is better to install YDL Linux than to use OS X. If you need excellent read performance, the maximum performance of your server will be up to 8 times better. If your server is only going to serve a limited number of users, YDL Linux will allow you to run with a less expensive system.

  If the usage pattern of your server is more OLTP, Transaction processing oriented, we give you the same advice. Our quick tests with InnoDB show the same kind of behavior and we have noticed very slow file system performance. At this point, we do not have enough data to be conclusive. We noticed, for example, that importing data in our database (via the ">" command) took up to 8 times longer.

Apple lost a sale last summer to one of my clients due to a lack of response on these articles. My Apple Xserve representative said that they were aware of the articles, but was told to say that there was no official response.

I still find the results quite mysterious -- is it only mySQL? Is it thread creation? What about Postgres which I don't believe uses threads.

MacOSHints OSX Server Forum

MacOSHints is a good general site for tricks and hints regarding OS X, but it also has a reasonably active sub-section, the OS X Server Forum. Unfortunately, although there is a titles-only RSS feed for the entire MacOSHints forums, there doesn't seem to be a way to get only the RSS for the the OS X Server Forum.

del.icio.us feed

I've added to the sidebar the 10 most recent items in del.icio.us tagged as "xserve".

AFP548 Community

The best online community for supporting the Xserve that I've found so far is AFP548. I've added their RSS feed in my sidebar here, but their forums are much more the place where the community gathers.