Sun Solaris
5
Caveat: I am a bigot - I used to work for Sun, in the Solaris group, and I have used Solaris for many years. On the other hand, I'm far from afraid to poke a stick at Sun when they do something I don't like.
I believe that Solaris 10 is the best server-side operating system in the world, full stop. It is years ahead of the Linux distros (which, by the way, I also really like) and is being continuously improved.
First, it should be noted that Solaris is based on the open-source OpenSolaris, so it is as "open-source" as Red Hat Linux and other Linux distros. And it is as "proprietary" as they are. Solaris 10 is to OpenSolaris as RHEL5 is to Fedora - you choose what you want.
Second, Solaris as a desktop or workstation OS is, well, weak. It's not bad, but it's not great, either. With the exception of OpenOffice/StarOffice, the desktop software available for Solaris tends to be backrev, which can cause problems (albeit relatively few), and there's not as much desktop software available for Solaris as for Linux, OS/X and, of course, Windows. The one thing about it - it almost never crashes. What it does, it does really, really well, but you wouldn't replace your Mac with a Solaris box.
Third, there's just no substitute for some of the unique functionality in Solaris 10 - Zones, DTrace, ZFS, etc. Some of these things can literally change the way a datacenter runs and save big $$$. Even if you're just doing relatively simple things (running a web server, for instance) the security capabilities of Solaris (privileges, for instance) can make a huge difference in keeping the server up and running smoothly.
Fourth, Solaris is free, as in free beer. Download either OpenSolaris or Solaris 10, put it on a server, and run. Support, of course, costs, but it's pretty reasonable.
I humbly submit that it's worth taking another look at Solaris.