What 64 bits can do for you

Recently we bought a DELL PowerEdge 2950 and meanwhile deployed it for our web hosting service. We installed Debian Etch, the first Debian release that officially and fully supports AMD64. So we had to decide were able to choose between installing a 32 or 64-bit version of our favourite OS. Now what is that decision about?

Intel vs. AMD

As you might know, already a few years ago Intel tried (with the help of Hewlett Packard) to develop a new "Intel 64-bit Architecture" (IA64), specifically for their customers asking for more computation power and more memory. However, the problem with IA64 is that it is incompatible with the "traditional" architecture and instruction set known as x86, requiring vast changes to make operating systems, compilers and applications work with it. Maybe not surprisingly, IA64 never gained the importance Intel hoped.

AMD, on the other hand, carefully chose to extend the established x86 architecture to make it 64-bit capable. They made registers 64-bit wide, added a few more of them to the processor and came up with some new CPU instructions to leverage 64-bit computation operations. Contrary to IA64, this architecture named AMD64 still supports the x86 instruction set which basically means that it is capable of falling back into 32-bit mode and run "old" code without modifications. AMD64 was rewarded with a lot of acceptance in the market Intel aimed at and eventually Intel chose to license some of AMD's ideas. Now more recent Xeon processors are "EM64T" capable where EM64T is the "extended memory 64-bit technology" which basically is Intel's name for the same thing.

If you're unsure whether your machine supports EM64T, cat /proc/cpuinfo and look for the "lm" flag. Now what difference does that all make?

64 > 2 * 32 ?

Switching to a 64-bit OS doesn't magically make your applications faster. Most probably you will benefit if you're doing complex numerical calculations or simulations and your software is capable of (has been optimized for) using the 64-bit instructions, allowing it to calculate or transport "more" data per CPU cycle.

Often the more interesting aspect of running a 64-bit OS is address space. With 32-bit registers, you can only store 2³² distinct values. If these values are memory addresses, you will end up with a 4 GB address space a single process can use. Once upon a time, "640 k ought to be enough for anyone", but nowadays, especially with large databases you want to be able to access more than 4GB of memory — mainly to cache data in memory instead of reading it from disk when needed. So 64-bit is a good thing, right?

But...

Now here's the "but": There are some details that should not hinder you from switching over to 64-bit, but you can save yourself (or your overworked sysadmin) some time and headaches if you consider them first. As you will see, these are merely practical aspects that will vanish as 64-bit systems are getting more and more commonplace.

• An application must (of course) be compiled for 64-bit to make use of it. The same holds for all the libraries your application depends upon - when the app is 64-bit, the lib must be it as well. Now that may be a problem if either of them is not available for the AMD64 platform, maybe because it's closed-source or a vendor-provided device driver that has not yet been 64-bit-proofed.

  Debian provides x86 and amd64 versions of a package and puts 64-bit libraries into /lib64 which /lib is a symlink to.

  Now when you've got a 32-bit application binary, you can run it on an amd64 arch system (Surprised? Scroll up!) as long as you got the 32-bit versions of all the libraries it depends on installed as well. On Debian, you can install the ia32-libs package that contains 32-bit versions of some often-used libraries. The dynamic loader will figure out that you're trying to run a 32-bit application and load the libraries from /lib32 which is a symlink to /emul/ia32-linux.

  Trouble starts when you need to install the 32-bit version of a certain Debian package on an amd64 arch system or even need to add 32-bit versions of some libraries yourself. At that point you're pretty much lost because all that will no longer work under the control of the APT package management system, so you will have to care for dependencies and upgrade tracking yourself.

• The other way round, 64-bit binaries do not work on 32-bit (x86) systems. So what?! Why am I telling this?

  Maybe you have been using Debian long enough so you learned that the versions of software packages you "officially" get are somewhat... err... older. Debian "stable" means stability in version numbers, not stability through most recent versions. And basically, this is a good thing because you don't really want to change library versions too often on a production system as every such change brings the risk of regressions in behaviour.

  However, in the past we had a few situations where we absolutely needed to get a certain bug fixed in the libc, libxml2, MySQL or just had to keep up with the current PHP versions. So we started to maintain some variants of the Debian packages ourselves and set up a server to deploy these variants to all our machines leveraging the APT package system. (Maybe interesting enough for another article?) Now running amd64 on some machines means having to build all these packages twice, once on x86 and once on amd64 (did anyone say "cross-compile"?) and manage both architectures in our package repository — quite some extra work.

• It may be that application developers made assumptions about the size of some structures (pointers or ints) that held true on the x86 platform but are wrong in the general case. That may lead to bugs that suddenly surface in applications and libraries you have been using for a long time without problems on x86. Sara Golemon told such stories at the PHP conference 2006 code camp: It was a single wrong data type declaration in the PHP innards that caused days of tracing and debugging when they moved some apps to 64-bit at Yahoo!

  It may take some time for applications to become as mature as they are on x86, so keep this in mind.

• 64-bit applications tend to be somewhat larger and have a bigger memory footprint, especially because pointers are larger (finally, this is what it's all about :-)). While size on disk should not really be an issue, the in-memory size may adversely affect the efficiency of your precious level-2 cache.

Conclusion

Clearly, the future will belong to 64-bit systems. But it's up to you to decide whether you want to start the transition right now — operating systems support it — or still wait a little. Before you step up, be aware of the differences and the problems you might get.

Especially, save yourself some grief and don't change just for sake of change. Find out what your applications need — are you number-crunching or just serving files? Can you leverage 64-bit power? Need much more memory per process?

If you are unsure about it, the best thing you can do is go and figure it out! We took ourselves the time and tried various options, conducting a lot of IO- and especially MySQL benchmarks. But benchmarking is a complex matter that deserves an article on its own — hopefully coming up soon on this website.

So what did we finally choose and does it work for us? See yourself:

mp:~$ uname -r ; uptime

2.6.18-4-amd64

19:35:07 up 72 days, 7:37, 1 user, load average: 0.06, 0.08, 0.08