Benchmarked: Ubuntu vs Vista vs Windows 7

Posted at 12:50pm on Tuesday February 3rd 2009

In depth: A lot of people have been chattering about the improvements Windows 7 brings for Windows users, but how does it compare to Ubuntu in real-world tests? We put Ubuntu 8.10, Windows Vista and Windows 7 through their paces in both 32-bit and 64-bit tests to see just how well Ubuntu faces the new contender. And, just for luck, we threw in a few tests using Jaunty Jackalope with ext4.

When Windows users say that Windows 7 is easier to install than ever, what do they really mean? When they say it's faster, is it just in their heads, or is Microsoft really making big strides forward? And, perhaps most importantly, when Linux benchmarkers show us how screamingly fast ext4 is compared to ext3, how well do those figures actually transfer to end users?

These are the questions we wanted to answer, so we asked Dell to provide us with a high-spec machine to give all the operating systems room to perform to their max. Our test machine packed an Intel Core i7 920, which in layman's terms has four cores running at 2.67GHz with hyperthreading and 8MB of L3 cache. It also had 6GB of RAM, plus two 500GB of hard drives with 16MB of cache.

The tests we wanted to perform for each operating system were:

• How long does each operating system take to install?
• How much disk space was used in the standard install?
• How long does boot up and shutdown take?
• How long does it take to copy files from USB to HD, and from HD to HD?
• How fast can it execute the Richards benchmark?

We also, just for the heck of it, kept track of how many mouse clicks it took to install each OS.

Before we jump into the results, there are a few things we should make clear:

• To ensure absolute fairness, install time was measured from the moment the computer was turned on until we reached a working desktop.
• The same computer hardware was used for all tests, and all operating systems were installed fresh for this article.
• We used the Ultimate versions of Windows Vista and Windows 7, simply because Windows 7 was provided only in this flavour.
• We used the Windows Vista SP1 disk to accurately reflect what users are likely to experience todaay.
• Our Windows 7 version is the open beta that Microsoft issued recently. It is probable Windows 7 will be at least this fast in the final build, if not faster.
• For Ubuntu 9.04 we used the daily build from January 22nd.
• All operating systems were installed using standard options; nothing was changed.
• After checking how much space was used during the initial install, each operating system was updated with all available patches before any other tests were performed.
• Our journalistic friends have informed us that Windows Vista (and, presumably, Windows 7 too) has technology to increase the speed of the system over time as it learns to cache programs intelligently. It also allows users to use flash drives to act as temporary storage to boost speed further. None of our tests are likely to show this technology in action, so please take that into account when reading the results.
• The filesystem, boot, shutdown and Richards benchmarks were performed three times each then averaged.

And, of course, there's the most important proviso of all: it is very, very likely that a few tweaks to any of these operating systems could have made a big difference to these results, but we're not too interested in that - these results reflect what you get you install a plain vanilla OS, like most users do.

Install time

Amount of time taken to install, from machine being turned on to working desktop. Measured in seconds; less is better.

At first glance, you might think that Ubuntu clearly installs far faster than either version of Windows, and while that's true there is one important mitigation: both Windows Vista and Windows 7 run system benchmarks part-way through the installation to determine the computer's capabilities.

A bit of a flippant one - just how many mouse clicks does it take to install an OS with the default options?

Surprisingly, Ubuntu 8.10 gets it done with half the clicks of Windows 7. NB: hopefully it's clear this doesn't make Ubuntu 8.04 twice as easy to install. Measured in, er, mouse clicks; fewer is better.

Disk space used immediately after a fresh install. Measured in gigabytes; less is better.

While some people might complain that we used the Ultimate editions of both Vista and Windows 7, they probably forget that the standard Ubuntu includes software such as an office suite as standard. NB: Vista failed to detect the network card during install, leaving us without an internet connection until a driver was downloaded on another computer.

Bootup and shutdown

Boot up time was also measured from the moment the machine was turned on, and the timer was stopped as soon as the desktop was reached. The Dell box does take about 20 seconds to get past POST, but to avoid questions about when to start the timer we just started it as soon as the power button was pressed.

Amount of time taken to boot, from machine being turned on to working desktop. Measured in seconds; less is better.

The 32-bit version of Windows 7 is the only one to beat the one-minute mark, but that advantage is quickly lost in the switch to 64-bit. Linux has always been rather slow to boot, but as we understand it reducing boot time is one of the goals of the Ubuntu 9.04 release.

Amount of time taken to shutdown, from button being clicked to machine powering off. Measured in seconds; less is better.

Windows lags a little behind the Linuxes, with 64-bit again proving a sticking point - this time for Windows Vista.

IO testing

To test filesystem performance, we ran four tests: copying large files from USB to HD, copying large files from HD to HD, copying small files from USB to HD, and copying small files from HD to HD. The HD to HD tests copied data from one part of the disk to another as opposed to copying to a different disk. For reference, the large file test comprised 39 files in 1 folder, making 399MB in total; the small file test comprised 2,154 files in 127 folders, making 603MB in total. Each of these tests were done with write caching disabled to ensure the full write had taken place.

Amount of time taken to copy the small files from a USB flash drive to hard disk. Measured in seconds; less is better.

Amount of time taken to copy the small files from one place to another on a single hard disk. Measured in seconds; less is better.

Let us take this opportunity to remind readers that Windows 7 is still at least nine months from release.

Amount of time taken to copy the large files from a USB flash drive to hard disk. Measured in seconds; less is better.

Amount of time taken to copy the large files from one place to another on a single hard disk. Measured in seconds; less is better.

With the exception of Windows 7 while copying larges files around a hard drive, Windows generally suffered compared to Linux in all of these tests. Obviously Windows does have to worry about some things that Linux doesn't, namely DRM checks, but these figures show a drastic performance difference between the two.

Notes: Vista and Windows 7 really seemed to struggle with copying lots of small files, but clearly it's something more than a dodgy driver because some of the large-file speeds are incredible in Windows 7.

Both Vista and Windows 7 seemed to introduce random delays when deleting files. For example, about one in three times when deleting the files from our filesystem benchmark, this screen below would appear and do nothing for 25-30 seconds before suddenly springing into action and deleting the files. However, this wasn't part of our benchmark, so isn't included in the numbers above.

This was very annoying.

Richards benchmark

Notes: This was done using the cross-platform Python port of Richards. For reference, Ubuntu 8.10 uses Python 2.5.2, Ubuntu 9.04 uses Python 2.5.4, and we used Python 2.5.4 on the Windows tests. Even though the 64-bit results for Linux and Windows don't look that far apart, we have to admit to being very impressed with the Windows tests - the deviation between tests was just 3ms on Vista, and 5ms on Windows 7, compared to 20ms on Linux.

Amount of time taken to execute the Python Richards benchmark. Measured in milliseconds; less is better.

It's clear from that graph that having a 64-bit OS can make a real difference in compute-intensive tasks, but it's not too pleasing to see Windows pip Linux to the post in nearly all results.

Switching to ext4

All the Linux benchmarks above were done using ext3, so what happens when we switch to ext4? Well, not a lot:

Boot, shutdown and filesystem tests for Ubuntu 9.04/x86-64 using ext3 (blue) and ext4 (red). Measured in seconds; less is better.

Although there's no difference in shutdown speed, the boot time using ext4 dropped by 8 seconds, which is a fair improvement. We can probably discount the the USB to HD tests simply out of error margin, which leaves the HD to HD tests, and there we find a very healthy boost: 3.7 seconds were shaved off the small files test, making ext4 about 25% faster. Our tests also showed an improvement in the large file test, but it's not as marked.

Conclusions

Benchmarks are always plagued with questions, uncertainties, error margins and other complexities, which is why we're not going to try to look too deeply into these figures. Obviously we're Linux users ourselves, but our tests have shown that there are some places where Windows 7 really is making some improvement and that's good for competition in the long term. However, Linux isn't sitting still: with ext4 now stable we expect it to be adopted into distros fairly quickly. Sadly it looks like Ubuntu 9.04 won't be among the first distros to make the switch, so users looking to get the best performance from their Linux boxes will either have to fiddle with the default options, have patience, or jump ship to Fedora - which will be switching to ext4 in the next release..