The HINT benchmark on different operating systems
Back in 1998, I used the HINT benchmark to compare the performance of a few operating system kernels on the x86 architecture.
The Setup
The following operating systems were compared:
Linux 2.0.34 (non-SMP, glibc 2.0.7)
Linux 2.1.108 (non-SMP, glibc 2.0.7)
Windows 95a
Windows NT 4.0, Service Pack 3
The comparision was done on a Pentium II system (266 MHz, 512 KB non-ECC second level Cache) with 128 MB 10ns SDRAM and a Buslogic Flashpoint SCSI Controller. The mainboard was an ASUS P2L97 (built around Intel's LX chipset).
Differences in the results of the HINT benchmark are mainly due to different memory management implementations on these operating systems, since identical machine code for the HINT kernel was used on all systems. In other words, neither the C compiler nor the C runtime library should influence the results below. The HINT benchmark was stopped before swapping started, so swapping (and hard disk) performance doesn't influence this comparision either.
The results
On each plattform, the HINT benchmark was run three times. The diagram below reproduces the results of the third run. It shows QIPS (Quality Improvements Per Second, the more the better) over the amount of memory used during the computation. All the results are collected here.
Some Remarks
In general, the HINT curves are typical for a machine with a 16 KB first level data cache and a 512 KB second level cache.
The memory management implementation (or perhaps something else) of Windows 95 is very poor. Probably, Windows 95 perpetually pollutes the first level cache. The performance breakdown just before 2K is exactly reproducable. Performance is even worse on CPUs with only 8 KB of first level data cache: The effect of the first level cache is hardly visible in the HINT curve.
NT's MM is much better than Windows 95. The increased computation speed compared to Windows 95 has already been noticed in high-level application benchmarks, so it does matter.
Both Linux 2.1.108 and 2.0.34 are faster than NT if the amounts of data are really huge. But in contrast to NT, it looks like the second level cache is not fully used, as suggested by the sudden decrease in QUIPS performance well before the 512 KB mark. This performance decrease differs a lot among different runs of the benchmark.
Linux 2.0.34 shows some strange, but reproducible peaks in the HINT curve.
The Linux MM hackers can be pretty proud of their achievements. Their code beats Windows NT asymptotically, even though they do not have the backing by Intel that Microsoft receives.
Technical details
The comparision was done using the platform-independent HINT benchmark, using the double data type. (The results for the int type were similar and are therefore not shown here). In order to get identical machine code for all the platforms, the HINT kernel (hkernel.c) was compiled with the Cygwin gcc (Beta 19) and the resulting assembly code was used in both the Linux and Win32 binaries. Under Linux, gcc 2.7.2.3 was used to compile the HINT driver, and Microsoft Visual C++ 4.0 was used for Win32. In addition, the FPU precision had to be set to 53 bits on Linux (as it is the default with Microsoft's runtime library), since 64 bit precision (the glibc default) is noticeable slower.
Historical Note
These results were published first during summer 1998. Apart from removing broken hyperlinks, reformatting, and the addition of this note and the leading text, the page has not been updated since then. The HINT benchmark no longer seems to be available to the general public.
Revisions
1998-08-01: published (approximately)
2002-02-17: republished
2003-07-30: reformatted