Benchmarking Microsoft Word 95 through Word 2007 - OpenOffice.org Ninja

Benchmarking Microsoft Word 95 through Word 2007

Posted by Andrew Z at Tuesday, July 22, 2008 | Permalink

The responses to benchmarking multiple versions of OpenOffice.org varied. Common responses were oversimplification of the results and some unrealistic expectations. To put that data into perspective, here is a benchmark for Microsoft Word 95 through 2007. That's over a decade years of releases, each of which has definitely become fatter. Before you read the benchmark results below, do you think over the years Word has become slower or faster?

Benchmark test environment

The modest test machine is about three years old.

  • Operating system: Windows XP SP3 (clean install)
  • CPU: AMD Athlon XP 3000+
  • RAM: 768 MB, DDR 333 (PC 2700)
  • HDD: Maxtor 6Y080L0, IDE, 7200 RPM, 80 GB
  • Video: Via VT8378 S3 Unichrome IGP at 1024x768

Though this is the same hardware as the OpenOffice.org benchmark, the results are not an apples-to-apples comparison between Microsoft Word and OpenOffice.org Writer because of differences in the test documents, operating system, and test procedure. Check back later for a direct comparison.

The Microsoft dilemma and the problem of quick starters

Benchmarking Microsoft Office has dilemmas. Microsoft Office 95 through Office XP install quick starters under various names in the Startup folder. These include Fast Start and Microsoft Office Startup Assistant (OSA). Their purpose is to improve startup performance by preloading Office into memory because hard drives are slow. Office 2003 and 2007 no longer load the OSA by default. The reason is likely because Windows XP introduced a prefetcher at the operating system level.

Quick starters don't really avoid any work. Quick starters simply hide the appearance of work by performing it before it is requested. To start the application, the operating system has to read the same amount of data from the hard drive (consuming disk I/O resources) and process it (consuming CPU resources). Being proactive is commendable, but there are drawbacks. First, quick starters use precious resources regardless of demand. If you don’t use the preloaded application, the resources that could improve other parts of the system are wasted. Speaking of precious resources, you probably experienced the second problem many times: the sluggish feeling of logging into a Windows XP session with many quick starters and system tray icons firing off simultaneously. Whether you notice them or not (because some load silently in the background), you may be waiting for Adobe Acrobat Reader Speed Launch, Quicktime, Winamp Agent, iTunes Helper, and others quick starters and services to load. It can easily be 30 seconds before the session is usable. A third problem with quick starters is they may be ineffective. Say you start Windows, then browse the web, and then open some PDFs, and then listen to some music. It's likely that by loading programs and accessing stored data, the operating system will eventually swap the preloaded application from RAM to the page file on the hard disk. Then, when you start the preloaded application, the operating system retrieves the file from the hard disk. In conclusion, quick starters are imperfect. Once we all get low-latency, low-seek-time solid-state drives so we can forget about quick starters, but these SSDs are only starting to enter the mainstream.

The goals of this benchmark include being realistic, being fair, being consistent, and measuring the amount of work as it relates to the user experience. If the quick starters were enabled, that would give Word 95 through XP an advantage over 2003 and 2007. Given the above problems and goals, I disabled Fast Start and OSA to better distinguish cold starts versus warm starts. I left Windows Prefetcher on its default settings. In the future when I benchmark OpenOffice.org on Windows, I will generally benchmark OpenOffice.org with its Quickstarter disabled too.

Benchmark test procedure

Very similar to the OpenOffice.org benchmark, this Microsoft Word benchmark uses automation to precisely measure the duration of a series of common operations: starting the application, opening a document, scrolling through from top to bottom, saving the document, and finally closing both the document and application. Automation is much more precise than a human with a stopwatch, and with the small durations in these tests, automation is necessary. Each of the five tests are repeated for 10 iterations. Before each set of 10 iterations, the system reboots. The purpose of rebooting is to measure the difference in cold start performance where information is not yet cached into fast memory. A reboot marks a pass, and there are 15 passes. That means for each version of Word, there are 150 iterations. Multiplying by 5 tests and by 6 versions of Microsoft Word yields 4500 total measurements collected for this article.

Benchmark results

A critical metric to the feeling of overall speed is starting an application on a cold start. The average cold start duration worsened 90% from Word 2003 to Word 2007. That's almost an order of magnitude. (In all the boxplots below, smaller is better.)

Notice the three outliers depicted as dots in the Word 2007 column. That is the Windows XP Prefetch progressively optimizing startup performance. The second cold start was slightly longer, but the third was about half. The fourth start was about a one-fourth of the first duration.

Warm starts from Word 2003 to Word 2007 deteriorated 247% from 0.13 seconds to 0.45 seconds. Also notice the difference for Word 2007 because cold and warm starts is 1.57 seconds - 0.45 seconds = 1.11 seconds. The difference of 1.11 seconds implies that work (71%) is done by the hard drive. To be fair, an application start under two seconds is quick.

Document tests were done with a special set of documents that use a wide variety of word processor features but are still backwards compatible to Word 95. Each version of Word used its own format for a total of three test documents: Word 95, Word 97/2000/XP/2003, and Word 2007.

The Word 97 format introduced multibyte encoding to support a wider variety of human languages: that made documents bigger and increased disk I/O (which is slow). That contributed to performance losses of 74% cold and 18% warm from Word 95 to 97. Word 2007 introduced the compressed .docx format which was smaller on the disk (so less disk I/O), but uncompressing and parsing XML requires much more CPU time. That contributed to performance degradation of 305% cold and 160% warm from Word 2003 to 2007.

Scrolling may measure screen painting, loading dynamic objects, and OLE automation. Again, Word 2007 is slower.

The times for exporting the document are all relatively fast, but performance still declined 114% cold and 58% warm from Word 2003 to 2007.

The times to closing the document and application are all reasonably quick, but guess what? Going from Word 2003 to Word 2007, it took 82% longer on a cold start and 75% longer on a warm start.

In the beginning, what was your guess: did you guess Word 2007 will be faster or slower? Well, here are all the tests rolled up together.

Word 2003 burned 7.05 seconds to run the gamut after a cold start and 6.17 seconds after a warm start. Word 2007 consumed 16.12 and 12.13 seconds (respectively).

Benchmark conclusion

Despite all the tweaks, tricks, hacks, and clever engineering, Microsoft's latest version of Word is a step back in performance. That's generally true of every new major release of any software. That's Wirth's Law. When Windows 95 came out, people joking exaggeratedly that "95" was the number of floppies Windows 95 shipped on. Though Windows 95 actually shipped on many fewer floppies, the point is that Windows 95 was a huge, slower beast compared to its predecessors. (Does anyone remember Windows 3.1, OS/2 Warp, and DESQview?) There were similar complaints for XP, and Vista's performance was the subject of the infamous "Vista Capable" class action lawsuit against Microsoft.

All that to say, whether you are upgrading to the latest Microsoft Office or OpenOffice.org on the same hardware year after year, someday something somewhere will have to give: you'll need to stop upgrading your software or start upgrading your hardware.

Related articles

This article is the fifth in an ongoing series on performance.

23 comments:

RTC said...

Nice benchmark. Thanks for putting this together.

Kozuch82 said...

The measurings just do not make sense. You should have installed Win95 for Word95, Windows XP for XP and Vista for 2007 etc. on a average hardware used those times. This numbers have no statistical value.

Anonymous said...

You should be commended for all the time and effort that went into that, but isn't the bottom line conclusion, literally, just stating the obvious?

"Upgrading to the latest Microsoft Office or OpenOffice.org on the same hardware year after year, someday something somewhere will have to give: you'll need to stop upgrading your software or start upgrading your hardware."

Heck, if I could use the same software (or OS) forever, some dead horses in the basement would still be in the race. I still do run Paradox 3.5 for DOS on current hardware, and it flies!

Andrew Z said...

Kozuch82: There is one reason to do that, but on the other hand that introduces a new variable. My point was to isolate Microsoft Word without skewing the results for differences in performance of Windows.

Anonymous: No, not obvious to all! My main reason for starting this article (which took weeks of hard work, by the way) was the reactions to the "Is OpenOffice.org Getting Faster?" benchmark article. I read through dozens of comments of people expecting new releases of the same software product to be faster. I hoped this Word benchmark would help those people.

RTC said...

I was going to mention, this had to have taken a long, long time. Just setting up all the software would take a couple of weeks. That must be why you haven't posted in a while?

Anyway, kozuch82, if you look back at the article looking at the same stats for OpenOffice, andrew didn't use different version of Linux (or Windows) for that article. He used the same versions. For comparability, it makes more sense to use a single OS version. Additionally, considering the purpose of the blog post is not to compare the speed on different platforms but the speed relative to each other, limiting the variables makes the most sense.

Your comment raises an interesting point: Is there optimization of versions of Word for different iterations of the operating system? Probably (andrew mentions some). But that's not what this article is about.

The only other thing I think would be helpful would be to quantify the changes in iterations of Word in terms of percentages and then overlay those on a graph with OpenOffice to see the two packages simultaneously. It would be a little tricky to do, but would put a nice finishing touch on this issue.

Robert said...

One idea to look at (especially between 2003 and 2007) would be the change to the docx extension. We saw a huge increase in file load time between 2003 and 2007. As I understand it, the 2007 files both compress and use XML to read/write files, which is much different than back in 2003.

Anonymous said...

> reactions to the "Is OpenOffice.org
> Getting Faster?" benchmark article. I
> read through dozens of comments of
> people expecting new releases of the
> same software product to be faster. I
> hoped this Word benchmark would help
> those people.

So..because Word gets slower, its okay for OOO to get slower, too? Great reasoning. Wow.

Please take a look at Softmaker Office.
OO may have some more features than this suite. But if you, once in your lifetime, go away from Windows and KDE and GNOME and the fat setups of the large Linux distributions, you'll notice that computers actually got *faster* over the last years.

OO is fucking slow. I only ever use it if I have to.

RTC said...

anonymous, nice advertisement. That wasn't a particularly useful comment.

Anonymous said...

Nice work! By any chance have you done something like this for outlook or Vista vs XP?

OlsonBW said...

"Does anyone remember Windows 3.1, OS/2 Warp, and DESQview?"

I remember all three.

One thing I remember in particular was having a 386 computer (I don't remember how much RAM).

I was given an additional 386 computer to get more work done. Both had Windows 3.1 on it. The new one was slightly faster with slightly more RAM than the "old" one which was about six months old at the most.

I had a spare hard drive and took out the HDD from the "old" computer and put it in and installed a beta for OS/2 2.0 and installed the Apps that I had on my Windows 3.1 box (WordPerfect, Lotus 1-2-3) and was stunned when the DOS and Windows 3.1 programs ran significantly faster on the the OS/2 box than on the new Windows 3.1 box.

One spreadsheet I had took over a minute in Windows 3.1 and took about 12 seconds in a Windows 3.1 box in OS/2.

I found that I could four or fives times the amount of things on OS/2 than I could with W3.1 alone. I installed it on the new box and stuck with OS/2 until it was just impossible due to W95 incompatibilities. Now OS ones but app incompatibilities where I couldn't open up documents anymore.

I didn't get rid of OS/2 at that point. One computer ran OS/2 for years after that and I was much happier using that than '95. It was actually when W2000 came out, which still wasn't as good as OS/2 but things had just come to a head and it just wasn't worth it anymore.

Like sucked until Mac OS X 10.2. Now I use that as much as possible (more so each day) along side my (at work only) XP box which is used less each day. I'm not going to bother getting Vista.

No I don't work for MS or Apple or IBM or any company named or remotely associated with named companies or products.

Rex Ballard said...

So the bottom line is:

Spend $millions per thousand users to upgrade their hardware/software, back up their data, and get them up to Vista/Office 2007, and my overall labor costs will DOUBLE!!!

Maybe that's why Microsoft had to hire a whole bunch more people and their costs relative to their income is getting out of control.

Such a bargain.

Maybe Open Office is "Good enough".

If Microsoft won't let me buy XP on my next machine, maybe I'll just install Linux, have VMWare Converter create a back-up of my old system, and I'll run that on the VMWare/Linux using VMWare Player or VMWare workstation.

Then I can use Linux/OpenOffice most of the time, and Windows only when I absolutely need it.

If I buy the machine with Vista, I'm legally allowed to downgrade.

But if Microsoft doesn't like that idea, maybe I should just scuttle Windows completely.

I'll bet there are quite a few CIOs thinking this way right now.

John Mann said...

How about a comparison of Word 2007 + .doc file versus Word 2007 + .docx file?

Is the Word 2003 to Word 2007 slowdown more to do with the application or the file format?

Andy Babiec said...

I think the interesting aspect is that performance improved (even if slightly) between 97 and 2003 - counter to the myth that every office version is slower then the previous. In fact, if you took out scrolling, there would be a bigger jump.

Andrew Z said...

Anonymous: Sorry, I don't plan to look at Outlook or Vista. By the way, I chose XP SP3 because that is what I had. Also, most computers today run XP, and there have been some controversies about Vista performance (though allegedly fixed in Vista SP1). Now, if someone wanted to donate a Mac, I could do a three way comparison of OpenOffice.org on three operating systems on the same hardware. :)

John Mann: Interesting question, but I may not be able to answer it as this blog focuses on OpenOffice.org.

Andy Babiec: I agree that Wirth's Law is not an absolute law, but there is definitely a trend. You can see the same exceptions in OpenOffice.org which significantly improved performance such as from 1.1.5 to 2.0.0. In "Is OpenOffice.org Getting Faster?" I suggested that these exceptions often indicate poor optimization in the older version.

Anonymous said...

This tells nothing. Honestly i even think it's stupid to compare everything this way.

sukru said...

Thanks for the nice work. I'm personally OK to spend 2 times more on disk activity, if Office 2007 can make my regular work (editing) faster (which it does).

Anyways, as someone else mentioned, a 2007 .doc format comparison would be nice.

Anonymous said...

Hmmm, what the benchmarks show--besides the slowdown in 2007--is that Microsoft Word 95-2003 really was simply a series of refinements of the same code base, i.e. minor upgrades, where Microsoft did not add many new features, instead focusing on bug-fixing, performance-tweaking, and fine-tuning.

Word 2007, in contrast, has been a rather marked break with the past--more so than any of the preceding versions listed were with their predecessors. It's as new as Word 6/95 was, and for this reason, slower. But if the tests here are any guide, we may expect future, incrementally improved versions of Word to be faster, at least until the next big release in, say, five to ten years.

Anonymous said...

Word 2007 is a bunch slower, but still quite a bit faster than OpenOffice 2.4. Then if we consider how fast Word 2003 is compared to both ... OO is even slower.

I'll admit to be slamming OO here, I read the "Is OpenOffice getting slower?" article at this site and the excuses. MS Office has improved dramatically over the years .. hopefully the next version of office will fix these performance regressions (outlook sure took a hit).

Anonymous said...

This is a nice comparison of Office performance. Can you provide details and the names of the tools you used to script the office benchmarking? I would like to run a similar benchmark against my own hardware and OS image.

Andrew Z said...

Anonymous: Good question. It took many months of work for me to create the tool. It is a heavily modified and specialized variation of the available tool called Jungletest QA (which contains no benchmarking features). I hope to publish the Jungletest benchmarking tools for OpenOffice.org and Microsoft Word this year.

Jezza said...

Nice comparison however, compared to your OOO comparison, MS Office 2003 is still faster than OOO and 200y is about the same speed as 2.4.

Each office suite is designed with the hardware of the time and running MS 2007 on a three year old machine will of course get you slow results and running MS 2003 on the same machine will get you faster results.

I know you are comparing the speed of each suite on the same platform however, it would be stupid of me to even try and run 2007 on my 5 year old PC (512MB RAM, 80GB IDE) when it runs much faster (obviously) on my newer PC (Dual core 1.8, 320GB SATA, 2GB RAM).

Personally, I'd rather use OOO then Office 2003 but on my older machine, 2003 runs much faster than OOO.

Andrew Z said...

Jezza: I think you are making my point without realizing it? :) Basically, my point is generally all software becomes slower over time.

Keith said...

Good stuffs, old but relevant to my fiddlings with my Eee. However surely it is necessary to clear the Windows prefetch folder before each new version is tested for cold start? I know that XP and 2003 share pretty much everything on the technical side so could the lack of performance increase between cold starts be explainable because it's using data in the prefetch from XP?