It's time for an OpenOffice.org benchmark rematch. Go-oo has been proudly boasting it is "Better, Faster, Freer," but last time when we tested OpenOffice.org 2.4, Go-oo came in fourth place out of four. Since then Go-oo developers addressed some problems, and now we're checking a new version 3.0 and adding benchmarks on Windows.
The following editions were tested. The abbreviated code identifies each edition in the charts.
Ubu Go-oo: Universal Linux build of Go-oo 3.0.0 on Ubuntu.
Ubu PPA: OpenOffice.org 3.0.1 on Ubuntu. Ubuntu 8.10 normally comes with OpenOffice.org 2.4, so this is the version from the popular OpenOffice.org PPA. Like the OpenOffice.org edition that ships with Ubuntu, PPA is based on Go-oo.
Ubu StarOffice: StarOffice 9 on Ubuntu with Java JRE 1.6.
Ubu Vanilla: Vanilla OpenOffice.org 3.0.1 on Ubuntu. Vanilla means it is the unmodified version available from www.openoffice.org.
Win Vanilla: Vanilla OpenOffice.org 3.0.1 on Windows without a Java JRE.
Win Portable: Portable OpenOffice.org 3.0.1 on Windows modifies the vanilla edition with executable compression, by removing some files, and by making other changes.
Win StarOffice: StarOffice 9 on Windows with Java JRE 1.6.
The test environment is a modest machine from several years ago:
- Operating system: Microsoft Windows XP SP3
- Operating system: Ubuntu 8.10 (Intrepid Ibex) with Linux kernel 2.6.27
- CPU: AMD Athlon XP 3000+ (32-bit, single core)
- RAM: 1.5 GB DDR 333 (PC 2700)
- Hard disk drive: SAMSUNG HD501LJ 500 GB SATA
- Video: Via VT8378 S3 Unichrome IGP at 1024x768
Like previous benchmarks, this OpenOffice.org benchmark uses automation to precisely measure the duration of a series of common operations: starting the application, opening a document, scrolling through from top to bottom, saving the document, and finally closing both the document and application. Automation is much more precise than a human with a stopwatch, and with the small durations in these tests, automation is necessary. Each of the five tests are repeated for 10 iterations. Before each set of 10 iterations, the system reboots. The purpose of rebooting is to measure the difference in cold start performance where information is not yet cached into fast memory. A reboot marks a pass, and there are 10 passes. That means for each edition of OpenOffice.org, there are 100 iterations. Multiplying by 5 tests and by 7 editions of OpenOffice.org yields 3500 total measurements collected for this article.
Some editions don't have a quickstarter, so I disabled it for all editions as in previous articles.
Two steps back
Speed doesn't tell the whole value of an application. People buy cars based on price, reliability, appearance, fuel economy, storage capacity, and so on. Likewise an office suite is measured by its compatibility, reliability, and productivity. This article focuses on one aspect—speed—but gathering these observations revealed a sea of regressions related to the automation necessary to gather precise performance measurements. All the OpenOffice.org 3.0 editions tested suffered from more bugs than OpenOffice.org 2.4, and StarOffice and Vanilla OpenOffice.org worked the best while OxygenOffice and Go-OOo contained the most difficult bugs. None of these bugs would affect a typical user, but it made benchmarking using automation difficult and, in some cases, impossible.
Let's quickly count these bugs
- Since before OpenOffice.org 3.0, automation sessions crash harmlessly on exist (issue 59026). No big deal.
- All Windows editions required a workaround for the error AttributeError: loadComponentFromURL (issue 90701).
- Go-OOo and OxygenOffice have a workaround for 90701, but it's worse on Windows.
- Using automation causes OxygenOffice and Go-OOo to crash on Fedora and Ubuntu (bug 479346). The bug is so severe it was not possible to test Go-OOo 3.0.1, OxygenOffice 3.0.0, or OxygenOffice 3.0.1. On Linux only Go-OOo 3.0.0 worked.
- A certain use of automation causes OxygenOffice 3.0 and Go-OOo 3.0 to crash on Windows (bug 487074) making them impractical to test on Windows.
- On Windows a regression prevented bootstrapping OOo (issue 99754). The workaround wasted a few hours.
on Portable OpenOffice.org would immediately quit on one of
my machines (the real test machine). The Windows Event Viewer logged theses
Dependent Assembly Microsoft.VC90.CRT could not be found and Last Error was The referenced assembly is not installed on your system.
Generate Activation Context failed for C:\OpenOfficePortable\App\openoffice\Basis\program\python-core-2.3.4\bin\python.exe.
Reference error message: The operation completed successfully.
- Portable OpenOffice.org froze four times during its 100 iterations.
While these regressions don't affect end-users directly, OOo should remember Steve Ballmer's mantra "Developers, developers, developers." It is the developers who build a rich ecosystem which makes OpenOffice.org attractive— especially to business users. Let's entice developers with an easy platform.
Cold start performance is the most critical factor to the perception of fast performance. While the Go-oo homepage brags about its fast startup, the results aren't a resounding win for Go-oo. Because of Go-oo bugs, it could not be tested on Windows, and on Linux it's a mixed bag for Go-oo where it takes first and last place. Clearly on Linux the PPA edition is the winner at 2.28 seconds, but it's the only one of the four Linux editions that can take advantage of system libraries already loaded by other programs such as GNOME. The three other editions (which ship with their own copies of system libraries to be compatible with different Linux distributions) are surprisingly indistinguishable considering the many performance factors such as features, patches, compiler version, and compiler options.
On Windows, I expected Portable OpenOffice.org to take the lead because of its use of compression, which minimizes disk I/O. (Disk I/O severely degrades cold start performance.) However, Vanilla OpenOffice on Windows won at 2.83 seconds. Vanilla's advantage over StarOffice may be a difference of features or that only StarOffice was tested with a Java JRE (which it installs by default).
Another important performance metric is warm start performance, which means starting OpenOffice.org a second time in a row while the disk information is cached in RAM. Because of the cache warm starts are always faster than cold starts. The dramatic difference between cold starts and warm starts shows the vital role of minimizing disk I/O. A quickstarter is intended to make all starts seem like warm starts, but quickstarters have their own drawbacks.
Like in cold starts, Linux PPA takes the lead in warm starts at 0.93 seconds. You can see by the outliers (indicated by dots) that Ubuntu PPA is unique: usually the second and all subsequent iterations have the same performance, but for Ubuntu PPA the second iteration scores halfway between the cold and warm. The three Windows editions are similarly grouped, and again, Go-oo takes last place.
The next step in the benchmark is opening reference document ODF_text_reference_v3.odt. When opening the document after a cold start notice how similarly Vanilla and StarOffice perform on each platform. Likewise, Go-oo and PPA perform similarly—probably because system libraries wouldn't give an advantage in this operation. Finally Portable has a 33% advantage over Vanilla—probably because of executable compression. StarOffice and Vanilla take last place with PPA again taking first place.
Don't put all your money on Ubuntu yet: things look completely different when opening a document after a warm start. Now Windows takes a big lead over Ubuntu, and Vanilla Windows shows a dramatic 44% edge over Vanilla Ubuntu. Portable OpenOffice.org's executable compression weighs it down only a fraction in this race where CPU speed is paramount. Go-oo's patches (which are also in PPA) help but not enough to crush Redmond. It's hard to say with certainty, but the credit could be due to efficiency in Microsoft Visual C++ 9 (used to compile OpenOffice.org on Windows) over the open source GCC (used to compile OpenOffice.org on Linux).
The scrolling test simulates someone pressing the down arrow key to scroll from top to bottom. This test is so synthetic it won't be included in the final tally. Still, you can see Microsoft has the upper hand, which I attribute to better video drivers. Also, the difference between cold and warm starts illustrates how OpenOffice.org dynamically loads code into memory as necessary to render objects on the screen.
If you don't like waiting to save a text document, don't use the Go-oo Universal Linux build. Otherwise, all editions scored similarly with StarOffice taking its first victory by a slight margin.
Now there is nothing left to do except close the document (which is already saved) and exit OpenOffice.org. Even here you can't shake the difference between cold and warm starts: OpenOffice.org is clearly accessing new data on the disk even while exiting. On Windows the warm start times are beautifully identical, and Portable's success over Vanilla in the cold start category implies OpenOffice.org is opening an internal DLL (perhaps to save settings to the OpenOffice.org registry). On Ubuntu, Go-oo takes first and last place after warm and cold starts (respectively). All these scores fit within my personal threshold for patience.
Counting all the races now, the champion of cold starts shouldn't surprise you if you read the last article. Last time the Fedora edition of OpenOffice.org licked its competitors on Fedora, and now Ubuntu PPA OpenOffice.org does the same on Ubuntu. Clearly there is a home field advantage on Linux: OpenOffice.org performs best when it avoids loading separate copies of libraries. As the Linux Standard Base (LSB) grows, maybe one day OpenOffice.org can avoid the library mess. Even so Intrepid Ibex squeezes out victory over Windows XP.
For people who use OpenOffice.org often, the warm start performance numbers carry more weight. Boosted by huge gains in the speed of opening a document, Windows XP whips the enduring Ibex.
Which OpenOffice.org edition is fastest? All OpenOffice.org editions and both operating systems performed well, and it's not possible to identify a single champion. Go-oo's tweaks often (but not always) gave it an advantage over Sun Microsystem's Vanilla edition, but OpenOffice.org PPA's system libraries gained the most substantial advantage.
OpenOffice.org 3.1 is just around the corner, and the rumor is the new performance improvements make is fast. I hope to see Go-oo and OxygenOffice fix the automation bugs, so they can be better represented in the next showdown. (I'd even more hope they would upstream all their patches despite the political drama, but that's another story.)
Subscribe to www.OOoNinja.com's news feed in the top right of the page. Upcoming articles will delve into practical steps you can take to improve performance while shedding light on some performance myths.