Responses to "Is getting faster?" - Ninja

Responses to "Is getting faster?"

Posted by Andrew Z at Wednesday, May 28, 2008 | Permalink

Perhaps I was naive when I wrote "Is getting faster?". Two responses especially surprised me.

PhireN on Digg claims the charts are misleading because the Y-axes do not start at zero. (Actually the last two charts do start at zero, but the first seven don't.) All the charts use the default axis starting point setting of R, the well-respected statistical software I had the pleasure of using to generate them.

If I wanted to be misleading or dishonest, I could have saved myself about two months (out of three) working on that article. After having the results ready in early April, I reworked a significant part of the benchmark system to minimize deviations in the data, which would have been hidden had I chosen a more common chart than a box plot. I repeated the tests to minimize noise and presented the charts as a box plot to show the spread of the data. I didn't throw out any versions, tests, or data points to present a glowing review. I didn't re-engineer the test document to make it load faster. Mark Asay [1] at CNET responds, "I'm not sure if this is supposed to count as advocacy, but it hardly sounds like a ringing endorsement." A ringing endorsement would have been nice: I probably would have gotten more attention and back links from types. How many benchmark articles go through all this trouble?

In any case, here are the same charts with the Y-axes forced to start at zero. benchmark: cold start open document

[1] Mark also wrote, "But perhaps we should be expecting OpenOffice's inefficiencies to be rectified and improved over time?" Of course they should improve, but did you see the article's second sentence with a link to a long list of performance improvements. Did you see the 50% improvement from 1.1.5 to 2.0.0? Performance is an ongoing effort with many successes. The latest versions are already rather efficient, so you won't see big wins like the jump from 1.1.5 to 2.0.0. A good outcome is not to become "too slow too quickly."

Mark, I tried not to overgeneralize any claims, yet you summarize the article with: "The spoiler? It's getting slower all the time." The real spoiler is omitting application start up times are the most common performance complaint, and has done a great job minimizing them. Many releases, including the latest two, have achieved the unnatural accomplishment of starting faster. I don't have any quantified data, but qualitatively the new Firefox 3.0 rc1 (touted for its performance improvements) feels very slow to start. Though Firefox is much smaller installed than, Firefox takes about as long to start.

Many responses have been positive. Dan Kegel writes, "I would like to see this benchmark run with multiple simultaneous users, e.g. simulating a busy LTSP server." Hmm. For a few years I actually ran on an LTSP server with over twenty users. Benchmarking speed on a terminal server is interesting, but it seems like the data would come out inconsistent. Personally I am curious how the memory usage would measure up because Linux shares certain memory across processes of the same application.

Ralwer writes, "One metric that would be interesting to compare is memory and IO use." Yes, those are great topics for more analysis. Already you can infer from the existing data here that is heavy on IO because of the differences between cold and warm measurements. Michael Meeks is among the intrepid hackers who have analyzed and improved IO. To this end, he has produced an interesting tool called iogrind.

Related articles


Lennart said...

Is there also an comparison to MS Word? It would be a help to decide which version should be used.

nadavkav said...

that's a beautiful article and a great effort you have here !!! wooo :-)

we use open office 2.2 and 2.4 on several (15+) LTSP servers and we defenetly feel performance improvements over the last versions. i hope 3.0 will not degrade in performance.

their is the multi-thread project in open office that is not fully integrated in the last version so i think when this piece of code will be included... an improvement might be noticed in a multi-process environments like LTSP servers.

if you have any ideas how to test the memory performance on an LTSP server please feel free to suggest them and i will run them on our computers.

Anonymous said...

hi again,

I guess this might be the right place to ask you how you manage to automate your tests on a gui application like oo?

Achim said...

I would like to know if the charts are made with OpenOffice or another program. The charts look very professional.

Andrew Z said...


Thank you. These charts were made with R. I chose it for several reasons:
1. R is easy (for me) to automate R from the command line. This saved me hours of work.
2. R natively supports boxplots (also called box-and-whisker plots) which I needed to describe the dispersion in the data.

mem said...

Any way you can do a memory usage trend of different versions of OpenOffice?

I like to see the kind of memory usage improvement like going from Firefox 2 to Firefox 3 ;)

Andrew Z said...

mem: I have thought about it, and may do it eventually. It has been a low priority because today computers have so much RAM that the main bottleneck is the HDD.

collinm said...

will you post new benchmark soon?

Andrew Z said...

collinm: Yes. I just posted "Benchmarking Microsoft Word 95 through Word 2007," and I will continue posting more benchmarks and articles in the performance series.

richlv said...

regarding memory usage, it would be wildly different for various tasks. i know memory usage when printing a document (espceially one containing images) increased dramatically from 1.1 to 2.0, even up to two times. no, i don't have tests in controlled environment & charts, but it has been confirmed by other users as well ;)

i can't query the issuezilla right now, but there's my report about writer crashing when printing a doc (turned out to run out of memory...), and i think somebody recently posted another report about huge memory grow for printing when going 1->2.

so i'd love to see printing included in memory benchmarks across various versions :)

savaş oyunu said...

ı have followed your writing for a long time.really you have given very successful information.
In spite of my english trouale,I am trying to read and understand your writing.
And ı am following frequently.I hope that you will be with us together with much more scharings.
I hope that your success will go on.