Phoronix Test Suite for virtual machine benchmarking

In my discussion of my home-based server, I wanted to test virtual machine performance in several environments (VirtualBox, KVM, ESXi, Hyper-V).

At first I thought I'd just do some seat-of-the-pants qualitative tests, but then I thought I'd look into using a more rigorous approach.

Without too much research, I uncovered the Phoronix Test Suite (PTS). This seems to be an industrial-strength, free "hardware" benchmarking tool with a long history and many features.

I also looked into the Inquisitor test system, and even downloaded and tried the Inquisitor live CD image. However, Inquisitor didn't seem as full-featured as PTS.

Like Inquisitor, PTS also has a live CD iso which you can download and test your "hardware" using standardized OS software. I created a VM which booted from the PTS live CD iso, but I couldn't find an easy way to save or upload results, so I just installed PTS in my Ubuntu guest image. Simply download the PTS .deb, and use dpkg -i to install the suite.

Installing PTS in my Ubuntu 12.04 guest image also made the most sense since I'm really concerned with performance on MY images, not performance on someone else's standard image.

PTS is very full featured. I found it had over 20 predefined suites like :

* database
* desktop-graphics
* favorites
* linux-system
* network
* server
* universe (ALL suites)
* universe-cli
* universe-x
* workstation

Each suite has maybe 10-20 tests in them. Note that many suites have overlapping tests so if you run a few suites you'll run some tests twice.

Be prepared to spend some time with this as "installing" a suite (downloading test artifacts) and running the suite each can take several hours.

PTS comes with very good documentation.

Here are some of the commands I found useful :

* $ phoronix-test-suite
- Returns command options (help)

* $ phoronix-test-suite install server
- Installs the suite SERVER (downloads test files etc.)
- This can take hours

* $ phoronix-test-suite run server
- Runs the suite SERVER (multiple runs for averages, saves results)
- This can take hours

* $ phoronix-test-suite make-download-cache
- Saves downloaded artifacts for later use on this or other systems
- Location is ~/.phoronix-test-suite/download-cache
- The cache is not small - mine was about 6 GB

* $ phoronix-test-suite upload-result TEST-RESULT
- Uploads result for viewing and comparison on openbenchmarking.org
- Test result is a name like "TESTY" which you supply when invoking a run

* $ phoronix-test-suite result-file-to-csv TEST-RESULT
- Test result is a name like "TESTY" which you supply when invoking a run

* $ touch ~/.phoronix-test-suite/skip-test
- Stops current test prematurely but continue the testing process

In my testing, at least one suite hung towards the end. You can try to finish the suite with finish-run or recover-run but that didn't seem to work for me.

I found I could create my own custom test suites by doing something like this :


$ cd ~/.phoronix-test-suite/test-suites
$ cp -r pts/favorites-1.0.0 local/my-favorites
$ vi local/my-favorites/suite-definition.xml

Then add or subtract tests as you want.

I found the following tests to run too long, and therefore unsuitable for my purposes :

* Flexible IO Tester (120 min)
* POV-ray (43 min)
* C-Ray (41 min)
* Minion (38 min)
* Dbench (37 min)
* Open FFM Nero2D (28 min)
* Smallpt (28 min)
* LZMA Compression (26 min)
* Build PHP (19 min)
* Dcraw (14 min)

Also, note the directory tree which will be created under ~/.phoronix-test-suite will get pretty large. I found my directory took up about 25 GB.