IO Performance Difference between VM and Container

Status
Not open for further replies.

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,038
556
113
We have just been discussing on IRC the pros and cons of different virtualisation types so I thought I would post the results of a couple of tests I just made.

I currently use Proxmox in a Ceph cluster, three nodes, replicating to all 3.

Here are the test results of two different technologies on the same host.

Here is the virtual machine:

Code:
Multiple IO Tester

This application emulates a busy server in several states by launching multiple
threads that do various types of IO. This allows us to see what the consequences
are of running in a multitasking environment. This test uses direct IO and
invalidates caches between tests, testing the disk, not the memory.

NOTE: You need at least 3GB of free space in your current working directory.

The following tests currently consist of:

  8 sequential readers
  8 sequential writers
  8 mixed seqential readers/writers (random choice per IO)
  8 random readers
  8 random writers
  8 mixed random readers/writers (random choice per IO)

Feel free to modify the script to meet your needs. Enjoy!

The test should take less than 2 minutes. Press <ENTER> to begin...

 running IO "sequential read" test...
        result is 632.25MB per second

 running IO "sequential write" test...
        result is 271.08MB per second

 running IO "seq read/seq write" test...
        result is 173.74MB/164.91MB per second

 running IO "random read" test...
        result is 91.21MB per second
        equals 23349.0 IOs per second

 running IO "random write" test...
        result is 54.95MB per second
        equals 14068.0 IOs per second

 running IO "rand read/rand write" test...
        result is 36.25MB/36.19MB per second
        equals 9279.8/9265.8 IOs per second

Here is an LXC container:

Code:
Multiple IO Tester

  This application emulates a busy server in several states by launching multiple
threads that do various types of IO. This allows us to see what the consequences
are of running in a multitasking environment. This test uses direct IO and
invalidates caches between tests, testing the disk, not the memory.

NOTE: You need at least 3GB of free space in your current working directory.

The following tests currently consist of:

  8 sequential readers
  8 sequential writers
  8 mixed seqential readers/writers (random choice per IO)
  8 random readers
  8 random writers
  8 mixed random readers/writers (random choice per IO)

Feel free to modify the script to meet your needs. Enjoy!

The test should take less than 2 minutes. Press <ENTER> to begin...

running IO "sequential read" test...
        result is 831.76MB per second

running IO "sequential write" test...
        result is 455.36MB per second

running IO "seq read/seq write" test...
        result is 312.55MB/265.15MB per second

running IO "random read" test...
        result is 184.23MB per second
        equals 47163.2 IOs per second

running IO "random write" test...
        result is 198.99MB per second
        equals 50941.8 IOs per second

running IO "rand read/rand write" test...
        result is 101.80MB/101.76MB per second
        equals 26062.0/26050.0 IOs per second

Take particular note of the last three random tests, these are the most important to us in a virtualisation workload.

As you can see, the container doesn't just beat the VM, it demolishes it!
 

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,383
364
83
Hi DigitalDaz, I find your tests very interesting. Here are the results from my VM runnng on Xen7.2:
Code:
Multiple IO Tester

  This application emulates a busy server in several states by launching multiple
threads that do various types of IO. This allows us to see what the consequences
are of running in a multitasking environment. This test uses direct IO and
invalidates caches between tests, testing the disk, not the memory.

NOTE: You need at least 3GB of free space in your current working directory.

The following tests currently consist of:

  8 sequential readers
  8 sequential writers
  8 mixed seqential readers/writers (random choice per IO)
  8 random readers
  8 random writers
  8 mixed random readers/writers (random choice per IO)

Feel free to modify the script to meet your needs. Enjoy!

The test should take less than 2 minutes. Press <ENTER> to begin...

 running IO "sequential read" test...
    result is 127.39MB per second

 running IO "sequential write" test...
    result is 202.10MB per second

 running IO "seq read/seq write" test...
    result is 36.70MB/39.06MB per second

 running IO "random read" test...
    result is 1.48MB per second
    equals 380.0 IOs per second

 running IO "random write" test...
    result is 2.67MB per second
    equals 684.2 IOs per second

 running IO "rand read/rand write" test...
    result is 1.22MB/1.06MB per second
    equals 311.5/272.5 IOs per second

The results are not that impressive are they...
The hardware is a Dual Intel Xeon 2.4GHz (12 cores/24 threads), 128GB Memory - 4TB Raid 5 Array.
 

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,038
556
113
@Adrian Fretwell That doesn't surprise me, RAID 5 is about as bad as you can get in terms of IOPs for an array. You will only get the IOPs of a single drive.

Don't be disheartened though, what I was testing on was a high perfomance all SSD setup. I'll be surprised to see anyone beat my figures.
 

Adrian Fretwell

Well-Known Member
Aug 13, 2017
1,383
364
83
@DigitalDaz Thank you for clarifying that. I was beginning to think I had done desperately wrong in my Xen setup!
You have certainly got me interested in LXC, I will have a play.
 

Jonathan R.

New Member
Nov 3, 2017
14
3
3
32
Standard PC on single SATA drive
On VMware 5.5

Code:
 running IO "sequential read" test...
        result is 110.55MB per second

 running IO "sequential write" test...
        result is 118.94MB per second

 running IO "seq read/seq write" test...
        result is 31.99MB/32.63MB per second

 running IO "random read" test...
        result is 711KB per second
        equals 177.8 IOs per second

 running IO "random write" test...
        result is 1.11MB per second
        equals 285.2 IOs per second

 running IO "rand read/rand write" test...
        result is 473KB/371KB per second
        equals 118.2/92.8 IOs per second
 

Jonathan R.

New Member
Nov 3, 2017
14
3
3
32
Vultr VPS in NJ

Code:
 running IO "sequential read" test...
        result is 463.93MB per second

 running IO "sequential write" test...
        result is 459.11MB per second

 running IO "seq read/seq write" test...
        result is 234.29MB/189.34MB per second

 running IO "random read" test...
        result is 204.07MB per second
        equals 52241.5 IOs per second

 running IO "random write" test...
        result is 188.91MB per second
        equals 48361.5 IOs per second

 running IO "rand read/rand write" test...
        result is 94.45MB/94.57MB per second
        equals 24179.5/24210.2 IOs per second
 
  • Like
Reactions: DigitalDaz

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,038
556
113
Vultr VPS in NJ

Code:
 running IO "sequential read" test...
        result is 463.93MB per second

 running IO "sequential write" test...
        result is 459.11MB per second

 running IO "seq read/seq write" test...
        result is 234.29MB/189.34MB per second

 running IO "random read" test...
        result is 204.07MB per second
        equals 52241.5 IOs per second

 running IO "random write" test...
        result is 188.91MB per second
        equals 48361.5 IOs per second

 running IO "rand read/rand write" test...
        result is 94.45MB/94.57MB per second
        equals 24179.5/24210.2 IOs per second
That's amazing figures for a VPS!!
 

DigitalDaz

Administrator
Staff member
Sep 29, 2016
3,038
556
113
I say my figures will be hard to beat but I expect that's probably not the case now with nvme.
 
Status
Not open for further replies.