On Sun, Jul 22, 2012 at 01:44:28PM -0700, Marc MERLIN wrote:
> On Sun, Jul 22, 2012 at 09:35:10PM +0200, Martin Steigerwald wrote:
> > Or use atop to have a complete system overview during the hdparm run. You
> > may want to use its default 10 seconds delay.
>
> atop looked totally fine and showed that a long dd took 6% CPU of one core.
>
> > Anyway, hdparm is only a very rough measurement. (Test time 2 / 3 seconds
> > is really short.)
> >
> > Did you repeat tests three or five times and looked at the deviation?
>
> Yes. I also ran dd with 1GB, and got the exact same number.
Oh my, the plot thickens.
I'm bringing this back on this list because
1) I've confirmed that I can get 500MB/s unencrypted reads (2GB file)
with btfrs on the SSD
2) Despite getting a measly 23MB/s reading from /dev/mapper/ssdcrypt,
once I mount it as a btrfs drive and I a read a file from it, I now
get 267MB/s
3) Stating a bunch of files on the ssd is very slow (5-6x slower than
stating the same files from disk). Using noatime did not help.
On an _unencrypted_ partition on the SSD, running du -sh on a directory
with 15K files, takes 23 seconds on unencrypted SSD and 4 secs on
encrypted spinning drive, both with a similar btrfs filesystem, and
the same kernel (3.4.4).
Since I'm getting some kind of "seek problem" on the ssd (yes, there are
no real seeks on SSD), it seems that I do have a problem with btrfs
that is at least not related to encryption.
Unencrypted btrfs on SSD:
gandalfthegreat:/mnt/mnt2# mount -o compress=lzo,discard,nossd,space_cache,noatime /dev/sda2 /mnt/mnt2
gandalfthegreat:/mnt/mnt2# echo 3 > /proc/sys/vm/drop_caches; time du -sh src
514M src
real 0m22.667s
Encrypted btrfs on spinning drive:
gandalfthegreat:/var/local# echo 3 > /proc/sys/vm/drop_caches; time du -sh src
514M src
real 0m3.881s
I've run this many times and get the same numbers.
I've tried deadline and noop on /dev/sda (the SSD) and du is just as slow.
I also tried with:
- space_cache and nospace_cache
- ssd and nossd
- noatime didn't seem to help even though I was hopeful on this one.
In all cases, I get:
gandalfthegreat:/mnt/mnt2# echo 3 > /proc/sys/vm/drop_caches; time du -sh src
514M src
real 0m22.537s
22 seconds for 15K files on an SSD while being 5 times slower than a
spinning disk with the same data.
What's going on?
More details below for anyone interested:
----------------------------------------------------------------------------
The test below shows that I can access the encrypted SSD 10x faster by talking
to it through a btrfs filesystem than doing a dd read from the device node.
So I suppose I could just ignore the dd device node thing, however not really.
Let's take a directory with some source files inside:
Let's start with the hard drive in my laptop (dmcrypt with btrtfs)
gandalfthegreat:/var/local# find src | wc -l
15261
gandalfthegreat:/var/local# echo 3 > /proc/sys/vm/drop_caches; time du -sh src
514M src
real 0m4.068s
So on an encrypted spinning disk, it takes 4 seconds.
On my SSD in the same machine with the same encryption and the same btrfs filesystem,
I get 5-6 times slower:
gandalfthegreat:/mnt/btrfs_pool1/var/local# time du -sh src
514M src
real 0m24.937s
Incidently 5x is also the speed difference between my encrypted HD and encrypted SSD
with dd.
Now, why du is 5x slower and dd of a file from the filesystem is 2.5x
faster, I have no idea :-/
See below:
1) drop caches
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=w2k-s001.vmdk of=/dev/null
2146631680 bytes (2.1 GB) copied, 8.03898 s, 267 MB/s
-> 267MB/s reading from the file through the encrypted filesystem. That's good.
For comparison
gandalfthegreat:/mnt/mnt2# dd if=w2k-s001.vmdk of=/dev/null
2146631680 bytes (2.1 GB) copied, 4.33393 s, 495 MB/s
-> almost 500MB/s reading through another unencrypted filesystem on the same SSD
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=/dev/mapper/ssdcrypt of=/dev/null bs=1M count=1000
1048576000 bytes (1.0 GB) copied, 45.1234 s, 23.2 MB/s
-> 23MB/s reading from the block device that my FS is mounted from. WTF?
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches; dd if=w2k-s001.vmdk of=test
2146631680 bytes (2.1 GB) copied, 17.9129 s, 120 MB/s
-> 120MB/s copying a file from the SSD to itself. That's not bad.
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# echo 3 > /proc/sys/vm/drop_caches; dd if=test of=/dev/null
2146631680 bytes (2.1 GB) copied, 8.4907 s, 253 MB/s
-> reading the new copied file still shows 253MB/s, good.
gandalfthegreat:/mnt/btrfs_pool1/var/local/VirtualBox VMs/w2k_virtual# dd if=test of=/dev/null
2146631680 bytes (2.1 GB) copied, 2.11001 s, 1.0 GB/s
-> reading without dropping the cache shows 1GB/s
I'm very lost now, any idea what's going on?
- big file copies from encrypted SSD are fast as expected
- raw device access is unexplainably slow (10x slower than btrfs on the
raw device)
- stating 15K files on the encrypted ssd with btrfs is super slow
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
