On I/O engines

[Martin Steigerwald <Martin@xxxxxxxxxxxx>]

Hi!

In order to understand I/O engines better, I like to summarize what I 
think to know at the moment. Maybe this can be a starting point for some 
additional documentation:

=== sync, psync, vsync ===

- all these are using synchronous Linux (POSIX) system calls
- is used by regular applications
- synchronous just refers to the system call interface: i.e. the when the 
system call returns to the application
- as far as I understand it returns when the I/O request is told to be 
completed
- it does not imply synchronous I/O aka O_SYNC which is way slower and 
enabled by sync=1
- thus it does not guarantee that the I/O has been physically written to 
the underlying device (see open(2))
- thus is only guarantees that the I/O request has been dealt with? what 
does this exactly mean?
- does it mean that this is I/O in the context of the process?
- it can be used with direct=1 to circumvent the pagecache


difference is the kind of system call used:
- sync uses read/write which read/write count bytes into from/to a buffer. 
Uses current file offset, changeable via fseek (or lseek, I did not find a 
manpage for fseek)
- psync uses pread/pwrite which read/write count bytes from given offset
- vsync uses readv/writev which read/writes count, i.e. mutiple buffers of 
given length in one call (struct iovec)

I am not sure on what performance difference to expect. I bet that 
sync/psync should perform roughly the same.


=== libaio ===

- this uses Linux asynchronous I/O calls[1]
- it uses libaio for that
- who else uses libaio? It systems application that are near to the 
system:

martin@merkaba:~> apt-cache rdepends libaio1
libaio1
Reverse Depends:
  fio
  qemu-kvm
  libdbd-oracle-perl
  zfs-fuse
  stressapptest
  qemu-kvm
  qemu-utils
  qemu-system
  multipath-tools
  ltp-kernel-test
  libaio1-dbg
  libaio-dev
  fio
  drizzle
  blktrace

- these calls allow applications to offload I/O calls to the background
- according to [1] this is only supported for direct I/O
- using anything else let it fall back to synchronous call behavior
- thus one sees this in combination with direct=1 in fio jobs
- does this mean that this is I/O outside the context of the process?


Question:
- what difference is between the following two other than the second one 
seems to be more popular in example job files?
1) ioengine=sync + direct=1
2) ioengine=libaio + direct=1

Current answer: It is that fio can issue further I/Os while the Linux 
kernels handles the I/O.



=== other I/O engines relevant to Linux ===
There seem to be some other I/O engines relevant to Linux and mass storage 
I/O:

== mmap ==
- maps the memory into files and uses memcpy
- used by quite some applications
- what else to note?

== syslet-rw ==
- make regular read/write asynchronous
- where is this used?
- what else to note?

Any others?


Is what I wrote correct so far?


I think I´d like to write something up about the different I/O concepts in 
Linux, if such a document doesn´t exist yet.


[1] http://lse.sourceforge.net/io/aio.html

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe fio" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html