On Mon, Oct 21, 2019 at 11:28:06AM -0700, Darrick J. Wong wrote:
> On Tue, Oct 15, 2019 at 11:42:40AM -0700, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@xxxxxx>
> >
> > Btrfs supports transparent compression: data written by the user can be
> > compressed when written to disk and decompressed when read back.
> > However, we'd like to add an interface to write pre-compressed data
> > directly to the filesystem, and the matching interface to read
> > compressed data without decompressing it. This adds support for
> > so-called "encoded I/O" via preadv2() and pwritev2().
> >
> > A new RWF_ENCODED flags indicates that a read or write is "encoded". If
> > this flag is set, iov[0].iov_base points to a struct encoded_iov which
> > is used for metadata: namely, the compression algorithm, unencoded
> > (i.e., decompressed) length, and what subrange of the unencoded data
> > should be used (needed for truncated or hole-punched extents and when
> > reading in the middle of an extent). For reads, the filesystem returns
> > this information; for writes, the caller provides it to the filesystem.
> > iov[0].iov_len must be set to sizeof(struct encoded_iov), which can be
> > used to extend the interface in the future. The remaining iovecs contain
> > the encoded extent.
> >
> > Filesystems must indicate that they support encoded writes by setting
> > FMODE_ENCODED_IO in ->file_open().
> >
> > Signed-off-by: Omar Sandoval <osandov@xxxxxx>
> > ---
> > include/linux/fs.h | 14 +++++++
> > include/uapi/linux/fs.h | 26 ++++++++++++-
> > mm/filemap.c | 82 ++++++++++++++++++++++++++++++++++-------
> > 3 files changed, 108 insertions(+), 14 deletions(-)
> >
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index e0d909d35763..54681f21e05e 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -175,6 +175,9 @@ typedef int (dio_iodone_t)(struct kiocb *iocb, loff_t offset,
> > /* File does not contribute to nr_files count */
> > #define FMODE_NOACCOUNT ((__force fmode_t)0x20000000)
> >
> > +/* File supports encoded IO */
> > +#define FMODE_ENCODED_IO ((__force fmode_t)0x40000000)
> > +
> > /*
> > * Flag for rw_copy_check_uvector and compat_rw_copy_check_uvector
> > * that indicates that they should check the contents of the iovec are
> > @@ -314,6 +317,7 @@ enum rw_hint {
> > #define IOCB_SYNC (1 << 5)
> > #define IOCB_WRITE (1 << 6)
> > #define IOCB_NOWAIT (1 << 7)
> > +#define IOCB_ENCODED (1 << 8)
> >
> > struct kiocb {
> > struct file *ki_filp;
> > @@ -3088,6 +3092,11 @@ extern int sb_min_blocksize(struct super_block *, int);
> > extern int generic_file_mmap(struct file *, struct vm_area_struct *);
> > extern int generic_file_readonly_mmap(struct file *, struct vm_area_struct *);
> > extern ssize_t generic_write_checks(struct kiocb *, struct iov_iter *);
> > +struct encoded_iov;
> > +extern int generic_encoded_write_checks(struct kiocb *, struct encoded_iov *);
> > +extern ssize_t check_encoded_read(struct kiocb *, struct iov_iter *);
> > +extern int import_encoded_write(struct kiocb *, struct encoded_iov *,
> > + struct iov_iter *);
> > extern int generic_remap_checks(struct file *file_in, loff_t pos_in,
> > struct file *file_out, loff_t pos_out,
> > loff_t *count, unsigned int remap_flags);
> > @@ -3403,6 +3412,11 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags)
> > return -EOPNOTSUPP;
> > ki->ki_flags |= IOCB_NOWAIT;
> > }
> > + if (flags & RWF_ENCODED) {
> > + if (!(ki->ki_filp->f_mode & FMODE_ENCODED_IO))
> > + return -EOPNOTSUPP;
> > + ki->ki_flags |= IOCB_ENCODED;
> > + }
> > if (flags & RWF_HIPRI)
> > ki->ki_flags |= IOCB_HIPRI;
> > if (flags & RWF_DSYNC)
> > diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
> > index 379a612f8f1d..ed92a8a257cb 100644
> > --- a/include/uapi/linux/fs.h
> > +++ b/include/uapi/linux/fs.h
> > @@ -284,6 +284,27 @@ struct fsxattr {
> >
> > typedef int __bitwise __kernel_rwf_t;
> >
> > +enum {
> > + ENCODED_IOV_COMPRESSION_NONE,
> > + ENCODED_IOV_COMPRESSION_ZLIB,
> > + ENCODED_IOV_COMPRESSION_LZO,
> > + ENCODED_IOV_COMPRESSION_ZSTD,
> > + ENCODED_IOV_COMPRESSION_TYPES = ENCODED_IOV_COMPRESSION_ZSTD,
> > +};
> > +
> > +enum {
> > + ENCODED_IOV_ENCRYPTION_NONE,
> > + ENCODED_IOV_ENCRYPTION_TYPES = ENCODED_IOV_ENCRYPTION_NONE,
> > +};
> > +
> > +struct encoded_iov {
> > + __u64 len;
> > + __u64 unencoded_len;
> > + __u64 unencoded_offset;
> > + __u32 compression;
> > + __u32 encryption;
>
> Can we add some must-be-zero padding space at the end here for whomever
> comes along next wanting to add more encoding info?
>From the commit message:
iov_len must be set to sizeof(struct encoded_iov), which can be used to
extend the interface in the future.
> (And maybe a manpage and some basic testing, to reiterate Dave...)
I sent a man page as part of this thread:
https://lore.kernel.org/linux-btrfs/c7e8f93596fee7bb818dc0edf29f484036be1abb.1571164851.git.osandov@xxxxxx/
See my reply to Dave, I have tests in my xfstests repo that I haven't
sent yet.
> --D
