thr3ads.net - Btrfs devel - [Ocfs2-devel] [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA

If this information is useful, please help other people find it:
Share via:

David Sterba

2013-Dec-12 15:26 UTC

[PATCH 3/4 v3] btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents

Set the EXTENT_DATA_COMPRESSED flag together with EXTENT_ENCODED as
defined by fiemap spec.

Signed-off-by: David Sterba <dsterba@suse.cz>
---
 fs/btrfs/extent_io.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 5ea0ef5..8a28f15 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -4158,6 +4158,7 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 
 	while (!end) {
 		u64 offset_in_extent = 0;
+		u64 em_phys_len;
 
 		/* break if the extent we found is outside the range */
 		if (em->start >= max || extent_map_end(em) < off)
@@ -4182,6 +4183,7 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 		em_end = extent_map_end(em);
 		em_len = em_end - em_start;
 		emflags = em->flags;
+		em_phys_len = em->len;
 		disko = 0;
 		flags = 0;
 
@@ -4220,9 +4222,12 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 
 			if (ref_cnt > 1)
 				flags |= FIEMAP_EXTENT_SHARED;
+			em_phys_len = em->block_len;
 		}
-		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags))
+		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags)) {
 			flags |= FIEMAP_EXTENT_ENCODED;
+			flags |= FIEMAP_EXTENT_DATA_COMPRESSED;
+		}
 
 		free_extent_map(em);
 		em = NULL;
@@ -4244,7 +4249,7 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 			end = 1;
 		}
 		ret = fiemap_fill_next_extent(fieinfo, em_start, disko,
-					      em_len, 0, flags);
+					      em_len, em_phys_len, flags);
 		if (ret)
 			goto out_free;
 	}
-- 
1.7.9

--
To unsubscribe from this list: send the line "unsubscribe
linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

David Sterba

2013-Dec-12 15:26 UTC

head link

[Ocfs2-devel] [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA_COMPRESSED flag

The original FIEMAP patch did not define this bit, btrfs will make use of
it.  The defined constant maintains the same value as originally proposed.

Currently, the 'filefrag' utility has no way to recognize and denote a
compressed extent. As implemented in btrfs right now, the compression step
splits a big extent into smaller chunks and this is reported as a heavily
fragmented file. Adding the flag to filefrag will at least give some
explanation why, this has been confusing users for some time already.

V3:
Based on feedback from Andreas, implement #1 from V2, current users of
fiemap_fill_next_extent (fs/, ext4, gfs2, ocfs2, nilfs2, xfs) updated
accordingly, no functional change.

V2:
Based on feedback from Andreas, the fiemap_extent is now able to hold the
physical extent length, to be filled by the filesystem callback.

The filesystems do not have access to the structure that is passed back to
userspace and are supposed to call fiemap_fill_next_extent, there's no
direct
way to fill fe_phys_length. There are two ways to pass it:

1) extend fiemap_fill_next_extent to take phys_length and update all
   users (ext4, gfs2, ocfs2, nilfs2, xfs)

2) add new function that takes arguments for all the fiemap_extent items,
   newly added phys_length compared to fiemap_fill_next_extent

David Sterba (4):
  fiemap: fix comment at EXTENT_DATA_ENCRYPTED
  fiemap: add EXTENT_DATA_COMPRESSED flag
  btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents
  Documentation/fiemap: Document the DATA_COMPRESSED flag

 Documentation/filesystems/fiemap.txt |   17 +++++++++++++----
 fs/btrfs/extent_io.c                 |    9 +++++++--
 fs/ext4/extents.c                    |    3 ++-
 fs/ext4/inline.c                     |    2 +-
 fs/gfs2/inode.c                      |    2 +-
 fs/ioctl.c                           |   18 ++++++++++++------
 fs/nilfs2/inode.c                    |    8 +++++---
 fs/ocfs2/extent_map.c                |    4 ++--
 fs/xfs/xfs_iops.c                    |    2 +-
 include/linux/fs.h                   |    2 +-
 include/uapi/linux/fiemap.h          |    8 ++++++--
 11 files changed, 51 insertions(+), 24 deletions(-)

-- 
1.7.9

David Sterba

2013-Dec-12 15:26 UTC

head link

[Ocfs2-devel] [PATCH 2/4 v3] fiemap: add EXTENT_DATA_COMPRESSED flag

This flag was not accepted when fiemap was proposed [2] due to lack of
in-kernel users. Btrfs has compression for a long time and we'd like to
see that an extent is compressed in the output of 'filefrag' utility
once it's taught about it.

For that purpose, a reserved field from fiemap_extent is used to let the
filesystem store along the physcial extent length when the flag is set.
This keeps compatibility with applications that use FIEMAP.

Extend arguments of fiemap_fill_next_extent and update all users.

[1] http://article.gmane.org/gmane.comp.file-systems.ext4/8871
[2] http://thread.gmane.org/gmane.comp.file-systems.ext4/8870
[3] http://thread.gmane.org/gmane.linux.file-systems/77632 (v1)
[4] http://www.spinics.net/lists/linux-fsdevel/msg69078.html (v2)

Cc: Al Viro <viro at zeniv.linux.org.uk>
CC: Andreas Dilger <adilger at dilger.ca>
CC: Christoph Hellwig <hch at infradead.org>
CC: Mark Fasheh <mfasheh at suse.com>
Signed-off-by: David Sterba <dsterba at suse.cz>
---
 fs/btrfs/extent_io.c        |    2 +-
 fs/ext4/extents.c           |    3 ++-
 fs/ext4/inline.c            |    2 +-
 fs/gfs2/inode.c             |    2 +-
 fs/ioctl.c                  |   18 ++++++++++++------
 fs/nilfs2/inode.c           |    8 +++++---
 fs/ocfs2/extent_map.c       |    4 ++--
 fs/xfs/xfs_iops.c           |    2 +-
 include/linux/fs.h          |    2 +-
 include/uapi/linux/fiemap.h |    6 +++++-
 10 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index ff43802..5ea0ef5 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -4244,7 +4244,7 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 			end = 1;
 		}
 		ret = fiemap_fill_next_extent(fieinfo, em_start, disko,
-					      em_len, flags);
+					      em_len, 0, flags);
 		if (ret)
 			goto out_free;
 	}
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 35f65cf..00ffd18 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -2224,6 +2224,7 @@ static int ext4_fill_fiemap_extents(struct inode *inode,
 				(__u64)es.es_lblk << blksize_bits,
 				(__u64)es.es_pblk << blksize_bits,
 				(__u64)es.es_len << blksize_bits,
+				0,
 				flags);
 			if (err < 0)
 				break;
@@ -4798,7 +4799,7 @@ static int ext4_xattr_fiemap(struct inode *inode,
 
 	if (physical)
 		error = fiemap_fill_next_extent(fieinfo, 0, physical,
-						length, flags);
+						length, 0, flags);
 	return (error < 0 ? error : 0);
 }
 
diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c
index bae9875..c5da773 100644
--- a/fs/ext4/inline.c
+++ b/fs/ext4/inline.c
@@ -1816,7 +1816,7 @@ int ext4_inline_data_fiemap(struct inode *inode,
 
 	if (physical)
 		error = fiemap_fill_next_extent(fieinfo, 0, physical,
-						length, flags);
+						length, 0, flags);
 	brelse(iloc.bh);
 out:
 	up_read(&EXT4_I(inode)->xattr_sem);
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index 7119504..86e9e9b 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -1817,7 +1817,7 @@ static int gfs2_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 			len = size - start;
 		if (start < size)
 			ret = fiemap_fill_next_extent(fieinfo, start, phys,
-						      len, flags);
+						      len, 0, flags);
 		if (ret == 1)
 			ret = 0;
 	} else {
diff --git a/fs/ioctl.c b/fs/ioctl.c
index 8ac3fad..e7902c4 100644
--- a/fs/ioctl.c
+++ b/fs/ioctl.c
@@ -70,6 +70,7 @@ static int ioctl_fibmap(struct file *filp, int __user *p)
  * @logical:	Extent logical start offset, in bytes
  * @phys:	Extent physical start offset, in bytes
  * @len:	Extent length, in bytes
+ * @phys_len:   Physical extent length in bytes
  * @flags:	FIEMAP_EXTENT flags that describe this extent
  *
  * Called from file system ->fiemap callback. Will populate extent
@@ -80,10 +81,11 @@ static int ioctl_fibmap(struct file *filp, int __user *p)
  * extent that will fit in user array.
  */
 #define SET_UNKNOWN_FLAGS	(FIEMAP_EXTENT_DELALLOC)
-#define SET_NO_UNMOUNTED_IO_FLAGS	(FIEMAP_EXTENT_DATA_ENCRYPTED)
+#define SET_NO_UNMOUNTED_IO_FLAGS	(FIEMAP_EXTENT_DATA_ENCRYPTED | \
+					 FIEMAP_EXTENT_DATA_COMPRESSED)
 #define SET_NOT_ALIGNED_FLAGS
(FIEMAP_EXTENT_DATA_TAIL|FIEMAP_EXTENT_DATA_INLINE)
 int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical,
-			    u64 phys, u64 len, u32 flags)
+			    u64 phys, u64 len, u64 phys_len, u32 flags)
 {
 	struct fiemap_extent extent;
 	struct fiemap_extent __user *dest = fieinfo->fi_extents_start;
@@ -110,6 +112,9 @@ int fiemap_fill_next_extent(struct fiemap_extent_info
*fieinfo, u64 logical,
 	extent.fe_length = len;
 	extent.fe_flags = flags;
 
+	if (flags & FIEMAP_EXTENT_DATA_COMPRESSED)
+		extent.fe_phys_length = phys_len;
+
 	dest += fieinfo->fi_extents_mapped;
 	if (copy_to_user(dest, &extent, sizeof(extent)))
 		return -EFAULT;
@@ -318,10 +323,11 @@ int __generic_block_fiemap(struct inode *inode,
 				flags = FIEMAP_EXTENT_MERGED|FIEMAP_EXTENT_LAST;
 				ret = fiemap_fill_next_extent(fieinfo, logical,
 							      phys, size,
-							      flags);
+							      0, flags);
 			} else if (size) {
 				ret = fiemap_fill_next_extent(fieinfo, logical,
-							      phys, size, flags);
+							      phys, size,
+							      0, flags);
 				size = 0;
 			}
 
@@ -347,7 +353,7 @@ int __generic_block_fiemap(struct inode *inode,
 			if (start_blk > last_blk && !whole_file) {
 				ret = fiemap_fill_next_extent(fieinfo, logical,
 							      phys, size,
-							      flags);
+							      0, flags);
 				break;
 			}
 
@@ -358,7 +364,7 @@ int __generic_block_fiemap(struct inode *inode,
 			if (size) {
 				ret = fiemap_fill_next_extent(fieinfo, logical,
 							      phys, size,
-							      flags);
+							      0, flags);
 				if (ret)
 					break;
 			}
diff --git a/fs/nilfs2/inode.c b/fs/nilfs2/inode.c
index 7e350c5..b03917a 100644
--- a/fs/nilfs2/inode.c
+++ b/fs/nilfs2/inode.c
@@ -1018,7 +1018,8 @@ int nilfs_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 			if (size) {
 				/* End of the current extent */
 				ret = fiemap_fill_next_extent(
-					fieinfo, logical, phys, size, flags);
+					fieinfo, logical, phys, size, 0,
+					flags);
 				if (ret)
 					break;
 			}
@@ -1068,7 +1069,8 @@ int nilfs_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 					flags |= FIEMAP_EXTENT_LAST;
 
 				ret = fiemap_fill_next_extent(
-					fieinfo, logical, phys, size, flags);
+					fieinfo, logical, phys, size, 0,
+					flags);
 				if (ret)
 					break;
 				size = 0;
@@ -1084,7 +1086,7 @@ int nilfs_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 					/* Terminate the current extent */
 					ret = fiemap_fill_next_extent(
 						fieinfo, logical, phys, size,
-						flags);
+						0, flags);
 					if (ret || blkoff > end_blkoff)
 						break;
 
diff --git a/fs/ocfs2/extent_map.c b/fs/ocfs2/extent_map.c
index 767370b..521b0f2 100644
--- a/fs/ocfs2/extent_map.c
+++ b/fs/ocfs2/extent_map.c
@@ -735,7 +735,7 @@ static int ocfs2_fiemap_inline(struct inode *inode, struct
buffer_head *di_bh,
 			phys += offsetof(struct ocfs2_dinode,
 					 id2.i_data.id_data);
 
-		ret = fiemap_fill_next_extent(fieinfo, 0, phys, id_count,
+		ret = fiemap_fill_next_extent(fieinfo, 0, phys, id_count, 0,
 					      flags);
 		if (ret < 0)
 			return ret;
@@ -809,7 +809,7 @@ int ocfs2_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
 		virt_bytes = (u64)le32_to_cpu(rec.e_cpos) <<
osb->s_clustersize_bits;
 
 		ret = fiemap_fill_next_extent(fieinfo, virt_bytes, phys_bytes,
-					      len_bytes, fe_flags);
+					      len_bytes, 0, fe_flags);
 		if (ret)
 			break;
 
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 27e0e54..31e9f53 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1000,7 +1000,7 @@ xfs_fiemap_format(
 		fiemap_flags |= FIEMAP_EXTENT_LAST;
 
 	error = fiemap_fill_next_extent(fieinfo, logical, physical,
-					length, fiemap_flags);
+					length, 0, fiemap_flags);
 	if (error > 0) {
 		error = 0;
 		*full = 1;	/* user array now full */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 121f11f..1a96f9b 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1479,7 +1479,7 @@ struct fiemap_extent_info {
 							fiemap_extent array */
 };
 int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical,
-			    u64 phys, u64 len, u32 flags);
+			    u64 phys, u64 len, u64 phys_len, u32 flags);
 int fiemap_check_flags(struct fiemap_extent_info *fieinfo, u32 fs_flags);
 
 /*
diff --git a/include/uapi/linux/fiemap.h b/include/uapi/linux/fiemap.h
index 93abfcd..0e32cae 100644
--- a/include/uapi/linux/fiemap.h
+++ b/include/uapi/linux/fiemap.h
@@ -19,7 +19,9 @@ struct fiemap_extent {
 	__u64 fe_physical; /* physical offset in bytes for the start
 			    * of the extent from the beginning of the disk */
 	__u64 fe_length;   /* length in bytes for this extent */
-	__u64 fe_reserved64[2];
+	__u64 fe_phys_length; /* physical length in bytes, undefined if
+			       * DATA_COMPRESSED not set */
+	__u64 fe_reserved64;
 	__u32 fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
 	__u32 fe_reserved[3];
 };
@@ -50,6 +52,8 @@ struct fiemap {
 						    * Sets EXTENT_UNKNOWN. */
 #define FIEMAP_EXTENT_ENCODED		0x00000008 /* Data can not be read
 						    * while fs is unmounted */
+#define FIEMAP_EXTENT_DATA_COMPRESSED	0x00000040 /* Data is compressed by fs.
+						    * Sets EXTENT_ENCODED */
 #define FIEMAP_EXTENT_DATA_ENCRYPTED	0x00000080 /* Data is encrypted by fs.
 						    * Sets EXTENT_ENCODED */
 #define FIEMAP_EXTENT_NOT_ALIGNED	0x00000100 /* Extent offsets may not be
-- 
1.7.9

Andreas Dilger

2013-Dec-12 22:20 UTC

head link

Re: [PATCH 3/4 v3] btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents

On Dec 12, 2013, at 8:26 AM, David Sterba <dsterba@suse.cz>
wrote:> Set the EXTENT_DATA_COMPRESSED flag together with EXTENT_ENCODED as
> defined by fiemap spec.
> 
> Signed-off-by: David Sterba <dsterba@suse.cz>
> ---
> fs/btrfs/extent_io.c |    9 +++++++--
> 1 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
> index 5ea0ef5..8a28f15 100644
> --- a/fs/btrfs/extent_io.c
> +++ b/fs/btrfs/extent_io.c
> 
> @@ -4220,9 +4222,12 @@ int extent_fiemap(struct inode *inode, struct
fiemap_extent_info *fieinfo,
> 
> 			if (ref_cnt > 1)
> 				flags |= FIEMAP_EXTENT_SHARED;
> +			em_phys_len = em->block_len;
> 		}
> -		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags))
> +		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags)) {
> 			flags |= FIEMAP_EXTENT_ENCODED;
> +			flags |= FIEMAP_EXTENT_DATA_COMPRESSED;
(minor nit) Could combine these (not sure it makes a difference to the
compiler):

			flags |= (FIEMAP_EXTENT_ENCODED |
				  FIEMAP_EXTENT_DATA_COMPRESSED);

Cheers, Andreas

Andreas Dilger

2013-Dec-12 22:22 UTC

head link

[Ocfs2-devel] [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA_COMPRESSED flag

On Dec 12, 2013, at 8:25 AM, David Sterba <dsterba at suse.cz>
wrote:> The original FIEMAP patch did not define this bit, btrfs will make use of
> it.  The defined constant maintains the same value as originally proposed.
> 
> Currently, the 'filefrag' utility has no way to recognize and
denote a
> compressed extent. As implemented in btrfs right now, the compression step
> splits a big extent into smaller chunks and this is reported as a heavily
> fragmented file. Adding the flag to filefrag will at least give some
> explanation why, this has been confusing users for some time already.
The whole series looks good to me (one minor nit if it needs to be resubmitted
for some reason).  You can add my:

Reviewed-by: Andreas Dilger <adilger at dilger.ca>
> V3:
> Based on feedback from Andreas, implement #1 from V2, current users of
> fiemap_fill_next_extent (fs/, ext4, gfs2, ocfs2, nilfs2, xfs) updated
> accordingly, no functional change.
> 
> V2:
> Based on feedback from Andreas, the fiemap_extent is now able to hold the
> physical extent length, to be filled by the filesystem callback.
> 
> The filesystems do not have access to the structure that is passed back to
> userspace and are supposed to call fiemap_fill_next_extent, there's no
direct
> way to fill fe_phys_length. There are two ways to pass it:
> 
> 1) extend fiemap_fill_next_extent to take phys_length and update all
>   users (ext4, gfs2, ocfs2, nilfs2, xfs)
> 
> 2) add new function that takes arguments for all the fiemap_extent items,
>   newly added phys_length compared to fiemap_fill_next_extent
> 
> David Sterba (4):
>  fiemap: fix comment at EXTENT_DATA_ENCRYPTED
>  fiemap: add EXTENT_DATA_COMPRESSED flag
>  btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents
>  Documentation/fiemap: Document the DATA_COMPRESSED flag
> 
> Documentation/filesystems/fiemap.txt |   17 +++++++++++++----
> fs/btrfs/extent_io.c                 |    9 +++++++--
> fs/ext4/extents.c                    |    3 ++-
> fs/ext4/inline.c                     |    2 +-
> fs/gfs2/inode.c                      |    2 +-
> fs/ioctl.c                           |   18 ++++++++++++------
> fs/nilfs2/inode.c                    |    8 +++++---
> fs/ocfs2/extent_map.c                |    4 ++--
> fs/xfs/xfs_iops.c                    |    2 +-
> include/linux/fs.h                   |    2 +-
> include/uapi/linux/fiemap.h          |    8 ++++++--
> 11 files changed, 51 insertions(+), 24 deletions(-)
> 
> -- 
> 1.7.9
> 

Cheers, Andreas





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP using GPGMail
Url :
http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20131212/6348d653/attachment.bin

Btrfs devel - Dec 2013 - [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA_COMPRESSED flag

[PATCH 3/4 v3] btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents

[Ocfs2-devel] [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA_COMPRESSED flag

[Ocfs2-devel] [PATCH 2/4 v3] fiemap: add EXTENT_DATA_COMPRESSED flag

Re: [PATCH 3/4 v3] btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents

[Ocfs2-devel] [PATCH 0/4 v3] fiemap: introduce EXTENT_DATA_COMPRESSED flag