thr3ads.net - freebsd stable - Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target [Nov 2015]

If this information is useful, please help other people find it:
Share via:

Christopher Forgeron

2015-Nov-17 22:08 UTC

Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target

I just submitted this as a bug:

( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 )

..but I thought I should bring it to the list's attention for more exposure
- If that's a no-no, let me know, as I have a few others that are related
to this that I'd like to discuss.

- - - -


Consider this scenario:

Virtual FreeBSD Machine, with a zpool created out of iSCSI disks.
Physical FreeBSD Machine, with a zpool holding a sparse file that is the
target for the iSCSI disk.

This setup works in an environment with all 10.1 machines, doesn't with all
10.2 machines.

- The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no
custom.
- The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no
custom.
- iSCSI is all CAM iSCSI, not the old istgt platform.
- The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target)

The target machine is the same physical machine, with the same zpools - I
either boot 10.1 or 10.2 for testing, and use the same zpool/disks

to ensure nothing is changing.

If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target,
TRIM doesn't work (shows as NONE below).
If I have a 10.2 iSCSI Initiator (client) connected to a 10.1 iSCSI Target,
TRIM does work.

(There is another bug with that last scenario as well, but I will open it
separately)

...for clarity, a 10.1 iSCSI Initiator connected to a 10.1 iSCSI Target
also works perfectly. I have ~20 of these in the field.

On the 10.1 / 10.2 Targets, the ctl.conf file is identical. Zpools are
identical, because they are shared between reboots of the same iSCSI

target machine.



On the 10.2 initiator machine, connected to a 10.2 Target machine:

# sysctl -a | grep cam.da

kern.cam.da.2.minimum_cmd_size: 6
kern.cam.da.2.delete_max: 131072
kern.cam.da.2.delete_method: NONE
kern.cam.da.1.error_inject: 0
kern.cam.da.1.sort_io_queue: 0
kern.cam.da.1.minimum_cmd_size: 6
kern.cam.da.1.delete_max: 131072
kern.cam.da.1.delete_method: NONE
kern.cam.da.0.error_inject: 0
kern.cam.da.0.sort_io_queue: -1
kern.cam.da.0.minimum_cmd_size: 6
kern.cam.da.0.delete_max: 131072
kern.cam.da.0.delete_method: NONE

Note the delete_method is NONE


# sysctl -a | grep trim
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.trim_on_init: 1
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 181
kstat.zfs.misc.zio_trim.success: 0
kstat.zfs.misc.zio_trim.bytes: 0

Note no trimmed bytes.


On the target machine, 10.1 and 10.2 share the same config file:
/etc/ctl.conf

portal-group pg0 {
        discovery-auth-group no-authentication
        listen 0.0.0.0
        listen [::]
}

        lun 0 {
                path /pool92/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92"
                option vendor "pool92"
                option insecure_tpc "on"
        }
}


target iqn.iscsi1.zvol {
        auth-group no-authentication
        portal-group pg0

        lun 0 {
                path /pool92_1/iscsi/iscsi.zvol
                blocksize 4K
                size 5T
                option unmap "on"
                option scsiname "pool92_1"
                option vendor "pool92_1"
                option insecure_tpc "on"
        }
}


When I boot a 10.1 Target server, the 10.2 initiator connects, and we do
see proper UNMAP ability:


kern.cam.da.2.minimum_cmd_size: 6
kern.cam.da.2.delete_max: 5497558138880
kern.cam.da.2.delete_method: UNMAP
kern.cam.da.1.error_inject: 0
kern.cam.da.1.sort_io_queue: 0
kern.cam.da.1.minimum_cmd_size: 6
kern.cam.da.1.delete_max: 5497558138880
kern.cam.da.1.delete_method: UNMAP
kern.cam.da.0.error_inject: 0
kern.cam.da.0.sort_io_queue: -1
kern.cam.da.0.minimum_cmd_size: 6
kern.cam.da.0.delete_max: 131072
kern.cam.da.0.delete_method: NONE


Please let me know what you'd like to know next.

Thanks.

Steven Hartland

2015-Nov-18 00:28 UTC

head link

Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target

On 17/11/2015 22:08, Christopher Forgeron wrote:> I just submitted this as a bug:
>
> ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 )
>
> ..but I thought I should bring it to the list's attention for more
exposure
> - If that's a no-no, let me know, as I have a few others that are
related
> to this that I'd like to discuss.
>
> - - - -
>
>
> Consider this scenario:
>
> Virtual FreeBSD Machine, with a zpool created out of iSCSI disks.
> Physical FreeBSD Machine, with a zpool holding a sparse file that is the
> target for the iSCSI disk.
>
> This setup works in an environment with all 10.1 machines, doesn't with
all
> 10.2 machines.
>
> - The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no
> custom.
> - The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no
> custom.
> - iSCSI is all CAM iSCSI, not the old istgt platform.
> - The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target)
>
> The target machine is the same physical machine, with the same zpools - I
> either boot 10.1 or 10.2 for testing, and use the same zpool/disks
>
> to ensure nothing is changing.
>
> If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target,
> TRIM doesn't work (shows as NONE below).
> If I have a 10.2 iSCSI Initiator (client) connected to a 10.1 iSCSI Target,
> TRIM does work.
>
> (There is another bug with that last scenario as well, but I will open it
> separately)
>
> ...for clarity, a 10.1 iSCSI Initiator connected to a 10.1 iSCSI Target
> also works perfectly. I have ~20 of these in the field.
>
> On the 10.1 / 10.2 Targets, the ctl.conf file is identical. Zpools are
> identical, because they are shared between reboots of the same iSCSI
>
> target machine.
>
>
>
> On the 10.2 initiator machine, connected to a 10.2 Target machine:
>
> # sysctl -a | grep cam.da
>
> kern.cam.da.2.minimum_cmd_size: 6
> kern.cam.da.2.delete_max: 131072
> kern.cam.da.2.delete_method: NONE
> kern.cam.da.1.error_inject: 0
> kern.cam.da.1.sort_io_queue: 0
> kern.cam.da.1.minimum_cmd_size: 6
> kern.cam.da.1.delete_max: 131072
> kern.cam.da.1.delete_method: NONE
> kern.cam.da.0.error_inject: 0
> kern.cam.da.0.sort_io_queue: -1
> kern.cam.da.0.minimum_cmd_size: 6
> kern.cam.da.0.delete_max: 131072
> kern.cam.da.0.delete_method: NONE
>
> Note the delete_method is NONE
>
>
> # sysctl -a | grep trim
> vfs.zfs.trim.max_interval: 1
> vfs.zfs.trim.timeout: 30
> vfs.zfs.trim.txg_delay: 32
> vfs.zfs.trim.enabled: 1
> vfs.zfs.vdev.trim_max_pending: 10000
> vfs.zfs.vdev.trim_max_active: 64
> vfs.zfs.vdev.trim_min_active: 1
> vfs.zfs.vdev.trim_on_init: 1
> kstat.zfs.misc.zio_trim.failed: 0
> kstat.zfs.misc.zio_trim.unsupported: 181
> kstat.zfs.misc.zio_trim.success: 0
> kstat.zfs.misc.zio_trim.bytes: 0
>
> Note no trimmed bytes.
>
>
> On the target machine, 10.1 and 10.2 share the same config file:
> /etc/ctl.conf
>
> portal-group pg0 {
>          discovery-auth-group no-authentication
>          listen 0.0.0.0
>          listen [::]
> }
>
>          lun 0 {
>                  path /pool92/iscsi/iscsi.zvol
>                  blocksize 4K
>                  size 5T
>                  option unmap "on"
>                  option scsiname "pool92"
>                  option vendor "pool92"
>                  option insecure_tpc "on"
>          }
> }
>
>
> target iqn.iscsi1.zvol {
>          auth-group no-authentication
>          portal-group pg0
>
>          lun 0 {
>                  path /pool92_1/iscsi/iscsi.zvol
>                  blocksize 4K
>                  size 5T
>                  option unmap "on"
>                  option scsiname "pool92_1"
>                  option vendor "pool92_1"
>                  option insecure_tpc "on"
>          }
> }
>
>
> When I boot a 10.1 Target server, the 10.2 initiator connects, and we do
> see proper UNMAP ability:
>
>
> kern.cam.da.2.minimum_cmd_size: 6
> kern.cam.da.2.delete_max: 5497558138880
> kern.cam.da.2.delete_method: UNMAP
> kern.cam.da.1.error_inject: 0
> kern.cam.da.1.sort_io_queue: 0
> kern.cam.da.1.minimum_cmd_size: 6
> kern.cam.da.1.delete_max: 5497558138880
> kern.cam.da.1.delete_method: UNMAP
> kern.cam.da.0.error_inject: 0
> kern.cam.da.0.sort_io_queue: -1
> kern.cam.da.0.minimum_cmd_size: 6
> kern.cam.da.0.delete_max: 131072
> kern.cam.da.0.delete_method: NONE
>
>
> Please let me know what you'd like to know next.
>Having a quick flick through the code it looks like umap is now only 
supported on dev backed and not file backed.

I believe the following commit is the cause:
https://svnweb.freebsd.org/base?view=revision&revision=279005

This was an MFC of:
https://svnweb.freebsd.org/base?view=revision&revision=278672

I'm guessing this was an unintentional side effect mav?

     Regards
     Steve

freebsd stable - Nov 2015 - Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target

Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target

Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target