zfs discuss - Nov 2006 - ZFS/iSCSI target integration

Rick McNeal and I have been working on building support for sharing ZVOLs
as iSCSI targets directly into ZFS. Below is the proposal I''ll be
submitting to PSARC. Comments and suggestions are welcome.

Adam

---8<---

iSCSI/ZFS Integration

A. Overview

The goal of this project is to couple ZFS with the iSCSI target in Solaris
specifically to make it as easy to create and export ZVOLs via iSCSI as it
is to create and export ZFS filesystems via NFS. We will add two new ZFS
properties to support this feature.

    shareiscsi

	Like the ''sharenfs'' property, ''shareiscsi''
indicates if a ZVOL should
	be exported as an iSCSI target. The acceptable values for this property
	are ''on'', ''off'', and
''direct''. In the future, we may support other
	target types (for example, ''tape''). The default is
''off''. This property
	may be set on filesystems, but has no direct effect; this is to allow
	ZVOLs created under the ZFS hierarchy to inherit a default. For
	example, an administrator may want ZVOLs to be shared by default, and
	so set ''shareiscsi=on'' for the pool.

    iscsioptions

	This property, which is hidden by default, is used by the iSCSI target
	daemon to store persistent information such as the IQN. The contents
	are not intended for users or external consumers.


B. Examples

iSCSI targets are simple to create with the zfs(1M) command:

# zfs create -V 100M pool/volumes/v1
# zfs set shareiscsi=on pool/volumes/v1
# iscsitadm list target
Target: pool/volumes/v1
    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
    Connections: 0

Renaming the ZVOL has the expected result for the iSCSI target:

# zfs rename pool/volumes/v1 pool/volumes/stuff     
# iscsitadm list target
Target: pool/volumes/stuff
    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
    Connections: 0

Note that per the iSCSI specification (RFC3720), the iSCSI Name is unchanged
after the ZVOL is renamed.

Exporting a pool containing a shared ZVOL will cause the target to be removed;
importing a pool containing a shared ZVOL will cause the target to be
shared:

# zpool export pool
# iscsitadm list target
# zpool import pool
# iscsitadm list target
Target: pool/volumes/stuff
    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
    Connections: 0

Note again that all configuration information is stored with the dataset. As
with NFS shared filesystems, iSCSI targets imported on a different system
will be shared appropriately.

---8<---

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Adam Leventhal wrote:
> iSCSI/ZFS Integration
> 
> A. Overview
> 
> The goal of this project is to couple ZFS with the iSCSI target in Solaris
> specifically to make it as easy to create and export ZVOLs via iSCSI as it
> is to create and export ZFS filesystems via NFS. We will add two new ZFS
> properties to support this feature.
Yeah!
>     shareiscsi
> 
> 	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> 	be exported as an iSCSI target. The acceptable values for this property
> 	are ''on'', ''off'', and
''direct''. In the future, we may support other
> 	target types (for example, ''tape''). The default is
''off''. This property
> 	may be set on filesystems, but has no direct effect; this is to allow
> 	ZVOLs created under the ZFS hierarchy to inherit a default. For
> 	example, an administrator may want ZVOLs to be shared by default, and
> 	so set ''shareiscsi=on'' for the pool.
Excellent, I particularly like the fact you can set shareiscsi=on at
the filesystem level to get inheritance.
>     iscsioptions
> 
> 	This property, which is hidden by default, is used by the iSCSI target
> 	daemon to store persistent information such as the IQN. The contents
> 	are not intended for users or external consumers.
What does "hidden" mean here ?  Is there a way to view it ?  Just 
curious more than anything I don''t see a problem.

The usage looks great.

BTW where is the iscsitadm(1M) man page it doesn''t seem to be in
snv_50.

-- 
Darren J Moffat

On Wed, Nov 01, 2006 at 01:33:33AM -0800, Adam Leventhal
wrote:
> Rick McNeal and I have been working on building support for sharing ZVOLs
> as iSCSI targets directly into ZFS. Below is the proposal I''ll be
> submitting to PSARC. Comments and suggestions are welcome.
It looks great and I''d love to see it implemented.

Ceri
-- 
That must be wonderful!  I don''t understand it at all.
                                                  -- Moliere
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061101/37037f40/attachment.bin>

On Wed, Nov 01, 2006 at 09:58:12AM +0000, Darren J Moffat
wrote:
> >    iscsioptions
> >
> >	This property, which is hidden by default, is used by the iSCSI 
> >	target
> >	daemon to store persistent information such as the IQN. The contents
> >	are not intended for users or external consumers.
> 
> What does "hidden" mean here ?  Is there a way to view it ?  Just
> curious more than anything I don''t see a problem.
If one were to type ''zfs get all <zvol>'', the
iscsioptions property would
not be shown. If you do ''zfs get iscsioptions <zvol>''
you can see it.
> BTW where is the iscsitadm(1M) man page it doesn''t seem to be in
snv_50.
Rick probably has an answer.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

> I''m assuming that growing the backing-store will be transparent,
say
> increasing a target from 500GB to 1TB.
Yes, if you were to set the ''volsize'' property, that change
would show up
in the target:

# iscsitadm list target -v
Target: pool/volumes/vol1
    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
    Alias: pool/volumes/vol1
	...
            Size:  100M
	...
# zfs set volsize=1G pool/volumes/vol1
# iscsitadm list target -v
Target: pool/volumes/vol1
    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
    Alias: pool/volumes/vol1
	...
            Size: 1.0G
	...

Look good? (Keep in mind you would probably want to change the reservation as
well in this case.)

Adam

On Wed, Nov 01, 2006 at 01:06:53AM -0800, Ben Rockwood
wrote:
> Two very enthusiast thumbs up.
> 
> I''m assuming that growing the backing-store will be transparent,
say
> increasing a target from 500GB to 1TB.
> 
> benr.
> 
> 
> Adam Leventhal wrote:
> >Rick McNeal and I have been working on building support for sharing
ZVOLs
> >as iSCSI targets directly into ZFS. Below is the proposal I''ll
be
> >submitting to PSARC. Comments and suggestions are welcome.
> >
> >Adam
> >
> >---8<---
> >
> >iSCSI/ZFS Integration
> >
> >A. Overview
> >
> >The goal of this project is to couple ZFS with the iSCSI target in
Solaris
> >specifically to make it as easy to create and export ZVOLs via iSCSI as
it
> >is to create and export ZFS filesystems via NFS. We will add two new
ZFS
> >properties to support this feature.
> >
> >    shareiscsi
> >
> >	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> >	be exported as an iSCSI target. The acceptable values for this 
> >	property
> >	are ''on'', ''off'', and
''direct''. In the future, we may support other
> >	target types (for example, ''tape''). The default is
''off''. This
> >	property
> >	may be set on filesystems, but has no direct effect; this is to allow
> >	ZVOLs created under the ZFS hierarchy to inherit a default. For
> >	example, an administrator may want ZVOLs to be shared by default, and
> >	so set ''shareiscsi=on'' for the pool.
> >
> >    iscsioptions
> >
> >	This property, which is hidden by default, is used by the iSCSI 
> >	target
> >	daemon to store persistent information such as the IQN. The contents
> >	are not intended for users or external consumers.
> >
> >
> >B. Examples
> >
> >iSCSI targets are simple to create with the zfs(1M) command:
> >
> ># zfs create -V 100M pool/volumes/v1
> ># zfs set shareiscsi=on pool/volumes/v1
> ># iscsitadm list target
> >Target: pool/volumes/v1
> >    iSCSI Name:
iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
> >    Connections: 0
> >
> >Renaming the ZVOL has the expected result for the iSCSI target:
> >
> ># zfs rename pool/volumes/v1 pool/volumes/stuff     
> ># iscsitadm list target
> >Target: pool/volumes/stuff
> >    iSCSI Name:
iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
> >    Connections: 0
> >
> >Note that per the iSCSI specification (RFC3720), the iSCSI Name is 
> >unchanged
> >after the ZVOL is renamed.
> >
> >Exporting a pool containing a shared ZVOL will cause the target to be 
> >removed;
> >importing a pool containing a shared ZVOL will cause the target to be
> >shared:
> >
> ># zpool export pool
> ># iscsitadm list target
> ># zpool import pool
> ># iscsitadm list target
> >Target: pool/volumes/stuff
> >    iSCSI Name:
iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
> >    Connections: 0
> >
> >Note again that all configuration information is stored with the
dataset.
> >As
> >with NFS shared filesystems, iSCSI targets imported on a different
system
> >will be shared appropriately.
> >
> >---8<---
> >
> >  
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Nov 01, 2006 at 10:05:01AM +0000, Ceri Davies
wrote:
> On Wed, Nov 01, 2006 at 01:33:33AM -0800, Adam Leventhal wrote:
> > Rick McNeal and I have been working on building support for sharing
ZVOLs
> > as iSCSI targets directly into ZFS. Below is the proposal
I''ll be
> > submitting to PSARC. Comments and suggestions are welcome.
> 
> It looks great and I''d love to see it implemented.
It''s implemented! This is the end of the process, not the beginning ;-)
I expect it will be in OpenSolaris by the end of November.

Adam


-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

>
> Note again that all configuration information is stored with the dataset.
As
> with NFS shared filesystems, iSCSI targets imported on a different system
> will be shared appropriately.

Does that mean that if I manage the iSCSI target via iscsitadm after
it is shared via
zfs shareiscsi=on and then ''zpool export'' and ''zpool
import'' on some other host
all the customization done via iscsitadm will be preserved ?



-- 
Regards,
        Cyril

On Wed, Nov 01, 2006 at 12:18:36PM +0200, Cyril Plisko
wrote:
> >
> >Note again that all configuration information is stored with the
dataset.
> >As
> >with NFS shared filesystems, iSCSI targets imported on a different
system
> >will be shared appropriately.
> 
> 
> Does that mean that if I manage the iSCSI target via iscsitadm after
> it is shared via
> zfs shareiscsi=on and then ''zpool export'' and
''zpool import'' on some other
> host
> all the customization done via iscsitadm will be preserved ?
No. Modifications to the target must be made through zfs(1M) not through
iscsitadm(1M) if you want them to be persistent. This is similar to sharing
ZFS filesystems via NFS: you can use share(1M), but it doesn''t affect
the
persistent properties of the dataset.

What properties are you specifically interested in modifying?

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Nov 01, 2006 at 02:14:24AM -0800, Adam Leventhal
wrote:
> On Wed, Nov 01, 2006 at 10:05:01AM +0000, Ceri Davies wrote:
> > On Wed, Nov 01, 2006 at 01:33:33AM -0800, Adam Leventhal wrote:
> > > Rick McNeal and I have been working on building support for
sharing ZVOLs
> > > as iSCSI targets directly into ZFS. Below is the proposal
I''ll be
> > > submitting to PSARC. Comments and suggestions are welcome.
> > 
> > It looks great and I''d love to see it implemented.
> 
> It''s implemented! This is the end of the process, not the
beginning ;-)
> I expect it will be in OpenSolaris by the end of November.
Oh, right.  For my next wish... :)

Ceri
-- 
That must be wonderful!  I don''t understand it at all.
                                                  -- Moliere
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061101/4971ca4d/attachment.bin>

On 11/1/06, Adam Leventhal <ahl at eng.sun.com>
wrote:
> On Wed, Nov 01, 2006 at 12:18:36PM +0200, Cyril Plisko wrote:
> > >
> > >Note again that all configuration information is stored with the
dataset.
> > >As
> > >with NFS shared filesystems, iSCSI targets imported on a different
system
> > >will be shared appropriately.
> >
> >
> > Does that mean that if I manage the iSCSI target via iscsitadm after
> > it is shared via
> > zfs shareiscsi=on and then ''zpool export'' and
''zpool import'' on some other
> > host
> > all the customization done via iscsitadm will be preserved ?
>
> No. Modifications to the target must be made through zfs(1M) not through
> iscsitadm(1M) if you want them to be persistent. This is similar to sharing
> ZFS filesystems via NFS: you can use share(1M), but it doesn''t
affect the
> persistent properties of the dataset.
>
> What properties are you specifically interested in modifying?
LUN for example. How would I configure LUN via zfs command ?

-- 
Regards,
        Cyril

On 01/11/06, Adam Leventhal <ahl at eng.sun.com>
wrote:
> Rick McNeal and I have been working on building support for sharing ZVOLs
> as iSCSI targets directly into ZFS. Below is the proposal I''ll be
> submitting to PSARC. Comments and suggestions are welcome.
>
> Adam
Am I right in thinking we''re effectively able to snapshot/clone iscsi
targets
now (by working on the underlying ZVOL)? And we''ll be able to use
sparse zvols
for this too (can''t think why we couldn''t, but it''d
be dead handy)?

This will be extremely useful, thanks.

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

On 01/11/06, Dick Davies <rasputnik at gmail.com>
wrote:
> And we''ll be able to use sparse zvols
> for this too (can''t think why we couldn''t, but
it''d be dead handy)?
Thinking about this, we won''t be able to (without some changes) -
I think a target is zero-filled before going online
(educated guess: it takes 5 minutes to ''online'' a target,
and it consumes virtually no space in the parent zvol if compression is on),
so a sparse zvol would exhaust zpool space.

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

On 11/1/06, Dick Davies <rasputnik at gmail.com>
wrote:
> On 01/11/06, Dick Davies <rasputnik at gmail.com> wrote:
> > And we''ll be able to use sparse zvols
> > for this too (can''t think why we couldn''t, but
it''d be dead handy)?
>
> Thinking about this, we won''t be able to (without some changes) -
> I think a target is zero-filled before going online
> (educated guess: it takes 5 minutes to ''online'' a target,
> and it consumes virtually no space in the parent zvol if compression is
on),
> so a sparse zvol would exhaust zpool space.
Looking at the code it doesn''t seem like the backing store being
zeroed.
In case of regular file a single sector (512 byte) of uninitialized data from
stack (bad practice ?) is written to the very end of the file. And in case
of character device it isn''t written at all. zvol should fall into char
device
category. See mgmt_create.c::setup_disk_backing()

Or did I miss something ?

-- 
Regards,
        Cyril

On 01/11/06, Cyril Plisko <cyril.plisko at mountall.com>
wrote:
> On 11/1/06, Dick Davies <rasputnik at gmail.com> wrote:
> > On 01/11/06, Dick Davies <rasputnik at gmail.com> wrote:
> > > And we''ll be able to use sparse zvols
> > > for this too (can''t think why we couldn''t, but
it''d be dead handy)?
> >
> > Thinking about this, we won''t be able to (without some
changes) -
> > I think a target is zero-filled before going online
> > (educated guess: it takes 5 minutes to ''online'' a
target,
> > and it consumes virtually no space in the parent zvol if compression
is on),
> > so a sparse zvol would exhaust zpool space.
>
> Looking at the code it doesn''t seem like the backing store being
zeroed.
> In case of regular file a single sector (512 byte) of uninitialized data
from
> stack (bad practice ?) is written to the very end of the file. And in case
> of character device it isn''t written at all. zvol should fall into
char device
> category. See mgmt_create.c::setup_disk_backing()
>
> Or did I miss something ?
I''m not the one to ask :)
I''m just saying what I''ve seen - it was SXCR b49, and a ZFS
filesystem, not a zvol as I said (seems iscsi targets are file backed
by default). Still took a few minutes to online a new target, so it was doing
something, but I don''t know what.

If it''s a non-issue that''d be great,


-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

On Wed, 1 Nov 2006, Adam Leventhal wrote:
> Rick McNeal and I have been working on building support for sharing ZVOLs
> as iSCSI targets directly into ZFS. Below is the proposal I''ll be
> submitting to PSARC. Comments and suggestions are welcome.
>
> Adam
>
> ---8<---
>
> iSCSI/ZFS Integration
>
> A. Overview
>
> The goal of this project is to couple ZFS with the iSCSI target in Solaris
> specifically to make it as easy to create and export ZVOLs via iSCSI as it
> is to create and export ZFS filesystems via NFS. We will add two new ZFS
> properties to support this feature.
>
>    shareiscsi
>
> 	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> 	be exported as an iSCSI target. The acceptable values for this property
> 	are ''on'', ''off'', and
''direct''. In the future, we may support other
> 	target types (for example, ''tape''). The default is
''off''. This property
> 	may be set on filesystems, but has no direct effect; this is to allow
> 	ZVOLs created under the ZFS hierarchy to inherit a default. For
> 	example, an administrator may want ZVOLs to be shared by default, and
> 	so set ''shareiscsi=on'' for the pool.
>
>    iscsioptions
>
> 	This property, which is hidden by default, is used by the iSCSI target
> 	daemon to store persistent information such as the IQN. The contents
> 	are not intended for users or external consumers.
>
>
> B. Examples
>
> iSCSI targets are simple to create with the zfs(1M) command:
>
> # zfs create -V 100M pool/volumes/v1
> # zfs set shareiscsi=on pool/volumes/v1
> # iscsitadm list target
> Target: pool/volumes/v1
>    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>    Connections: 0
>
> Renaming the ZVOL has the expected result for the iSCSI target:
>
> # zfs rename pool/volumes/v1 pool/volumes/stuff
> # iscsitadm list target
> Target: pool/volumes/stuff
>    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>    Connections: 0
>
> Note that per the iSCSI specification (RFC3720), the iSCSI Name is
unchanged
> after the ZVOL is renamed.
>
> Exporting a pool containing a shared ZVOL will cause the target to be
removed;
> importing a pool containing a shared ZVOL will cause the target to be
> shared:
>
> # zpool export pool
> # iscsitadm list target
> # zpool import pool
> # iscsitadm list target
> Target: pool/volumes/stuff
>    iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>    Connections: 0
>
> Note again that all configuration information is stored with the dataset.
As
> with NFS shared filesystems, iSCSI targets imported on a different system
> will be shared appropriately.
This is super useful! Will ACLs and aliases be stored as properties? Could 
you post the list of available iSCSI properties to the list?

Thanks,
- Ryan
--
UNIX Administrator
http://prefetch.net

> >What properties are you specifically interested in modifying?
> 
> LUN for example. How would I configure LUN via zfs command ?
You can''t. Forgive my ignorance about how iSCSI is deployed, but why
would
you want/need to change the LUN?

Adam

On Wed, Nov 01, 2006 at 01:36:05PM +0200, Cyril Plisko
wrote:
> On 11/1/06, Adam Leventhal <ahl at eng.sun.com> wrote:
> >On Wed, Nov 01, 2006 at 12:18:36PM +0200, Cyril Plisko wrote:
> >> >
> >> >Note again that all configuration information is stored with
the
> >dataset.
> >> >As
> >> >with NFS shared filesystems, iSCSI targets imported on a
different
> >system
> >> >will be shared appropriately.
> >>
> >>
> >> Does that mean that if I manage the iSCSI target via iscsitadm
after
> >> it is shared via
> >> zfs shareiscsi=on and then ''zpool export'' and
''zpool import'' on some
> >other
> >> host
> >> all the customization done via iscsitadm will be preserved ?
> >
> >No. Modifications to the target must be made through zfs(1M) not
through
> >iscsitadm(1M) if you want them to be persistent. This is similar to
sharing
> >ZFS filesystems via NFS: you can use share(1M), but it doesn''t
affect the
> >persistent properties of the dataset.
> >
> >What properties are you specifically interested in modifying?
> 
> LUN for example. How would I configure LUN via zfs command ?
> 
> -- 
> Regards,
>        Cyril
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Adam Leventhal wrote:
>
> Exporting a pool containing a shared ZVOL will cause the target to be
removed;
> importing a pool containing a shared ZVOL will cause the target to be
> shared:
>
>   

Is there going to be a method to override that on the import? I can see 
a situation where you want to import the pool for some kind of 
maintenance procedure but you don''t want the iSCSI target to fire up 
automagically.

Also, what if I don''t have the iSCSI target packages on the node
I''m
importing to? Error messages? Nothing?

On Wed, Nov 01, 2006 at 01:17:02PM -0500, Torrey McMahon
wrote:
> Is there going to be a method to override that on the import? I can see 
> a situation where you want to import the pool for some kind of 
> maintenance procedure but you don''t want the iSCSI target to fire
up
> automagically.
There isn''t -- to my knowledge -- a way to do this today for NFS
shares.
This would be a reasonable RFE that would apply to both NFS and iSCSI.
> Also, what if I don''t have the iSCSI target packages on the node
I''m
> importing to? Error messages? Nothing?
You''ll get an error message reporting that it could not be shared.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Nov 01, 2006 at 10:57:27AM -0800, Adam Leventhal
wrote:
> On Wed, Nov 01, 2006 at 01:17:02PM -0500, Torrey McMahon wrote:
> > Is there going to be a method to override that on the import? I can
see
> > a situation where you want to import the pool for some kind of 
> > maintenance procedure but you don''t want the iSCSI target to
fire up
> > automagically.
> 
> There isn''t -- to my knowledge -- a way to do this today for NFS
shares.
> This would be a reasonable RFE that would apply to both NFS and iSCSI.
svcadm disable nfs/server ?

Ceri
-- 
That must be wonderful!  I don''t understand it at all.
                                                  -- Moliere
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061101/9985d226/attachment.bin>

On Wed, Nov 01, 2006 at 12:22:46PM -0500, Matty wrote:
> This is super useful! Will ACLs and aliases be stored as properties? Could 
> you post the list of available iSCSI properties to the list?
We''re still investigating ACL and iSNS support. The alias name will
always be
the name of the dataset, but we''ve considered making that an option you
could
set in the ''shareiscsi'' property
(''alias=<blah>'' for example). The iSCSI
properties I was referring to are the private meta data for the target daemon
such as IQN.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Nov 01, 2006 at 07:00:52PM +0000, Ceri Davies
wrote:
> > There isn''t -- to my knowledge -- a way to do this today for
NFS shares.
> > This would be a reasonable RFE that would apply to both NFS and iSCSI.
> 
> svcadm disable nfs/server ?
That won''t to it because the service will be automatically enabled. You
can set the application/auto_enable property to false for the nfs/server
service and the same will be true for the iscsitgt service.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Adam Leventhal wrote:
> On Wed, Nov 01, 2006 at 01:17:02PM -0500, Torrey McMahon wrote:
> > Is there going to be a method to override that on the import? I can
see
> > a situation where you want to import the pool for some kind of 
> > maintenance procedure but you don''t want the iSCSI target to
fire up
> > automagically.
> 
> There isn''t -- to my knowledge -- a way to do this today for NFS
shares.
> This would be a reasonable RFE that would apply to both NFS and iSCSI.
In the case of NFS, this can be dangerous if the "rest" of the NFS
server is allowed to come up and serve other filesystems.  The non-shared
filesystem will end up returning ESTALE errors to clients that are
active on that filesystem.  It should be an all or nothing selection...

Spencer
> 
> > Also, what if I don''t have the iSCSI target packages on the
node I''m
> > importing to? Error messages? Nothing?
> 
> You''ll get an error message reporting that it could not be shared.
> 
> Adam
> 
> -- 
> Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On 11/1/06, Adam Leventhal <ahl at eng.sun.com>
wrote:
> > >What properties are you specifically interested in modifying?
> >
> > LUN for example. How would I configure LUN via zfs command ?
>
> You can''t. Forgive my ignorance about how iSCSI is deployed, but
why would
> you want/need to change the LUN?
Well, with iSCSI specifically it is of less importance, since one can easily
created multiple units identified by other means, than LUN.
I, however, trying to look forward for FC SCSI target functionality mirroring
that of the iSCSI (AFAIK it is on the Rick'' roadmap [and I really do
not
mind helping]). In FC world it is essentially the only way to have multiple
units on a particular FC port.

Can we do something similar to NFS case, where sharenfs can be
"on", "off", or something else, in which case it is a list
of options ?
Would this technique be applicable to shareiscsi too ?

-- 
Regards,
        Cyril

On Wed, Nov 01, 2006 at 09:25:26PM +0200, Cyril Plisko
wrote:
> On 11/1/06, Adam Leventhal <ahl at eng.sun.com> wrote:
> >> >What properties are you specifically interested in modifying?
> >>
> >> LUN for example. How would I configure LUN via zfs command ?
> >
> >You can''t. Forgive my ignorance about how iSCSI is deployed,
but why would
> >you want/need to change the LUN?
> 
> Well, with iSCSI specifically it is of less importance, since one can
easily
> created multiple units identified by other means, than LUN.
> I, however, trying to look forward for FC SCSI target functionality 
> mirroring
> that of the iSCSI (AFAIK it is on the Rick'' roadmap [and I really
do not
> mind helping]). In FC world it is essentially the only way to have multiple
> units on a particular FC port.
> 
> Can we do something similar to NFS case, where sharenfs can be
> "on", "off", or something else, in which case it is a
list of options ?
> Would this technique be applicable to shareiscsi too ?
Absolutely. We would, however, like to be conservative about adding options
only doing so when it meets a specific need. As you noted, there''s no
real
requirement to be able to set the LUN.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On 11/1/06, Adam Leventhal <ahl at eng.sun.com>
wrote:
> On Wed, Nov 01, 2006 at 09:25:26PM +0200, Cyril Plisko wrote:
> > On 11/1/06, Adam Leventhal <ahl at eng.sun.com> wrote:
> > >> >What properties are you specifically interested in
modifying?
> > >>
> > >> LUN for example. How would I configure LUN via zfs command ?
> > >
> > >You can''t. Forgive my ignorance about how iSCSI is
deployed, but why would
> > >you want/need to change the LUN?
> >
> > Well, with iSCSI specifically it is of less importance, since one can
easily
> > created multiple units identified by other means, than LUN.
> > I, however, trying to look forward for FC SCSI target functionality
> > mirroring
> > that of the iSCSI (AFAIK it is on the Rick'' roadmap [and I
really do not
> > mind helping]). In FC world it is essentially the only way to have
multiple
> > units on a particular FC port.
> >
> > Can we do something similar to NFS case, where sharenfs can be
> > "on", "off", or something else, in which case it
is a list of options ?
> > Would this technique be applicable to shareiscsi too ?
>
> Absolutely. We would, however, like to be conservative about adding options
> only doing so when it meets a specific need. As you noted, there''s
no real
> requirement to be able to set the LUN.
...to be able to set the LUN *for iSCSI*. That would be more concise
formulation.

I wonder why you would like to be conservative. Can you please explain
your considerations ? How would it be different from NFS ? It seems that
in NFS case one can put whatever options she likes in "sharenfs="
attribute.

Also can you please elaborate more on the hidden attribute ? Which
values would be stored there ? Maybe the idea was to put all the options
into that attribute, thus making the "zfs get" output prettier (iSCSI
options
tend to have massive character count).
In that case why not to make something similar with NFS too ?

-- 
Regards,
        Cyril

Spencer Shepler wrote:
> On Wed, Adam Leventhal wrote:
>   
>> On Wed, Nov 01, 2006 at 01:17:02PM -0500, Torrey McMahon wrote:
>>     
>>> Is there going to be a method to override that on the import? I can
see
>>> a situation where you want to import the pool for some kind of 
>>> maintenance procedure but you don''t want the iSCSI target
to fire up
>>> automagically.
>>>       
>> There isn''t -- to my knowledge -- a way to do this today for
NFS shares.
>> This would be a reasonable RFE that would apply to both NFS and iSCSI.
>>     
>
> In the case of NFS, this can be dangerous if the "rest" of the
NFS
> server is allowed to come up and serve other filesystems.  The non-shared
> filesystem will end up returning ESTALE errors to clients that are
> active on that filesystem.  It should be an all or nothing selection...
>   
Lets say server A has the pool with NFS shared, or iSCSI shared, 
volumes. Server A exports the pool or goes down. Server B imports the pool.

Which clients would still be active on the filesystem(s)? The ones that 
were mounting it when it was on Server A?

On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon
wrote:
> Lets say server A has the pool with NFS shared, or iSCSI shared, 
> volumes. Server A exports the pool or goes down. Server B imports the pool.
> 
> Which clients would still be active on the filesystem(s)? The ones that 
> were mounting it when it was on Server A?
Clients would need to explicitly change the server they''re contacting
unless
that new server also took over the IP address, hostname, etc.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Adam Leventhal wrote:
> On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon wrote:
>> Lets say server A has the pool with NFS shared, or iSCSI shared, 
>> volumes. Server A exports the pool or goes down. Server B imports the
pool.
>>
>> Which clients would still be active on the filesystem(s)? The ones that
>> were mounting it when it was on Server A?
> 
> Clients would need to explicitly change the server they''re
contacting unless
> that new server also took over the IP address, hostname, etc.
Does this imply that using Sun Cluster, which transfers (logical) IP
addresses, would easily provide an HA-iSCSI service?  I''ll admit that
I still don''t understand the iSCSI naming and security schemes.
  -- richard

Cyril Plisko wrote:
>> > Can we do something similar to NFS case, where sharenfs can be
>> > "on", "off", or something else, in which case
it is a list of options ?
>> > Would this technique be applicable to shareiscsi too ?
>>
>> Absolutely. We would, however, like to be conservative about adding 
>> options
>> only doing so when it meets a specific need. As you noted,
there''s no
>> real
>> requirement to be able to set the LUN.
> 
> ...to be able to set the LUN *for iSCSI*. That would be more concise
> formulation.
> 
> I wonder why you would like to be conservative. Can you please explain
> your considerations ? How would it be different from NFS ? It seems that
> in NFS case one can put whatever options she likes in "sharenfs="
> attribute.
 From my point of view I would like to be conservative to prevent the 
addition of a cool new feature that six months down the road we find 
that it''s only used in one particular case. Once added it''s
difficult to
remove because of our backwards compatibility requirement.
> Also can you please elaborate more on the hidden attribute ? Which
> values would be stored there ? Maybe the idea was to put all the options
> into that attribute, thus making the "zfs get" output prettier
(iSCSI
> options
> tend to have massive character count).
> In that case why not to make something similar with NFS too ?
The target daemon stores the IQN name which is about 60 characters, the 
GUID which is 32 characters, the emulation type, plus a few other 
miscellaneous bits of information. The zfs command goes through some 
effort to determine the longest property name and value. It then 
displays that information in uniform column widths. The iscsi property 
value blows that out of the water.

-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Richard Elling - PAE wrote:
> Adam Leventhal wrote:
>> On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon wrote:
>>> Lets say server A has the pool with NFS shared, or iSCSI shared, 
>>> volumes. Server A exports the pool or goes down. Server B imports
the
>>> pool.
>>>
>>> Which clients would still be active on the filesystem(s)? The ones 
>>> that were mounting it when it was on Server A?
>>
>> Clients would need to explicitly change the server they''re
contacting
>> unless
>> that new server also took over the IP address, hostname, etc.
> 
> Does this imply that using Sun Cluster, which transfers (logical) IP
> addresses, would easily provide an HA-iSCSI service?  I''ll admit
that
> I still don''t understand the iSCSI naming and security schemes.
The simple answer is yes. For a complete solution the iSCSI target 
project needs to support iSNS and the administrator should setup RADIUS 
to enable simplified authentication administration.
>  -- richard
> _______________________________________________
> storage-discuss mailing list
> storage-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/storage-discuss
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Adam Leventhal wrote:
> On Wed, Nov 01, 2006 at 09:58:12AM +0000, Darren J Moffat wrote:
>>>    iscsioptions
>>>
>>> 	This property, which is hidden by default, is used by the iSCSI 
>>> 	target
>>> 	daemon to store persistent information such as the IQN. The
contents
>>> 	are not intended for users or external consumers.
>> What does "hidden" mean here ?  Is there a way to view it ? 
Just
>> curious more than anything I don''t see a problem.
> 
> If one were to type ''zfs get all <zvol>'', the
iscsioptions property would
> not be shown. If you do ''zfs get iscsioptions
<zvol>'' you can see it.
> 
>> BTW where is the iscsitadm(1M) man page it doesn''t seem to be
in snv_50.
> 
> Rick probably has an answer.
The man page is scheduled to go into build 52. It''s been available for 
some time, but because of vacation schedules and documentation gate 
schedules the man page hasn''t made it into the release yet.
> Adam
> 
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Instead of replying to each of the messages I''ll just reply to the
last,
but answer the questions that each of the three have raised.

Dick Davies wrote:
> On 01/11/06, Cyril Plisko <cyril.plisko at mountall.com> wrote:
>> On 11/1/06, Dick Davies <rasputnik at gmail.com> wrote:
>> > On 01/11/06, Dick Davies <rasputnik at gmail.com> wrote:
>> > > And we''ll be able to use sparse zvols
>> > > for this too (can''t think why we couldn''t,
but it''d be dead handy)?
>> >
>> > Thinking about this, we won''t be able to (without some
changes) -
>> > I think a target is zero-filled before going online
>> > (educated guess: it takes 5 minutes to ''online''
a target,
>> > and it consumes virtually no space in the parent zvol if
compression
>> is on),
>> > so a sparse zvol would exhaust zpool space.
>>
I should change the code and look to see if the backing store is a 
character device. If so, there''s no need to initialize the backing
store
to verify that the space is available.
>> Looking at the code it doesn''t seem like the backing store
being zeroed.
>> In case of regular file a single sector (512 byte) of uninitialized 
>> data from
>> stack (bad practice ?) is written to the very end of the file. And in 
>> case
I hang my head in shame. I''ve fixed the code.
>> of character device it isn''t written at all. zvol should fall
into
>> char device
>> category. See mgmt_create.c::setup_disk_backing()
>>
>> Or did I miss something ?
>
The routine that you''re looking at is primarily dealing with the 
condition when the administrator has provided a backing store. The code 
that you''re looking provides the ability to specify a backing store
that
doesn''t exist, but a size must also be given.

At the end of create_lun() there''s a call to create_lun_common(), not 
very original and a little misleading. create_lun_common() starts the 
thread which initializes the backing store. This is done for disk and 
tapes, but not for raw devices. I''m going to change create_lun_common()
to ignore backing store which points at character devices.
> I''m not the one to ask :)
> I''m just saying what I''ve seen - it was SXCR b49, and a
ZFS
> filesystem, not a zvol as I said (seems iscsi targets are file backed
> by default). Still took a few minutes to online a new target, so it was 
> doing
> something, but I don''t know what.
> 
The daemon does use regular files by default. This was done to make the 
daemon as flexible as possible.

When using regular files the daemon must zero fill the backing store so 
that initiators will not get a write error because the underlying file 
system has filled up. The daemon is able to create logical units that 
are hole-y files, but doing so requires a special XML tag to be added. 
I''ve been thinking about adding this as an option to the CLI if enough 
folks would find it useful.

Hope this helps.
> If it''s a non-issue that''d be great,
> 
> 
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Cyril Plisko wrote:
> On 11/1/06, Dick Davies <rasputnik at gmail.com> wrote:
>> On 01/11/06, Dick Davies <rasputnik at gmail.com> wrote:
>> > And we''ll be able to use sparse zvols
>> > for this too (can''t think why we couldn''t, but
it''d be dead handy)?
>>
>> Thinking about this, we won''t be able to (without some
changes) -
>> I think a target is zero-filled before going online
>> (educated guess: it takes 5 minutes to ''online'' a
target,
>> and it consumes virtually no space in the parent zvol if compression 
>> is on),
>> so a sparse zvol would exhaust zpool space.
> 
> Looking at the code it doesn''t seem like the backing store being
zeroed.
> In case of regular file a single sector (512 byte) of uninitialized data 
> from
> stack (bad practice ?) is written to the very end of the file. And in case
> of character device it isn''t written at all. zvol should fall into
char
> device
> category. See mgmt_create.c::setup_disk_backing()
In another email I indicated that the routine create_lun_common() was 
where the initialization was done and that I didn''t check for a 
character device. The code already does check for character devices and 
ignores them.
> Or did I miss something ?
I too must be missing something. I can''t imagine why it would take 5 
minutes to online a target. A ZVOL should automatically be brought 
online since now initialization is required.


-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Rick McNeal wrote:
>>> Looking at the code it doesn''t seem like the backing store
being zeroed.
>>> In case of regular file a single sector (512 byte) of uninitialized
>>> data from
>>> stack (bad practice ?) is written to the very end of the file. And
in
>>> case
> 
> I hang my head in shame. I''ve fixed the code.
NB. ZFS compression will very nicely compress zeros.  I suppose one could
consider this to be sparse.  One might wish to turn compression off for
iSCSI zvols if they also don''t wish a sparse-like behaviour.
  -- richard

On 01/11/06, Rick McNeal <Rick.McNeal at sun.com> wrote:
> I too must be missing something. I can''t imagine why it would take
5
> minutes to online a target. A ZVOL should automatically be brought
> online since now initialization is required.
s/now/no/ ?

Thanks for the explanation. The ''5 minute online'' issue I had
was
with a file-based target (which happened to be on a ZFS
filesystem).
>From what you say, it should be a non-issue
with a zvol-backed target.

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

Adam Leventhal wrote:
> Rick McNeal and I have been working on building support for sharing ZVOLs
> as iSCSI targets directly into ZFS. Below is the proposal I''ll be
> submitting to PSARC. Comments and suggestions are welcome.
> 
> Adam
> 
> ---8<---
> 
> iSCSI/ZFS Integration
> 
> A. Overview
> 
> The goal of this project is to couple ZFS with the iSCSI target in Solaris
> specifically to make it as easy to create and export ZVOLs via iSCSI as it
> is to create and export ZFS filesystems via NFS. We will add two new ZFS
> properties to support this feature.
> 
>     shareiscsi
> 
> 	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> 	be exported as an iSCSI target. The acceptable values for this property
> 	are ''on'', ''off'', and
''direct''. In the future, we may support other
> 	target types (for example, ''tape''). The default is
''off''. This property
> 	may be set on filesystems, but has no direct effect; this is to allow
> 	ZVOLs created under the ZFS hierarchy to inherit a default. For
> 	example, an administrator may want ZVOLs to be shared by default, and
> 	so set ''shareiscsi=on'' for the pool.
hey adam, what''s "direct" mean?

eric
> 
>     iscsioptions
> 
> 	This property, which is hidden by default, is used by the iSCSI target
> 	daemon to store persistent information such as the IQN. The contents
> 	are not intended for users or external consumers.
> 
> 
> B. Examples
> 
> iSCSI targets are simple to create with the zfs(1M) command:
> 
> # zfs create -V 100M pool/volumes/v1
> # zfs set shareiscsi=on pool/volumes/v1
> # iscsitadm list target
> Target: pool/volumes/v1
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Renaming the ZVOL has the expected result for the iSCSI target:
> 
> # zfs rename pool/volumes/v1 pool/volumes/stuff     
> # iscsitadm list target
> Target: pool/volumes/stuff
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Note that per the iSCSI specification (RFC3720), the iSCSI Name is
unchanged
> after the ZVOL is renamed.
> 
> Exporting a pool containing a shared ZVOL will cause the target to be
removed;
> importing a pool containing a shared ZVOL will cause the target to be
> shared:
> 
> # zpool export pool
> # iscsitadm list target
> # zpool import pool
> # iscsitadm list target
> Target: pool/volumes/stuff
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Note again that all configuration information is stored with the dataset.
As
> with NFS shared filesystems, iSCSI targets imported on a different system
> will be shared appropriately.
> 
> ---8<---
>

Hello Richard,

Wednesday, November 1, 2006, 11:36:14 PM, you wrote:

REP> Adam Leventhal wrote:
>> On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon wrote:
>>> Lets say server A has the pool with NFS shared, or iSCSI shared, 
>>> volumes. Server A exports the pool or goes down. Server B imports
the pool.
>>>
>>> Which clients would still be active on the filesystem(s)? The ones
that
>>> were mounting it when it was on Server A?
>> 
>> Clients would need to explicitly change the server they''re
contacting unless
>> that new server also took over the IP address, hostname, etc.
REP> Does this imply that using Sun Cluster, which transfers (logical) IP
REP> addresses, would easily provide an HA-iSCSI service?  I''ll
admit that
REP> I still don''t understand the iSCSI naming and security schemes.

I haven''t played with iSCSI (yet) but is really hostname necessary?
What else? Putting zpool under SC with an IP address in SC group
should be enough I hope - or maybe not?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

On Thu, Nov 02, 2006 at 12:10:06AM -0800, eric kustarz
wrote:
> >	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> >	be exported as an iSCSI target. The acceptable values for this 
> >	property
> >	are ''on'', ''off'', and
''direct''. In the future, we may support other
> >	target types (for example, ''tape''). The default is
''off''. This
> >	property
> >	may be set on filesystems, but has no direct effect; this is to allow
> >	ZVOLs created under the ZFS hierarchy to inherit a default. For
> >	example, an administrator may want ZVOLs to be shared by default, and
> >	so set ''shareiscsi=on'' for the pool.
> 
> hey adam, what''s "direct" mean?
It''s iSCSI target lingo for vanilla disk emulation.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon
wrote:
> Spencer Shepler wrote:
> >On Wed, Adam Leventhal wrote:
> >  
> >>On Wed, Nov 01, 2006 at 01:17:02PM -0500, Torrey McMahon wrote:
> >>    
> >>>Is there going to be a method to override that on the import? I
can see
> >>>a situation where you want to import the pool for some kind of 
> >>>maintenance procedure but you don''t want the iSCSI
target to fire up
> >>>automagically.
> >>>      
> >>There isn''t -- to my knowledge -- a way to do this today
for NFS shares.
> >>This would be a reasonable RFE that would apply to both NFS and
iSCSI.
> >>    
> >
> >In the case of NFS, this can be dangerous if the "rest" of
the NFS
> >server is allowed to come up and serve other filesystems.  The
non-shared
> >filesystem will end up returning ESTALE errors to clients that are
> >active on that filesystem.  It should be an all or nothing selection...
> >  
> 
> Lets say server A has the pool with NFS shared, or iSCSI shared, 
> volumes. Server A exports the pool or goes down. Server B imports the pool.
> 
> Which clients would still be active on the filesystem(s)? The ones that 
> were mounting it when it was on Server A?
For NFS, it''s possible (but likely suboptimal) for clients to be
configured to mount the filesystem from server A and fail over to
server B, assuming that the pool import can happen quickly enough for
them not to receive ENOENT.

Ceri
-- 
That must be wonderful!  I don''t understand it at all.
                                                  -- Moliere
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061102/ed2f25fb/attachment.bin>

Ceri Davies wrote:
> For NFS, it''s possible (but likely suboptimal) for clients to be
> configured to mount the filesystem from server A and fail over to
> server B, assuming that the pool import can happen quickly enough for
> them not to receive ENOENT.
IIRC NFS client side failover is really only intended for read-only 
mounts.  I can''t remember though if this is enforced or not though.

-- 
Darren J Moffat

On Thu, Darren J Moffat wrote:
> Ceri Davies wrote:
> >For NFS, it''s possible (but likely suboptimal) for clients to
be
> >configured to mount the filesystem from server A and fail over to
> >server B, assuming that the pool import can happen quickly enough for
> >them not to receive ENOENT.
> 
> IIRC NFS client side failover is really only intended for read-only 
> mounts.  I can''t remember though if this is enforced or not
though.
NFS client side failover is for read-only exports.  No way to strictly
enforce since the NFSv2/v3 protocols don''t have support.  The client
attempts to ensure that active files "look" the same when failing
over.

NFSv4 has migration support such that a filesystem can move between
servers but the administrative model is for that: movement and not
server failover.

Spencer

This thread has diverged a bit but I''m still a little paranoid that a 
sysadmin is going to move a pool from one host to an other and all of a 
sudden the new system is serving NFS shares and iSCSI LUNs all of a 
sudden when really they just wanted to copy some data or fix a problem.

In a lot of ways this behavior is very useful and I''m sure someone is 
going, "Ooooh. Poor man''s cluster". (With a Homer Simpson
voice of
course.) However, I''m still worried that there is no way no stop or 
override it when necessary. At the very least the docs should lay out 
this particular behavior in very certain terms.

-- 
Torrey McMahon
Sun Microsystems Inc.

Torrey McMahon wrote:
> This thread has diverged a bit but I''m still a little paranoid
that a
> sysadmin is going to move a pool from one host to an other and all of a 
> sudden the new system is serving NFS shares and iSCSI LUNs all of a 
> sudden when really they just wanted to copy some data or fix a problem.
There has just been a PSARC case approved for disabling user delegation 
when a pool is imported.

It seems like what is needed here is two other pool properties that can
be set on import so that sharenfs and shareiscsi [and in the future 
sharecifs :-)] are not honoured.

-- 
Darren J Moffat

Cyril Plisko wrote:
> On 11/1/06, Adam Leventhal <ahl at eng.sun.com> wrote:
>> > >What properties are you specifically interested in modifying?
>> >
>> > LUN for example. How would I configure LUN via zfs command ?
>>
>> You can''t. Forgive my ignorance about how iSCSI is deployed,
but why
>> would
>> you want/need to change the LUN?
> 
> Well, with iSCSI specifically it is of less importance, since one can 
> easily
> created multiple units identified by other means, than LUN.
> I, however, trying to look forward for FC SCSI target functionality 
> mirroring
> that of the iSCSI (AFAIK it is on the Rick'' roadmap [and I really
do not
> mind helping]). In FC world it is essentially the only way to have multiple
> units on a particular FC port.
The administration of FC devices for the target mode needs some serious 
thinking so that we don''t end up with a real nightmare on our hands.

As you point out the FC world doesn''t separate the port address from
the
target name. Therefore each FC target must support thousands of LUs. We 
also need to support LU masking. I have no plans to support LU mapping 
since that was created to support a certain OS which could only boot off 
of LUN 0.

Also need to support linking iSCSI targets with FC LUs. The emulation 
code doesn''t care about the transport layer so there''s no
reason why a
logical unit can''t be exposed via iSCSI and FC.

Lot''s of things to worry about.
> Can we do something similar to NFS case, where sharenfs can be
> "on", "off", or something else, in which case it is a
list of options ?
> Would this technique be applicable to shareiscsi too ?
> 
That''s how the shareiscsi property works today.

-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Dick Davies wrote:
> On 01/11/06, Rick McNeal <Rick.McNeal at sun.com> wrote:
> 
>> I too must be missing something. I can''t imagine why it would
take 5
>> minutes to online a target. A ZVOL should automatically be brought
>> online since now initialization is required.
> 
> s/now/no/ ?
Correct. That should have been ''no''.
> Thanks for the explanation. The ''5 minute online'' issue I
had was
> with a file-based target (which happened to be on a ZFS filesystem).
>  From what you say, it should be a non-issue
> with a zvol-backed target.
Ah. I missed that for some reason. I thought you had set the backing 
store to be a ZVOL.
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Adam Leventhal wrote:
> On Thu, Nov 02, 2006 at 12:10:06AM -0800, eric kustarz wrote:
>>> 	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
>>> 	be exported as an iSCSI target. The acceptable values for this 
>>> 	property
>>> 	are ''on'', ''off'', and
''direct''. In the future, we may support other
>>> 	target types (for example, ''tape''). The default
is ''off''. This
>>> 	property
>>> 	may be set on filesystems, but has no direct effect; this is to
allow
>>> 	ZVOLs created under the ZFS hierarchy to inherit a default. For
>>> 	example, an administrator may want ZVOLs to be shared by default,
and
>>> 	so set ''shareiscsi=on'' for the pool.
>> hey adam, what''s "direct" mean?
> 
> It''s iSCSI target lingo for vanilla disk emulation.
If it''s easier for folks we could change the accepted value to be 
"disk". "direct" is a term that comes from the T10 world.
Other values
are sequential for tapes, printer, scanner, etc..
> Adam
> 
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

On 11/2/06, Rick McNeal <Rick.McNeal at sun.com>
wrote:
> >
>
> The administration of FC devices for the target mode needs some serious
> thinking so that we don''t end up with a real nightmare on our
hands.
>
> As you point out the FC world doesn''t separate the port address
from the
> target name. Therefore each FC target must support thousands of LUs. We
> also need to support LU masking. I have no plans to support LU mapping
> since that was created to support a certain OS which could only boot off
> of LUN 0.
IMO there are more applications for LU mapping, than simple boot support.
Some systems are unable to address more than very modest (7!) number
of LU. I do not feel like ignoring LU mapping would be wise. Moreover
I believe that LU masking can be seen as a particular case of LU mapping.
>
> Also need to support linking iSCSI targets with FC LUs. The emulation
> code doesn''t care about the transport layer so there''s no
reason why a
> logical unit can''t be exposed via iSCSI and FC.
Absolutely. That is quite logical thing to do.

I think I''ll start another thread on this particular subject - FC
target.
> Lot''s of things to worry about.
>
> > Can we do something similar to NFS case, where sharenfs can be
> > "on", "off", or something else, in which case it
is a list of options ?
> > Would this technique be applicable to shareiscsi too ?
> >
>
> That''s how the shareiscsi property works today.
So, why manipulating LUN is impossible via zfs ???

-- 
Regards,
        Cyril

comment below...

Robert Milkowski wrote:
> Hello Richard,
> 
> Wednesday, November 1, 2006, 11:36:14 PM, you wrote:
> 
> REP> Adam Leventhal wrote:
>>> On Wed, Nov 01, 2006 at 04:00:43PM -0500, Torrey McMahon wrote:
>>>> Lets say server A has the pool with NFS shared, or iSCSI
shared,
>>>> volumes. Server A exports the pool or goes down. Server B
imports the pool.
>>>>
>>>> Which clients would still be active on the filesystem(s)? The
ones that
>>>> were mounting it when it was on Server A?
>>> Clients would need to explicitly change the server they''re
contacting unless
>>> that new server also took over the IP address, hostname, etc.
> 
> REP> Does this imply that using Sun Cluster, which transfers (logical)
IP
> REP> addresses, would easily provide an HA-iSCSI service?  I''ll
admit that
> REP> I still don''t understand the iSCSI naming and security
schemes.
> 
> I haven''t played with iSCSI (yet) but is really hostname
necessary?
> What else? Putting zpool under SC with an IP address in SC group
> should be enough I hope - or maybe not?
I haven''t played with iSCSI lately, but it is unlikely I would have
used a
hostname.  They just aren''t worth setting up on a test net.

For an HA cluster, the hostname/IP address mapping is usually done in a
name service for clients.  Since the IP address fails over, there is no
need to change the hostname/IP address association.  There are, of course,
a number of corner cases that bite occasionally, but if you KISS, life will
be more enjoyable.
  -- richard

Cyril Plisko wrote:
> On 11/2/06, Rick McNeal <Rick.McNeal at sun.com> wrote:
>> >
>>
>> The administration of FC devices for the target mode needs some serious
>> thinking so that we don''t end up with a real nightmare on our
hands.
>>
>> As you point out the FC world doesn''t separate the port
address from the
>> target name. Therefore each FC target must support thousands of LUs. We
>> also need to support LU masking. I have no plans to support LU mapping
>> since that was created to support a certain OS which could only boot
off
>> of LUN 0.
> 
> IMO there are more applications for LU mapping, than simple boot support.
> Some systems are unable to address more than very modest (7!) number
> of LU. I do not feel like ignoring LU mapping would be wise. Moreover
> I believe that LU masking can be seen as a particular case of LU mapping.
I clearly need to get more input on the FC side of things. Regarding 
hosts that can only access 7 LUs I would wonder if they aren''t already 
at capacity. If that''s true would they even care about such a product.
>>
>> Also need to support linking iSCSI targets with FC LUs. The emulation
>> code doesn''t care about the transport layer so
there''s no reason why a
>> logical unit can''t be exposed via iSCSI and FC.
> 
> Absolutely. That is quite logical thing to do.
> 
> I think I''ll start another thread on this particular subject - FC
target.
> 
I too feel a FC target thread would be helpful.
>> Lot''s of things to worry about.
>>
>> > Can we do something similar to NFS case, where sharenfs can be
>> > "on", "off", or something else, in which case
it is a list of options ?
>> > Would this technique be applicable to shareiscsi too ?
>> >
>>
>> That''s how the shareiscsi property works today.
> 
> So, why manipulating LUN is impossible via zfs ???
> 
A ZVOL is a single LU, so there''s nothing to manipulate. Could you give
me an example of what you think should/could be changed?

-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

On 11/2/06, Rick McNeal <Rick.McNeal at sun.com>
wrote:
>
> >> That''s how the shareiscsi property works today.
> >
> > So, why manipulating LUN is impossible via zfs ???
> >
>
> A ZVOL is a single LU, so there''s nothing to manipulate. Could you
give
> me an example of what you think should/could be changed?
I was thinking about manipulating Logical Unit _Number_  (LUN)
for that particular Logical Unit (LU). iscsitadm allows me to set a LUN
for the target (LU) in hand and I thought that if we are going to
squeeze LU parameters into shareiscsi, then why LUN is excluded ?


-- 
Regards,
        Cyril

Cyril Plisko wrote:
> On 11/2/06, Rick McNeal <Rick.McNeal at sun.com> wrote:
>>
> 
>> >> That''s how the shareiscsi property works today.
>> >
>> > So, why manipulating LUN is impossible via zfs ???
>> >
>>
>> A ZVOL is a single LU, so there''s nothing to manipulate. Could
you give
>> me an example of what you think should/could be changed?
> 
> I was thinking about manipulating Logical Unit _Number_  (LUN)
> for that particular Logical Unit (LU). iscsitadm allows me to set a LUN
> for the target (LU) in hand and I thought that if we are going to
> squeeze LU parameters into shareiscsi, then why LUN is excluded ?
This administrative model takes the approach of one LU per Target. Since 
SAM requires LUN 0 there''s no point in allowing someone to change the
value.

If you wish to have multiple LU''s per target then the iscsitadm 
interface is available and you can still use ZVOL''s as the backing
store.
> 
-- 
----
Rick McNeal

A good friend will come and bail you out of jail...but, a true
friend will be sitting next to you saying, "Damn...that was fun!"

Thanks for all the feedback. This PSARC case was approved yesterday and
will be integrated relatively soon.

Adam

On Wed, Nov 01, 2006 at 01:33:33AM -0800, Adam Leventhal
wrote:
> Rick McNeal and I have been working on building support for sharing ZVOLs
> as iSCSI targets directly into ZFS. Below is the proposal I''ll be
> submitting to PSARC. Comments and suggestions are welcome.
> 
> Adam
> 
> ---8<---
> 
> iSCSI/ZFS Integration
> 
> A. Overview
> 
> The goal of this project is to couple ZFS with the iSCSI target in Solaris
> specifically to make it as easy to create and export ZVOLs via iSCSI as it
> is to create and export ZFS filesystems via NFS. We will add two new ZFS
> properties to support this feature.
> 
>     shareiscsi
> 
> 	Like the ''sharenfs'' property,
''shareiscsi'' indicates if a ZVOL should
> 	be exported as an iSCSI target. The acceptable values for this property
> 	are ''on'', ''off'', and
''direct''. In the future, we may support other
> 	target types (for example, ''tape''). The default is
''off''. This property
> 	may be set on filesystems, but has no direct effect; this is to allow
> 	ZVOLs created under the ZFS hierarchy to inherit a default. For
> 	example, an administrator may want ZVOLs to be shared by default, and
> 	so set ''shareiscsi=on'' for the pool.
> 
>     iscsioptions
> 
> 	This property, which is hidden by default, is used by the iSCSI target
> 	daemon to store persistent information such as the IQN. The contents
> 	are not intended for users or external consumers.
> 
> 
> B. Examples
> 
> iSCSI targets are simple to create with the zfs(1M) command:
> 
> # zfs create -V 100M pool/volumes/v1
> # zfs set shareiscsi=on pool/volumes/v1
> # iscsitadm list target
> Target: pool/volumes/v1
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Renaming the ZVOL has the expected result for the iSCSI target:
> 
> # zfs rename pool/volumes/v1 pool/volumes/stuff     
> # iscsitadm list target
> Target: pool/volumes/stuff
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Note that per the iSCSI specification (RFC3720), the iSCSI Name is
unchanged
> after the ZVOL is renamed.
> 
> Exporting a pool containing a shared ZVOL will cause the target to be
removed;
> importing a pool containing a shared ZVOL will cause the target to be
> shared:
> 
> # zpool export pool
> # iscsitadm list target
> # zpool import pool
> # iscsitadm list target
> Target: pool/volumes/stuff
>     iSCSI Name: iqn.1986-03.com.sun:02:4db92521-f5dc-cde4-9cd5-a3f6f567220a
>     Connections: 0
> 
> Note again that all configuration information is stored with the dataset.
As
> with NFS shared filesystems, iSCSI targets imported on a different system
> will be shared appropriately.
> 
> ---8<---
> 
> -- 
> Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On 10/11/06, Adam Leventhal <ahl at eng.sun.com>
wrote:
> Thanks for all the feedback. This PSARC case was approved yesterday and
> will be integrated relatively soon.
>
> Adam
I get the impression this is now integrated - could you tell me when
(i.e. what build of SXCR/Opensolaris)?

Thanks.

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

Dick Davies wrote:
> On 10/11/06, Adam Leventhal <ahl at eng.sun.com> wrote:
>> Thanks for all the feedback. This PSARC case was approved yesterday and
>> will be integrated relatively soon.
>>
>> Adam
> 
> I get the impression this is now integrated - could you tell me when
> (i.e. what build of SXCR/Opensolaris)?
http://opensolaris.org/os/community/on/flag-days/51-55/

Shows that PSARC/2006/622 integrated on Nov 15 into build 54.

If you look at the build schedule:

http://opensolaris.org/os/community/on/onnv_schedule.txt

You will see that build 54 is still open.

-- 
Darren J Moffat

On Mon, 20 Nov 2006, Darren J Moffat wrote:
> Dick Davies wrote:
> > On 10/11/06, Adam Leventhal <ahl at eng.sun.com> wrote:
> >> Thanks for all the feedback. This PSARC case was approved
yesterday and
> >> will be integrated relatively soon.
> >>
> >> Adam
> >
> > I get the impression this is now integrated - could you tell me when
> > (i.e. what build of SXCR/Opensolaris)?
>
> http://opensolaris.org/os/community/on/flag-days/51-55/
>
> Shows that PSARC/2006/622 integrated on Nov 15 into build 54.
             ^^^^^^^^^^^^^^
But that link (http://opensolaris.org/os/community/arc/caselog/2006/622/)
does not resolve!
> If you look at the build schedule:
>
> http://opensolaris.org/os/community/on/onnv_schedule.txt
>
> You will see that build 54 is still open.
>
> --
> Darren J Moffat
>
Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
             OpenSolaris Governing Board (OGB) Member - Feb 2006

On 20/11/06, Darren J Moffat <Darren.Moffat at sun.com>
wrote:
> Dick Davies wrote:
> > On 10/11/06, Adam Leventhal <ahl at eng.sun.com> wrote:
> >> Thanks for all the feedback. This PSARC case was approved
yesterday and
> >> will be integrated relatively soon.
> >>
> >> Adam
> >
> > I get the impression this is now integrated - could you tell me when
> > (i.e. what build of SXCR/Opensolaris)?
>
> http://opensolaris.org/os/community/on/flag-days/51-55/
>
> Shows that PSARC/2006/622 integrated on Nov 15 into build 54.
>
> If you look at the build schedule:
>
> http://opensolaris.org/os/community/on/onnv_schedule.txt
>
> You will see that build 54 is still open.
Ah, thanks. Haven''t seen that page before.
So Opensolaris/Nevada b53 is out today?

What''s the gap between WOS delivery and a SXCR ISO - about
a fortnight?

Thanks again.

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/

On 20/11/06, Dick Davies <rasputnik at gmail.com> wrote:
> Ah, thanks. Haven''t seen that page before.
> So Opensolaris/Nevada b53 is out today?
>
> What''s the gap between WOS delivery and a SXCR ISO - about
> a fortnight?
Don''t worry, found this:

  http://whacked.net/2005/06/21/confused-so-was-i

-- 
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/