Shweta Krishnan
2007-May-14 06:54 UTC
[zfs-discuss] Re: ZFS over a layered driver interface
I ran zpool with truss, and here is the system call trace. (again, zfs_lyr is
the layered driver I am trying to use to talk to the ramdisk driver).
When I compared it to a successful zpool creation, the culprit is the last
failing ioctl
i.e. ioctl(3, ZFS_IOC_CREATE_POOL, <address>)
I tried looking at the source code for the failing ioctl, but didn''t
get any hints there.
Guess I must try dtrace (which I am about to learn!).
bash-3.00# truss -f zpool create adsl-pool /devices/pseudo/zfs_lyr\@1\:zfsminor1
2> /var/tmp/zpool.truss
bash-3.00# grep Err /var/tmp/zpool.truss
2232: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
2232: xstat(2, "/lib/libdiskmgt.so.1", 0x080469C8) Err#2 ENOENT
2232: xstat(2, "/lib/libxml2.so.2", 0x08046868) Err#2 ENOENT
2232: xstat(2, "/lib/libz.so.1", 0x08046868) Err#2 ENOENT
2232: stat64("/devices/pseudo/zfs_lyr at 1:zfsminor1s2", 0x080429E0)
Err#2 ENOENT
2232: modctl(MODSIZEOF_DEVID, 0x03740001, 0x080429BC, 0x08071714, 0x00000000)
Err#22 EINVAL
2232: mkdir("/var/run/sysevent_channels/syseventd_channel", 0755)
Err#17 EEXIST
2232: unlink("/var/run/sysevent_channels/syseventd_channel/17")
Err#2 ENOENT
2232/1:
umount2("/var/run/sysevent_channels/syseventd_channel/17", 0x00000000)
Err#22 EINVAL
2232/1: ioctl(7, I_CANPUT, 0x00000000) Err#89 ENOSYS
2232/1: stat64("/adsl-pool", 0x08043330) Err#2
ENOENT
2232/1: ioctl(3, ZFS_IOC_POOL_CREATE, 0x08041BC4) Err#22 EINVAL
This message posted from opensolaris.org
Shweta Krishnan wrote:> I ran zpool with truss, and here is the system call trace. (again, zfs_lyr is the layered driver I am trying to use to talk to the ramdisk driver). > > When I compared it to a successful zpool creation, the culprit is the last failing ioctl > i.e. ioctl(3, ZFS_IOC_CREATE_POOL, <address>) > > I tried looking at the source code for the failing ioctl, but didn''t get any hints there. > Guess I must try dtrace (which I am about to learn!). > > bash-3.00# truss -f zpool create adsl-pool /devices/pseudo/zfs_lyr\@1\:zfsminor1 2> /var/tmp/zpool.truss > bash-3.00# grep Err /var/tmp/zpool.truss > 2232: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT > 2232: xstat(2, "/lib/libdiskmgt.so.1", 0x080469C8) Err#2 ENOENT > 2232: xstat(2, "/lib/libxml2.so.2", 0x08046868) Err#2 ENOENT > 2232: xstat(2, "/lib/libz.so.1", 0x08046868) Err#2 ENOENT > 2232: stat64("/devices/pseudo/zfs_lyr at 1:zfsminor1s2", 0x080429E0) Err#2 ENOENT > 2232: modctl(MODSIZEOF_DEVID, 0x03740001, 0x080429BC, 0x08071714, 0x00000000) Err#22 EINVALMODSIZEOF_DEVID is 10. $ dtrace -n ''syscall::modctl:entry{trace(arg0); ustack();}'' The relevant stack is the following. 0 71587 modctl:entry 10 libc.so.1`modctl+0x15 zpool`make_disks+0x1bf zpool`make_disks+0x72 zpool`make_root_vdev+0x56 zpool`zpool_do_create+0x1c4 zpool`main+0xa2 zpool`_start+0x7a make_disks() calls devid_get() which calls modctl(MODSIZEOF_DEVID). This fails. http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/zpool/zpool_vdev.c#959 The code however, seems to ignore this. So this might not be the issue.> 2232: mkdir("/var/run/sysevent_channels/syseventd_channel", 0755) Err#17 EEXIST > 2232: unlink("/var/run/sysevent_channels/syseventd_channel/17") Err#2 ENOENT > 2232/1: umount2("/var/run/sysevent_channels/syseventd_channel/17", 0x00000000) Err#22 EINVAL > 2232/1: ioctl(7, I_CANPUT, 0x00000000) Err#89 ENOSYS > 2232/1: stat64("/adsl-pool", 0x08043330) Err#2 ENOENT > 2232/1: ioctl(3, ZFS_IOC_POOL_CREATE, 0x08041BC4) Err#22 EINVALZFS_IOC_POOL_CREATE is failing. I am not sure if the problem has already happened or if it happens during this ioctl. But you could try dtracing this ioctl and see where EINVAL is being set. $ dtrace -n ''fbt:zfs:zfs_ioc_pool_create:entry{self->t=1;} \ fbt:zfs::return/self->t && arg1 == 22/{stack(); exit(0);} \ fbt:zfs:zfs_ioc_pool_create:return{self->t=0;}'' If it does not provide a clue, you could try the following trace with is more heavy weight. Warning: it could generate a lot of output. :) $ dtrace -n ''fbt:zfs:zfs_ioc_pool_create:entry{self->t=1;} \ fbt:zfs::entry/self->t/{} fbt:zfs::return/self->t/{trace(arg1);} \ fbt:zfs:zfs_ioc_pool_create:return{self->t=0;}'' Perhaps there are folks on this list who know what the problem is without all the dtracing that I am suggesting. But this is what I would try. Good luck! :) -Manoj PS: When running the above scripts, run it on one telnet/ssh/xterm window. Run ''zpool create'' on another.
This is likely because ldi_get_size() is failing for your device. We''ve seen this before on 3rd party devices, and have been meaning to create a special errno (instead of EINVAL) to give a more helpful message in this case. - Eric On Sun, May 13, 2007 at 11:54:45PM -0700, Shweta Krishnan wrote:> I ran zpool with truss, and here is the system call trace. (again, zfs_lyr is the layered driver I am trying to use to talk to the ramdisk driver). > > When I compared it to a successful zpool creation, the culprit is the last failing ioctl > i.e. ioctl(3, ZFS_IOC_CREATE_POOL, <address>) > > I tried looking at the source code for the failing ioctl, but didn''t get any hints there. > Guess I must try dtrace (which I am about to learn!). > > bash-3.00# truss -f zpool create adsl-pool /devices/pseudo/zfs_lyr\@1\:zfsminor1 2> /var/tmp/zpool.truss > bash-3.00# grep Err /var/tmp/zpool.truss > 2232: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT > 2232: xstat(2, "/lib/libdiskmgt.so.1", 0x080469C8) Err#2 ENOENT > 2232: xstat(2, "/lib/libxml2.so.2", 0x08046868) Err#2 ENOENT > 2232: xstat(2, "/lib/libz.so.1", 0x08046868) Err#2 ENOENT > 2232: stat64("/devices/pseudo/zfs_lyr at 1:zfsminor1s2", 0x080429E0) Err#2 ENOENT > 2232: modctl(MODSIZEOF_DEVID, 0x03740001, 0x080429BC, 0x08071714, 0x00000000) Err#22 EINVAL > 2232: mkdir("/var/run/sysevent_channels/syseventd_channel", 0755) Err#17 EEXIST > 2232: unlink("/var/run/sysevent_channels/syseventd_channel/17") Err#2 ENOENT > 2232/1: umount2("/var/run/sysevent_channels/syseventd_channel/17", 0x00000000) Err#22 EINVAL > 2232/1: ioctl(7, I_CANPUT, 0x00000000) Err#89 ENOSYS > 2232/1: stat64("/adsl-pool", 0x08043330) Err#2 ENOENT > 2232/1: ioctl(3, ZFS_IOC_POOL_CREATE, 0x08041BC4) Err#22 EINVAL > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On Mon, May 14, 2007 at 11:55:28AM -0500, Swetha Krishnan wrote:> Thanks Eric and Manoj. > > Here''s what ldi_get_size() returns: > bash-3.00# dtrace -n ''fbt::ldi_get_size:return{trace(arg1);}'' -c ''zpool > create adsl-pool /dev/layerzfsminor1'' dtrace: description > ''fbt::ldi_get_size:return'' matched 1 probe > cannot create ''adsl-pool'': invalid argument for this pool operation > dtrace: pid 2582 has exited > CPU ID FUNCTION:NAME > 0 20927 ldi_get_size:return 4294967295 > > > This is strange because I looked at the code for ldi_get_size() and the > only possible return values in the code are DDI_SUCCESS (0) and > DDI_FAILURE(-1). > > Looks like what I''m looking at either isn''t the return value, or some > bad address is being reached. Any hints?That is actually -1 (2^32 - 1). The default type for ''argN'' in DTrace is unsigned. If you do ''trace((int)arg1)'', you''ll see -1. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
Edward Pilatowicz
2007-May-14 21:56 UTC
[zfs-discuss] Re: ZFS over a layered driver interface
i''ve seen this ldi_get_size() failure before and it usually occurs on drivers that don''t implement their prop_op(9E) entry point correctly or that don''t implement the dynamic [Nn]blocks/[Ss]size property correctly. what does your layered driver do in it''s prop_op(9E) entry point? also, what driver is your layered driver layered over? ed On Mon, May 14, 2007 at 09:37:51AM -0700, Eric Schrock wrote:> This is likely because ldi_get_size() is failing for your device. We''ve > seen this before on 3rd party devices, and have been meaning to create a > special errno (instead of EINVAL) to give a more helpful message in this > case. > > - Eric > > On Sun, May 13, 2007 at 11:54:45PM -0700, Shweta Krishnan wrote: > > I ran zpool with truss, and here is the system call trace. (again, zfs_lyr is the layered driver I am trying to use to talk to the ramdisk driver). > > > > When I compared it to a successful zpool creation, the culprit is the last failing ioctl > > i.e. ioctl(3, ZFS_IOC_CREATE_POOL, <address>) > > > > I tried looking at the source code for the failing ioctl, but didn''t get any hints there. > > Guess I must try dtrace (which I am about to learn!). > > > > bash-3.00# truss -f zpool create adsl-pool /devices/pseudo/zfs_lyr\@1\:zfsminor1 2> /var/tmp/zpool.truss > > bash-3.00# grep Err /var/tmp/zpool.truss > > 2232: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT > > 2232: xstat(2, "/lib/libdiskmgt.so.1", 0x080469C8) Err#2 ENOENT > > 2232: xstat(2, "/lib/libxml2.so.2", 0x08046868) Err#2 ENOENT > > 2232: xstat(2, "/lib/libz.so.1", 0x08046868) Err#2 ENOENT > > 2232: stat64("/devices/pseudo/zfs_lyr at 1:zfsminor1s2", 0x080429E0) Err#2 ENOENT > > 2232: modctl(MODSIZEOF_DEVID, 0x03740001, 0x080429BC, 0x08071714, 0x00000000) Err#22 EINVAL > > 2232: mkdir("/var/run/sysevent_channels/syseventd_channel", 0755) Err#17 EEXIST > > 2232: unlink("/var/run/sysevent_channels/syseventd_channel/17") Err#2 ENOENT > > 2232/1: umount2("/var/run/sysevent_channels/syseventd_channel/17", 0x00000000) Err#22 EINVAL > > 2232/1: ioctl(7, I_CANPUT, 0x00000000) Err#89 ENOSYS > > 2232/1: stat64("/adsl-pool", 0x08043330) Err#2 ENOENT > > 2232/1: ioctl(3, ZFS_IOC_POOL_CREATE, 0x08041BC4) Err#22 EINVAL > > > > > > This message posted from opensolaris.org > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > -- > Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss