John McCullough
2010-Apr-14 00:20 UTC
[Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
I have been working with a colleague to get a large number of small VMs running on a single system. We were hoping for at least 100, but we seem to be topping out around 81. Each VM has a disk image and a swap image. It seemed like we were hitting a blktap limit, so we tried bumping up the MAX macros in tool/blktap2 and the linux driver, with no change. (Though we haven''t hit the theoretical 256 blktap devices yet). (Initially we were only able to get 64 VMs until we bumped CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). To isolate the problem, I tried creating a large number of blktap devices in the dom0 with no guests running and I ran into the same ceiling (162 total devices). Commands to reproduce the problem follow: echo 9 > /sys/class/blktap2/verbosity for x in `seq 0 163`; do if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 seek=1M 2> /dev/null); then echo "Qemu fail on $x"; exit 1 fi if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then echo "blktap fail on $x"; exit 1 fi done The result: ... /dev/xen/blktap-2/tapdev159 /dev/xen/blktap-2/tapdev160 /dev/xen/blktap-2/tapdev161 /dev/xen/blktap-2/tapdev162 unrecognized child response blktap fail on 163 Dmesg output associated with 163: [ 1288.839978] blktap_sysfs_create: adding attributes for dev ffff88019e4d1e00 [ 1288.840947] blktap_sysfs_destroy (Output for the prior devices includes processing a request, and a blktap_device_finish_request) No related xm dmesg output. $ hg tip changeset: 21091:f28f1ee587c8 tag: tip user: Keir Fraser <keir.fraser@citrix.com> date: Wed Apr 07 12:38:28 2010 +0100 summary: Added signature for changeset 484179b2be5d $ uname -a Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 x86_64 GNU/Linux Has anyone had contrary experience? Does anyone know where the 162 max is coming from? Thanks, John _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Apr-14 03:37 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
Hi. Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. Where N is the number of devices you desire. As for the apparently missing big fat complaint you should have seen pop in syslog, I''ll keep it in mind for the next update. :} Cheers, Daniel On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote:> I have been working with a colleague to get a large number of small VMs > running on a single system. We were hoping for at least 100, but we > seem to be topping out around 81. Each VM has a disk image and a swap > image. It seemed like we were hitting a blktap limit, so we tried > bumping up the MAX macros in tool/blktap2 and the linux driver, with no > change. (Though we haven''t hit the theoretical 256 blktap devices yet). > > (Initially we were only able to get 64 VMs until we bumped > CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). > > To isolate the problem, I tried creating a large number of blktap > devices in the dom0 with no guests running and I ran into the same > ceiling (162 total devices). Commands to reproduce the problem follow: > > echo 9 > /sys/class/blktap2/verbosity > > for x in `seq 0 163`; do > if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 > seek=1M 2> /dev/null); then > echo "Qemu fail on $x"; exit 1 > fi > if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then > echo "blktap fail on $x"; exit 1 > fi > done > > The result: > ... > /dev/xen/blktap-2/tapdev159 > /dev/xen/blktap-2/tapdev160 > /dev/xen/blktap-2/tapdev161 > /dev/xen/blktap-2/tapdev162 > unrecognized child response > blktap fail on 163 > > Dmesg output associated with 163: > [ 1288.839978] blktap_sysfs_create: adding attributes for dev > ffff88019e4d1e00 > [ 1288.840947] blktap_sysfs_destroy > > (Output for the prior devices includes processing a request, and a > blktap_device_finish_request) > > No related xm dmesg output. > > $ hg tip > changeset: 21091:f28f1ee587c8 > tag: tip > user: Keir Fraser <keir.fraser@citrix.com> > date: Wed Apr 07 12:38:28 2010 +0100 > summary: Added signature for changeset 484179b2be5d > > $ uname -a > Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 > x86_64 GNU/Linux > > Has anyone had contrary experience? Does anyone know where the 162 max > is coming from? > > Thanks, > John > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John McCullough
2010-Apr-14 05:40 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
Daniel, That did the trick and got us up to 256, Thanks! Out of curiosity, what''s standing in the way of more devices? We tried raising the MAX_*_DEVICES constants in these files to 512, but didn''t have any luck: linux-2.6-pvops.git/drivers/xen/blktap/blktap.h tools/blktap2/include/blktaplib.h tools/blktap/lib/blktaplib.h (The error is now "vbd open failed: -6") I noticed an artificial limit of 26*26 in the tapdev naming scheme, but I didn''t look very thoroughly. Thanks again, John Daniel Stodden wrote:> Hi. > > Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) > to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. > > Where N is the number of devices you desire. > > As for the apparently missing big fat complaint you should have seen pop > in syslog, I''ll keep it in mind for the next update. :} > > Cheers, > Daniel > > On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote: > >> I have been working with a colleague to get a large number of small VMs >> running on a single system. We were hoping for at least 100, but we >> seem to be topping out around 81. Each VM has a disk image and a swap >> image. It seemed like we were hitting a blktap limit, so we tried >> bumping up the MAX macros in tool/blktap2 and the linux driver, with no >> change. (Though we haven''t hit the theoretical 256 blktap devices yet). >> >> (Initially we were only able to get 64 VMs until we bumped >> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). >> >> To isolate the problem, I tried creating a large number of blktap >> devices in the dom0 with no guests running and I ran into the same >> ceiling (162 total devices). Commands to reproduce the problem follow: >> >> echo 9 > /sys/class/blktap2/verbosity >> >> for x in `seq 0 163`; do >> if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 >> seek=1M 2> /dev/null); then >> echo "Qemu fail on $x"; exit 1 >> fi >> if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then >> echo "blktap fail on $x"; exit 1 >> fi >> done >> >> The result: >> ... >> /dev/xen/blktap-2/tapdev159 >> /dev/xen/blktap-2/tapdev160 >> /dev/xen/blktap-2/tapdev161 >> /dev/xen/blktap-2/tapdev162 >> unrecognized child response >> blktap fail on 163 >> >> Dmesg output associated with 163: >> [ 1288.839978] blktap_sysfs_create: adding attributes for dev >> ffff88019e4d1e00 >> [ 1288.840947] blktap_sysfs_destroy >> >> (Output for the prior devices includes processing a request, and a >> blktap_device_finish_request) >> >> No related xm dmesg output. >> >> $ hg tip >> changeset: 21091:f28f1ee587c8 >> tag: tip >> user: Keir Fraser <keir.fraser@citrix.com> >> date: Wed Apr 07 12:38:28 2010 +0100 >> summary: Added signature for changeset 484179b2be5d >> >> $ uname -a >> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 >> x86_64 GNU/Linux >> >> Has anyone had contrary experience? Does anyone know where the 162 max >> is coming from? >> >> Thanks, >> John >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Apr-14 08:24 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote:> Daniel, > > That did the trick and got us up to 256, Thanks! > > Out of curiosity, what''s standing in the way of more devices?I must admit I never tried. Lack of maybe a couple sparse tables here and there?> We tried raising the MAX_*_DEVICES constants in these files to 512, but > didn''t have any luck: > linux-2.6-pvops.git/drivers/xen/blktap/blktap.h > tools/blktap2/include/blktaplib.h > tools/blktap/lib/blktaplib.h > > (The error is now "vbd open failed: -6")That would be an ENXIO, probably while trying to open the ring (can you verify that with an strace -f?) Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs. I don''t think this fails in the ring code, we return ENODEV there (which is a bug) and the kernel won''t correct that code. Daniel> I noticed an artificial limit of 26*26 in the tapdev naming scheme, but > I didn''t look very thoroughly. > > Thanks again, > John > > > Daniel Stodden wrote: > > Hi. > > > > Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) > > to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. > > > > Where N is the number of devices you desire. > > > > As for the apparently missing big fat complaint you should have seen pop > > in syslog, I''ll keep it in mind for the next update. :} > > > > Cheers, > > Daniel > > > > On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote: > > > >> I have been working with a colleague to get a large number of small VMs > >> running on a single system. We were hoping for at least 100, but we > >> seem to be topping out around 81. Each VM has a disk image and a swap > >> image. It seemed like we were hitting a blktap limit, so we tried > >> bumping up the MAX macros in tool/blktap2 and the linux driver, with no > >> change. (Though we haven''t hit the theoretical 256 blktap devices yet). > >> > >> (Initially we were only able to get 64 VMs until we bumped > >> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). > >> > >> To isolate the problem, I tried creating a large number of blktap > >> devices in the dom0 with no guests running and I ran into the same > >> ceiling (162 total devices). Commands to reproduce the problem follow: > >> > >> echo 9 > /sys/class/blktap2/verbosity > >> > >> for x in `seq 0 163`; do > >> if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 > >> seek=1M 2> /dev/null); then > >> echo "Qemu fail on $x"; exit 1 > >> fi > >> if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then > >> echo "blktap fail on $x"; exit 1 > >> fi > >> done > >> > >> The result: > >> ... > >> /dev/xen/blktap-2/tapdev159 > >> /dev/xen/blktap-2/tapdev160 > >> /dev/xen/blktap-2/tapdev161 > >> /dev/xen/blktap-2/tapdev162 > >> unrecognized child response > >> blktap fail on 163 > >> > >> Dmesg output associated with 163: > >> [ 1288.839978] blktap_sysfs_create: adding attributes for dev > >> ffff88019e4d1e00 > >> [ 1288.840947] blktap_sysfs_destroy > >> > >> (Output for the prior devices includes processing a request, and a > >> blktap_device_finish_request) > >> > >> No related xm dmesg output. > >> > >> $ hg tip > >> changeset: 21091:f28f1ee587c8 > >> tag: tip > >> user: Keir Fraser <keir.fraser@citrix.com> > >> date: Wed Apr 07 12:38:28 2010 +0100 > >> summary: Added signature for changeset 484179b2be5d > >> > >> $ uname -a > >> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 > >> x86_64 GNU/Linux > >> > >> Has anyone had contrary experience? Does anyone know where the 162 max > >> is coming from? > >> > >> Thanks, > >> John > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > >> > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John McCullough
2010-Apr-14 16:30 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
On 04/14/2010 01:24 AM, Daniel Stodden wrote:> On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote: > >> Daniel, >> >> That did the trick and got us up to 256, Thanks! >> >> Out of curiosity, what''s standing in the way of more devices? >> > I must admit I never tried. Lack of maybe a couple sparse tables here > and there? > > >> We tried raising the MAX_*_DEVICES constants in these files to 512, but >> didn''t have any luck: >> linux-2.6-pvops.git/drivers/xen/blktap/blktap.h >> tools/blktap2/include/blktaplib.h >> tools/blktap/lib/blktaplib.h >> >> (The error is now "vbd open failed: -6") >> > That would be an ENXIO, probably while trying to open the ring (can you > verify that with an strace -f?) >Looks like it is ENXIO: ... [pid 4154] open("/dev/xen/blktap-2/blktap256", O_RDWR <unfinished ...> [pid 4153] close(4) = 0 [pid 4153] fcntl(3, F_GETFL) = 0 (flags O_RDONLY) [pid 4153] brk(0) = 0xa1e000 [pid 4153] brk(0xa3f000) = 0xa3f000 [pid 4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 [pid 4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000 [pid 4153] lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) [pid 4153] read(3, <unfinished ...> [pid 4154] <... open resumed> ) = -1 ENXIO (No such device or address) [pid 4154] gettimeofday({1271262219, 398530}, NULL) = 0 [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, MSG_NOSIGNAL, NULL, 0) = 115 [pid 4154] close(3) = 0 [pid 4154] gettimeofday({1271262219, 398836}, NULL) = 0 [pid 4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, MSG_NOSIGNAL, NULL, 0) = 109 [pid 4154] gettimeofday({1271262219, 399161}, NULL) = 0 [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 [pid 4154] write(1, "-6: vbd open failed: -6\n", 24) = 24 ... (err, ioctl to control, close and exit) Thanks, John> Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs. > > I don''t think this fails in the ring code, we return ENODEV there (which > is a bug) and the kernel won''t correct that code. > > Daniel > > >> I noticed an artificial limit of 26*26 in the tapdev naming scheme, but >> I didn''t look very thoroughly. >> >> Thanks again, >> John >> >> >> Daniel Stodden wrote: >> >>> Hi. >>> >>> Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) >>> to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. >>> >>> Where N is the number of devices you desire. >>> >>> As for the apparently missing big fat complaint you should have seen pop >>> in syslog, I''ll keep it in mind for the next update. :} >>> >>> Cheers, >>> Daniel >>> >>> On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote: >>> >>> >>>> I have been working with a colleague to get a large number of small VMs >>>> running on a single system. We were hoping for at least 100, but we >>>> seem to be topping out around 81. Each VM has a disk image and a swap >>>> image. It seemed like we were hitting a blktap limit, so we tried >>>> bumping up the MAX macros in tool/blktap2 and the linux driver, with no >>>> change. (Though we haven''t hit the theoretical 256 blktap devices yet). >>>> >>>> (Initially we were only able to get 64 VMs until we bumped >>>> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). >>>> >>>> To isolate the problem, I tried creating a large number of blktap >>>> devices in the dom0 with no guests running and I ran into the same >>>> ceiling (162 total devices). Commands to reproduce the problem follow: >>>> >>>> echo 9> /sys/class/blktap2/verbosity >>>> >>>> for x in `seq 0 163`; do >>>> if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 >>>> seek=1M 2> /dev/null); then >>>> echo "Qemu fail on $x"; exit 1 >>>> fi >>>> if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then >>>> echo "blktap fail on $x"; exit 1 >>>> fi >>>> done >>>> >>>> The result: >>>> ... >>>> /dev/xen/blktap-2/tapdev159 >>>> /dev/xen/blktap-2/tapdev160 >>>> /dev/xen/blktap-2/tapdev161 >>>> /dev/xen/blktap-2/tapdev162 >>>> unrecognized child response >>>> blktap fail on 163 >>>> >>>> Dmesg output associated with 163: >>>> [ 1288.839978] blktap_sysfs_create: adding attributes for dev >>>> ffff88019e4d1e00 >>>> [ 1288.840947] blktap_sysfs_destroy >>>> >>>> (Output for the prior devices includes processing a request, and a >>>> blktap_device_finish_request) >>>> >>>> No related xm dmesg output. >>>> >>>> $ hg tip >>>> changeset: 21091:f28f1ee587c8 >>>> tag: tip >>>> user: Keir Fraser<keir.fraser@citrix.com> >>>> date: Wed Apr 07 12:38:28 2010 +0100 >>>> summary: Added signature for changeset 484179b2be5d >>>> >>>> $ uname -a >>>> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 >>>> x86_64 GNU/Linux >>>> >>>> Has anyone had contrary experience? Does anyone know where the 162 max >>>> is coming from? >>>> >>>> Thanks, >>>> John >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >>>> >>>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Apr-14 20:12 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
On Wed, 2010-04-14 at 12:30 -0400, John McCullough wrote:> On 04/14/2010 01:24 AM, Daniel Stodden wrote: > > On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote: > > > >> Daniel, > >> > >> That did the trick and got us up to 256, Thanks! > >> > >> Out of curiosity, what''s standing in the way of more devices? > >> > > I must admit I never tried. Lack of maybe a couple sparse tables here > > and there? > > > > > >> We tried raising the MAX_*_DEVICES constants in these files to 512, but > >> didn''t have any luck: > >> linux-2.6-pvops.git/drivers/xen/blktap/blktap.h > >> tools/blktap2/include/blktaplib.h > >> tools/blktap/lib/blktaplib.h > >> > >> (The error is now "vbd open failed: -6") > >> > > That would be an ENXIO, probably while trying to open the ring (can you > > verify that with an strace -f?) > > > > Looks like it is ENXIO: > > ... > [pid 4154] open("/dev/xen/blktap-2/blktap256", O_RDWR <unfinished ...> > > [pid 4153] close(4) = 0 > [pid 4153] fcntl(3, F_GETFL) = 0 (flags O_RDONLY) > [pid 4153] brk(0) = 0xa1e000 > [pid 4153] brk(0xa3f000) = 0xa3f000 > [pid 4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 > [pid 4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000 > [pid 4153] lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) > [pid 4153] read(3, <unfinished ...> > > [pid 4154] <... open resumed> ) = -1 ENXIO (No such device or > address) > [pid 4154] gettimeofday({1271262219, 398530}, NULL) = 0 > [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, > MSG_NOSIGNAL, NULL, 0) = 115 > [pid 4154] close(3) = 0 > [pid 4154] gettimeofday({1271262219, 398836}, NULL) = 0 > [pid 4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, > MSG_NOSIGNAL, NULL, 0) = 109 > [pid 4154] gettimeofday({1271262219, 399161}, NULL) = 0 > [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, > MSG_NOSIGNAL, NULL, 0) = 86 > [pid 4154] write(1, "-6: vbd open failed: -6\n", 24) = 24 > ... (err, ioctl to control, close and exit)Yep. So that''s the ring device, which was apparently created ok but remains somehow inaccessible. Like I said, I think this fails earlier in the call stack than juyst blktap_ring_file_operations. Just from looking at some code I don''t immediately see the issue. I guess this presently returns only halfway through chrdev_open. Broken device registration somewhere? Does it take more magic to register minors beyond 8 bit? Thanks for the trace. Can''t spent much time on it right now. Want to take it on? Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2010-Apr-14 20:27 UTC
Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13
On Wed, 2010-04-14 at 16:12 -0400, Daniel Stodden wrote:> On Wed, 2010-04-14 at 12:30 -0400, John McCullough wrote: > > On 04/14/2010 01:24 AM, Daniel Stodden wrote: > > > On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote: > > > > > >> Daniel, > > >> > > >> That did the trick and got us up to 256, Thanks! > > >> > > >> Out of curiosity, what''s standing in the way of more devices? > > >> > > > I must admit I never tried. Lack of maybe a couple sparse tables here > > > and there? > > > > > > > > >> We tried raising the MAX_*_DEVICES constants in these files to 512, but > > >> didn''t have any luck: > > >> linux-2.6-pvops.git/drivers/xen/blktap/blktap.h > > >> tools/blktap2/include/blktaplib.h > > >> tools/blktap/lib/blktaplib.h > > >> > > >> (The error is now "vbd open failed: -6") > > >> > > > That would be an ENXIO, probably while trying to open the ring (can you > > > verify that with an strace -f?) > > > > > > > Looks like it is ENXIO: > > > > ... > > [pid 4154] open("/dev/xen/blktap-2/blktap256", O_RDWR <unfinished ...> > > > > [pid 4153] close(4) = 0 > > [pid 4153] fcntl(3, F_GETFL) = 0 (flags O_RDONLY) > > [pid 4153] brk(0) = 0xa1e000 > > [pid 4153] brk(0xa3f000) = 0xa3f000 > > [pid 4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 > > [pid 4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000 > > [pid 4153] lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) > > [pid 4153] read(3, <unfinished ...> > > > > [pid 4154] <... open resumed> ) = -1 ENXIO (No such device or > > address) > > [pid 4154] gettimeofday({1271262219, 398530}, NULL) = 0 > > [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, > > MSG_NOSIGNAL, NULL, 0) = 115 > > [pid 4154] close(3) = 0 > > [pid 4154] gettimeofday({1271262219, 398836}, NULL) = 0 > > [pid 4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, > > MSG_NOSIGNAL, NULL, 0) = 109 > > [pid 4154] gettimeofday({1271262219, 399161}, NULL) = 0 > > [pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, > > MSG_NOSIGNAL, NULL, 0) = 86 > > [pid 4154] write(1, "-6: vbd open failed: -6\n", 24) = 24 > > ... (err, ioctl to control, close and exit) > > Yep. So that''s the ring device, which was apparently created ok but > remains somehow inaccessible. > > Like I said, I think this fails earlier in the call stack than juyst > blktap_ring_file_operations. Just from looking at some code I don''t > immediately see the issue. > > I guess this presently returns only halfway through chrdev_open. > Broken device registration somewhere? Does it take more magic to > register minors beyond 8 bit?Oh. --snip * This function registers a range of 256 minor numbers. The first minor number * is 0. */ int register_chrdev(unsigned int major, const char *name, const struct file_operations *fops) --snip-- Judging from the rest of the kernel, we''ll need to reimplement the registration somewhat different that that. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel