Hello , All , when I start two vms used VHD as virtual disk concurrently , the two return errors sometimes , they return different errors : one is : Error: Device 51952 (tap2) could not be connected. Hotplug scripts not working. the other is : Error: Device 0 (vif) could not be connected. Hotplug scripts not working. if I start one after another , it works well . how does it happen ? xen : 4.1.2 dom0 : centos5.6 domU : centos5.6 Thanks in advance ! Bryant _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2011-Dec-22 04:51 UTC
Re: why errors happen when two vms start concurrently ?
On Wed, Dec 21, 2011 at 7:03 PM, Bryant Kobe <shijie1234321@gmail.com>wrote:> Hello , All , > > when I start two vms used VHD as virtual disk concurrently , the two > return errors sometimes , > they return different errors : > > one is : > Error: Device 51952 (tap2) could not be connected. Hotplug scripts not > working. > > the other is : > Error: Device 0 (vif) could not be connected. Hotplug scripts not working. > > if I start one after another , it works well . > how does it happen ? > > xen : 4.1.2 > dom0 : centos5.6 > domU : centos5.6 > > Thanks in advance ! > Bryant >Yeah, I battled that forever and finally gave up. It gets better if you''re using LVM for disks (and you''re not limited to somewhere around 100 disks). I finally put a 30 second wait time in my scripts that started my VMs because I often started 50 at one time. This means it would take 25 minutes to start them but they''d start reliably. I also added a wait in there so every 4 that started it would wait until they were booted up before going on. I''ve since moved to XCP and I can start all 50 at the same time without issues and it doesn''t seem to matter what kind of disks I have. I can now start 50 and have the prompt back in a matter of minutes. Sorry I don''t have a better answer for you. Sometimes after I''d get that error I wouldn''t be able to start ANYTHING again until a reboot. That was on Xen 3.4. Grant McWilliams http://grantmcwilliams.com/ Some people, when confronted with a problem, think "I know, I''ll use Windows." Now they have two problems. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
hello, Grant , other vms can also be started after these two errors occurred in my server , even the vm just error happened can also be started if you reexecute " xm start *** ". I have printed many logs in xend source code , but find no implies. I''ve no idea what the next direction I should go to solve the problem. I don''t want to change my system . Bryant 2011/12/22 Grant McWilliams <grantmasterflash@gmail.com>> > On Wed, Dec 21, 2011 at 7:03 PM, Bryant Kobe <shijie1234321@gmail.com>wrote: > >> Hello , All , >> >> when I start two vms used VHD as virtual disk concurrently , the two >> return errors sometimes , >> they return different errors : >> >> one is : >> Error: Device 51952 (tap2) could not be connected. Hotplug scripts not >> working. >> >> the other is : >> Error: Device 0 (vif) could not be connected. Hotplug scripts not working. >> >> if I start one after another , it works well . >> how does it happen ? >> >> xen : 4.1.2 >> dom0 : centos5.6 >> domU : centos5.6 >> >> Thanks in advance ! >> Bryant >> > > > Yeah, I battled that forever and finally gave up. It gets better if you''re > using LVM for disks (and you''re not limited to somewhere around 100 disks). > I finally put a 30 second wait time in my scripts that started my VMs > because I often started 50 at one time. This means it would take 25 minutes > to start them but they''d start reliably. I also added a wait in there so > every 4 that started it would wait until they were booted up before going > on. > > I''ve since moved to XCP and I can start all 50 at the same time without > issues and it doesn''t seem to matter what kind of disks I have. I can now > start 50 and have the prompt back in a matter of minutes. > > Sorry I don''t have a better answer for you. Sometimes after I''d get that > error I wouldn''t be able to start ANYTHING again until a reboot. That was > on Xen 3.4. > > > Grant McWilliams > http://grantmcwilliams.com/ > > Some people, when confronted with a problem, think "I know, I''ll use > Windows." > Now they have two problems. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Grant McWilliams
2011-Dec-22 07:05 UTC
Re: why errors happen when two vms start concurrently ?
On Wed, Dec 21, 2011 at 9:55 PM, Bryant Kobe <shijie1234321@gmail.com>wrote:> hello, Grant , > other vms can also be started after these two errors occurred in > my server , even the vm just error happened can also be started > if you reexecute " xm start *** ". > I have printed many logs in xend source code , but find no implies. > I''ve no idea what the next direction I should go to solve the problem. > I don''t want to change my system . > > Bryant > >It was always my theory that there should be some locking somewhere but it isn''t there. For instance if you start two VMs and both access the same resource at the same time one tramples on the other. This was just a gut feeling though. It just seemed that xm needed to lock the resources when creating the disk/net interfaces then releasing it so the next process could have it. Again I worked around it after fighting it for months. I found that for the first 15 VMs they''d all start fine but after that the system would start to really slow down (since a lot of VMs are hitting their disks at the same time) and the errors would start. If I started them slowly one at a time they worked fine. It seems that this problem happens right at the beginning of the xm create process. Moving to LVM for disks did improve it for me. Again I''d guess that it''s just because the process needed to use an LV is a lot shorter than to create a tapdisk to access a file. Grant McWilliams _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
George Shuklin
2011-Dec-22 17:45 UTC
Re: why errors happen when two vms start concurrently ?
On 22.12.2011 08:51, Grant McWilliams wrote:> I''ve since moved to XCP and I can start all 50 at the same time > without issues and it doesn''t seem to matter what kind of disks I > have. I can now start 50 and have the prompt back in a matter of minutes. > > Sorry I don''t have a better answer for you. Sometimes after I''d get > that error I wouldn''t be able to start ANYTHING again until a reboot. > That was on Xen 3.4. > >Sorry, but you able to start 50 VM on same host same time? Or you are talking about 50 VM on 50 hosts in pool in same time? Or you just send 50 vm-start and all vm starts? It''s really different cases, and here my data for XCP parallelism: Host can perform one vm-start/reboot task at time. All others are waiting in queue and will not process until other tasks are done. pool master can send operations to different hosts (but it do this not for ''speed'', but for memory balance, so some vm will wait in queue of one (busy) host and others will start on ''not busy hosts'' without significant delay). pool master do have limit on amount of tasks in queue, but I can''t say exact number.