Nathan Valentine
2006-Nov-08 17:44 UTC
[Xen-users] domU''s dropping network config under heavy disk and network I/O
dom0: FC5 running kernel 2.6.17-1.2187_FC5xen0 domU: FC5 running kernel 2.6.17-1.2187_FC5xenU Both are patched up to the latest and greatest. I''ve been trying to rsync about 100GBs of data to the domU but every hour or so the domU drops its network config. There''s nothing in the logs on either the dom0 or domU to indicate any kind of problem with the network or the storage subsystem. The domU does not drop network config under any other circumstances. Any ideas? -- --- Nathan Valentine - nvalentine@discoverymining.com Systems/Network Admin 415.561.6780 Discovery Mining, Inc. - http://www.discoverymining.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jerry Amundson
2006-Nov-08 19:36 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
On 11/8/06, Nathan Valentine <nvalentine@discoverymining.com> wrote:> > dom0: FC5 running kernel 2.6.17-1.2187_FC5xen0 > domU: FC5 running kernel 2.6.17-1.2187_FC5xenU > > Both are patched up to the latest and greatest. > > I''ve been trying to rsync about 100GBs of data to the domU but every > hour or so the domU drops its network config. There''s nothing in the > logs on either the dom0 or domU to indicate any kind of problem with the > network or the storage subsystem. The domU does not drop network config > under any other circumstances.Sorry, can''t help with the rsync issue. But... why use rsync? The data is local to dom0? Can''t you present it as storage (lvm, etc) to the domU? I''m missing something here?? jerry _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Valentine
2006-Nov-08 20:47 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
> But... why use rsync? The data is local to dom0? Can''t you present it > as storage (lvm, etc) to the domU? I''m missing something here??The data isn''t local to the dom0. -- --- Nathan Valentine - nvalentine@discoverymining.com Systems/Network Admin 415.561.6780 Discovery Mining, Inc. - http://www.discoverymining.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Luke Crawford
2006-Nov-08 21:13 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
what does "dropping network config" mean? what does ifconfig look like before and after? anything relevant in dmesg or syslog? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jerry Amundson
2006-Nov-09 02:40 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
On 11/8/06, Nathan Valentine <nvalentine@discoverymining.com> wrote:> > > > But... why use rsync? The data is local to dom0? Can''t you present it > > as storage (lvm, etc) to the domU? I''m missing something here?? > > The data isn''t local to the dom0.OK, so where is it such that it''s can''t be provided to the domU the same way? jerry _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-Nov-09 07:18 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
Jerry Amundson wrote:> On 11/8/06, Nathan Valentine <nvalentine@discoverymining.com> wrote: >> >> dom0: FC5 running kernel 2.6.17-1.2187_FC5xen0 >> domU: FC5 running kernel 2.6.17-1.2187_FC5xenU >> >> Both are patched up to the latest and greatest. >> >> I''ve been trying to rsync about 100GBs of data to the domU but every >> hour or so the domU drops its network config. There''s nothing in the >> logs on either the dom0 or domU to indicate any kind of problem with the >> network or the storage subsystem. The domU does not drop network config >> under any other circumstances. > > Sorry, can''t help with the rsync issue. > But... why use rsync? The data is local to dom0? Can''t you present it > as storage (lvm, etc) to the domU? I''m missing something here?? > > jerry > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-usersI''m sure that we can think of many workarounds, but it''s an acceptable expectation that a domU would continue to run under normal uses like transferring a lot of data and then storing it to disk. I don''t think that''s out of the ordinary at all. :) I''m curious about what "drop network config" means, as well. -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Valentine
2006-Nov-09 16:10 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
It was as if the domU lost its DHCP lease...which, after more testing, is precisely what seems to have happened. No IP address, no routes, etc etc. I changed the domU from DHCP to static addressing and re-ran the data copies and they completed as expected. It seems that perhaps under high disk or network I/O the DHCP client on the domU stops renewing/honoring leases. This could be a problem with the process not getting scheduled (it hasn''t died) but I suspect that it has more to do with the Xen Ethernet bridge flaking out. Not sure how to debug that though and it is only a hunch. As I said before, neither the logs on the domU nor on the dom0 show anything unusual. At least not that I''ve found so far. On Wed, 2006-11-08 at 13:13 -0800, Luke Crawford wrote:> what does "dropping network config" mean? > > what does ifconfig look like before and after? anything relevant in dmesg > or syslog? > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- --- Nathan Valentine - nvalentine@discoverymining.com Systems/Network Admin 415.561.6780 Discovery Mining, Inc. - http://www.discoverymining.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Christopher G. Stach II
2006-Nov-09 20:50 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
Nathan Valentine wrote:> It was as if the domU lost its DHCP lease...which, after more testing, > is precisely what seems to have happened. No IP address, no routes, etc > etc. I changed the domU from DHCP to static addressing and re-ran the > data copies and they completed as expected. > > It seems that perhaps under high disk or network I/O the DHCP client on > the domU stops renewing/honoring leases. This could be a problem with > the process not getting scheduled (it hasn''t died) but I suspect that it > has more to do with the Xen Ethernet bridge flaking out. > > Not sure how to debug that though and it is only a hunch. As I said > before, neither the logs on the domU nor on the dom0 show anything > unusual. At least not that I''ve found so far.strace dhcpcd, run your rsync, and see what happens. :) -- Christopher G. Stach II _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ulrich Windl
2006-Nov-10 07:37 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
On 9 Nov 2006 at 8:10, Nathan Valentine wrote:> > It was as if the domU lost its DHCP lease...which, after more testing, > is precisely what seems to have happened. No IP address, no routes, etc > etc. I changed the domU from DHCP to static addressing and re-ran the > data copies and they completed as expected.Even with no load DHCP requests (or replies/offers) seem to get lost somewhere. Then significant time later it suddenly works. I was wondering whether a possible overflow in the bridge''s MAC table might be a reason.... [...] Regards, Ulrich _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nathan Valentine
2006-Nov-10 17:41 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
> Even with no load DHCP requests (or replies/offers) seem to get lost somewhere. > Then significant time later it suddenly works. I was wondering whether a possible > overflow in the bridge''s MAC table might be a reason....Sorry, I''m relatively new to the Xen community. Is this a known issue/bug? -- --- Nathan Valentine - nvalentine@discoverymining.com Systems/Network Admin 415.561.6780 Discovery Mining, Inc. - http://www.discoverymining.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ulrich Windl
2006-Nov-13 07:42 UTC
Re: [Xen-users] domU''s dropping network config under heavy disk and network I/O
On 10 Nov 2006 at 9:41, Nathan Valentine wrote:> > > Even with no load DHCP requests (or replies/offers) seem to get lost somewhere. > > Then significant time later it suddenly works. I was wondering whether a possible > > overflow in the bridge''s MAC table might be a reason.... > > Sorry, I''m relatively new to the Xen community. Is this a known > issue/bug?It seems like it: German iX magazine (iX 11/2006, page 86) has a review of XEN in it and they report the same (or very similar) problems. Regards, Ulrich _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users