Hi all, Is it possible to re-create the XEN network bridge? On one of my XEN servers, somce domU''s can''t connect to the outside world, yet I can connect to the VPS'' from the internet on SSH. It seems that outgoing traffic is not working as it should, but incoming is, and I think it might have todo with the bridge. How would I re-create the bridge to see if it sovles the problem? -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-06 09:48 UTC
Re: [Xen-users] howto re-create XEN network bridge?
On Fri, Nov 6, 2009 at 6:58 AM, Rudi Ahlers <Rudi@softdux.com> wrote:> Hi all, > > Is it possible to re-create the XEN network bridge? On one of my XEN > servers, somce domU''s can''t connect to the outside world, yet I can > connect to the VPS'' from the internet on SSH. It seems that outgoing > traffic is not working as it should, but incoming is, and I think it > might have todo with the bridge. How would I re-create the bridge to > see if it sovles the problem?Take a look at the output of "brctl show", and also see the contents of "/etc/xen/scripts/network-bridge". IMHO you should first make sure that it''s not iptables problem. See the content of /proc/sys/net/ipv4/netfilter/ip_conntrack_count /proc/sys/net/ipv4/netfilter/ip_conntrack_max if count is close to max, you could either: - increase ip_conntrack_max, or - disable iptables for bridged traffic (echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables) -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 11:48 AM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Fri, Nov 6, 2009 at 6:58 AM, Rudi Ahlers <Rudi@softdux.com> wrote: >> Hi all, >> >> Is it possible to re-create the XEN network bridge? On one of my XEN >> servers, somce domU''s can''t connect to the outside world, yet I can >> connect to the VPS'' from the internet on SSH. It seems that outgoing >> traffic is not working as it should, but incoming is, and I think it >> might have todo with the bridge. How would I re-create the bridge to >> see if it sovles the problem? > > Take a look at the output of "brctl show", and also see the contents > of "/etc/xen/scripts/network-bridge". > IMHO you should first make sure that it''s not iptables problem. See > the content of > /proc/sys/net/ipv4/netfilter/ip_conntrack_count > /proc/sys/net/ipv4/netfilter/ip_conntrack_max > > if count is close to max, you could either: > - increase ip_conntrack_max, or > - disable iptables for bridged traffic (echo 0 > > /proc/sys/net/bridge/bridge-nf-call-iptables) > > -- > Fajar > > _______________________________________________Thank you Fajar, I tried to look at the ip_contrack files that you suggested, but there are no such files on the server: root@zaxen01:[~]$ locate ip_conntrack_count root@zaxen01:[~]$ locate ip_conntrack_max root@zaxen01:[~]$ root@zaxen01:[~]$ uname -a Linux zaxen01.softdux.com 2.6.18-128.1.16.el5xen #1 SMP Tue Jun 30 06:39:23 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux root@zaxen01:[~]$ cat /etc/redhat-release CentOS release 5.3 (Final) Also, there''s no firewall running ont he dom0 that blocks anything for the domU''s - I don''t quite know how todo this even. Any other suggestions? -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-06 10:38 UTC
Re: [Xen-users] howto re-create XEN network bridge?
On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@softdux.com> wrote:>> IMHO you should first make sure that it''s not iptables problem. See >> the content of >> /proc/sys/net/ipv4/netfilter/ip_conntrack_count >> /proc/sys/net/ipv4/netfilter/ip_conntrack_max> I tried to look at the ip_contrack files that you suggested, but there > are no such files on the server: > > > root@zaxen01:[~]$ locate ip_conntrack_count > root@zaxen01:[~]$ locate ip_conntrack_maxerr ... I DIDN''T tell you to use "locate". I DID give you the exact file path.> Also, there''s no firewall running ont he dom0 that blocks anything for > the domU''s - I don''t quite know how todo this even.The default xen bridge setup adds some iptables rules. Even when you don''t explicitly set them up. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 12:38 PM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@softdux.com> wrote: > >>> IMHO you should first make sure that it''s not iptables problem. See >>> the content of >>> /proc/sys/net/ipv4/netfilter/ip_conntrack_count >>> /proc/sys/net/ipv4/netfilter/ip_conntrack_max > > >> I tried to look at the ip_contrack files that you suggested, but there >> are no such files on the server: >> >> >> root@zaxen01:[~]$ locate ip_conntrack_count >> root@zaxen01:[~]$ locate ip_conntrack_max > > err ... I DIDN''T tell you to use "locate". > I DID give you the exact file path.Well, no need to be rude, but to me more specific, those files don''t exist: root@zaxen01:[~]$ more /proc/sys/net/ipv4/netfilter/ip_conntrack_count /proc/sys/net/ipv4/netfilter/ip_conntrack_count: No such file or directory root@zaxen01:[~]$ locate ip_conntrack_count Nor is there a /proc/sys/net/ipv4/netfilter/ folder..... ll /proc/sys/net/ipv4/netfilter/ ls: /proc/sys/net/ipv4/netfilter/: No such file or directory> >> Also, there''s no firewall running ont he dom0 that blocks anything for >> the domU''s - I don''t quite know how todo this even. > > The default xen bridge setup adds some iptables rules. Even when you > don''t explicitly set them up.AND, as I said in my first post, the firewall is disabled: root@zaxen01:[~]$ iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination> > -- > Fajar > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 12:38 PM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Fri, Nov 6, 2009 at 5:25 PM, Rudi Ahlers <Rudi@softdux.com> wrote: > >>> IMHO you should first make sure that it''s not iptables problem. See >>> the content of >>> /proc/sys/net/ipv4/netfilter/ip_conntrack_count >>> /proc/sys/net/ipv4/netfilter/ip_conntrack_max > > >> I tried to look at the ip_contrack files that you suggested, but there >> are no such files on the server: >> >> >> root@zaxen01:[~]$ locate ip_conntrack_count >> root@zaxen01:[~]$ locate ip_conntrack_max > > err ... I DIDN''T tell you to use "locate". > I DID give you the exact file path. > >> Also, there''s no firewall running ont he dom0 that blocks anything for >> the domU''s - I don''t quite know how todo this even. > > The default xen bridge setup adds some iptables rules. Even when you > don''t explicitly set them up. > > -- > Fajar > > _______________________________________________Here''s the output of brctl show: root@zaxen01:[~]$ brctl show bridge name bridge id STP enabled interfaces xenbr0 8000.feffffffffff no vifpict0 vifandr0 vifzafe0 vifserv0 vifraft0 vifplut0 viferis0 vifbyra0 vifanim0 vifacti0 peth0 vif0.0 All the interfaces that should be there, is there. -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-06 10:59 UTC
Re: [Xen-users] howto re-create XEN network bridge?
On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@softdux.com> wrote:> Here's the output of brctl show: > > > root@zaxen01:[~]$ brctl show > bridge name bridge id STP enabled interfaces > xenbr0 8000.feffffffffff no vifpict0 > vifandr0 > vifzafe0 > vifserv0 > vifraft0 > vifplut0 > viferis0 > vifbyra0 > vifanim0 > vifacti0 > peth0 > vif0.0 >You can try by sniffing packets on uplink physical interface (should be peth0 in your case). Test with simple packets (like ping) and snoop it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That would at least tell you whether packets can get to physical interface or not. If the traffic is there but somehow you still have problems, the source of problem might be elsewhere and not in your bridge setup. Might be switch blocking ports with many MACs, or router caching old MAC addresses. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@fajar.net> wrote:> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@softdux.com> wrote: > >> Here''s the output of brctl show: >> >> >> root@zaxen01:[~]$ brctl show >> bridge name bridge id STP enabled interfaces >> xenbr0 8000.feffffffffff no vifpict0 >> vifandr0 >> vifzafe0 >> vifserv0 >> vifraft0 >> vifplut0 >> viferis0 >> vifbyra0 >> vifanim0 >> vifacti0 >> peth0 >> vif0.0 >> > > You can try by sniffing packets on uplink physical interface (should > be peth0 in your case). Test with simple packets (like ping) and snoop > it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That > would at least tell you whether packets can get to physical interface > or not. > > If the traffic is there but somehow you still have problems, the > source of problem might be elsewhere and not in your bridge setup. > Might be switch blocking ports with many MACs, or router caching old > MAC addresses. > > -- > Fajar > > _______________________________________________Hi Fajar, The packets does indeed get to the physical interface, but I get no return ICMP packets, but it''s almost asif that originating packets are incomplete: I got this from "tcpdump -n -i peth0 icmp" on the dom0 hostnode: 18:06:56.043744 IP 196.34.136.58 > 66.197.167.226: ICMP echo request, id 34310, seq 1, length 64 18:06:57.053851 IP 196.34.136.58 > 66.197.167.226: ICMP echo request, id 34310, seq 2, length 64 Yet, when I ping another VPS on the same dom0, I see an icmp reply: [root@vps ~]# ping -c2 196.34.136.51 PING 196.34.136.51 (196.34.136.51) 56(84) bytes of data. 64 bytes from 196.34.136.51: icmp_seq=1 ttl=64 time=0.903 ms 64 bytes from 196.34.136.51: icmp_seq=2 ttl=64 time=0.115 ms These IP''s is completely open on the network firewall as well. -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 6:19 PM, Rudi Ahlers <Rudi@softdux.com> wrote:> On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@fajar.net> wrote: >> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@softdux.com> wrote: >> >>> Here''s the output of brctl show: >>> >>> >>> root@zaxen01:[~]$ brctl show >>> bridge name bridge id STP enabled interfaces >>> xenbr0 8000.feffffffffff no vifpict0 >>> vifandr0 >>> vifzafe0 >>> vifserv0 >>> vifraft0 >>> vifplut0 >>> viferis0 >>> vifbyra0 >>> vifanim0 >>> vifacti0 >>> peth0 >>> vif0.0 >>> >> >> You can try by sniffing packets on uplink physical interface (should >> be peth0 in your case). Test with simple packets (like ping) and snoop >> it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That >> would at least tell you whether packets can get to physical interface >> or not. >> >> If the traffic is there but somehow you still have problems, the >> source of problem might be elsewhere and not in your bridge setup. >> Might be switch blocking ports with many MACs, or router caching old >> MAC addresses. >> >> -- >> Fajar >> >> _______________________________________________ > > Hi Fajar, > > The packets does indeed get to the physical interface, but I get no > return ICMP packets, but it''s almost asif that originating packets are > incomplete: > > I got this from "tcpdump -n -i peth0 icmp" on the dom0 hostnode: > > 18:06:56.043744 IP 196.34.136.58 > 66.197.167.226: ICMP echo request, > id 34310, seq 1, length 64 > 18:06:57.053851 IP 196.34.136.58 > 66.197.167.226: ICMP echo request, > id 34310, seq 2, length 64 > > > Yet, when I ping another VPS on the same dom0, I see an icmp reply: > > [root@vps ~]# ping -c2 196.34.136.51 > PING 196.34.136.51 (196.34.136.51) 56(84) bytes of data. > 64 bytes from 196.34.136.51: icmp_seq=1 ttl=64 time=0.903 ms > 64 bytes from 196.34.136.51: icmp_seq=2 ttl=64 time=0.115 ms > > > > > These IP''s is completely open on the network firewall as well. > > > > --I see now that none of the VPS''s can establish any outbound connections, yet the server itself can. I have just upgraded the OS, as follows: root@zaxen01:[~]$ uname -a Linux zaxen01.softdux.com 2.6.18-164.6.1.el5xen #1 SMP Tue Nov 3 16:48:13 EST 2009 x86_64 x86_64 x86_64 GNU/Linux root@zaxen01:[~]$ xm dmesg __ __ _____ _ ____ _ __ _ _ __ _ _ ____ \ \/ /___ _ __ |___ / / | |___ \ / |/ /_ | || | / /_ / | ___| | ___| \ // _ \ \047_ \ |_ \ | | __) |__| | \047_ \| || |_| \047_ \ | | / _ \ |___ \ / \ __/ | | | ___) || |_ / __/|__| | (_) |__ _| (_) || || __/ |___) | /_/\_\___|_| |_| |____(_)_(_)_____| |_|\___/ |_|(_)___(_)_(_)___|_|____/ http://www.cl.cam.ac.uk/netos/xen University of Cambridge Computer Laboratory Xen version 3.1.2-164.6.1.el5 (mockbuild@centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) Tue Nov 3 16:04:14 EST 2009 Latest ChangeSet: unavailable -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Fri, Nov 6, 2009 at 8:28 PM, Rudi Ahlers <Rudi@softdux.com> wrote:> On Fri, Nov 6, 2009 at 6:19 PM, Rudi Ahlers <Rudi@softdux.com> wrote: >> On Fri, Nov 6, 2009 at 12:59 PM, Fajar A. Nugraha <fajar@fajar.net> wrote: >>> On Fri, Nov 6, 2009 at 5:50 PM, Rudi Ahlers <Rudi@softdux.com> wrote: >>> >>>> Here''s the output of brctl show: >>>> >>>> >>>> root@zaxen01:[~]$ brctl show >>>> bridge name bridge id STP enabled interfaces >>>> xenbr0 8000.feffffffffff no vifpict0 >>>> vifandr0 >>>> vifzafe0 >>>> vifserv0 >>>> vifraft0 >>>> vifplut0 >>>> viferis0 >>>> vifbyra0 >>>> vifanim0 >>>> vifacti0 >>>> peth0 >>>> vif0.0 >>>> >>> >>> You can try by sniffing packets on uplink physical interface (should >>> be peth0 in your case). Test with simple packets (like ping) and snoop >>> it with tcpdump (something like "tcpdump -n -i peth0 icmp"). That >>> would at least tell you whether packets can get to physical interface >>> or not. >>> >>> If the traffic is there but somehow you still have problems, the >>> source of problem might be elsewhere and not in your bridge setup. >>> Might be switch blocking ports with many MACs, or router caching old >>> MAC addresses. >>> >>> -- >>> Fajar >>> >>> _______________________________________________ >>Are there any limits on the amount of network interfaces one can setup on XEN? I''m really at whit''s end with this problem and don''t know how to fix it. -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-09 02:20 UTC
Re: [Xen-users] howto re-create XEN network bridge?
On Sat, Nov 7, 2009 at 1:28 AM, Rudi Ahlers <Rudi@softdux.com> wrote:> I see now that none of the VPS''s can establish any outbound > connections, yet the server itself can.Are the domUs still accessible from outside world? You mentioned earlier that you can ssh to domU, but domU can''t access anywhere. If that''s the case I suggest you try using bridge + NAT. You should have a bridge called virbr0 by default on RHEL/Centos (created by libvirt), with NAT already setup (check with "iptables -nL -t nat"). It should show something like this # iptables -nL -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination After that, create a domU that uses virbr0 bridge (use DHCP in domU). Then check whether you can access outside world from domU. If you can, then what I said earlier is probably true : the source of problem might be elsewhere and not in your bridge setup. It''s also possible that your ISP/uplink provider is blocking packets from your domU IP address. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users