thr3ads.net - Xen users - finding the source VM of local ip [Apr 2013]

If this information is useful, please help other people find it:
Share via:

Mofta7y

2013-Apr-26 03:36 UTC

finding the source VM of local ip

Hi e very one

I am having an interesting issue.

it seems that one or more VMs users in an Xen server has configured a
local ip range for communication between VMs

We never gave our customers any local IPs and we give them public IPs only.

The issue is that those local IPs are flooding each other !! by TCP_SYN.
This in turn cause dom0 cpu usage to go high (si goes up to 36%) and as
a result we get a lot of lost packets and very high ping time

now my main issue is to find out which VMs are using these local IPs

I tried arping those ips and got their MAC address but this mac address
is not the mac address of any network interface in the server !!!

we are using bridged domu networking

anyone knows of any way to find which VMs are using these local IPs ?
Also if there is no way to find who is using them can we just prevent
them from communicating with each other through dom0 ?

why dom0 will almost stop handle traffic of domus when having a flood of
only 10K tcp connections (I know the number is not normal but i guess
dom0 should handle it especially dom0 has 6 dedicated cores)

Thanks

Mike

2013-Apr-26 04:17 UTC

head link

Re: finding the source VM of local ip

Mofta7y wrote:> it seems that one or more VMs users in an Xen server has configured a 
> local ip range for communication between VMs
> 
> now my main issue is to find out which VMs are using these local IPs
> 
> I tried arping those ips and got their MAC address but this mac 
> address is not the mac address of any network interface in the server 
Not sure what you mean.  The MACs will probably be assigned to the 
domU-internal interfaces, not to any dom0 interface.
> we are using bridged domu networking
> 
> anyone knows of any way to find which VMs are using these local IPs ? 
Try "brctl showmacs" on the domUs''s bridge.  That should get
you the
port(s) of the MACs that you''re seeing.  (I''d expect all those
MACs to
be non-local.)
> Also if there is no way to find who is using them can we just prevent 
> them from communicating with each other through dom0 ?
Iptables rules can handle that.  I find vif-bridge''s handle_iptable 
rules to be too generous, so I use a modified script and code my own 
rules.

Mike

2013-Apr-27 00:09 UTC

head link

Re: finding the source VM of local ip

Top-quoting is confusing.

Mofta7y wrote:> I managed to see the MAC addresses of these local IPs using the brctl
> showmacs command.
> But these Macs are assigned to port 1.
> 
> looking at dmesg output it seems port 1 is the peth0 interface.
> 
> So I still couldnt find which VM/s is/are using those local IPs
Command output would be a lot more useful.  Are the MACs non-local, like 
I supposed they''d be?  That sounds like those MACs are elsewhere on the
network.  I can''t imagine why your machine would be processing them, 
then, unless you (mis)configured a bridge loop.

I''d need to see some output to make sense of this.  Start with the 
TCP_SYN communication, show some representative tcpdump output.  "brctl 
show" and the various "brctl showmacs", "ip link show",
"ip address
show", "ip route show".  /etc/network/interfaces.  Sounds like
you''re
using the deprecated network-bridge script.  You haven''t mentioned what
OS distribution or version of Xen you''re using.
> Mike wrote:
>> Mofta7y wrote:
>>> it seems that one or more VMs users in an Xen server has configured
>>> a local ip range for communication between VMs
>>>
>>> now my main issue is to find out which VMs are using these local
IPs
>>>
>>> I tried arping those ips and got their MAC address but this mac 
>>> address is not the mac address of any network interface in the
server
>> 
>> Not sure what you mean.  The MACs will probably be assigned to the 
>> domU-internal interfaces, not to any dom0 interface.
>> 
>>> we are using bridged domu networking
>>>
>>> anyone knows of any way to find which VMs are using these local IPs
?
>> 
>> Try "brctl showmacs" on the domUs''s bridge.  That
should get you the
>> port(s) of the MACs that you''re seeing.  (I''d expect
all those MACs to
>> be non-local.)
>> 
>>> Also if there is no way to find who is using them can we just 
>>> prevent them from communicating with each other through dom0 ?
>> 
>> Iptables rules can handle that.  I find vif-bridge''s
handle_iptable
>> rules to be too generous, so I use a modified script and code my own 
>> rules.

Mike

2013-May-05 20:43 UTC

head link

Re: finding the source VM of local ip

On Apr 29, moftah moftah wrote:> here are the outputs
> 
> This email will be huge
> 
> sorry for that but it is the only way to send all data
pastebin?
> 1- showmacs output of brctl
> brctl showmacs eth0
> port no mac addr                is local?       ageing timer
>   1     00:14:f2:87:20:de       no                 1.30
All the XEN OUI MACs below, probably assigned to VM veth interfaces.  I 
assume you''re using veths in your VMs.
>  69     00:16:3e:04:86:94       no                 3.02
>  44     00:16:3e:05:23:45       no                 1.67
>  18     00:16:3e:07:83:af       no                 0.19
>  36     00:16:3e:0c:8e:c0       no                 0.16
>  46     00:16:3e:0d:2d:1b       no                 0.02
>  53     00:16:3e:0f:3b:1e       no                 0.18
>  11     00:16:3e:13:b6:29       no                 3.57
>   4     00:16:3e:15:74:ac       no               130.60
>  22     00:16:3e:15:fe:1e       no                 0.25
>  27     00:16:3e:1d:1a:14       no                 0.69
>  16     00:16:3e:1e:e7:fe       no                 0.00
>  61     00:16:3e:1f:62:59       no                11.56
>  17     00:16:3e:21:7b:98       no                 0.18
>  35     00:16:3e:24:fd:39       no               191.02
>  12     00:16:3e:26:21:af       no                77.98
>  75     00:16:3e:29:c6:6c       no               288.85
>  58     00:16:3e:2b:ad:2e       no                 1.42
>  54     00:16:3e:30:aa:14       no                 3.78
>  24     00:16:3e:34:89:ba       no               181.59
>  51     00:16:3e:3b:5a:4f       no                45.65
>  33     00:16:3e:3c:66:8c       no                12.68
>  60     00:16:3e:3f:aa:50       no               151.09
>   7     00:16:3e:45:0a:cf       no                 0.60
>  20     00:16:3e:45:ea:73       no                 0.15
>   6     00:16:3e:46:95:95       no                23.50
>  21     00:16:3e:47:5e:ed       no                 1.05
>  29     00:16:3e:4c:c0:b8       no                 0.98
>  57     00:16:3e:4f:71:d9       no                43.07
>  62     00:16:3e:54:9f:17       no                 0.02
>  39     00:16:3e:56:60:f1       no               213.88
>  40     00:16:3e:58:b3:b0       no                17.20
>  37     00:16:3e:59:91:30       no                 0.38
>  14     00:16:3e:63:b2:95       no                45.98
>  41     00:16:3e:64:4a:95       no                14.60
>  48     00:16:3e:66:40:22       no               152.58
>  23     00:16:3e:6b:f2:9b       no                 0.05
>  28     00:16:3e:72:12:76       no                 1.75
>   5     00:16:3e:72:44:2e       no                71.37
>  64     00:16:3e:72:98:d5       no                 0.18
>  45     00:16:3e:75:37:cd       no               161.67
>  55     00:16:3e:75:fc:8a       no                43.47
>   3     00:16:3e:76:b3:1d       no                33.75
>  13     00:16:3e:78:f6:53       no               165.33
>   8     00:16:3e:7b:d0:05       no                16.54
>  38     00:16:3e:82:2c:d3       no                 0.02
>  50     00:16:3e:84:5e:7f       no                34.90
>  63     00:16:3e:8c:e4:94       no                 0.06
>  59     00:16:3e:8e:a4:14       no                42.15
> 106     00:16:3e:98:10:57       no                 6.57
>  52     00:16:3e:9d:f1:0c       no                32.99
>  31     00:16:3e:ab:01:ea       no                31.35
>  19     00:16:3e:b9:02:30       no                 0.63
>   2     00:16:3e:c0:a1:56       no               200.78
>  78     00:16:3e:ce:0e:7b       no                 0.16
>  42     00:16:3e:ce:34:6c       no                 1.91
>  34     00:16:3e:cf:f5:56       no                 0.07
>  82     00:16:3e:d5:80:c1       no                 0.15
>  94     00:16:3e:d5:e2:34       no                 0.33
>  30     00:16:3e:df:41:05       no                 0.07
>  49     00:16:3e:e3:a3:75       no               120.84
>  15     00:16:3e:e3:c2:e8       no                 1.46
>  79     00:16:3e:e7:ac:59       no                73.66
>  70     00:16:3e:eb:c3:ed       no                34.88
>  43     00:16:3e:f1:69:06       no               283.33
>  10     00:16:3e:f4:e7:e0       no                 6.43
>  32     00:16:3e:fb:20:5c       no                 0.02
>  26     00:16:3e:fc:0c:a2       no                40.38
>   9     00:16:3e:fc:5b:6d       no                 0.02
>   1     00:17:c5:51:eb:41       no                45.46
>   1     00:22:4d:55:0a:01       no                45.26
>   1     00:23:9c:13:d6:01       no                 0.00
>   1     00:24:b2:ba:6c:1e       no                17.39
>   1     00:25:90:56:ac:f8       no                22.52
>   1     00:25:90:56:ac:f9       no                22.36
>   1     00:25:90:57:d5:44       no               109.90
>   1     00:25:90:57:d5:45       no                 1.67
>   1     00:30:48:f5:ed:ec       yes                0.00
Above is peth0, as seen from the "ip link show" below.  It''s
local, as
expected.  So all the other non-local port 1 MACs must be from 
interfaces elsewhere on your LAN.
>   7     da:3c:0e:f1:cc:d9       yes                0.00
Above is "tap172.0".  What is that device?
>   5     fe:ff:ff:ff:ff:ff       yes                0.00
Above is, I guess, all of your VIFs?  Mapped to one port, because they 
share the default MAC?  I don''t know how that works.  I''m
accustomed to
setting them explicitly.
> 2- arping output of the proplimatic ips
Why are these problematic?  You didn''t attach any tcpdump or anything
to
support your claim of TCP_SYN flooding.
> arping 192.168.2.13
> ARPING 192.168.2.13 from 68.XX.XX.XX eth0
> Unicast reply from 192.168.2.13 [00:25:90:55:36:58]  1.455ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  1.743ms
I find it odd that it switches MACs here.
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  0.811ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  0.850ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  0.982ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  4.539ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  0.835ms
> Unicast reply from 192.168.2.13 [00:25:90:55:36:59]  0.873ms
> Sent 7 probes (1 broadcast(s))
> Received 8 response(s)
> # arping 192.168.2.14
> ARPING 192.168.2.14 from 68.XX.XX.XX eth0
> Unicast reply from 192.168.2.14 [00:25:90:55:36:80]  1.514ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  1.632ms
...and here.
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.750ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.739ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.732ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.808ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.708ms
> Unicast reply from 192.168.2.14 [00:25:90:55:36:81]  0.720ms
> Sent 7 probes (1 broadcast(s))
> Received 8 response(s)
> 
> 
> 3- after doing the last 2 arping commands I got these new entries in 
> brctl showmacs
>   1     00:25:90:55:36:80       no                44.57
>   1     00:25:90:55:36:81       no                38.55
>   1     00:25:90:56:a9:c4       no                29.50
>   1     00:25:90:56:ac:f8       no                89.13
>   1     00:25:90:56:ac:f9       no                87.13
>   1     00:25:90:57:d2:db       no                39.27
>   1     00:25:90:57:d5:44       no                16.08
>   1     00:25:90:57:d5:45       no                99.29
I''d have expected to see 00:25:90:55:36:58 and 00:25:90:55:36:59 as 
well.

The first two map from 192.168.2.14, and they''re non-local, and on port
1 of the eth0 bridge, whose local interface is peth0.  From that I''d 
surmise that those MACs are also on another machine on your network.
> 4- to see which interface port 1 of the bridge is i see
>  dmesg | grep "port 1("
> eth0: port 1(peth0) entering forwarding state
Same conclusion, but I use the "ip link show" output below.
> 5- brctl show
I''m wondering if this is your problem, that you have STP disabled on 
your bridge, if not your network:
> eth0            8000.003048f5edec       no              vifvm341.0
>                                                         vifvm339.0
>                                                         vifvm157.0
>                                                         vifvm305.0
>                                                         vifvm121.0
>                                                         vifvm139.0
>                                                         vifvm256.0
>                                                         vifvm257.0
>                                                         vifvm176.0
>                                                         vifvm237.0
>                                                         vifvm220.0
>                                                         vifvm351.0
>                                                         vifvm335.0
>                                                         vifvm297.0
>                                                         vifvm163.0
>                                                         vifvm294.0
>                                                         vifvm348.0
>                                                         vifvm245.0
>                                                         vifvm394
>                                                         tap172.0
What is this tap device?
>                                                         vifvm165.0
>                                                         vifvm498
>                                                         vifvm274.0
>                                                         vifvm355.0
>                                                         vifvm353.0
>                                                         vifvm354.0
>                                                         vifvm346.0
>                                                         vifvm344.0
>                                                         vifvm340.0
>                                                         vifvm332.0
>                                                         vifvm325.0
>                                                         vifvm299.0
>                                                         vifvm295.0
>                                                         vifvm292.0
>                                                         vifvm291.0
>                                                         vifvm319.0
>                                                         vifvm279.0
>                                                         vifvm277.0
>                                                         vifvm102.0
>                                                         vifvm269.0
>                                                         vifvm447
>                                                         vifvm260.0
>                                                         vifvm258.0
>                                                         vifvm341
>                                                         vifvm455
>                                                         vifvm252.0
>                                                         vifvm445
>                                                         vifvm332
>                                                         vifvm235.0
>                                                         vifvm164
>                                                         vifvm232.0
>                                                         vifvm187
>                                                         vifvm216.0
>                                                         vifvm154
>                                                         vifvm178.0
>                                                         vifvm298
>                                                         vifvm177.0
>                                                         vifvm174.0
>                                                         vifvm481
>                                                         vifvm170.0
>                                                         vifvm168.0
>                                                         vifvm475
>                                                         vifvm490
>                                                         vifvm137.0
>                                                         vifvm411
>                                                         vifvm113.0
>                                                         vifvm103.0
>                                                         vifvm513
>                                                         vifvm412
>                                                         vifvm279
>                                                         peth0
So eth0 is your bridge and, I assume, peth0 is your physical.
> 6- ip link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> 2: peth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen
> 1000
>     link/ether 00:30:48:f5:ed:ec brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:30:48:f5:ed:ed brd ff:ff:ff:ff:ff:ff
What does eth1 connect to?
> 4: vif0.0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 5: veth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 6: vif0.1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 7: veth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 8: vif0.2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 9: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 10: vif0.3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 11: veth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 12: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>     link/ether 00:30:48:f5:ed:ec brd ff:ff:ff:ff:ff:ff
> 30: vifvm279: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
> qlen 500
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
[snip a lot of these vifvmNNN interfaces]
> 209: tap172.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
> qlen 500
>     link/ether da:3c:0e:f1:cc:d9 brd ff:ff:ff:ff:ff:ff
> 7-  ip address show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
> 2: peth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
qlen
> 1000
>     link/ether 00:30:48:f5:ed:ec brd ff:ff:ff:ff:ff:ff
> 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
>     link/ether 00:30:48:f5:ed:ed brd ff:ff:ff:ff:ff:ff
> 4: vif0.0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 5: veth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 6: vif0.1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 7: veth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 8: vif0.2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 9: veth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 10: vif0.3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
> 11: veth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 12: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>     link/ether 00:30:48:f5:ed:ec brd ff:ff:ff:ff:ff:ff
>     inet 68.XX.XX.XX/27 brd 68.XX.XX.XX scope global eth0
> 30: vifvm279: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
> qlen 500
>     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
[snip a lot of these vifvmNNN interfaces]
> 209: tap172.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast
> qlen 500
>     link/ether da:3c:0e:f1:cc:d9 brd ff:ff:ff:ff:ff:ff
> 8- ip route show
> 68.XX.XX.XX/27 dev eth0  proto kernel  scope link  src 68.XX.XX.XX
> XX.XX.0.0/16 dev eth0  scope link
> default via 68.XX.XX.XX dev eth0
> 
> 9- xm info
> host                   : XXX.localdomain.server
> release                : 2.6.18-348.3.1.el5xen
My, that''s old.
> version                : #1 SMP Mon Mar 11 20:28:48 EDT 2013
> machine                : x86_64
> nr_cpus                : 24
> nr_nodes               : 1
> cores_per_socket       : 12
> threads_per_core       : 1
> cpu_mhz                : 2100
> hw_caps                :
> 178bf3ff:efd3fbff:00000000:00000310:00802001:00000000:000837ff:00000000
> virt_caps              : hvm
> total_memory           : 114686
> free_memory            : 49764
> node_to_cpu            : node0:0-23
> node_to_memory         : node0:49764
> xen_major              : 3
> xen_minor              : 4
> xen_extra              : .4
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
> cc_compile_by          : root
> cc_compile_domain      : soluslabs.net
> cc_compile_date        : Thu Nov 22 06:14:22 EST 2012
> xend_config_format     : 4

Xen users - Apr 2013 - finding the source VM of local ip

finding the source VM of local ip

Re: finding the source VM of local ip

Re: finding the source VM of local ip

Re: finding the source VM of local ip