libvirt users - Jan 2013 - http problem with (a particular url) and default (nat) networking

At work I have a script that provisions a vm for use by employees.  One
step in this process is to fetch hadoop, which we happen to get from
cloudera.  I noticed the script always failed when I used libvirt's default
networking (nat) but worked fine when I used user mode networking. My
instinct is that this is related to (potentially uncommon) network traffic
from the server in question, and the iptables rules added by libvirt.

Repro steps:

1. Create a vm (I tested with linux and freebsd guests) using default
libvirt networking settings (<interface type='network'>).
2. wget, curl, fetch:
http://archive.cloudera.com/one-click-install/lucid/cdh3-repository_1.0_all.deb

Observe it will "hang". If you use strace you'll see it block on
the select
call.

My particular host is using a virbr0 network bridge, with the following
iptables rules:

$ iptables -S -v -Z
-P INPUT ACCEPT -c 404828 91071544
-P FORWARD ACCEPT -c 0 0
-P OUTPUT ACCEPT -c 402905 45139291
-A INPUT -i virbr0 -p udp -m udp --dport 53 -c 26 1703 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -c 0 0 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -c 70 22960 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -c 0 0 -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state
RELATED,ESTABLISHED -c 1191 1495856 -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -c 853 64266 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -c 6 1968 -j ACCEPT
-A FORWARD -o virbr0 -c 0 0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -c 0 0 -j REJECT --reject-with icmp-port-unreachable
Zeroing chain `INPUT'
Zeroing chain `FORWARD'
Zeroing chain `OUTPUT'

I'm not sure how best to diagnose this problem. Any ideas or tips?

Thanks!

John M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20130126/849ce289/attachment.htm>

On 01/26/2013 07:30 PM, John McFarlane wrote:
> At work I have a script that provisions a vm for use by employees.
>  One step in this process is to fetch hadoop, which we happen to get
> from cloudera.  I noticed the script always failed when I used
> libvirt's default networking (nat) but worked fine when I used user
> mode networking. My instinct is that this is related to (potentially
> uncommon) network traffic from the server in question, and the
> iptables rules added by libvirt. 
>
> Repro steps:
>
> 1. Create a vm (I tested with linux and freebsd guests) using default
> libvirt networking settings (<interface type='network'>).
> 2. wget, curl,
> fetch:
http://archive.cloudera.com/one-click-install/lucid/cdh3-repository_1.0_all.deb
>
> Observe it will "hang". If you use strace you'll see it block
on the
> select call.
I just tried getting that file using wget from a RHEL6 guest on a Fedora
17 system with libvirt from "recent" git (within the last week), and
it
was able to retrieve the file with no problems and no delay. (the guest
is connected using the "default" network, i.e. NAT and virbr0.

I think there is something going on here beyond the iptables rules added
by libvirt.

Here are a few things I would try:

1) restart the libvirt daemon and see if the behavior changes.
restarting libvirtd causes all libvirt-created iptables rules to be
reloaded, so if the ordering had been messed up by some other
application, that would be corrected.

2) run "tcpdump -i eth0 -n -vvv host 184.73.217.71" on the host before
attempting the command in the guest (assuming the host's default route
goes out eth0), then see if maybe there is an incoming packet with the
SYN bit set or something (in my case, there wasn't anything like that -
it was a vanilla tcp session on port 80)

3) at the same time, run "tcpdump -i virbr0 -vvv host 184.73.217.71"
and
look for differences between what you see on eth0 and what you see on
virbr0 (possibly a packet isn't being NATed when it should be?)

4) along with looking at the filter table, look at the nat table:
"iptables -S -t nat", maybe you'll see something revealing there.
>
> My particular host is using a virbr0 network bridge, with the
> following iptables rules:
>
> $ iptables -S -v -Z
> -P INPUT ACCEPT -c 404828 91071544
> -P FORWARD ACCEPT -c 0 0
> -P OUTPUT ACCEPT -c 402905 45139291
> -A INPUT -i virbr0 -p udp -m udp --dport 53 -c 26 1703 -j ACCEPT
> -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -c 0 0 -j ACCEPT
> -A INPUT -i virbr0 -p udp -m udp --dport 67 -c 70 22960 -j ACCEPT
> -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -c 0 0 -j ACCEPT
> -A FORWARD -d 192.168.122.0/24 <http://192.168.122.0/24> -o virbr0 -m
> state --state RELATED,ESTABLISHED -c 1191 1495856 -j ACCEPT
> -A FORWARD -s 192.168.122.0/24 <http://192.168.122.0/24> -i virbr0 -c
> 853 64266 -j ACCEPT
> -A FORWARD -i virbr0 -o virbr0 -c 6 1968 -j ACCEPT
> -A FORWARD -o virbr0 -c 0 0 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i virbr0 -c 0 0 -j REJECT --reject-with icmp-port-unreachable
> Zeroing chain `INPUT'
> Zeroing chain `FORWARD'
> Zeroing chain `OUTPUT'
>
> I'm not sure how best to diagnose this problem. Any ideas or tips?
>
> Thanks!
>
> John M.
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users at redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users