asterisk users - Feb 2009 - sendto syscall: EPERM (Operation not permitted)

A bizarre problem on this host running Asterisk.  I don't actually think
this is Asterisk's problem, but I don't know who to ask, so if this is
OT,
please redirect me.

CentOS 5.2 64-bit Xen domU running Asterisk 1.4.22 32-bit with a number of
SIP phones and a few SIP<->PSTN gateways.

When asterisk tries to send a packet over UDP 5060 to one (and only one)
of the gateways, the syscall returns "EPERM (Operation not
permitted)".
Strangely, sending to UDP 5060 on the other gateway on the same network or
any of the SIP phones works just fine, sent from the same socket, only
difference is the IP address and the packet contents.

Following is an strace of Asterisk starting up with only the syscalls
relevant to the SIP socket.  Also, some network information showing that
there's nothing funny about the routes and no firewall rules to get in the
way.  No packets are actually sent out the wire to that gateway, though
the gateway is pingable, and I can get a response from the same asterisk
host using sipsak.

Also, oddly, SIP sessions initiated from the gateway to asterisk succeed.

There is very little information about EPERM errors in reference to
SOCK_DGRAM type sockets on google, so hopefully there's someone here who
can point me the right direction (even if it's to another, more
appropriate list).

    John

[root at pbx0 ~]# strace -f /usr/sbin/asterisk -U asterisk -G asterisk \
    -C /etc/asterisk/asterisk.conf -g -p -T
[...]
[Initial socket setup]
[pid  5006] socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 9
[pid  5006] setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid  5006] setsockopt(9, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
[pid  5006] bind(9, {sa_family=AF_INET, sin_port=htons(5060), \
    sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid  5006] setsockopt(9, SOL_IP, IP_TOS, [96], 4) = 0
[...]
[This is the gateway, same network, that asterisk successfully connects to]
[pid  5015] sendto(9, "OPTIONS sip:192.168.3.21 SIP/2.0"..., 485, 0, \
    {sa_family=AF_INET, sin_port=htons(5060), \
    sin_addr=inet_addr("192.168.3.21")}, 16) = 485
[...]
[The response from that gateway]
[pid  5015] recvfrom(9, "SIP/2.0 200 OK\r\nTo: <sip:101 at 192"...,
4095, 0, \
    {sa_family=AF_INET, sin_port=htons(5060), \
    sin_addr=inet_addr("192.168.3.51")}, [16]) = 319
[...]
[An example failed sendto syscall to the affected gateway]
[pid  5015] sendto(9, "OPTIONS sip:192.168.3.20 SIP/2.0"..., 485, 0, \
    {sa_family=AF_INET, sin_port=htons(5060), \
    sin_addr=inet_addr("192.168.3.20")}, 16) = -1 EPERM (Operation not
\
    permitted)


[root at pbx0 asterisk]# netstat -rn
Kernel IP routing table
Destination  Gateway      Genmask        Flags  MSS Window  irtt Iface
192.168.3.0  0.0.0.0      255.255.255.0  U        0 0          0 eth0
0.0.0.0      192.168.3.1  0.0.0.0        UG       0 0          0 eth0



[root at pbx0 asterisk]# iptables -L -v
Chain INPUT (policy ACCEPT 7974 packets, 1803K bytes)
 pkts bytes target     prot opt in     out     source    destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source    destination

Chain OUTPUT (policy ACCEPT 7950 packets, 2369K bytes)
 pkts bytes target     prot opt in     out     source    destination


[root at pbx0 asterisk]# tcpdump -nn host 192.168.3.20& sleep 1;
\
> sipsak -T -s sip:foo at 192.168.3.20; kill %
[1] 5223
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
warning: IP extract from warning activated to be more informational
17:04:44.511831 IP 192.168.3.19.42231 > 192.168.3.20.5060: SIP, length: 361
17:04:44.523252 IP 192.168.3.20.5060 > 192.168.3.19.42231: SIP, length: 381
0: ?? (11.231 ms) SIP/2.0 200 OK
        without Contact header