CentOS - Feb 2009 - Keepalive...

Hi,

I''ve been trying to setup keepalive (vrrp) with LVS in direct routing
on Centos 5 and I am stuck...
Installed ipvsadm from the repo (had to touch /etc/sysconfig/ipvsadm because it
complained it was missing)
Compiled keepalived-1.1.16.tar.gz

Test setup:

             main switch (192.168.16.0/20)
                                     |
             eth0: 192.168.28.[226|227]
                   VIP=192.168.16.123
                2 lvs/keepalived servers
                       eth1: 10.0.0.[1|2]
                                     |
                    test switch (10.0.0.0/8)
                                     |
                         10.0.0.[11|12]
                    VIP=192.168.16.123
                
test servers (real servers)
                    192.168.16.[228|229]
                                     |
                     
back to main switch

But let''s focus on the 2 lvs servers...

keepalived.conf:

global_defs {
   notification_email {
     root at localhost
   }
   notification_email_from keepalived at lvs1.iper
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_MASTER          ### BACKUP on the backup ###
}

vrrp_instance VI_1 {
    state MASTER          ### BACKUP on the backup ###
    interface eth0
    lvs_sync_daemon_interface eth1
    virtual_router_id 51
    priority 100                 ### 99 on the backup ###
    advert_int 1
    smtp_alert
    virtual_ipaddress {
        192.168.16.123
    }
}

virtual_server 192.168.16.123 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 50
    protocol TCP
    real_server 10.0.0.11 80 {
        weight 1
    }
    real_server 10.0.0.12 80 {
        weight 1
    }
    sorry_server 127.0.0.1 80
}

/var/log/messages:

Feb 20 15:01:09 lvs1 Keepalived: Starting Keepalived v1.1.16 (02/19,2009) 
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Registering Kernel netlink reflector
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Registering Kernel netlink command channel
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Registering gratutious ARP shared channel
Feb 20 15:01:09 lvs1 Keepalived: Starting VRRP child process, pid=19385
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Opening file
''/etc/keepalived/keepalived.conf''.
Feb 20 15:01:09 lvs1 Keepalived_vrrp: Configuration is using : 35092 Bytes
Feb 20 15:01:09 lvs1 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112),
fd(9,10)]

And that''s it... no VIP showing up... no error message...
Looking at the (outdated) log example from the keepalived doc, I seem to miss
all the "parsing" of the configuration.
A bit like if it was stuck in the vrrp child...

On the backup, I get these extra:
Feb 20 15:01:18 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Feb 20 15:01:18 lvs2 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112),
fd(9,10)]
Feb 20 15:01:18 lvs2 Keepalived_vrrp: Remote SMTP server [127.0.0.1:25]
connected.
Feb 20 15:01:18 lvs2 Keepalived_vrrp: SMTP alert successfully sent.

ps auxfw:

root     19384  0.0  0.0   4796   624 ?        Ss   15:01   0:00 keepalived -D
root     19385  0.0  0.1   4864  1048 ?        S    15:01   0:00  \_ keepalived
-D

lvs1, no VIP:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
    link/ether 00:04:23:9e:f3:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.226/20 brd 192.168.31.255 scope global eth0
    inet6 fe80::204:23ff:fe9e:f374/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
100
    link/ether 00:04:23:9e:f3:75 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/8 brd 10.255.255.255 scope global eth1
    inet6 fe80::204:23ff:fe9e:f375/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop 
    link/sit 0.0.0.0 brd 0.0.0.0

lvs2, no VIP:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
    link/ether 00:04:23:b2:65:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.227/20 brd 192.168.31.255 scope global eth0
    inet6 fe80::204:23ff:feb2:6502/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
100
    link/ether 00:04:23:b2:65:03 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/8 brd 10.255.255.255 scope global eth1
    inet6 fe80::204:23ff:feb2:6503/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop 
    link/sit 0.0.0.0 brd 0.0.0.0

ipvsadm -l -n shows nothing
tcpdump -i eth1 -p vrrp shows nothing

Any idea why keepalived would not bring the VIP up and setup vrrp?

Thx,
JD

On Fri, 20 Feb 2009, John Doe wrote:
> I''ve been trying to setup keepalive (vrrp) with LVS in direct
routing on
> Centos 5 and I am stuck... Installed ipvsadm from the repo (had to touch 
> /etc/sysconfig/ipvsadm because it complained it was missing) Compiled 
> keepalived-1.1.16.tar.gz [...]
I have also had similar (albeit intermittent) problems with 1.1.16 as 
well; try 1.1.15.

Steve

> vrrp_instance VI_1 {
>     state MASTER          ### BACKUP on the backup ###
>     interface eth0
>     lvs_sync_daemon_interface eth1
>     virtual_router_id 51
>     priority 100                 ### 99 on the backup ###
>     advert_int 1
>     smtp_alert
>     virtual_ipaddress {
>         192.168.16.123
>     }
> }
Suddenly, I went a bit further...
I saw a different (from the outdated doc) config on the web and tried it
succesfully:

     virtual_ipaddress {
         192.168.16.123/32 brd 192.168.31.255 scope global
     }

And get these extra bits:

Feb 20 16:19:56 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER
STATE
Feb 20 16:19:57 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 20 16:19:57 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 20 16:19:57 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous
ARPs on eth0 for 192.168.16.123
Feb 20 16:19:57 lvs1 Keepalived_vrrp: Remote SMTP server [127.0.0.1:25]
connected.
Feb 20 16:19:57 lvs1 Keepalived_vrrp: SMTP alert successfully sent.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
    link/ether 00:04:23:9e:f3:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.226/20 brd 192.168.31.255 scope global eth0
    inet 192.168.16.123/32 brd 192.168.31.255 scope global eth0
    inet6 fe80::204:23ff:fe9e:f374/64 scope link 
       valid_lft forever preferred_lft forever

But not sure both lvs are talking vrrp to each other; tcpdump does not intercept
anything...

JD

On Fri, Feb 20, 2009 at 3:58 PM, John Doe <jdmls at yahoo.com>
wrote:
>
> I''ve been trying to setup keepalive (vrrp) with LVS in direct
routing on Centos 5 and I am stuck...
> Installed ipvsadm from the repo (had to touch /etc/sysconfig/ipvsadm
because it complained it was missing)
> Compiled keepalived-1.1.16.tar.gz
>
...snip...

Install keepalived from rpmforge instead of compiling your own.
Because mostlikely your build of keepalived is without LVS support.
(Check the output of the ./configure command to verify).

I''ve always used the rpmforge package and it works for me.

Regards,
Tim

P.S. : Dag, your keepalived package is a bit outdated. Could you update it ?

-- 
Tim Verhoeven - tim.verhoeven.be at gmail.com - 0479 / 88 11 83

Hoping the problem  magically goes away  by ignoring it is the
"microsoft approach to programming" and should never be allowed.
(Linus Torvalds)

From: Tim Verhoeven <tim.verhoeven.be at
gmail.com>
> Install keepalived from rpmforge instead of compiling your own.
> Because mostlikely your build of keepalived is without LVS support.
> (Check the output of the ./configure command to verify).
> 
> I''ve always used the rpmforge package and it works for me.
> 
> Regards,
> Tim
> 
> P.S. : Dag, your keepalived package is a bit outdated. Could you update it
?
Bingo!
I did not see that it was not finding the kernel headers...

../configure
....
checking for kernel version... 0.0.0
configure: WARNING: Cannot determine Linux Kernel version.
....

I created a linux symlink and it worked out.
Tcpdump reports the VRRP mcasts.
And I can browse my web server through my VIP
Thx guys!
Now, gonna test the failover and the sorry server...

JD

On Fri, 20 Feb 2009, Tim Verhoeven wrote:
> On Fri, Feb 20, 2009 at 3:58 PM, John Doe <jdmls at yahoo.com> wrote:
>>
>> I''ve been trying to setup keepalive (vrrp) with LVS in direct
routing on Centos 5 and I am stuck...
>> Installed ipvsadm from the repo (had to touch /etc/sysconfig/ipvsadm
because it complained it was missing)
>> Compiled keepalived-1.1.16.tar.gz
>>
> ...snip...
>
> Install keepalived from rpmforge instead of compiling your own.
> Because mostlikely your build of keepalived is without LVS support.
> (Check the output of the ./configure command to verify).
>
> I''ve always used the rpmforge package and it works for me.
>
> P.S. : Dag, your keepalived package is a bit outdated. Could you update it
?
I am not using it, so I find it too risky to do it myself. Time for 
someone to step up and maintain it ?

-- 
--   dag wieers,  dag at centos.org,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]

Dag Wieers napsal(a):
> I am not using it, so I find it too risky to do it myself. Time for 
> someone to step up and maintain it ?
> 
I''m prepared to step in. See
http://fs12.vsb.cz/hrb33/el5/hrb-fw/stable/i386/repodata/
http://fs12.vsb.cz/hrb33/el5/hrb-fw/stable/x86_64/repodata/
David Hrb??

> Test setup:
>
>       main switch (192.168.16.0/20)
>                     |
>         eth0: 192.168.28.[226|227]
>             VIP=192.168.16.123
>          2 lvs/keepalived servers
>             eth1: 10.0.0.[1|2]
>                     |
>         test switch (10.0.0.0/8)
>                     |
>               10.0.0.[11|12]
>             VIP=192.168.16.123
>         test servers (real servers)
>            192.168.16.[228|229]
>                     |
>            back to main switch
Hi again,

I stopped the servers for the week-end...  Restarted them on monday and... it
did not work anymore.
Tried 1.1.15 as suggested, same.
So, I installed the keepalived-1.1.16-1.el5.hrb rpm David kindly built.
And, not really better.
My config more or less work... More ''less'' than
''more'' sadly...
I have many "random" problems and weird behaviors, that fix themselves
after a few restarts/reboots, without changing anything in my conf.  And they
will be back at the next restart...

Once, it is the vrrp stuff that do not seem to work.
I say seem because, even if tcpdump does not show any vrrp packets (it does
other times), sometimes the backup catches the master that was brought down and
switches to master state. And, at other times, both detects nothing at all...  A
few restarts and it works again until next failure.  And at other times, I can
see the packets...

There were times when both would be master...

Another time, keepalive does not seem to check the webservers as regularly as
other times.
I say again seem because, while the accesslog of my webserver does not display
any recent entry from keepalive (hash) checks, keepalive still detects that one
web server was brought down and that it temporarly removes it from its list... 
And I see nothing in keepalived logs about this...  Except once in a while.
By example, right now lvs1 is master, and I see only lvs2 checks in my web logs.
But if I bring down web1, lvs1 catches it and removes it until I bring it back
up...

Another time, the arp resolution on my client for the VIP is
"incomplete".  Fixed after a few restarts.
Many times, the master gets stuck on "VRRP sockpool".

Each time there is a problem, I checked and both my web servers are accessible
from the 2 lvs servers and from outside through the exit IPs
(192.168.16.[228|229]).

Also, when I use service restart, it will fail once out of 3 times with
"Keepalived: daemon is already running"...

Am I the only one having all these unstabilities?

Thx,
JD

I went a bit further...

lvs1# service keepalived stop
lvs2# service keepalived stop
lvs1# service network restart
lvs2# service network restart

Clean start

lvs1# service keepalived start

Feb 25 15:03:18 lvs1 Keepalived: Starting Keepalived v1.1.16 (02/17,2009) 
Feb 25 15:03:18 lvs1 Keepalived: Starting Healthcheck child process, pid=9511
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Using MII-BMSR NIC polling
thread...
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Netlink reflector reports IP
192.168.28.226 added
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Netlink reflector reports IP
10.0.0.1 added
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Registering Kernel netlink
reflector
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Registering Kernel netlink
command channel
Feb 25 15:03:18 lvs1 Keepalived: Starting VRRP child process, pid=9512
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Netlink reflector reports IP
192.168.28.226 added
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Netlink reflector reports IP 10.0.0.1
added
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Registering Kernel netlink reflector
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Registering Kernel netlink command channel
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Registering gratutious ARP shared channel
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Opening file
''/etc/keepalived/keepalived.conf''.
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Configuration is using : 13235
Bytes
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Activating healtchecker for
service [10.0.0.11:80]
Feb 25 15:03:18 lvs1 Keepalived_healthcheckers: Activating healtchecker for
service [10.0.0.12:80]
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Opening file
''/etc/keepalived/keepalived.conf''.
Feb 25 15:03:18 lvs1 Keepalived_vrrp: Configuration is using : 34062 Bytes
Feb 25 15:03:18 lvs1 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112),
fd(10,11)]

No VIP and no checks on the web servers...

lvs2# service keepalived start

Feb 25 15:05:23 lvs2 Keepalived: Starting Keepalived v1.1.16 (02/17,2009) 
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Using MII-BMSR NIC polling
thread...
Feb 25 15:05:23 lvs2 Keepalived: Starting Healthcheck child process, pid=8718
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Using MII-BMSR NIC polling thread...
Feb 25 15:05:23 lvs2 Keepalived: Starting VRRP child process, pid=8719
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Netlink reflector reports IP
192.168.28.227 added
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Netlink reflector reports IP
10.0.0.2 added
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Registering Kernel netlink
reflector
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Registering Kernel netlink
command channel
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Netlink reflector reports IP
192.168.28.227 added
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Netlink reflector reports IP 10.0.0.2
added
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Registering Kernel netlink reflector
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Registering Kernel netlink command channel
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Registering gratutious ARP shared channel
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Opening file
''/etc/keepalived/keepalived.conf''.
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Configuration is using : 13233
Bytes
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Activating healtchecker for
service [10.0.0.11:80]
Feb 25 15:05:23 lvs2 Keepalived_healthcheckers: Activating healtchecker for
service [10.0.0.12:80]
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Opening file
''/etc/keepalived/keepalived.conf''.
Feb 25 15:05:23 lvs2 Keepalived_vrrp: Configuration is using : 34060 Bytes
Feb 25 15:05:23 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Feb 25 15:05:23 lvs2 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112),
fd(10,11)]

No VIP and only one check on the web servers...

lvs1# service keepalived stop

Feb 25 15:07:30 lvs1 Keepalived: Terminating on signal
Feb 25 15:07:30 lvs1 Keepalived: Stopping Keepalived v1.1.16 (02/17,2009) 
Feb 25 15:07:30 lvs1 Keepalived_vrrp: Terminating VRRP child process on signal
Feb 25 15:07:30 lvs1 Keepalived_healthcheckers: Terminating Healthchecker child
process on signal

And nothing else (lvs2 does not become MASTER)...

lvs1# service keepalived start

Nothing much...

lvs2# service keepalived stop
lvs2# service keepalived start

Nothing and no checks on the web servers...

lvs1# service keepalived stop
lvs1# service keepalived start

Nothing and no checks on the web servers...

lvs1# service keepalived stop
lvs1# service keepalived start

Nothing and only one check on the web servers...
Always stuck on "VRRP sockpool"

By the way, a restart or a stop+restart too fast too often leads to a failed
start with "daemon is already running"

lvs1# service keepalived restart

Nothing and no checks on the web servers...

lvs1# service keepalived restart

Nothing and no checks on the web servers...

lvs1# service keepalived restart

Nothing and no checks on the web servers...

lvs1# service keepalived restart

Baam, suddenly many vrrp packets, and one web servers check

Feb 25 15:15:11 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio
advert, forcing new election
Feb 25 15:15:11 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous
ARPs on eth0 for 192.168.16.123
Feb 25 15:15:11 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous
ARPs on eth0 for 192.168.16.123
Feb 25 15:15:16 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio
advert, forcing new election
Feb 25 15:15:16 lvs1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous
ARPs on eth0 for 192.168.16.123

Feb 25 15:14:50 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER
STATE
Feb 25 15:14:50 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio
advert
Feb 25 15:14:50 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Feb 25 15:14:55 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER
STATE
Feb 25 15:14:55 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio
advert
Feb 25 15:14:55 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

The web servers are correctly accessed from outside in rr; but there are still
no web checks from the keepalives...

lvs1# ipvsadm

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.16.123:http rr
  -> 10.0.0.12:http               Route   1      0          28
  -> 10.0.0.11:http               Route   1      0          28

lvs2# ipvsadm

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.16.123:http rr
  -> 10.0.0.12:http               Route   1      0          0         
  -> 10.0.0.11:http               Route   1      0          0

lvs1# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
    link/ether 00:04:23:9e:f3:74 brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.226/20 brd 192.168.31.255 scope global eth0
    inet 192.168.16.123/32 scope global eth0
    inet6 fe80::204:23ff:fe9e:f374/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
100
    link/ether 00:04:23:9e:f3:75 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/8 brd 10.255.255.255 scope global eth1
    inet6 fe80::204:23ff:fe9e:f375/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0: <NOARP> mtu 1480 qdisc noop 
    link/sit 0.0.0.0 brd 0.0.0.0

No VIP on lvs2 (BACKUP state)

lvs1# service keepalived stop

Feb 25 15:29:06 lvs2 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER
STATE
tcpdump => VRRP.MCAST.NET: VRRPv2, Advertisement, vrid 51, prio 0, authtype
none, intvl 1s, length 20
No VIP on lvs1 and lvs2, ARP resolution for VIP incomplete...

lvs2# ip a add dev eth0 local 192.168.16.123/32 scope global

Baam, suddenly vrrp packets, and one round (only) of web server checks
15:33:18.639546 IP lvs2.iper > VRRP.MCAST.NET: VRRPv2, Advertisement, vrid
51, prio 99, authtype none, intvl 1s, length 20
15:33:19.641002 IP lvs2.iper > VRRP.MCAST.NET: VRRPv2, Advertisement, vrid
51, prio 99, authtype none, intvl 1s, length 20

lvs1# service keepalived start

Nothing...

lvs2# service keepalived stop

Baam, suddenly vrrp packets, and one round (only) of web server checks
The web servers are correctly accessed from outside in rr...

lvs2# service keepalived start

Nothing, other than Entering BACKUP STATE
Both lvs have the VIP up...

lvs1# service keepalived stop

Same as above, except the VIP is up on lvs2 and down on lvs1, and no
webchecks...
The web servers are correctly accessed from outside in rr...

lvs1# service keepalived start

Nothing...
lvs1 "stuck" on VRRP sockpool, while lvs2 is still MASTER
VIP down on lvs1 and up on lvs2

lvs2# service keepalived stop

Baam, suddenly vrrp packets, no web server checks at all
The web servers are correctly accessed from outside in rr...
Both lvs have the VIP up

lvs1# service keepalived stop
lvs1# service keepalived start
lvs2# service keepalived stop

Same as above except that there are webchecks from lvs1 now...

lvs2# service keepalived start

backup state, no webchecks from lvs2

lvs1# service keepalived stop

lvs2 => MASTER
VIP is up on lvs2, down on lvs1
Everything is stuck for like 30s... and then web servers are accessible.

lvs1# service keepalived start

Nothing...
lvs1 "stuck" on VRRP sockpool, while lvs2 is still MASTER
VIP down on lvs1 and up on lvs2

lvs2# service network restart

baam, vrrp packets, lvs1 transition to MASTER and sends ARPs
And I get regular webchecks from both lvs...
And if I bring down one web server, it is correctly removed from the services.
2mns later, no more web checks...

lvs1# service keepalived stop

lvs2 => MASTER
VIP is down on both lvs...  ARP is incomplete.
Everything is stuck for ever...

lvs2# ip a add dev eth0 local 192.168.16.123/32 scope global

baam, vrrp packets, lvs1 entering MASTER state and sends ARPs
I caught this: Netlink: error: File exists, type=(20), seq=1235574458, pid=0

Looking for errors in the logs, I found:

Feb 23 16:20:20 lvs1 Keepalived_vrrp: Netlink: filter function error
Feb 23 16:20:20 lvs1 Keepalived_healthcheckers: Netlink: filter function error
Feb 23 16:42:58 lvs1 Keepalived_vrrp: Netlink: filter function error
Feb 23 16:42:58 lvs1 Keepalived_healthcheckers: Netlink: filter function error
Feb 25 12:00:50 lvs1 kernel: IPVS: ip_vs_send_async error
Feb 25 12:12:04 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:04 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:05 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth0 failed: Input/output
error
Feb 25 12:12:05 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:05 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:05 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:06 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth0 failed: Input/output
error
Feb 25 12:12:06 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:06 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:06 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:07 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth0 failed: Input/output
error
Feb 25 12:12:07 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:07 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:07 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:08 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth0 failed: Input/output
error
Feb 25 12:12:08 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:08 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:08 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:09 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth0 failed: Input/output
error
Feb 25 12:12:09 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:09 lvs1 Keepalived_vrrp: SIOCGMIIREG on eth1 failed: Input/output
error
Feb 25 12:12:09 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:10 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:10 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:11 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:11 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:12 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:12 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:13 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:13 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:14 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:14 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:15 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:16 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:12:16 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth0 failed:
Input/output error
Feb 25 12:12:17 lvs1 Keepalived_healthcheckers: SIOCGMIIREG on eth1 failed:
Input/output error
Feb 25 12:33:39 lvs1 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235561506, pid=0
Feb 25 12:39:11 lvs1 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235561507, pid=0
Feb 25 12:40:10 lvs1 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235561508, pid=0
Feb 25 12:40:52 lvs1 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235561509, pid=0

Feb 23 16:20:16 lvs2 Keepalived_vrrp: Netlink: filter function error
Feb 23 16:20:16 lvs2 Keepalived_healthcheckers: Netlink: filter function error
Feb 23 16:42:46 lvs2 Keepalived_vrrp: Netlink: filter function error
Feb 23 16:42:46 lvs2 Keepalived_healthcheckers: Netlink: filter function error
Feb 23 17:35:36 lvs2 Keepalived_healthcheckers: Netlink: filter function error
Feb 23 17:35:36 lvs2 Keepalived_vrrp: Netlink: filter function error
Feb 25 12:25:22 lvs2 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235560956, pid=0
Feb 25 12:30:50 lvs2 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235561435, pid=0
Feb 25 15:33:18 lvs2 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235570954, pid=0
Feb 25 16:12:02 lvs2 Keepalived_vrrp: Netlink: error: Cannot assign requested
address, type=(21), seq=1235574457, pid=0
Feb 25 16:29:11 lvs2 Keepalived_vrrp: Netlink: error: File exists, type=(20),
seq=1235574458, pid=0

Do you have any idea about what could be causing these problems?

Thx,
JD

I changed my apparently faulty nic, but it does not change anything...

I can reproduce this:

I bring down keepalived on lvs1.
Keepalived on lvs2 says it switched to MASTER but does nothing.
2 possibilities:
- If no VIP, as soon as I manualy add the VIP, everything is unlocked and he
forward the requests...
- If there is a VIP, a ifdown+ifup of the interface linked to the web servers
will unlock keepalived.
I bring up keepalived on lvs1, and it does not setup the VIP.
On lvs2, it says: "removing protocol VIPs", but the VIP is still
there.
A ifdown+ifup on lvs2 of the interface linked to the web servers will unlock
keepalived on lvs1...
And, from times to times, it will work as expected...

One thing I really do not understand is why restarting the nic linked to the
webservers will unlock the vrrp on the other nic...

The only thing that almost never work is the service checks...
After one check, or a few at best, they just disapear...

JD