Xen devel - May 2011 - Problem Xen 4.0-amd64 + Heartbeat 3.0.3-2 with Debian Squeeze

Hi!Hello! Can know. I have the solution Xen 4.0 amd64 + Heartbeat 3.0.3-2 on
Debian Squeeze installed using apt-get. I have the eth1 and eth2 used by DRBD
Heartbeat and used by the other three interfaces used by the domus as a bridge.
The log of the heartbeat is generating this message: May 25 08:23:28 inga
heartbeat: [26789]:
WARN: Gmain_timeout_dispatch: Dispatch function for send local status took too
long to execute: 310 ms (> 50 ms) (GSource: 0x1dae330) May 25 08:28:26 inga
heartbeat: [26789]:
WARN: Gmain_timeout_dispatch: Dispatch function for send local status took too
long to execute: 90 ms (> 50 ms) (GSource: 0x1dae330) A few more days and had
this message and then migrated to the server1 to server2 domus. I changed the
values ​​of ha.cf as below. deadtime 30 warntime 15. The
Server1 (dom0) 8 domU , already Server2 (dom0) has 3 domU.Configuration Servers
dom02 Processor Quad Core 2.432 GB ram6
Gigabit Ethernet5 HD SAS 300 GB 15 K raid 5Thanks.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi!Hello! Can know. I have the solution Xen 4.0 amd64 + Heartbeat 3.0.3-2 on
Debian Squeeze installed using apt-get. I have the eth1 and eth2 used by DRBD
Heartbeat
and used by the other three interfaces used by the domus as a bridge. The log of
the
heartbeat is generating this message:May 25 08:23:28 inga heartbeat:
[26789]: WARN: Gmain_timeout_dispatch: Dispatch function for send local status
took too
long to execute: 310 ms (> 50 ms) (GSource: 0x1dae330)May 25 08:28:26 inga
heartbeat: [26789]: WARN: Gmain_timeout_dispatch: Dispatch function for send
local
status took too long to execute: 90 ms (> 50 ms) (GSource: 0x1dae330)A
few more days and had this message and then migrated to the server1 to server2
domus.I changed the values ​​of ha.cf as below.deadtime
30warntime
15.The Server1 (dom0) 8 domU , already Server2 (dom0) has 3 domU.Configuration
Servers dom02 Processor Quad Core 2.432 GB ram6
Gigabit Ethernet5 HD SAS 300 GB 15 K raid 5Thanks.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I saw your previous message and to be honest it wasn''t clear if you
were
asking for help with an issue or providing a solution you had
discovered.

Since you have repeated it I assume the former. If you don''t initially
get a response to a mail then simply repeating it verbatim in a new
thread may not help (and can be considered rude) -- you should consider
why people may not have responded. i.e. whether the mail needs
clarifying or additional information etc.

Anyway, this seems like an issue with heartbeat, perhaps you should ask
for help on the heartbeat mailing list, since there you are more likely
to find experts in that field there. In terms of Xen I think this
question would have been better suited to xen-users@ in the first
instance.

Ian.

On Sat, 2011-05-28 at 01:25 +0100, gilmarlinux@agrovale.com.br
wrote:
> Hi!
> Hello! Can know. I have the solution Xen 4.0 amd64 + Heartbeat 3.0.3-2
> on Debian Squeeze installed using apt-get. I have the eth1 and eth2
> used by DRBD Heartbeat and used by the other three interfaces used by
> the domus as a bridge. The log of the heartbeat is generating this
> message:
> 
> May 25 08:23:28 inga heartbeat: [26789]: WARN: Gmain_timeout_dispatch:
> Dispatch function for send local status took too long to execute: 310
> ms (> 50 ms) (GSource: 0x1dae330)
> May 25 08:28:26 inga heartbeat: [26789]: WARN: Gmain_timeout_dispatch:
> Dispatch function for send local status took too long to execute: 90
> ms (> 50 ms) (GSource: 0x1dae330)
> 
> A few more days and had this message and then migrated to the server1
> to server2 domus.
> I changed the values ​​of ha.cf as below.
> deadtime 30
> warntime 15.
> 
> The Server1 (dom0) 8 domU , already Server2 (dom0) has 3 domU.
> 
> Configuration Servers dom0
> 
> 2 Processor Quad Core 2.4
> 32 GB ram
> 6 Gigabit Ethernet
> 5 HD SAS 300 GB 15 K raid 5
> Thanks.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Sorry. Also sent to the list of the heartbeat, but found this link below: And it
seems that the same problem. http://www.mailinglistarchive.com/xen-devel @
lists.xensource.com/msg59416.html> I saw your
previous message and to be honest it wasn''t clear if you were>
asking for help
with an issue or providing a solution you had> discovered.> >
Since you have repeated it I assume the former. If you don''t
initially> get a
response to a mail then simply repeating it verbatim in a new> thread may not
help (and can be considered rude) -- you should consider> why people may not
have responded. i.e. whether the mail needs> clarifying or additional
information etc.> > Anyway, this seems like an issue with heartbeat,
perhaps you should ask> for help on the heartbeat mailing list, since there
you
are more likely> to find experts in that field there. In terms of Xen I think
this> question would have been better suited to xen-users@ in the first>
instance.> > Ian.> > On Sat, 2011-05-28 at
01:25 +0100, gilmarlinux@agrovale.com.br wrote:>> Hi!>> Hello!
Can know. I have the solution Xen 4.0 amd64 + Heartbeat 3.0.3-2>> onDebian
Squeeze installed using apt-get. I have the eth1 and eth2>> used by
DRBD Heartbeat and used by the other three interfaces used by>> the domus
as
a bridge. The log of the heartbeat is generating this>>
message:>>>> May 25 08:23:28 inga heartbeat: [26789]: WARN:
Gmain_timeout_dispatch:>> Dispatch function for send local status took too
long to execute: 310>> ms (> 50 ms) (GSource: 0x1dae330)>>
May 25 08:28:26 inga heartbeat: [26789]: WARN: Gmain_timeout_dispatch:>>
Dispatch function for send local status took too long to execute: 90>> ms
(> 50 ms) (GSource: 0x1dae330)>>>> A few more days and had
this message and then migrated to the server1>> to server2 domus.>>
I changed the values ​​of ha.cf as
below.>> deadtime 30>> warntime 15.>>>>
The Server1 (dom0) 8 domU , already Server2 (dom0) has 3 domU.>>>>
Configuration Servers dom0>>>> 2 Processor Quad Core
2.4>> 32 GB ram>> 6 Gigabit Ethernet>> 5 HD SAS 300
GB 15 K raid 5>> Thanks.> > >


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ok, Thanks.I thought it might be something I / O xen causing this in
heartbeat.> I saw your
previous message and to be honest it wasn''t clear if you were>
asking for help
with an issue or providing a solution you had> discovered.> >
Since you have repeated it I assume the former. If you don''t
initially> get a
response to a mail then simply repeating it verbatim in a new> thread may not
help (and can be considered rude) -- you should consider> why people may not
have responded. i.e. whether the mail needs> clarifying or additional
information etc.> > Anyway, this seems like an issue with heartbeat,
perhaps you should ask> for help on the heartbeat mailing list, since there
you
are more likely> to find experts in that field there. In terms of Xen I think
this> question would have been better suited to xen-users@ in the first>
instance.> > Ian.> > On Sat, 2011-05-28 at
01:25 +0100, gilmarlinux@agrovale.com.br wrote:>> Hi!>> Hello!
Can know. I have the solution Xen 4.0 amd64 + Heartbeat 3.0.3-2>> onDebian
Squeeze installed using apt-get. I have the eth1 and eth2>> used by
DRBD Heartbeat and used by the other three interfaces used by>> the domus
as
a bridge. The log of the heartbeat is generating this>>
message:>>>> May 25 08:23:28 inga heartbeat: [26789]: WARN:
Gmain_timeout_dispatch:>> Dispatch function for send local status took too
long to execute: 310>> ms (> 50 ms) (GSource: 0x1dae330)>>
May 25 08:28:26 inga heartbeat: [26789]: WARN: Gmain_timeout_dispatch:>>
Dispatch function for send local status took too long to execute: 90>> ms
(> 50 ms) (GSource: 0x1dae330)>>>> A few more days and had
this message and then migrated to the server1>> to server2 domus.>>
I changed the values ​​of ha.cf as
below.>> deadtime 30>> warntime 15.>>>>
The Server1 (dom0) 8 domU , already Server2 (dom0) has 3 domU.>>>>
Configuration Servers dom0>>>> 2 Processor Quad Core
2.4>> 32 GB ram>> 6 Gigabit Ethernet>> 5 HD SAS 300
GB 15 K raid 5>> Thanks.> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 2011-05-28 at 22:21 +0100, gilmarlinux@agrovale.com.br
wrote:
> I thought it might be something I / O xen causing this in heartbeat.
"I / O"? I''m not sure what this means.

I wonder if this is just simply down to scheduling latencies? i.e. dom0
simply isn''t getting scheduled frequently enough to meet the
requirements of the heartbeat process.

I assume this doesn''t happen when there is no domU workload to contend
with dom0? What sort of workloads are your guests running? How many
VCPUs do you have in total?

As an experiment you could try playing with the scheduling priorities
for dom0, e.g. giving it more weight.

Another experiment you could try is pinning a dom0 VCPU to a PCPU,
pinning the heartbeat process to that VCPU and excluding domU VCPUs from
contention by pinning them to the remaining PCPUs.

IIRC these sorts of latency sensitive workloads are somewhere that the
existing credit scheduler doesn''t do so well and is something which
George is looking to address with credit2. His canonical workload of
this type is audio playback but I suppose heartbeat falls into the same
broad class. So it might also be worth trying the credit2 scheduler.

You could also try Xen 4.1, IIRC some band-aids were applied to the
credit1 scheduler between 4.0 and 4.1.

Lastly you could presumably configure heartbeat to be less sensitive to
these latencies? I also hope that the heartbeat process sets itself up
with the Linux scheduler to be a fairly high priority process within
dom0?

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

(please try not to top post and always leave the CC to xen-devel in
tact)

On Tue, 2011-05-31 at 15:45 +0100, gilmarlinux@agrovale.com.br
wrote:
> Thank you for your help.
> Yesterday increase the values ​​of the heartbeat configuration as
> below.
> 
> /etc/ha.d/ha.cf
> keepalive 15
> deadtime 50
> warntime 30
What are the units of these values? I seem to recall it was complaining
of delays of the order 300ms, so you''d presumably want to be sure they
were longer than that.

Ian.

> 
> Liberie also the suggestion, I will now await the return message
> appears in the logs or not.
> 
> > On Sat, 2011-05-28 at 22:21 +0100, gilmarlinux@agrovale.com.br
> wrote:
> >> I thought it might be something I / O xen causing this in
> heartbeat.
> > 
> > "I / O"? I''m not sure what this means.
> > 
> > I wonder if this is just simply down to scheduling latencies? i.e.
> dom0
> > simply isn''t getting scheduled frequently enough to meet the
> > requirements of the heartbeat process.
> > 
> > I assume this doesn''t happen when there is no domU workload
to
> contend
> > with dom0? What sort of workloads are your guests running? How many
> > VCPUs do you have in total?
> > 
> > As an experiment you could try playing with the scheduling
> priorities
> > for dom0, e.g. giving it more weight.
> > 
> > Another experiment you could try is pinning a dom0 VCPU to a PCPU,
> > pinning the heartbeat process to that VCPU and excluding domU VCPUs
> from
> > contention by pinning them to the remaining PCPUs.
> > 
> > IIRC these sorts of latency sensitive workloads are somewhere that
> the
> > existing credit scheduler doesn''t do so well and is something
which
> > George is looking to address with credit2. His canonical workload of
> > this type is audio playback but I suppose heartbeat falls into the
> same
> > broad class. So it might also be worth trying the credit2 scheduler.
> > 
> > You could also try Xen 4.1, IIRC some band-aids were applied to the
> > credit1 scheduler between 4.0 and 4.1.
> > 
> > Lastly you could presumably configure heartbeat to be less sensitive
> to
> > these latencies? I also hope that the heartbeat process sets itself
> up
> > with the Linux scheduler to be a fairly high priority process within
> > dom0?
> > 
> > Ian.
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, May 31, 2011 at 9:30 AM, Ian Campbell
<Ian.Campbell@eu.citrix.com> wrote:
> IIRC these sorts of latency sensitive workloads are somewhere that the
> existing credit scheduler doesn''t do so well and is something
which
> George is looking to address with credit2. His canonical workload of
> this type is audio playback but I suppose heartbeat falls into the same
> broad class. So it might also be worth trying the credit2 scheduler.
It is quite possible to get 50-100ms delays in scheduling with credit1
if the system is fairly busy.  What kind of results does xentop
report?  You might consider moving to 4.1 and testing the credit2
scheduler.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel