[STD]Ein
2008-Jun-11 02:32 UTC
RE: [Xen-users] Missed interrupt. Increasing latency to 5 ms in orderto compensate.
Is it possible the cpu(s) are being taxed? If they are under heavy load (from another domain most likely) it may cause the domain to not respond to interrupts in a timely manner. If this is the case, you might try changing the scheduling weights of your domains (search the list). _____ From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Jamie J. Begin Sent: June 10, 2008 1:12 PM To: xen-users@lists.xensource.com Subject: [Xen-users] Missed interrupt. Increasing latency to 5 ms in orderto compensate. I am using pciback to pass a Digium PCIX analog telephony card to an Asterisk domU. Seemingly at random times, the card wigs out with the following appended to my /var/log/messages: Jun 10 12:17:19 asterisk01 kernel: wctdm24xxp0: Missed interrupt. Increasing latency to 4 ms in order to compensate. Jun 10 14:01:01 asterisk01 kernel: wctdm24xxp0: Missed interrupt. Increasing latency to 5 ms in order to compensate. This is becoming a major problem because it will drop any active calls and refuse to accept new calls until I stop Asterisk, remove and then re-add the kernel module, and finally restart Asterisk. I don''t know whether the drivers, Xen, the motherboard/BIOS, or Asterisk is to blame. However, this did not occur until after I virtualized the server. Does anyone have any suggestions? This is killing me here. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
C. Mylonas (Mr VoIP)
2008-Jun-11 05:57 UTC
Re: [Xen-users] Missed interrupt. Increasing latency to 5 ms in orderto compensate.
Dear Jamie, I would *not* be putting telephony onto virtual infrastructure in a production environment period. If you are looking after this system, and are able to put it back onto a physical server, it''s not worth the stress of it, especially when using the switching network through telephony interface cards. As soon as your system chokes, you''ll have XX amount of users hurling abuse at the phones or yourself, and management will look badly on the whole virtualisation thing you''re trying to achieve with Xen. Chris On Wed, Jun 11, 2008 at 12:32 PM, [STD]Ein <ein@anti-logic.com> wrote:> Is it possible the cpu(s) are being taxed? If they are under heavy load > (from another domain most likely) it may cause the domain to not respond to > interrupts in a timely manner. If this is the case, you might try > changing the scheduling weights of your domains (search the list). > > > ------------------------------ > > *From:* xen-users-bounces@lists.xensource.com [mailto: > xen-users-bounces@lists.xensource.com] *On Behalf Of *Jamie J. Begin > *Sent:* June 10, 2008 1:12 PM > *To:* xen-users@lists.xensource.com > *Subject:* [Xen-users] Missed interrupt. Increasing latency to 5 ms in > orderto compensate. > > > > I am using pciback to pass a Digium PCIX analog telephony card to an > Asterisk domU. Seemingly at random times, the card wigs out with the > following appended to my /var/log/messages: > > > > Jun 10 12:17:19 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 4 ms in order to compensate. > > Jun 10 14:01:01 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 5 ms in order to compensate. > > > > This is becoming a major problem because it will drop any active calls and > refuse to accept new calls until I stop Asterisk, remove and then re-add the > kernel module, and finally restart Asterisk. I don''t know whether the > drivers, Xen, the motherboard/BIOS, or Asterisk is to blame. However, this > did not occur until after I virtualized the server. Does anyone have any > suggestions? This is killing me here… > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Kind Regards, Chris Mylonas Engineer _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jamie J. Begin
2008-Jun-11 06:09 UTC
RE: [Xen-users] Missed interrupt. Increasing latency to 5 ms inorderto compensate.
> -----Original Message----- > ________________________________ > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Jamie J. Begin > Sent: June 10, 2008 1:12 PM > To: xen-users@lists.xensource.com > Subject: [Xen-users] Missed interrupt. Increasing latency to 5 ms in > orderto compensate. > > I am using pciback to pass a Digium PCIX analog telephony card to an > Asterisk domU. Seemingly at random times, the card wigs out with the > following appended to my /var/log/messages: > > Jun 10 12:17:19 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 4 ms in order to compensate. > Jun 10 14:01:01 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 5 ms in order to compensate. > > This is becoming a major problem because it will drop any active calls and > refuse to accept new calls until I stop Asterisk, remove and then re-add > the kernel module, and finally restart Asterisk. I don''t know whether the > drivers, Xen, the motherboard/BIOS, or Asterisk is to blame. However, > this did not occur until after I virtualized the server. Does anyone have > any suggestions? This is killing me here.> From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of [STD]Ein > Sent: Tuesday, June 10, 2008 10:32 PM > To: ''Jamie J. Begin''; xen-users@lists.xensource.com > Subject: RE: [Xen-users] Missed interrupt. Increasing latency to 5 ms > inorderto compensate. > > Is it possible the cpu(s) are being taxed? If they are under heavy load > (from another domain most likely) it may cause the domain to not respond > to interrupts in a timely manner. If this is the case, you might try > changing the scheduling weights of your domains (search the list). >Thanks for the suggestion; unfortunately I wish it were that easy. The physical machine is a brand new Dell Poweredge 1650 III with a quad-core Xeon and 2 GB of RAM. The Asterisk domU is the only domU currently running. It is allocated two of the cores and rarely gets above a 0.04 load average throughout the day since it''s only processing 2-3 voice channels at any given time. I''ve been in this business long enough to know that "random" problems are rarely random but I''m having difficulty connecting the dots. This problem can occur during the weekend when the system is idle, or in the middle of a phone call on a Monday afternoon. I''m thinking it may somehow be an IRQ conflict. The card doesn''t seem to share an IRQ with anything; but it looks to have been assigned IRQ 18, which seems rather odd. I haven''t played around with any of the APIC kernel options because I''m not that familiar with them. Here''s some output that may be useful: "lspci" from the dom0: 0b:08.0 Ethernet controller: Digium, Inc. Unknown device 8002 (rev 11) Subsystem: Digium, Inc. Unknown device 8002 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- Latency: 32 (16000ns min, 32000ns max), Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 18 Region 0: I/O ports at dc00 [size=256] Region 1: Memory at fc7dfc00 (32-bit, non-prefetchable) [size=1K] Expansion ROM at fc7e0000 [disabled] [size=128K] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- "cat /proc/interrupts" from dom0. CPU0 CPU1 CPU2 CPU3 1: 2 0 0 0 Phys-irq i8042 8: 0 0 0 0 Phys-irq rtc 9: 0 0 0 0 Phys-irq acpi 12: 4 0 0 0 Phys-irq i8042 17: 31664532 0 0 94 Phys-irq megasas, peth0 19: 446716 2870 5 36 Phys-irq eth2 20: 25613484 0 1305 0 Phys-irq uhci_hcd:usb1, ehci_hcd:usb2 256: 102837808 0 0 0 Dynamic-irq timer0 257: 2439233 0 0 0 Dynamic-irq resched0 258: 43 0 0 0 Dynamic-irq callfunc0 259: 0 634445 0 0 Dynamic-irq resched1 260: 0 126 0 0 Dynamic-irq callfunc1 261: 0 13329219 0 0 Dynamic-irq timer1 262: 0 0 559915 0 Dynamic-irq resched2 263: 0 0 120 0 Dynamic-irq callfunc2 264: 0 0 6024225 0 Dynamic-irq timer2 265: 0 0 0 1626891 Dynamic-irq resched3 266: 0 0 0 101 Dynamic-irq callfunc3 267: 0 0 0 31030283 Dynamic-irq timer3 268: 4234 0 0 0 Dynamic-irq xenbus 269: 0 0 0 0 Dynamic-irq console 270: 378 0 0 0 Dynamic-irq pciback 271: 89831 2843 2883 15 Dynamic-irq blkif-backend 272: 4164194 24455 22095 4105 Dynamic-irq vif1.0 273: 60758 7524 5516 50 Dynamic-irq blkif-backend 274: 694799 3497036 2722 8 Dynamic-irq vif6.0 NMI: 0 0 0 0 LOC: 0 0 0 0 "cat /proc/interrupts" from the Asterisk domU: CPU0 CPU1 18: 577477891 0 Phys-irq wctdm24xxp0 256: 78034763 0 Dynamic-irq timer0 257: 59661 0 Dynamic-irq resched0 258: 38 0 Dynamic-irq callfunc0 259: 409 0 Dynamic-irq xenbus 260: 0 73103 Dynamic-irq resched1 261: 0 70 Dynamic-irq callfunc1 262: 0 3942339 Dynamic-irq timer1 263: 409 1 Dynamic-irq xencons 264: 113336 10469 Dynamic-irq blkif 265: 8317410 0 Dynamic-irq eth0 NMI: 0 0 LOC: 0 0 ERR: 0 MIS: 0 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jamie J. Begin
2008-Jun-11 06:15 UTC
RE: [Xen-users] Missed interrupt. Increasing latency to 5 ms in orderto compensate.
Yes, it''s a possibility that I have reluctantly considered. My reason for virtualizing it is primarily because the hardware is going to be so underutilized as simply a VoIP server--and I seem to have come so close to making it work properly. :-\ But as you correctly guessed, I am walking a management tightrope here with this project. It may just not be worth it. Thanks for the input.> -----Original Message----- > From: C. Mylonas (Mr VoIP) [mailto:chris@mrvoip.com.au] > Sent: Wednesday, June 11, 2008 1:58 AM > To: Jamie J. Begin > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Missed interrupt. Increasing latency to 5 ms in > orderto compensate. > > Dear Jamie, > > I would not be putting telephony onto virtual infrastructure in a > production environment period. > If you are looking after this system, and are able to put it back onto a > physical server, it''s not worth the stress of it, especially when using > the switching network through telephony interface cards. > > As soon as your system chokes, you''ll have XX amount of users hurling > abuse at the phones or yourself, and management will look badly on the > whole virtualisation thing you''re trying to achieve with Xen. > > Chris > > > On Wed, Jun 11, 2008 at 12:32 PM, [STD]Ein <ein@anti-logic.com> wrote: > > > Is it possible the cpu(s) are being taxed? If they are under heavy > load (from another domain most likely) it may cause the domain to not > respond to interrupts in a timely manner. If this is the case, you might > try changing the scheduling weights of your domains (search the list). > > > > > ________________________________ > > > From: xen-users-bounces@lists.xensource.com [mailto:xen-users- > bounces@lists.xensource.com] On Behalf Of Jamie J. Begin > Sent: June 10, 2008 1:12 PM > To: xen-users@lists.xensource.com > Subject: [Xen-users] Missed interrupt. Increasing latency to 5 ms in > orderto compensate. > > > > I am using pciback to pass a Digium PCIX analog telephony card to an > Asterisk domU. Seemingly at random times, the card wigs out with the > following appended to my /var/log/messages: > > > > Jun 10 12:17:19 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 4 ms in order to compensate. > > Jun 10 14:01:01 asterisk01 kernel: wctdm24xxp0: Missed interrupt. > Increasing latency to 5 ms in order to compensate. > > > > This is becoming a major problem because it will drop any active > calls and refuse to accept new calls until I stop Asterisk, remove and > then re-add the kernel module, and finally restart Asterisk. I don''t know > whether the drivers, Xen, the motherboard/BIOS, or Asterisk is to blame. > However, this did not occur until after I virtualized the server. Does > anyone have any suggestions? This is killing me here. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > > > > > -- > Kind Regards, > Chris Mylonas > Engineer_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users