Hi, I would like to ask the proper or recommended way of upgrading xen...? I started with 3.1.0 then upgraded to 3.2.1 but now I am not able to use the original system. Since I am not satisfied[*] with the current setup I would try to upgrade to 3.3.1 but it would be nice if something went wrong I could downgrade it to 3.2.x... It is a pitty that booting the previous xen image does not work because of the changed libs. [*] The problem is that under heavy load in dom0 (or IO operation like performing the daily backups) I get a lot of "Time went backwards, etc." and sometimes (per 1-2 months) it crashes (freezes). The hw is a TYAN VX-50 w/ 4 Dual-Opteron 8220 CPUs and 4 500G WD RAID ED. discs (pairwise in software raid1, so the total capacity is ~1TB). Unfortunately the onboard NV SATA controller did not work under XEN so an ADAPTEC PCI card had to be installed. So any hints or URLs are welcome, -- Zsolt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, We saw time keeping issues with the Tyan B4989 Quads Opterons, I think it is fairly common. In the end we found using the /acpi=on (Might want to try numa=on as well) in the bootloader and setting hpet=0 in the VM config file reduced the problem significantly and NTP on the actual VMs could then compensate for small differences. This problem sounds similar to the one you are having in Dom 0, try changing the ACPI HPET option in the bios (If the VX50 has it). Some advice from the Xen developers suggested the ACPI HPET timer code might be a bit shaky (3.3.1 might fix it for all I know, I''m currently on 3.3.0). I''m still unsure how to tell the timer mode in Dom0 though. As for upgrading with rollback, this can be done by building the new kernel and adding it as a separate option in your bootloader (eg. Grub). I''m quite new to Kernel building and have only just got my first builds working so you might want to wait for advice from a more experienced user on that. Rob -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of SZABO Zsolt Sent: 27 January 2009 11:09 To: xen-users@lists.xensource.com Subject: [Xen-users] howto upgrade Hi, I would like to ask the proper or recommended way of upgrading xen...? I started with 3.1.0 then upgraded to 3.2.1 but now I am not able to use the original system. Since I am not satisfied[*] with the current setup I would try to upgrade to 3.3.1 but it would be nice if something went wrong I could downgrade it to 3.2.x... It is a pitty that booting the previous xen image does not work because of the changed libs. [*] The problem is that under heavy load in dom0 (or IO operation like performing the daily backups) I get a lot of "Time went backwards, etc." and sometimes (per 1-2 months) it crashes (freezes). The hw is a TYAN VX-50 w/ 4 Dual-Opteron 8220 CPUs and 4 500G WD RAID ED. discs (pairwise in software raid1, so the total capacity is ~1TB). Unfortunately the onboard NV SATA controller did not work under XEN so an ADAPTEC PCI card had to be installed. So any hints or URLs are welcome, -- Zsolt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users The SAQ Group Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ SAQ is the trading name of SEMTEC Limited. Registered in England & Wales Company Number: 06481952 http://www.saqnet.co.uk AS29219 SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. ISPA Member Find us in http://www.thebestof.co.uk/petersfield _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 27 Jan 2009, Robert Dunkley wrote:> We saw time keeping issues with the Tyan B4989 Quads Opterons, I think > it is fairly common. In the end we found using the /acpi=on (Might want > to try numa=on as well) in the bootloader and setting hpet=0 in the VM > config file reduced the problem significantly and NTP on the actual VMs > could then compensate for small differences. This problem sounds similar > to the one you are having in Dom 0, try changing the ACPI HPET option in > the bios (If the VX50 has it).Hi Rob, thanks for the answer! Here are some new questions... :-) What does HPET actually mean? In the BIOS I have found a Hammer Configuration menu in the Advanced section. In that there is a "ACPI 2.0 Static Resources Affinity Table for ccNUMA systems" option which I have switched off (till this time it was Enabled). I am curious if that helps... The timer problem often appeared already during the boot process of xen and sometimes it led to system hangup with "BUG: soft lockup detected on CPU#2" (the number 2 is just an example) messages... Here you can find some typical logs: http://www.mm.bme.hu/~szazs/tmp/xenlogs.php (there was no backup after midnight between Jan 07 and Jan 25 because of a typo in the dirvish.conf) On Jan 27 I could not reboot XEN because the backup partition get corrupted, so I have rebooted a simple debian kernel (2.6.24-etchnhalf-amd64) to rebuild-tree and to do some maintenance... and I saw that raid1 has problem with sdd3 so I removed it from the mirror. Since the reboot this morning it runs "smoothly" but I do not know whether it is because of either the BIOS change or the rebuilt backup partition or the kicked off mirror partition...> As for upgrading with rollback, this can be done by building the new > kernel and adding it as a separate option in your bootloader (eg. Grub). > I''m quite new to Kernel building and have only just got my first builds > working so you might want to wait for advice from a more experienced > user on that.So the changed libs cannot be a problem...? Are there any special things when considering an upgrade? (from 3.1.x to 3.2.x the network bridge setup scripts have changed, so I needed some time to figure out the new syntax) TIA, -- Zsolt _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Robert Dunkley
2009-Jan-28 11:28 UTC
[Xen-devel] RE: [Xen-users] howto upgrade - APIC / Timer issue??
Hi Zsolt, Your problem is different to the one I faced, it seems the older VX50 is not even using HPET timer but using the older ACPI PM timer instead. I would enable the ACPI option in the bios and try my boot options. Remember you can always add the new boot options as a separate menu entry in Grub in case they cause problems. The APIC debug option should provide more info in dmesg too. I think the problem is that only one APIC is being used for a 4 socket system (IMHO this is weird) so if the above boot options don''t help then I would post your system config, errors and xm dmesg ouput to the xen-devel mailing list, they could likely hint to where the problem is. I''ve copied this to the devel list for you. Rob -----Original Message----- From: SZABO Zsolt [mailto:szazs@mm.bme.hu] Sent: 28 January 2009 11:12 To: Robert Dunkley Subject: RE: [Xen-users] howto upgrade On Wed, 28 Jan 2009, Robert Dunkley wrote:> I don''t think the NUMA ACPI table is involved with this problem. Have > you tried enabling NUMA and ACPI in grub boot line?Not yet...> Below is my Kernel boot line: > title CentOS (2.6.18-92.1.6.el5xen330) > root (hd0,0) > kernel /xen.gz-3.3.0 apic_verbosity=debug dom0_mem=2048Macpi=on> numa=on cpufreq=dom0-kernelOh, should I directly enable it? I will try that tomorrow...> Notice I have enabled ACPI, Powernow and NUMA (NUMA requires a setDom0> memory and also disabling ballooning in xend-config.sxp) , this shouldHmm.. I do not understand well thet ballooning, too... (reading the other thread about high load)> be optimal for Opterons systems but enabling or disabling ACPI willhave> likely change the timer method entirely on your setup (Whichconsidering> your problems might well help). > > Can you please post the output from "xm dmesg" > > I''m wondering if yours contains something like this: > (XEN) Using local APIC timer interrupts. > (XEN) calibrating APIC timer ... > (XEN) Platform timer is 25.000MHz HPETNow (with disabled APIC SRAT in the BIOS), as I see, no... Here is the grub entry for xen: title Xen 3.2.1 / Debian GNU/Linux, kernel 2.6.18.8-xen root (hd0,1) kernel /boot/xen-3.2.1.gz console=vga vga=gfx-1024x768x8,keep module /boot/vmlinuz-2.6.18.8-xen root=/dev/md0 ro console=tty0 module /boot/initrd.img-2.6.18.8-xen (Btw., I am not satisfied with the console setup, too... maybe the keep option is not needed, but I did not do much trial with it) -- Zsolt __ __ _____ ____ _ \ \/ /___ _ __ |___ / |___ \ / | \ // _ \ ''_ \ |_ \ __) | | | / \ __/ | | | ___) | / __/ _| | /_/\_\___|_| |_| |____(_)_____(_)_| (XEN) Xen version 3.2.1 (root@mm.bme.hu) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) Tue Aug 5 18:23:52 CEST 2008 (XEN) Latest ChangeSet: unavailable (XEN) Command line: console=vga vga=gfx-1024x768x8,keep (XEN) Video information: (XEN) VGA is graphics mode 1024x768, 8 bpp (XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds (XEN) EDID info not retrieved because no DDC retrieval method detected (XEN) Disc information: (XEN) Found 4 MBR signatures (XEN) Found 4 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009cc00 (usable) (XEN) 000000000009cc00 - 00000000000a0000 (reserved) (XEN) 00000000000ce000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000007ff90000 (usable) (XEN) 000000007ff90000 - 000000007ff97000 (ACPI data) (XEN) 000000007ff97000 - 0000000080000000 (ACPI NVS) (XEN) 0000000080000000 - 00000000cdf00000 (usable) (XEN) 00000000cdf00000 - 00000000ce000000 (reserved) (XEN) 00000000e0000000 - 00000000f0000000 (reserved) (XEN) 00000000fec00000 - 00000000fec10000 (reserved) (XEN) 00000000fee00000 - 00000000fee01000 (reserved) (XEN) 00000000fff80000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 0000000432000000 (usable) (XEN) System RAM: 16382MB (16775344kB) (XEN) Xen heap: 14MB (14568kB) (XEN) Domain heap initialised: DMA width 32 bits (XEN) Processor #0 15:1 APIC version 16 (XEN) Processor #1 15:1 APIC version 16 (XEN) Processor #2 15:1 APIC version 16 (XEN) Processor #3 15:1 APIC version 16 (XEN) Processor #4 15:1 APIC version 16 (XEN) Processor #5 15:1 APIC version 16 (XEN) Processor #6 15:1 APIC version 16 (XEN) Processor #7 15:1 APIC version 16 (XEN) IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-23 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 2813.022 MHz processor. (XEN) AMD SVM: ASIDs disabled. (XEN) HVM: SVM enabled (XEN) CPU0: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 1/1 eip 8c000 (XEN) AMD: Disabling C1 Clock Ramping Node #0 (XEN) AMD: Disabling C1 Clock Ramping Node #1 (XEN) AMD: Disabling C1 Clock Ramping Node #2 Disabling C1 Clock Ramping Node #3 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU1: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 2/2 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU2: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 3/3 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU3: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 4/4 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU4: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 5/5 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU5: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 6/6 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU6: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Booting processor 7/7 eip 8c000 (XEN) AMD SVM: ASIDs disabled. (XEN) CPU7: AMD Dual-Core AMD Opteron(tm) Processor 8220 stepping 03 (XEN) Total of 8 processors activated. (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) Platform timer overflows in 234 jiffies. (XEN) Platform timer is 3.579MHz ACPI PM Timer (XEN) Brought up 8 CPUs (XEN) AMD IOMMU: Disabled (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff805b8c8c (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000420000000->0000000424000000 (4084129 pages to be allocated) (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff80200000->ffffffff805b8c8c (XEN) Init. ramdisk: ffffffff805b9000->ffffffff814c8800 (XEN) Phys-Mach map: ffffffff814c9000->ffffffff83411d08 (XEN) Start info: ffffffff83412000->ffffffff834124a4 (XEN) Page tables: ffffffff83413000->ffffffff83432000 (XEN) Boot stack: ffffffff83432000->ffffffff83433000 (XEN) TOTAL: ffffffff80000000->ffffffff83800000 (XEN) ENTRY ADDRESS: ffffffff80200000 (XEN) Dom0 has maximum 8 VCPUs (XEN) Initrd len 0xf0f800, start at 0xffffffff805b9000 (XEN) Scrubbing Free RAM: .done. (XEN) Xen trace buffers: disabled (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) Xen is keeping VGA console. (XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times to switch input to Xen) (XEN) Freed 100kB init memory. The SAQ Group Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ SAQ is the trading name of SEMTEC Limited. Registered in England & Wales Company Number: 06481952 http://www.saqnet.co.uk AS29219 SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. ISPA Member Find us in http://www.thebestof.co.uk/petersfield _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel