I always found the lvm2 ''snapshot'' terminology confusing - the
thing
created as a ''snapshot'' is what accepts changes while a backup
is made
of the original volume. However I have sucessfully followed the
following recipe (essentially as suggested in this list) for creating
lvm2 things which can be used as copy-on-write file systems for xenU
domains:
-----------------------------------------------------------------
# create a volume group
vgcreate vmgroup /dev/xxx /dev/yyy
# create a logical volume for a common root filesystem
lvcreate -L6G -n root_file_system vmgroup
mkfs -t ext3 /dev/vmgroup/root_file_system
# initialise the root_file_system - eg copy an existing install
mount /dev/vmgroup/root_file_system /mnt/new_root/
mount [partition containing pristine root fs] /dev/pristine_root
cp -ax /mnt/pristine_root/* /mnt/new_root
# create lvm2 ''snapshots'' allowing (eg) 512M of modification
each
lvcreate -L512M -s -n u1 /dev/vmgroup/root_file_system
lvcreate -L512M -s -n u2 /dev/vmgroup/root_file_system
lvcreate -L512M -s -n u3 /dev/vmgroup/root_file_system
# now we have 3 lvm2 things to use as COW file systems for xenU domains
black-magic-to-configure-and-start-a-domain u1
black-magic-to-configure-and-start-a-domain u2
black-magic-to-configure-and-start-a-domain u3
# drat - I needed another domain
lvcreate -L512M -s -n u4 /dev/vmgroup/root_file_system
... nasty messages .... all xenU domains dead ....
... lmv2 system in inconsistent state ...
... /dev/vmgroup/u4 doesn''t exist ...
... /dev/mapper/root_file_system-u4 does exist ...
# no other way of recovering that I know of
reboot
------------------------------------------------------------------------
After the reboot, it seems that the new ''snapshot'' u4 was
created, and
is in fact usable.
The nasty messages mention running out of memory, and complain that the
previously created ''snapshot'' (/dev/vmgroup/u3) is not
available.
The problem is that the ''snapshot'' cows hold onto each
other''s tails -
they seem to be held in a list linked (I think) from the original
logical volume (here /dev/vmgroup/root_file_system). For their intended
use as enabling backup, this seems to be meant to allow writes to the
original volume to be propagated to all ''snapshots'' created
against that
volume - there are comments about getting rid of the
''snapshots'' after
the backup has been done because this propagation of writes hits
performance.
For my requirements, and I imagine for most others reading this list,
all of this is superfluous. I don''t need
original -> snap1 -> snap2 -> snap3 ...
so that I can''t create a new snap4 while any of the others are in use.
I just need
original <- cow1
original <- cow2
original <- cow3
original <- cow4
...
where A ''<-'' B means B is a cow image of A, and where each
of the cows
is independent of the others so that a new cow can be created at any
time, regardless how many others are active.
Have I missed something?
In the worst case, this means defining a new kind of logical volume with
its own driver, and modifying the user space tools to deal with it.
Alternatively it may be possible to tweak only the lvm2 userspace tools.
Or has Bin Ren''s 2.4 series code been reconstituted for lvm2 and 2.6?
There was also mention of a cow file handler tweak to the kernel
root-nfs handler - did anything come of that?
In user-mode-linux and qemu, cow file handling is incorporated into the
emulator-system interface. Is there any scope for doing something like
that in xen0?
I haven''t yet looked at migration (partly because I don''t know
where to
find information about it). Does the choice of root file system
technology interact with migration? If so, is this an argument for
pushing cow stuff into the system so as to ensure the cows can migrate?
Incidentally I now get a string of messages about nbdNNN during a
reboot/shutdown - I didn''t get a chance to note them precisely. Is that
nbd as in network block device?
This message has got rather long. Thanks anyway.
Peri
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sun, Sep 26, 2004 at 12:38:06PM +0100, Peri Hankey wrote:> I always found the lvm2 ''snapshot'' terminology confusing - the thing > created as a ''snapshot'' is what accepts changes while a backup is made > of the original volume.I don''t think that''s the terminology the LVM2 people use. The regular use is to create a snapshot and backup this snapshot while you keep using the original.> # drat - I needed another domain > lvcreate -L512M -s -n u4 /dev/vmgroup/root_file_system > ... nasty messages .... all xenU domains dead .... > ... lmv2 system in inconsistent state ... > ... /dev/vmgroup/u4 doesn''t exist ... > ... /dev/mapper/root_file_system-u4 does exist ...This should work, if it doesn''t then it would seem to be a bug in LVM2. Since you mention out of memory error messages, are you sure that you''re not running out of memory in dom0?> The problem is that the ''snapshot'' cows hold onto each other''s tails - > they seem to be held in a list linked (I think) from the original > logical volume (here /dev/vmgroup/root_file_system). For their intended > use as enabling backup, this seems to be meant to allow writes to the > original volume to be propagated to all ''snapshots'' created against that > volume - there are comments about getting rid of the ''snapshots'' after > the backup has been done because this propagation of writes hits > performance. > > For my requirements, and I imagine for most others reading this list, > all of this is superfluous. I don''t need > > original -> snap1 -> snap2 -> snap3 ...This is not the layout LVM2 uses. If you look at the output of ``dmsetup table'''', you''ll see that each snapshot is independent and only refers to the device it is a snapshot of and to its cow device which will hold modifications.> so that I can''t create a new snap4 while any of the others are in use. > > I just need > > original <- cow1 > original <- cow2 > original <- cow3 > original <- cow4 > ... > > where A ''<-'' B means B is a cow image of A, and where each of the cows > is independent of the others so that a new cow can be created at any > time, regardless how many others are active.This is the layout LVM2 uses. And it is indeed simple (and should be quite robust) as long as you don''t want to write to the original. If you write to the original, you will have to copy the changed blocks to every snapshot''s cow device. I think I''ve seen this fail when having multiple snapshots and writing to the original. But since you didn''t write to the original (and one generally doesn''t need/want to write to the original in our case), that problem is unlikely to be relevant to the failure you''ve seen. christian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
It''s very encouraging to know that I misunderstood the situation. But I do have a problem: It may be that I am genuinely running out of memory, but I am also getting a lot of messages about nbdNNN, and the overall effect is pretty disastrous. I successfully created 2 new snapshots while another was in use as the root file system of a xenU domain. Previously this had failed immediately. Then the next lvcreate -s failed with a message about running out of memory, as before. I wonder whether the timing problem I have which affects rpm is also causing some problem? The relevant bit of /var/log/messages is: ... lots of nbdNNN messages ... Sep 26 19:47:33 a4 kernel: nbd126: Request when not-ready Sep 26 19:47:33 a4 kernel: end_request: I/O error, dev nbd126, sector 0 Sep 26 19:47:33 a4 kernel: nbd127: Request when not-ready Sep 26 19:47:33 a4 kernel: end_request: I/O error, dev nbd127, sector 0 Sep 26 19:48:26 a4 net.agent[7343]: add event not handled Sep 26 19:48:26 a4 kernel: device vif6.0 entered promiscuous mode Sep 26 19:48:27 a4 kernel: xen-br0: port 3(vif6.0) entering learning state Sep 26 19:48:27 a4 kernel: xen-br0: topology change detected, propagating Sep 26 19:48:27 a4 kernel: xen-br0: port 3(vif6.0) entering forwarding state Sep 26 19:50:00 a4 CROND[7360]: (mail) CMD (/usr/bin/python -S /usr/lib/mailman/cron/gate_news) Sep 26 19:53:40 a4 kernel: nbd0: Request when not-ready Sep 26 19:53:40 a4 kernel: end_request: I/O error, dev nbd0, sector 0 Sep 26 19:53:40 a4 kernel: nbd1: Request when not-ready Sep 26 19:53:40 a4 kernel: end_request: I/O error, dev nbd1, sector 0 Sep 26 19:53:40 a4 kernel: nbd2: Request when not-ready ... lots more nbdNNN messages ... Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd124, sector 0 Sep 26 19:53:44 a4 kernel: nbd125: Request when not-ready Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd125, sector 0 Sep 26 19:53:44 a4 kernel: nbd126: Request when not-ready Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd126, sector 0 Sep 26 19:53:44 a4 kernel: nbd127: Request when not-ready Sep 26 19:53:44 a4 kernel: end_request: I/O error, dev nbd127, sector 0 Sep 26 19:53:44 a4 kernel: lvcreate: page allocation failure. order:0, mode:0xd0 Sep 26 19:53:44 a4 kernel: [__alloc_pages+824/842] __alloc_pages+0x338/0x34a Sep 26 19:53:44 a4 kernel: [<c013a16f>] __alloc_pages+0x338/0x34a Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [kmem_cache_alloc+89/100] kmem_cache_alloc+0x59/0x64 Sep 26 19:53:44 a4 kernel: [<c013eb87>] kmem_cache_alloc+0x59/0x64 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [alloc_pl+48/76] alloc_pl+0x30/0x4c Sep 26 19:53:44 a4 kernel: [<c0328a54>] alloc_pl+0x30/0x4c Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [client_alloc_pages+40/129] client_alloc_pages+0x28/0x81 Sep 26 19:53:44 a4 kernel: [<c0328ba7>] client_alloc_pages+0x28/0x81 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [vmalloc+32/36] vmalloc+0x20/0x24 Sep 26 19:53:44 a4 kernel: [<c014f4d1>] vmalloc+0x20/0x24 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [kcopyd_client_create+104/199] kcopyd_client_create+0x68/0xc7 Sep 26 19:53:44 a4 kernel: [<c03296cd>] kcopyd_client_create+0x68/0xc7 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [dm_create_persistent+199/320] dm_create_persistent+0xc7/0x140 Sep 26 19:53:44 a4 kernel: [<c032b769>] dm_create_persistent+0xc7/0x140 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [snapshot_ctr+715/892] snapshot_ctr+0x2cb/0x37c Sep 26 19:53:44 a4 kernel: [<c0329f7f>] snapshot_ctr+0x2cb/0x37c Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [dm_table_add_target+323/464] dm_table_add_target+0x143/0x1d0 Sep 26 19:53:44 a4 kernel: [<c0324aa7>] dm_table_add_target+0x143/0x1d0 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [populate_table+125/216] populate_table+0x7d/0xd8 Sep 26 19:53:44 a4 kernel: [<c0327105>] populate_table+0x7d/0xd8 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [table_load+103/309] table_load+0x67/0x135 Sep 26 19:53:44 a4 kernel: [<c03271c7>] table_load+0x67/0x135 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [ctl_ioctl+236/330] ctl_ioctl+0xec/0x14a Sep 26 19:53:44 a4 kernel: [<c03279d3>] ctl_ioctl+0xec/0x14a Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [table_load+0/309] table_load+0x0/0x135 Sep 26 19:53:44 a4 kernel: [<c0327160>] table_load+0x0/0x135 Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [ctl_ioctl+0/330] ctl_ioctl+0x0/0x14a Sep 26 19:53:44 a4 kernel: [<c03278e7>] ctl_ioctl+0x0/0x14a Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [sys_ioctl+484/670] sys_ioctl+0x1e4/0x29e Sep 26 19:53:44 a4 kernel: [<c0167606>] sys_ioctl+0x1e4/0x29e Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: [syscall_call+7/11] syscall_call+0x7/0xb Sep 26 19:53:44 a4 kernel: [<c010d967>] syscall_call+0x7/0xb Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: device-mapper: : Could not create kcopyd client Sep 26 19:53:44 a4 kernel: Sep 26 19:53:44 a4 kernel: device-mapper: error adding target to table ... at this point lvm2 is unusable until after a reboot (unless some service can be restarted?) and xenU domains (or at least those with lvm2 snapshot based file systems) are dead. I certainly have no intention of writing to the original image. Thanks Peri Christian Limpach wrote:>On Sun, Sep 26, 2004 at 12:38:06PM +0100, Peri Hankey wrote: > > >>I always found the lvm2 ''snapshot'' terminology confusing - the thing >>created as a ''snapshot'' is what accepts changes while a backup is made >>of the original volume. >> >> > >I don''t think that''s the terminology the LVM2 people use. The regular >use is to create a snapshot and backup this snapshot while you keep >using the original. > > > >># drat - I needed another domain >>lvcreate -L512M -s -n u4 /dev/vmgroup/root_file_system >>... nasty messages .... all xenU domains dead .... >>... lmv2 system in inconsistent state ... >>... /dev/vmgroup/u4 doesn''t exist ... >>... /dev/mapper/root_file_system-u4 does exist ... >> >> > >This should work, if it doesn''t then it would seem to be a bug in >LVM2. Since you mention out of memory error messages, are you sure >that you''re not running out of memory in dom0? > > > >>The problem is that the ''snapshot'' cows hold onto each other''s tails - >>they seem to be held in a list linked (I think) from the original >>logical volume (here /dev/vmgroup/root_file_system). For their intended >>use as enabling backup, this seems to be meant to allow writes to the >>original volume to be propagated to all ''snapshots'' created against that >>volume - there are comments about getting rid of the ''snapshots'' after >>the backup has been done because this propagation of writes hits >>performance. >> >>For my requirements, and I imagine for most others reading this list, >>all of this is superfluous. I don''t need >> >> original -> snap1 -> snap2 -> snap3 ... >> >> > >This is not the layout LVM2 uses. If you look at the output of >``dmsetup table'''', you''ll see that each snapshot is independent >and only refers to the device it is a snapshot of and to its cow >device which will hold modifications. > > > >>so that I can''t create a new snap4 while any of the others are in use. >> >>I just need >> >> original <- cow1 >> original <- cow2 >> original <- cow3 >> original <- cow4 >> ... >> >>where A ''<-'' B means B is a cow image of A, and where each of the cows >>is independent of the others so that a new cow can be created at any >>time, regardless how many others are active. >> >> > >This is the layout LVM2 uses. And it is indeed simple (and should be >quite robust) as long as you don''t want to write to the original. >If you write to the original, you will have to copy the changed >blocks to every snapshot''s cow device. I think I''ve seen this >fail when having multiple snapshots and writing to the original. >But since you didn''t write to the original (and one generally doesn''t >need/want to write to the original in our case), that problem >is unlikely to be relevant to the failure you''ve seen. > > christian > > > >------------------------------------------------------- >This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 >Project Admins to receive an Apple iPod Mini FREE for your judgement on >who ports your project to Linux PPC the best. Sponsored by IBM. >Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I successfully created 2 new snapshots while another was in use as the > root file system of a xenU domain. Previously this had failed > immediately. Then the next lvcreate -s failed with a message about > running out of memory, as before. > > I wonder whether the timing problem I have which affects rpm is also > causing some problem? > > The relevant bit of /var/log/messages is: > > ... lots of nbdNNN messages ...Are you using NBD? Our default kernel has it compiled in, but it looks like some of your start up scripts are trying to initialise it, or you have some daemon that''s polling the devices. It looks like the error messages are hammering the machine pretty hard from time to time. It''s definitely worth figuring out what''s causing it. Or, compile nbd out of the kernel...> Sep 26 19:53:44 a4 kernel: lvcreate: page allocation failure. order:0, > mode:0xd0 > Sep 26 19:53:44 a4 kernel: [__alloc_pages+824/842]Seeing the output of /proc/slabinfo and /proc/meminfo might be interesting. And idea why the machine might be under memory pressure? Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
I''m not aware of anything that is trying to use NBD - I did experiment a bit a while back, but didn''t get anywhere. I thought xen might be using it behind the scenes in some way. I have attached some tar-zipped output which may help. lvm2-nbd-problem-20040916/ lvm2-nbd-problem-20040916/messages-log # messages during vchange -ay lvm2-nbd-problem-20040916/services-list # services that were enabled lvm2-nbd-problem-20040916/syslog-start # syslog start at reboot lvm2-nbd-problem-20040916/mem-slab-1 # before vchange -ay lvm2-nbd-problem-20040916/mem-slab-2 # after vchange -ay lvm2-nbd-problem-20040916/vgchange-ay-result # response to command lvm2-nbd-problem-20040916/cmdline # xen0 kernel command line xen version date: xen-2.0-20040924. I am puzzled that in /var/log/messages it almost immediately starts trying to access nbd devices - none are listed as existing in /dev. Also I''m not sure what the ''['' relates to. There is a note about the symbols matching 2.6.8, but I am running 2.6.8.1. Also it says kernel modules ere not enabled, but lsmod shows: ... # lsmod Module Size Used by nfsd 91720 8 exportfs 5248 1 nfsd Should I be doing anything about kernel symbols? I build and install straight out of the box with no special config: make world (as orndinary user) make install (as root) Too late. I hope this helps. Thanks Peri>>I successfully created 2 new snapshots while another was in use as the >>root file system of a xenU domain. Previously this had failed >>immediately. Then the next lvcreate -s failed with a message about >>running out of memory, as before. >> >>I wonder whether the timing problem I have which affects rpm is also >>causing some problem? >> >>The relevant bit of /var/log/messages is: >> >>... lots of nbdNNN messages ... >> >> > >Are you using NBD? Our default kernel has it compiled in, but it >looks like some of your start up scripts are trying to initialise >it, or you have some daemon that''s polling the devices. > >It looks like the error messages are hammering the machine pretty >hard from time to time. It''s definitely worth figuring out what''s >causing it. Or, compile nbd out of the kernel... > > > >>Sep 26 19:53:44 a4 kernel: lvcreate: page allocation failure. order:0, >>mode:0xd0 >>Sep 26 19:53:44 a4 kernel: [__alloc_pages+824/842] >> >> > >Seeing the output of /proc/slabinfo and /proc/meminfo might be >interesting. And idea why the machine might be under memory pressure? > >Ian > > >------------------------------------------------------- >This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 >Project Admins to receive an Apple iPod Mini FREE for your judgement on >who ports your project to Linux PPC the best. Sponsored by IBM. >Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >
> I''m not aware of anything that is trying to use NBD - I did experiment a > bit a while back, but didn''t get anywhere. I thought xen might be using > it behind the scenes in some way.Xen doesn''t use nbd. I bet you have something in /etc/rc.d/init.d that is starting nbd. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
I rebuilt with todays xen-2.0-src: ... untar and rename todays download ... make mrproper make world su make install reboot The first few lines in /var/log/messages show that something is immediately trying to use nbd devices, way before any service in /etc/init.d/ can get a sniff at it. I think nbd used to be a module the xen0 config, but I see it''s now compiled into the kernel - could there be a device number confusion somewhere? According to devices.txt network block devices have block major 43. ---- start of boot sequence in /var/log/messages ---- ---- I don''t know where ''or 0'' comes from ------------ Sep 27 08:23:29 a4 syslogd 1.4.1: restart. Sep 27 08:23:29 a4 kernel: klogd 1.4.1, log source = /proc/kmsg started. Sep 27 08:23:29 a4 kernel: Inspecting /boot/System.map-2.6.8.1-xen0 Sep 27 08:23:29 a4 partmon: Checking if partitions have enough free diskspace: Sep 27 08:23:30 a4 kernel: Loaded 19997 symbols from /boot/System.map-2.6.8.1-xen0. Sep 27 08:23:30 a4 kernel: Symbols match kernel version 2.6.8. Sep 27 08:23:30 a4 kernel: No module symbols loaded - kernel modules not enabled. Sep 27 08:23:30 a4 kernel: or 0 Sep 27 08:23:30 a4 kernel: nbd60: Request when not-ready Sep 27 08:23:30 a4 kernel: end_request: I/O error, dev nbd60, sector 0 Sep 27 08:23:30 a4 kernel: nbd61: Request when not-ready Sep 27 08:23:30 a4 kernel: end_request: I/O error, dev nbd61, sector 0 ... ------------------------------------------------------------------------------------ Thanks Peri Ian Pratt wrote:>>I''m not aware of anything that is trying to use NBD - I did experiment a >>bit a while back, but didn''t get anywhere. I thought xen might be using >>it behind the scenes in some way. >> >> > >Xen doesn''t use nbd. I bet you have something in >/etc/rc.d/init.d that is starting nbd. > >Ian > > >------------------------------------------------------- >This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 >Project Admins to receive an Apple iPod Mini FREE for your judgement on >who ports your project to Linux PPC the best. Sponsored by IBM. >Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
A further point - I mentioned in my last mail that attempts to access an nbd device start very early in the boot sequence. I now also notice after looking at several boot sequences that the first seems always to be an attempt to access nbd60, sector 0 (but the sector 0 may be a red herring). I suspect something is trying to acccess block device 43,60 hoping to find something other than a non-existent nbd device). -- Peri Ian Pratt wrote:>>I''m not aware of anything that is trying to use NBD - I did experiment a >>bit a while back, but didn''t get anywhere. I thought xen might be using >>it behind the scenes in some way. >> >> > >Xen doesn''t use nbd. I bet you have something in >/etc/rc.d/init.d that is starting nbd. > >Ian > > >------------------------------------------------------- >This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 >Project Admins to receive an Apple iPod Mini FREE for your judgement on >who ports your project to Linux PPC the best. Sponsored by IBM. >Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > > >------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> The first few lines in /var/log/messages show that something is > immediately trying to use nbd devices, way before any service in > /etc/init.d/ can get a sniff at it.I can''t deduce that from your output, which doesn''t contain other kernel messages to put it in context.> I think nbd used to be a module the xen0 config, but I see it''s now > compiled into the kernel - could there be a device number confusion > somewhere? According to devices.txt network block devices have block > major 43.I don''t believe we''re using block major 43. I still think these messages are most likely being caused by something in your command line, init.d, fstab etc. Ian> ---- start of boot sequence in /var/log/messages ---- > ---- I don''t know where ''or 0'' comes from ------------ > Sep 27 08:23:29 a4 syslogd 1.4.1: restart. > Sep 27 08:23:29 a4 kernel: klogd 1.4.1, log source = /proc/kmsg started. > Sep 27 08:23:29 a4 kernel: Inspecting /boot/System.map-2.6.8.1-xen0 > Sep 27 08:23:29 a4 partmon: Checking if partitions have enough free > diskspace: > Sep 27 08:23:30 a4 kernel: Loaded 19997 symbols from > /boot/System.map-2.6.8.1-xen0. > Sep 27 08:23:30 a4 kernel: Symbols match kernel version 2.6.8. > Sep 27 08:23:30 a4 kernel: No module symbols loaded - kernel modules not > enabled. > Sep 27 08:23:30 a4 kernel: or 0 > Sep 27 08:23:30 a4 kernel: nbd60: Request when not-ready > Sep 27 08:23:30 a4 kernel: end_request: I/O error, dev nbd60, sector 0 > Sep 27 08:23:30 a4 kernel: nbd61: Request when not-ready > Sep 27 08:23:30 a4 kernel: end_request: I/O error, dev nbd61, sector 0 > ...------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
That looks like it. The xen0 default kernel config used to build nbd as a module. That meant that there were no nbd devices available at boot or at any time until you explictly loaded the nbd module. So they weren''t visible to the lvm code that scans block devices. With the nbd devices available, the lvm stuff starts happily scanning and rescanning all 128 of them. So far I have managed to show this by inadvertently putting a syntax error in lvm.conf - my attempts to reject nb* and accept others continued to scan everything. But I think I''m on the right track. Thanks -- Peri Wm wrote:>On Mon, Sep 27, 2004 at 09:40:49AM +0100, Peri Hankey wrote: > > >>A further point - I mentioned in my last mail that attempts to access an >>nbd device start very early in the boot sequence. I now also notice >>after looking at several boot sequences that the first seems always to >>be an attempt to access nbd60, sector 0 (but the sector 0 may be a red >>herring). I suspect something is trying to acccess block device 43,60 >>hoping to find something other than a non-existent nbd device). >> >> > >Could it be the lvm / device mapper initalistation scripts seaching for >block devices with pvs on them? > >Can you have a look in /etc/lvm/lvm.conf in there you can tell >lvm to only scan devices that you know might have pvs on them. > > >bill > > > >------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Although I did get quite a lot wrong in my analysis, I come back to the
point that as lvm2 creates additional snapshots of a the same original,
it adds them to a list of snapshots of that original. If something goes
wrong while a new snapshot is being added, the effect in my experience
is pretty bad: all of the other snapshots of the same original become
unusable and you lose control of xenU domains that have their root
filesystems in those snapshots.
There may be a bug in lvm2 in the handling of error conditions when
adding a new snapshot. But if there are linkages between snapshots of
the same original it is going to be pretty hard to guarantee that the
snapshots are properly independent of each other and proof against this
kind of snarlup.
You can see what I mean if you look at the code:
drivers/md/snap.c: line 76
/*
* One of these per registered origin, held in the snapshot_origins hash
*/
struct origin {
/* The origin device */
struct block_device *bdev;
struct list_head hash_list;
/* List of snapshots for this origin */
struct list_head snapshots;
};
drivers/md/snap.c: line 922 +
static int __origin_write(struct list_head *snapshots, struct bio *bio)
{
... code which propagates a write to the original to all the
snapshots on that original ...
}
Of course I would be delighted to be wrong, or to find that a little
bugfix or two in the lvm code will do the trick. But it feels wrong to
have any connection between the independent cow images, and it probably
explains why we can''t have cow image of cow images, which
don''t really
present a problem if all the linkages refer backwards from the cow image
to the original. Not that I see a violent need for cow pyramid structures.
-- Peri
Christian Limpach wrote:
>On Sun, Sep 26, 2004 at 12:38:06PM +0100, Peri Hankey wrote:
>
>
>>I always found the lvm2 ''snapshot'' terminology
confusing - the thing
>>created as a ''snapshot'' is what accepts changes while
a backup is made
>>of the original volume.
>>
>>
>
>I don''t think that''s the terminology the LVM2 people use.
The regular
>use is to create a snapshot and backup this snapshot while you keep
>using the original.
>
>
>
>># drat - I needed another domain
>>lvcreate -L512M -s -n u4 /dev/vmgroup/root_file_system
>>... nasty messages .... all xenU domains dead ....
>>... lmv2 system in inconsistent state ...
>>... /dev/vmgroup/u4 doesn''t exist ...
>>... /dev/mapper/root_file_system-u4 does exist ...
>>
>>
>
>This should work, if it doesn''t then it would seem to be a bug in
>LVM2. Since you mention out of memory error messages, are you sure
>that you''re not running out of memory in dom0?
>
>
>
>>The problem is that the ''snapshot'' cows hold onto each
other''s tails -
>>they seem to be held in a list linked (I think) from the original
>>logical volume (here /dev/vmgroup/root_file_system). For their intended
>>use as enabling backup, this seems to be meant to allow writes to the
>>original volume to be propagated to all ''snapshots''
created against that
>>volume - there are comments about getting rid of the
''snapshots'' after
>>the backup has been done because this propagation of writes hits
>>performance.
>>
>>For my requirements, and I imagine for most others reading this list,
>>all of this is superfluous. I don''t need
>>
>> original -> snap1 -> snap2 -> snap3 ...
>>
>>
>
>This is not the layout LVM2 uses. If you look at the output of
>``dmsetup table'''', you''ll see that each snapshot
is independent
>and only refers to the device it is a snapshot of and to its cow
>device which will hold modifications.
>
>
>
>>so that I can''t create a new snap4 while any of the others are
in use.
>>
>>I just need
>>
>> original <- cow1
>> original <- cow2
>> original <- cow3
>> original <- cow4
>> ...
>>
>>where A ''<-'' B means B is a cow image of A, and
where each of the cows
>>is independent of the others so that a new cow can be created at any
>>time, regardless how many others are active.
>>
>>
>
>This is the layout LVM2 uses. And it is indeed simple (and should be
>quite robust) as long as you don''t want to write to the original.
>If you write to the original, you will have to copy the changed
>blocks to every snapshot''s cow device. I think I''ve seen
this
>fail when having multiple snapshots and writing to the original.
>But since you didn''t write to the original (and one generally
doesn''t
>need/want to write to the original in our case), that problem
>is unlikely to be relevant to the failure you''ve seen.
>
> christian
>
>
>
>
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
On Mon, Sep 27, 2004 at 02:35:49PM +0100, Peri Hankey wrote:> Although I did get quite a lot wrong in my analysis, I come back to the > point that as lvm2 creates additional snapshots of a the same original, > it adds them to a list of snapshots of that original. If something goes > wrong while a new snapshot is being added, the effect in my experience > is pretty bad: all of the other snapshots of the same original become > unusable and you lose control of xenU domains that have their root > filesystems in those snapshots. > > There may be a bug in lvm2 in the handling of error conditions when > adding a new snapshot. But if there are linkages between snapshots of > the same original it is going to be pretty hard to guarantee that the > snapshots are properly independent of each other and proof against this > kind of snarlup. > > You can see what I mean if you look at the code:As said before, this is needed to allow writes to the original and it has to be a bug in error handling if snapshot creation failing renders all snapshots of the same original lv unusable. It seems to me that you have 2 options: - fix the error handling - create a new snapshot module which doesn''t allow writes to the original I don''t see the benefits of the 2nd option, but nobody''s stopping you ;-) christian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel