Xen users - Sep 2008 - Increase size of file-based diskimage (with MBR, partitions + fs)

Hello all,

I have been struggling with a problem for some time and haven''t quite
found a working solution. Tried to search the mailing list and other
online resources but still got stuck.

What I have: a diskimage stored in the fs on dom0. The image is not
just a filesystem, but contains a full disk presented to the domU. Thus
it contains an MBR, partition table and a single ext3 filesystem. No 
LVM is not used in the domU. Both dom0 and domU is Centos 5.2.

What I want to accomplish: increase the size of this diskimage, resize
the partition and filesystem.

Some basic facts and proof of valid diskimage from dom0:

 # Partition table
 
 > sfdisk -l xvda.img
 [...]
 Disk xvda.img: 195 cylinders, 255 heads, 63 sectors/track
 Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
    Device Boot Start     End   #cyls    #blocks   Id  System
    xvda.img1   *      0+    194     195-   1566306   83  Linux
 [...]

 # Verify partition table

 > losetup /dev/loop15 xvda.img
 > sfdisk -V /dev/loop15
 /dev/loop15: OK
 > losetup -d /dev/loop15

 # Verify filesystem

 > losetup /dev/loop16 -o `expr 63 \* 512` xvda.img
 > e2fsck -f /dev/loop16
 e2fsck 1.39 (29-May-2006)
 Pass 1: Checking inodes, blocks, and sizes
 Pass 2: Checking directory structure
 Pass 3: Checking directory connectivity
 Pass 4: Checking reference counts
 Pass 5: Checking group summary information
 /: 32573/391680 files (2.8% non-contiguous), 265879/391576 blocks
 > losetup -d /dev/loop16

I can boot this image correctly using pygrub. The configuration is
simple (xvda is the fs, xvdb is the swap):

 # /etc/xen/test.cfg
 name = "test"
 memory = 256
 disk = [ ''tap:aio:/var/lib/xen/images/test/xvda.img,xvda,w'',
          ''tap:aio:/var/lib/xen/images/test/xvdb.img,xvdb,w'',
]
 vif = [ ''mac=00:16:3E:4B:79:7B,bridge=xenbr1'', ]
 bootloader = "/usr/bin/pygrub"
 vcpus = 1
 on_reboot = ''restart''
 on_crash = ''restart''

Ok, so now on to what I have tried and failed (with a known good image
as verified above):

 # Extend the size of diskimage from 1.5GB to 3GB

 > ls -la xvda.img 
 -rw-r--r-- 1 root root 1610612736 Oct  1 00:16 xvda.img
 > dd if=/dev/zero of=xvda.img bs=1 count=1 seek=3G conv=notrunc
 1+0 records in
 1+0 records out
 1 byte (1 B) copied, 0.00506 seconds, 0.2 kB/s
 > ls -la xvda.img
 -rw-r--r-- 1 root root 3221225473 Oct  1 00:25 xvda.img

 # Now rewrite the partition table to span the entire disk, also
 # make sure that the partition is still bootable

 > sfdisk -q --no-reread xvda.img << EOF
 ,,,*
 EOF
 > sfdisk -l xvda.img
 Disk xvda.img: 391 cylinders, 255 heads, 63 sectors/track
 Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
   Device Boot Start     End   #cyls    #blocks   Id  System
   xvda.img1   *      0+    390     391-   3140707   83  Linux

 # Resize the filesystem to span the entire partition, also verify the
 # filesystem

 > losetup /dev/loop16 -o `expr 63 \* 512` xvda.img
 > e2fsck -f /dev/loop16
 [...passed]
 > resize2fs /dev/loop16
 resize2fs 1.39 (29-May-2006)
 Resizing the filesystem on /dev/loop16 to 786424 (4k) blocks.
 The filesystem on /dev/loop16 is now 786424 blocks long.
 > e2fsck -f /dev/loop16
 [...passed]
 > sfdisk -V xvda.img
 xvda.img: OK

So far so good! I can mount the filesystem and verify it to be 3GB. Now
on to trying to boot the domU again:

 > xm create -c hc24.cfg
 Using config file "./test.cfg".
 Traceback (most recent call last):
   File "/usr/bin/pygrub", line 651, in ?
       fs = fsimage.open(file, get_fs_offset(file))
       IOError: [Errno 95] Operation not supported
       No handlers could be found for logger "xend"
       Error: Boot loader didn''t return any data!

If I understand this correctly the diskimage does not contain a valid
MBR that satisfies pygrub. I have tried two things to solve this:

TEST1

 # Try to copy the MBR from backup to the new diskimage

 > dd if=oldxvda.img of=xvda.img count=1 bs=446 conv=notrunc

As I understand it the first 446 bytes contain the MBR, while the first
512 bytes would also copy the old partition table. Trying to boot this
disk image gives the same error message as before.

TEST2

 # Try to create a new MBR on the disk, but for whatever reason GRUB
 # cannot recognize the filesystem, even though it finds the partition
 # type

 > losetup /dev/loop15 xvda.img
 > grub --batch --device-map=/dev/null
 [...]
 grub> device (hd0) /dev/loop15
 device (hd0) /dev/loop15
 grub> root (hd0,0)
  Filesystem type unknown, partition type 0x83
 grub> setup (hd0)
  Error 17: Cannot mount selected partition
 grub> quit
 > losetup -d /dev/loop15

So this fails miserably and would of course not start either.

Folks, what am I missing?

Thanks,
Henrik

PS. Thanks for reading this whole thing... ;(

-- 
Henrik Holmboe, Stockholm, Sweden
<http://henrik.holmboe.se/contact/>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Sep 30, 2008 at 6:54 PM, Henrik Holmboe <henrik@holmboe.se>
wrote:
> Hello all,
>
> I have been struggling with a problem for some time and haven''t
quite
> found a working solution. Tried to search the mailing list and other
> online resources but still got stuck.
>
> What I have: a diskimage stored in the fs on dom0. The image is not
> just a filesystem, but contains a full disk presented to the domU. Thus
> it contains an MBR, partition table and a single ext3 filesystem. No
> LVM is not used in the domU. Both dom0 and domU is Centos 5.2.
>
> What I want to accomplish: increase the size of this diskimage, resize
> the partition and filesystem.
>
There are a couple ways that are slight variations on the things that you
have tried that have worked for me.

Simplest way, if it works, is to simply create a "disk image" with dd
and append it onto your disk image.

Something like:

dd if=/dev/zero of=space.img bs=1M seek=1024k count=1
cat space.img >> your_small_image.img

Then what you have is unallocated space at the end of your
image file, which you can then resize the partitions the way
you want to get them to use the newly added space

For linux file systems that can be as easy as using losetup
and/or kpartx to associate the disk partitions with device
nodes and then using standard resize tools. Alternatively
gparted should also be able to handle the virtual device
after it has been associated with a device node, if you
prefer a graphically tool. For windows file systems you
can simply use the Manage Computer option (right click
My Computer --> Manage) and then resize the partitions
with the wizard.

If that way doesn''t work, using kpartx/losetup to associate
the image is again a good first step. Then, create a new
blank bigger image and use losetup to associate it with
a loop device, use fdisk to partition it, and then use dd
with the "if" equal to the source disk image and "of"
equal to the destination disk image. Using conv=notrunc

You may then also need to do resizing of partition as above.
>From dd man page:
Specify the output pathname; the default is standard output. If the
seek=expr conversion is not also specified, the output file will be
truncated before the copy begins, unless conv=notrunc is specified. If
seek=expr is specified, but conv=notrunc is not, the effect of the
copy will be to preserve the blocks in the output file over which dd
seeks, but no other portion of the output file will be preserved. (If
the size of the seek plus the size of the input file is less than the
previous size of the output file, the output file will be shortened by
the copy.)

Hope that helps some,
Cheers,
Todd

-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello again,

and thanks for your answers. I reviewed the differences in your methods
and mine, and noticed two big differences. One was the way that you
appended extra space at the end of the diskimage, and the second was how
you repartitioned the diskimage afterwards. See below.

Short version: use fdisk instead of sfdisk.

++ 01/10/08 00:54 +0200 - Henrik Holmboe:

[...]
>What I have: a diskimage stored in the fs on dom0. The image is not
>just a filesystem, but contains a full disk presented to the domU. Thus
>it contains an MBR, partition table and a single ext3 filesystem. No 
>LVM is not used in the domU. Both dom0 and domU is Centos 5.2.
>
>What I want to accomplish: increase the size of this diskimage, resize
>the partition and filesystem.
[...]
>Ok, so now on to what I have tried and failed (with a known good image
>as verified above):
>
> # Extend the size of diskimage from 1.5GB to 3GB
>
> > ls -la xvda.img 
> -rw-r--r-- 1 root root 1610612736 Oct  1 00:16 xvda.img
> > dd if=/dev/zero of=xvda.img bs=1 count=1 seek=3G conv=notrunc
> 1+0 records in
> 1+0 records out
> 1 byte (1 B) copied, 0.00506 seconds, 0.2 kB/s
> > ls -la xvda.img
> -rw-r--r-- 1 root root 3221225473 Oct  1 00:25 xvda.img
I noticed that both Franz and used something similar ot

 dd if=/dev/zero of=space.img bs=1 count=1 seek=1536M
 cat space.img >> xvda.img

rather than the method I used above. I have verified that both methods
work similarly and produce the same results.
> # Now rewrite the partition table to span the entire disk, also
> # make sure that the partition is still bootable
>
> > sfdisk -q --no-reread xvda.img << EOF
> ,,,*
> EOF
> > sfdisk -l xvda.img
> Disk xvda.img: 391 cylinders, 255 heads, 63 sectors/track
> Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
>   Device Boot Start     End   #cyls    #blocks   Id  System
>   xvda.img1   *      0+    390     391-   3140707   83  Linux
The solution to the problem was here. I used sfdisk, and when using
fdisk the diskimage is actually bootable. So instead of the above, I did
this

 # Get the number of cylinders (in my case 391)
 sfdisk -l xvda.img

 # Delete the partition, recreate a bigger partition using the same
 # start cylinder, and finally write the partition table. My diskimage
 # only contains a single partition (no swap, no additional filesystem
 # partitions), so the steps in my case were this

 fdisk xvda.img
 # Enter expert mode
 x
 # Specify number of cylinders
 c
 391
 # Return to main menu
 r
 # Delete partition
 d
 # New primary partition, starts on cylinder 1, ends on cylinder 391
 # which is the entire available space
 n
 p
 1
 1
 391
 # Make this partition bootable
 a
 1
 # Write and exit
 w

After this I had to extend the filesystem to span the added space to the
partition. Nothing changed here that affected the result from my initial
procedure, but for completeness sake I will include it here.

 losetup /dev/loop15 xvda.img
 kpartx -a /dev/loop15
 e2fsck -f /dev/mapper/loop15p1
 resize2fs /dev/mapper/loop15p1
 e2fsck -f /dev/mapper/loop15p1
 kpartx -d /dev/loop15
 losetup -d /dev/loop15

The next step would be to see why sfdisk is not working correctly. The
diskimage verified correctly but pygrub couldn''t read it. Weird.

Thanks again!

Henrik

-- 
Henrik Holmboe, Stockholm, Sweden
<http://henrik.holmboe.se/contact/>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users