Xen devel - May 2006 - [PATCH] [XM-TEST] block device write verify test 2nd attempt

The only difference from the last version of the patch is that the minor
version number in configure.ac is incremented.
>From the patch:
+# This test imports a ram disk device as a physical device into a domU.
+# The domU initialises the ram disk with data from /dev/urandom and
calculates
+# the md5 checksum of the data (using tee as it is written so as to
avoid
+# reading it back from the device which might potentially mask
problems).
+# The domU is stopped and the md5 checksum of the data on the device is
+# calculated by dom0.  The test succeeds if the checksums match,
indicating
+# that all the data written by domU was sucessfully committed to the
device.
+

This patch also enables tee and fancy head in busybox on the ramdisk.  I
have tested the patch with both `make existing'' where the tests run but
the new test fails because the ramdisk is missing tee and fancy head and
`make` where the test passes successfully.

The patch also installs the block-integrity tests in the default test
set so they get executed.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, May 23, 2006 at 11:35:55AM +0100, Harry Butterworth wrote:
> The only difference from the last version of the patch is that the minor
> version number in configure.ac is incremented.
> 
> >From the patch:
> 
> +# This test imports a ram disk device as a physical device into a domU.
> +# The domU initialises the ram disk with data from /dev/urandom and
> calculates
> +# the md5 checksum of the data (using tee as it is written so as to
> avoid
> +# reading it back from the device which might potentially mask
> problems).
> +# The domU is stopped and the md5 checksum of the data on the device is
> +# calculated by dom0.  The test succeeds if the checksums match,
> indicating
> +# that all the data written by domU was sucessfully committed to the
> device.
> +
> 
> This patch also enables tee and fancy head in busybox on the ramdisk.  I
> have tested the patch with both `make existing'' where the tests
run but
> the new test fails because the ramdisk is missing tee and fancy head and
> `make` where the test passes successfully.
Why don''t you use dd instead of head -c?

Why don''t you just fix the size of the datablock that you write to the
ramdisk, instead of determining the current size of the ramdisk with cat
/dev/hda1 | wc -c?

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2006-05-23 at 12:40 +0100, Ewan Mellor wrote:
> On Tue, May 23, 2006 at 11:35:55AM +0100, Harry Butterworth wrote:
> 
> > The only difference from the last version of the patch is that the
minor
> > version number in configure.ac is incremented.
> > 
> > >From the patch:
> > 
> > +# This test imports a ram disk device as a physical device into a
domU.
> > +# The domU initialises the ram disk with data from /dev/urandom and
> > calculates
> > +# the md5 checksum of the data (using tee as it is written so as to
> > avoid
> > +# reading it back from the device which might potentially mask
> > problems).
> > +# The domU is stopped and the md5 checksum of the data on the device
is
> > +# calculated by dom0.  The test succeeds if the checksums match,
> > indicating
> > +# that all the data written by domU was sucessfully committed to the
> > device.
> > +
> > 
> > This patch also enables tee and fancy head in busybox on the ramdisk. 
I
> > have tested the patch with both `make existing'' where the
tests run but
> > the new test fails because the ramdisk is missing tee and fancy head
and
> > `make` where the test passes successfully.
> 
> Why don''t you use dd instead of head -c?
I tried using dd with a block size of 1 and a count of the right number
of bytes but the test was very slow.  I didn''t want to assume a 512b
block size and I''m not very good at shell script so didn''t
manage to
work out how to do it better.
> Why don''t you just fix the size of the datablock that you write to
the
> ramdisk, instead of determining the current size of the ramdisk with cat
> /dev/hda1 | wc -c?
I wanted to test writing at the device limits.  Sometimes there are off
by one errors that mean you can''t write the last sector of a block
device.

cat | wc -c was the most robust way I could think of for getting the
size.
> 
> Ewan.
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Signed-off-by: Harry Butterworth <butterwo@uk.ibm.com>

On Tue, 2006-05-23 at 11:35 +0100, Harry Butterworth
wrote:
> The only difference from the last version of the patch is that the minor
> version number in configure.ac is incremented.
> 
> >From the patch:
> 
> +# This test imports a ram disk device as a physical device into a domU.
> +# The domU initialises the ram disk with data from /dev/urandom and
> calculates
> +# the md5 checksum of the data (using tee as it is written so as to
> avoid
> +# reading it back from the device which might potentially mask
> problems).
> +# The domU is stopped and the md5 checksum of the data on the device is
> +# calculated by dom0.  The test succeeds if the checksums match,
> indicating
> +# that all the data written by domU was sucessfully committed to the
> device.
> +
> 
> This patch also enables tee and fancy head in busybox on the ramdisk.  I
> have tested the patch with both `make existing'' where the tests
run but
> the new test fails because the ramdisk is missing tee and fancy head and
> `make` where the test passes successfully.
> 
> The patch also installs the block-integrity tests in the default test
> set so they get executed.
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, May 23, 2006 at 01:35:35PM +0100, Harry Butterworth wrote:
> On Tue, 2006-05-23 at 12:40 +0100, Ewan Mellor wrote:
> > On Tue, May 23, 2006 at 11:35:55AM +0100, Harry Butterworth wrote:
> > 
> > > The only difference from the last version of the patch is that
the minor
> > > version number in configure.ac is incremented.
> > > 
> > > >From the patch:
> > > 
> > > +# This test imports a ram disk device as a physical device into
a domU.
> > > +# The domU initialises the ram disk with data from /dev/urandom
and
> > > calculates
> > > +# the md5 checksum of the data (using tee as it is written so as
to
> > > avoid
> > > +# reading it back from the device which might potentially mask
> > > problems).
> > > +# The domU is stopped and the md5 checksum of the data on the
device is
> > > +# calculated by dom0.  The test succeeds if the checksums match,
> > > indicating
> > > +# that all the data written by domU was sucessfully committed to
the
> > > device.
> > > +
> > > 
> > > This patch also enables tee and fancy head in busybox on the
ramdisk.  I
> > > have tested the patch with both `make existing'' where
the tests run but
> > > the new test fails because the ramdisk is missing tee and fancy
head and
> > > `make` where the test passes successfully.
> > 
> > Why don''t you use dd instead of head -c?
> 
> I tried using dd with a block size of 1 and a count of the right number
> of bytes but the test was very slow.  I didn''t want to assume a
512b
> block size and I''m not very good at shell script so
didn''t manage to
> work out how to do it better.
dd bs=<number> count=1 should do just fine.
> > Why don''t you just fix the size of the datablock that you
write to the
> > ramdisk, instead of determining the current size of the ramdisk with
cat
> > /dev/hda1 | wc -c?
> 
> I wanted to test writing at the device limits.  Sometimes there are off
> by one errors that mean you can''t write the last sector of a block
> device.
> 
> cat | wc -c was the most robust way I could think of for getting the
> size.
How about blockdev --getsize64 /dev/ram1?

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2006-05-24 at 11:20 +0100, Ewan Mellor wrote:
> On Tue, May 23, 2006 at 01:35:35PM +0100, Harry Butterworth wrote:
> 
> > On Tue, 2006-05-23 at 12:40 +0100, Ewan Mellor wrote:
> > > On Tue, May 23, 2006 at 11:35:55AM +0100, Harry Butterworth
wrote:
> > > 
> > > > The only difference from the last version of the patch is
that the minor
> > > > version number in configure.ac is incremented.
> > > > 
> > > > >From the patch:
> > > > 
> > > > +# This test imports a ram disk device as a physical device
into a domU.
> > > > +# The domU initialises the ram disk with data from
/dev/urandom and
> > > > calculates
> > > > +# the md5 checksum of the data (using tee as it is written
so as to
> > > > avoid
> > > > +# reading it back from the device which might potentially
mask
> > > > problems).
> > > > +# The domU is stopped and the md5 checksum of the data on
the device is
> > > > +# calculated by dom0.  The test succeeds if the checksums
match,
> > > > indicating
> > > > +# that all the data written by domU was sucessfully
committed to the
> > > > device.
> > > > +
> > > > 
> > > > This patch also enables tee and fancy head in busybox on the
ramdisk.  I
> > > > have tested the patch with both `make existing''
where the tests run but
> > > > the new test fails because the ramdisk is missing tee and
fancy head and
> > > > `make` where the test passes successfully.
> > > 
> > > Why don''t you use dd instead of head -c?
> > 
> > I tried using dd with a block size of 1 and a count of the right
number
> > of bytes but the test was very slow.  I didn''t want to assume
a 512b
> > block size and I''m not very good at shell script so
didn''t manage to
> > work out how to do it better.
> 
> dd bs=<number> count=1 should do just fine.
The ram disk is 64MB, I thought that this would require a 64MB buffer in
domU which I thought was too big.  It would be better to use dd with the
correct block size and count of blocks.
> 
> > > Why don''t you just fix the size of the datablock that
you write to the
> > > ramdisk, instead of determining the current size of the ramdisk
with cat
> > > /dev/hda1 | wc -c?
> > 
> > I wanted to test writing at the device limits.  Sometimes there are
off
> > by one errors that mean you can''t write the last sector of a
block
> > device.
> > 
> > cat | wc -c was the most robust way I could think of for getting the
> > size.
> 
> How about blockdev --getsize64 /dev/ram1?
blockdev and stat are not available in the ramdisk.  I could use
blockdev to get the blocksize and number of blocks in dom0 and then pass
the values to domU but it would be better to obtain all the information
in domU since that would test whether the size of the device is
correctly passed from the back to the front by the blk driver.

The test as it stands does check that aspect of the blk driver operation
and is only a factor of 2 slower than the limit and the python code is
trivial so I thought it was reasonable.

Can you think of any way to get the block size and number of blocks
using tools available on the ramdisk?
> 
> Ewan.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, May 24, 2006 at 12:08:28PM +0100, Harry Butterworth wrote:
> > > > Why don''t you just fix the size of the datablock
that you write to the
> > > > ramdisk, instead of determining the current size of the
ramdisk with cat
> > > > /dev/hda1 | wc -c?
> > > 
> > > I wanted to test writing at the device limits.  Sometimes there
are off
> > > by one errors that mean you can''t write the last sector
of a block
> > > device.
> > > 
> > > cat | wc -c was the most robust way I could think of for getting
the
> > > size.
> > 
> > How about blockdev --getsize64 /dev/ram1?
> 
> blockdev and stat are not available in the ramdisk.  I could use
> blockdev to get the blocksize and number of blocks in dom0 and then pass
> the values to domU but it would be better to obtain all the information
> in domU since that would test whether the size of the device is
> correctly passed from the back to the front by the blk driver.
> 
> The test as it stands does check that aspect of the blk driver operation
> and is only a factor of 2 slower than the limit and the python code is
> trivial so I thought it was reasonable.
> 
> Can you think of any way to get the block size and number of blocks
> using tools available on the ramdisk?
dd if=/dev/urandom of=/dev/ram1 bs=512 count=`cat /sys/block/ram1/size`

The value in /sys/block/ram1/size is defined to be in multiples of 512 bytes,
as rd.c shows:

		/* rd_size is given in kB */
                ...
		set_capacity(disk, rd_size * 2);


Catting the whole device to determine its size is vile, and would slow down
the regression suite unnecessarily.  Modifying the ramdisk is not verboten,
but it is a right pain in the backside for everyone, particularly me because I
need to build a new one and upload it and all of its source (because of the
GPL) to xm-test.xensource.com.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2006-05-24 at 13:37 +0100, Ewan Mellor wrote:
> On Wed, May 24, 2006 at 12:08:28PM +0100, Harry Butterworth wrote:
> 
> > > > > Why don''t you just fix the size of the
datablock that you write to the
> > > > > ramdisk, instead of determining the current size of the
ramdisk with cat
> > > > > /dev/hda1 | wc -c?
> > > > 
> > > > I wanted to test writing at the device limits.  Sometimes
there are off
> > > > by one errors that mean you can''t write the last
sector of a block
> > > > device.
> > > > 
> > > > cat | wc -c was the most robust way I could think of for
getting the
> > > > size.
> > > 
> > > How about blockdev --getsize64 /dev/ram1?
> > 
> > blockdev and stat are not available in the ramdisk.  I could use
> > blockdev to get the blocksize and number of blocks in dom0 and then
pass
> > the values to domU but it would be better to obtain all the
information
> > in domU since that would test whether the size of the device is
> > correctly passed from the back to the front by the blk driver.
> > 
> > The test as it stands does check that aspect of the blk driver
operation
> > and is only a factor of 2 slower than the limit and the python code is
> > trivial so I thought it was reasonable.
> > 
> > Can you think of any way to get the block size and number of blocks
> > using tools available on the ramdisk?
> 
> dd if=/dev/urandom of=/dev/ram1 bs=512 count=`cat /sys/block/ram1/size`
> The value in /sys/block/ram1/size is defined to be in multiples of 512
bytes,
> as rd.c shows:
> 
> 		/* rd_size is given in kB */
>                 ...
> 		set_capacity(disk, rd_size * 2);
OK, thanks I''ll use that.
> Modifying the ramdisk is not verboten,
> but it is a right pain in the backside for everyone, particularly me
because I
> need to build a new one and upload it and all of its source (because of the
> GPL) to xm-test.xensource.com.
The current test is modifying the ramdisk to add `tee` (as well as fancy
head which we don''t need anymore).

Are you going to accept the patch if it requires the addition of `tee`
to the ramdisk?  Or should I change the test to read the data back in
domU---which I was specifically trying to avoid.

The problem with reading the data back in domU after having written it
is that it might mask bugs in the block driver.  Reading the data back
might cause it to be committed to the device where otherwise a bug might
cause it to be lost.

Harry.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, May 24, 2006 at 02:09:49PM +0100, Harry Butterworth wrote:
> > Modifying the ramdisk is not verboten,
> > but it is a right pain in the backside for everyone, particularly me
because I
> > need to build a new one and upload it and all of its source (because
of the
> > GPL) to xm-test.xensource.com.
> 
> The current test is modifying the ramdisk to add `tee` (as well as fancy
> head which we don''t need anymore).
> 
> Are you going to accept the patch if it requires the addition of `tee`
> to the ramdisk?  Or should I change the test to read the data back in
> domU---which I was specifically trying to avoid.
> 
> The problem with reading the data back in domU after having written it
> is that it might mask bugs in the block driver.  Reading the data back
> might cause it to be committed to the device where otherwise a bug might
> cause it to be lost.
I think that''s a fair point, so yes, we''ll have to rev the
ramdisk anyway.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel