On 8/1/20 12:39 PM, Richard W.M. Jones wrote:> On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote: >> >> One thing I noticed which is a bit odd is: >> >> $ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file >> 2048 512 >> $ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file >> 3968 512 >> >> The second method is how we currently create the file. Since looking >> through the history there seems to be no reason for that I'm going to >> push a commit which changes file creation to the first method, and it >> may be slightly faster too. >> >> However it makes me wonder if the file is not laid out in a single >> extent and if that might be causing our problems. Being only able >> to reproduce this on Koji makes a bit tedious to test theories :-( > > I pushed this patch and did another test build in Koji, but > the issue is still not fixed.It looks like the real issue at hand is that stat is indeed reporting allocated size caused by the filesystem pre-emptively over-allocating based on access patterns (more so when creating the first file, especially when reopening the file; less so when copying as the source file size is now stabilized so the copy has less reason to overshoot). But since the real crux of the test is whether we managed to punch holes, would it be sufficient to take note of the original sizes, and merely check that the resulting size has either shrunk (where the file should now be sparser) or remained unchanged? I'll push a patch along those lines, but as you're the one doing the koji runs, there's obviously another feedback cycle to go through. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Mon, Aug 03, 2020 at 04:21:13PM -0500, Eric Blake wrote:> On 8/1/20 12:39 PM, Richard W.M. Jones wrote: > >On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote: > >> > >>One thing I noticed which is a bit odd is: > >> > >>$ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file > >>2048 512 > >>$ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file > >>3968 512 > >> > >>The second method is how we currently create the file. Since looking > >>through the history there seems to be no reason for that I'm going to > >>push a commit which changes file creation to the first method, and it > >>may be slightly faster too. > >> > >>However it makes me wonder if the file is not laid out in a single > >>extent and if that might be causing our problems. Being only able > >>to reproduce this on Koji makes a bit tedious to test theories :-( > > > >I pushed this patch and did another test build in Koji, but > >the issue is still not fixed. > > It looks like the real issue at hand is that stat is indeed > reporting allocated size caused by the filesystem pre-emptively > over-allocating based on access patterns (more so when creating the > first file, especially when reopening the file; less so when copying > as the source file size is now stabilized so the copy has less > reason to overshoot). But since the real crux of the test is whether > we managed to punch holes, would it be sufficient to take note of > the original sizes, and merely check that the resulting size has > either shrunk (where the file should now be sparser) or remained > unchanged? > > I'll push a patch along those lines, but as you're the one doing the > koji runs, there's obviously another feedback cycle to go through.Sure I'll give your patch a go once I see it in git, thanks! Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
On Mon, Aug 03, 2020 at 10:48:54PM +0100, Richard W.M. Jones wrote:> On Mon, Aug 03, 2020 at 04:21:13PM -0500, Eric Blake wrote: > > On 8/1/20 12:39 PM, Richard W.M. Jones wrote: > > >On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote: > > >> > > >>One thing I noticed which is a bit odd is: > > >> > > >>$ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file > > >>2048 512 > > >>$ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file > > >>3968 512 > > >> > > >>The second method is how we currently create the file. Since looking > > >>through the history there seems to be no reason for that I'm going to > > >>push a commit which changes file creation to the first method, and it > > >>may be slightly faster too. > > >> > > >>However it makes me wonder if the file is not laid out in a single > > >>extent and if that might be causing our problems. Being only able > > >>to reproduce this on Koji makes a bit tedious to test theories :-( > > > > > >I pushed this patch and did another test build in Koji, but > > >the issue is still not fixed. > > > > It looks like the real issue at hand is that stat is indeed > > reporting allocated size caused by the filesystem pre-emptively > > over-allocating based on access patterns (more so when creating the > > first file, especially when reopening the file; less so when copying > > as the source file size is now stabilized so the copy has less > > reason to overshoot). But since the real crux of the test is whether > > we managed to punch holes, would it be sufficient to take note of > > the original sizes, and merely check that the resulting size has > > either shrunk (where the file should now be sparser) or remained > > unchanged? > > > > I'll push a patch along those lines, but as you're the one doing the > > koji runs, there's obviously another feedback cycle to go through. > > Sure I'll give your patch a go once I see it in git, thanks!Unfortunately no this didn't fix it. The log is: https://kojipkgs.fedoraproject.org//work/tasks/9782/48609782/build.log nozero2.img (for example) was originally: nozero2.img: 2048 allocated blocks of size 512 bytes, total size 1048576 but when we test it later it has grown to 4096 blocks: + for i in {2..6} ++ stat -c %b nozero2.img + test 4096 '!=' 2048 + echo 'nozero2.img was trimmed by mistake' nozero2.img was trimmed by mistake + fail=1 So it has been "trimmed" from 2048 to 4096 blocks. Perhaps we should use -lt in that test to detect if the file got smaller? Also I was interested in what filesystem the builders are using (since I still cannot reproduce any of this without Koji). So I added ‘stat -f .’ in the %check section. It printed: File: "." ID: fc0200000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 33538048 Free: 25172957 Available: 25172957 Inodes: Total: 67108864 Free: 66469270 I believe the builder is running Fedora 32, at least going by the kernel version. (This surprised me as I thought they were running RHEL.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top