One thing I noticed which is a bit odd is:
$ rm file; for f in {0..1023}; do printf '%1024s' .; done > file;
stat -c "%b %B" file
2048 512
$ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done
; stat -c "%b %B" file
3968 512
The second method is how we currently create the file. Since looking
through the history there seems to be no reason for that I'm going to
push a commit which changes file creation to the first method, and it
may be slightly faster too.
However it makes me wonder if the file is not laid out in a single
extent and if that might be causing our problems. Being only able
to reproduce this on Koji makes a bit tedious to test theories :-(
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote:> > One thing I noticed which is a bit odd is: > > $ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file > 2048 512 > $ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file > 3968 512 > > The second method is how we currently create the file. Since looking > through the history there seems to be no reason for that I'm going to > push a commit which changes file creation to the first method, and it > may be slightly faster too. > > However it makes me wonder if the file is not laid out in a single > extent and if that might be causing our problems. Being only able > to reproduce this on Koji makes a bit tedious to test theories :-(I pushed this patch and did another test build in Koji, but the issue is still not fixed. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On 8/1/20 12:39 PM, Richard W.M. Jones wrote:> On Sat, Aug 01, 2020 at 08:46:11AM +0100, Richard W.M. Jones wrote: >> >> One thing I noticed which is a bit odd is: >> >> $ rm file; for f in {0..1023}; do printf '%1024s' .; done > file; stat -c "%b %B" file >> 2048 512 >> $ rm file; for f in {0..1023}; do printf '%1024s' . >> file; done ; stat -c "%b %B" file >> 3968 512 >> >> The second method is how we currently create the file. Since looking >> through the history there seems to be no reason for that I'm going to >> push a commit which changes file creation to the first method, and it >> may be slightly faster too. >> >> However it makes me wonder if the file is not laid out in a single >> extent and if that might be causing our problems. Being only able >> to reproduce this on Koji makes a bit tedious to test theories :-( > > I pushed this patch and did another test build in Koji, but > the issue is still not fixed.It looks like the real issue at hand is that stat is indeed reporting allocated size caused by the filesystem pre-emptively over-allocating based on access patterns (more so when creating the first file, especially when reopening the file; less so when copying as the source file size is now stabilized so the copy has less reason to overshoot). But since the real crux of the test is whether we managed to punch holes, would it be sufficient to take note of the original sizes, and merely check that the resulting size has either shrunk (where the file should now be sparser) or remained unchanged? I'll push a patch along those lines, but as you're the one doing the koji runs, there's obviously another feedback cycle to go through. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org