hello! i have a strange problem with writing data to my ufs2+su filesystem. 1. i made a 1T gpt partition on my storage server, and formatted it: newfs -U -m 0 -o time -i 32768 /dev/da1p3a 2. i tried to move data from other servers to this filesystem, total size of files is slightly less than 1T 3. i encountered a 'No space left on device' while i still have 11G of free space and about 13 million free inodes on the filesystem: #df -ih Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/da1p3a 1.0T 1.0T 11G 99% 20397465 13363173 60% /mnt/45_114 all i want to know is whether this is a bug or a feature? and if such a behavior is well-known, where can i read about it? -- SY, Marat -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3221 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090705/bf7122dc/smime.bin
2009/7/5 Marat N.Afanasyev <amarat@ksu.ru>:> hello! > > i have a strange problem with writing data to my ufs2+su filesystem. > > 1. i made a 1T gpt partition on my storage server, and formatted it: > newfs -U -m 0 -o time -i 32768 /dev/da1p3a > > 2. i tried to move data from other servers to this filesystem, total size of > files is slightly less than 1T > > 3. i encountered a 'No space left on device' while i still have 11G of free > space and about 13 million free inodes on the filesystem: > > #df -ih > Filesystem ? ? Size ? ?Used ? Avail Capacity ?iused ? ?ifree %iused Mounted > on > /dev/da1p3a ? ?1.0T ? ?1.0T ? ? 11G ? ?99% 20397465 13363173 ? 60% > /mnt/45_114 > > all i want to know is whether this is a bug or a feature? and if such a > behavior is well-known, where can i read about it?By default, a part of a filesystem is reserved, the amount reserved has historically varied between 5-8%. This is adjustable. See the "-m" switch to tunefs. - Sincerely, Dan Naumov
Ralf Folkerts wrote:> Marat N.Afanasyev wrote: >> hello! >> >> i have a strange problem with writing data to my ufs2+su filesystem. >> >> 1. i made a 1T gpt partition on my storage server, and formatted it: >> newfs -U -m 0 -o time -i 32768 /dev/da1p3a >> >> 2. i tried to move data from other servers to this filesystem, total >> size of files is slightly less than 1T >> >> 3. i encountered a 'No space left on device' while i still have 11G of >> free space and about 13 million free inodes on the filesystem: >> >> #df -ih >> Filesystem Size Used Avail Capacity iused ifree %iused >> Mounted on >> /dev/da1p3a 1.0T 1.0T 11G 99% 20397465 13363173 60% >> /mnt/45_114 >> >> all i want to know is whether this is a bug or a feature? and if such >> a behavior is well-known, where can i read about it? > Hi Marat, > > just a guess: Are there sparse Files on the Source System that are not > being detected by the Tool you used to restore the data? If you used > (bsd)tar, did you try -S? > > A while ago I ran into a similar Problem when copying > Oracle-Database-Files (on Linux, though) and the -S - Option came to > rescue. > > Cheers, > _ralf_i have a huge amount of small files on the source systems, as you can see they have about 20 million files and almost each of them is jpeg or gif. afaik, there are no sparse files at all. i still cannot figure out what is it: a free space leak in ufs2+su or bug in statfs(3), that is used in df, or something else. -- SY, Marat -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3221 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090706/cfe596e6/smime.bin
On Tue, Jul 07, 2009 at 12:15:46AM +0400, Marat N.Afanasyev wrote:> Kostik Belousov wrote: > >On Mon, Jul 06, 2009 at 09:45:45PM +0400, Marat N.Afanasyev wrote: > >>i have a huge amount of small files on the source systems, as you can > >>see they have about 20 million files and almost each of them is jpeg or > >>gif. afaik, there are no sparse files at all. > >> > >>i still cannot figure out what is it: a free space leak in ufs2+su or > >>bug in statfs(3), that is used in df, or something else. > > > >My guess that it is due to fragmentation. > >As an experiment, try to create 1-byte file. Does it work on the filesystem > >in described state ? > > I can create small files, as many as i have patience, maximum size of > such "small file" is 14336, so. it seems that if file is no greater than > (block_size-2048) it can be created. larger file cannot be created. > > imho, fragmentation on filesystem should be very low, there were no > deletions on it, just creations.The fragmentation on UFS usually means using fragments for the file tails, not having file sequential blocks allocated in the non-sequential disk blocks. You experiment confirms my hypothesis. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090706/5d9fac15/attachment.pgp
Kostik Belousov wrote:> On Tue, Jul 07, 2009 at 12:15:46AM +0400, Marat N.Afanasyev wrote: >> Kostik Belousov wrote: >>> On Mon, Jul 06, 2009 at 09:45:45PM +0400, Marat N.Afanasyev wrote: >>>> i have a huge amount of small files on the source systems, as you can >>>> see they have about 20 million files and almost each of them is jpeg or >>>> gif. afaik, there are no sparse files at all. >>>> >>>> i still cannot figure out what is it: a free space leak in ufs2+su or >>>> bug in statfs(3), that is used in df, or something else. >>> My guess that it is due to fragmentation. >>> As an experiment, try to create 1-byte file. Does it work on the filesystem >>> in described state ? >> I can create small files, as many as i have patience, maximum size of >> such "small file" is 14336, so. it seems that if file is no greater than >> (block_size-2048) it can be created. larger file cannot be created. >> >> imho, fragmentation on filesystem should be very low, there were no >> deletions on it, just creations. > > The fragmentation on UFS usually means using fragments for the file tails, > not having file sequential blocks allocated in the non-sequential disk > blocks. > > You experiment confirms my hypothesis.is there any way to change this behavior? and i wonder why even if i can create a reasonable number of small files i cannot create a slightly larger file joining two unallocated parts of different blocks even if no fully free block exists? ;) -- SY, Marat -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3221 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090706/5e0af2ec/smime.bin
On 2009-Jul-05 21:56:23 +0400, "Marat N.Afanasyev" <amarat@ksu.ru> wrote:>i have a strange problem with writing data to my ufs2+su filesystem.Overall, I think you are pushing UFS and have hit one of its limits.>1. i made a 1T gpt partition on my storage server, and formatted it: >newfs -U -m 0 -o time -i 32768 /dev/da1p3a'-m 0' is undesirable. The default 8% free space is intended to hide the sort of effect you've run into. Unless you can find a slightly bigger disk, I suggest you try building an 8K/1K filesystem by adding '-b 8192 -f 1024' - this will be slightly slower than the default 16K/2K but will hopefully allow you to fit slightly more files in.>2. i tried to move data from other servers to this filesystem, total >size of files is slightly less than 1TCan you be more accurate than "slightly less than 1T" (ie how many MiB) and what is the size distribution of the files? The average size is 48KiB but are they all roughly the same size or do they vary widely? Are there many files <16KiB?>3. i encountered a 'No space left on device' while i still have 11G of >free space and about 13 million free inodes on the filesystem: > >#df -ih >Filesystem Size Used Avail Capacity iused ifree %iused >Mounted on >/dev/da1p3a 1.0T 1.0T 11G 99% 20397465 13363173 60% >/mnt/45_114More significant figures are needed. How about 'df -im' or 'df -ik'. How close are you to fitting in all the files you want?>all i want to know is whether this is a bug or a feature? and if such a >behavior is well-known, where can i read about it?Have a read of /usr/share/doc/smm/05.fastfs/paper.ascii.gz - especially the bits about "fragments". Whilst this paper talks about the original UFS and a 4K/1K configuration, the principles remain the same in UFS2. -- Peter Jeremy -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4029 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090707/2eaf789e/smime-0001.bin
Marat N.Afanasyev wrote:> > all i want to know is whether this is a bug or a feature? and if such a > behavior is well-known, where can i read about it? >thanks everybody, i have resolved this as my misunderstanding of UFS basics, so it was pretty good lesson for me :) -- SY, Marat -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3221 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090712/833c858a/smime.bin