samba - Jun 2015 - Zero-blocks in large files (smb in libvirt/kvm RHEL)

Hi!

I'm having a very strange problem, and I'm running out of ideas how to
debug it :)

Whenever I copy large files (several hundred MB/GB per file), it happens
every now and then that some of them have blocks of zeroes in them. If it
happens, it's always between around 50kB and 150kB.

The problem occurs when I copy files from Debian6 to Samba on RHEL6.5,
using rsync.
It happens frequently, but not always...


The system setup is rather complex, since it's a network-based
multi-server 300 TB mass storage. That's why there's Linux Kernel
software
RAID, libvirt/KVM, GlusterFS, etc involved.

I've narrowed it down pretty much to Samba on the RHEL6.5 KVM-Client.
(Details upon request)


If anyone has any ideas how to pin this issue down, I'd be very grateful!

Thanks a lot in advance,
Peter B.




== Versions:
Client (Debian 6)
- rsync v3.0.7
- smbd v3.5.6

Server (RHEL6.5 KVM-client):
- smbd v3.6.23-12.el6 (RHEL 6.5)

Am 01.06.2015 um 12:21 schrieb Peter B.:
> I'm having a very strange problem, and I'm running out of ideas how
to
> debug it :)
>
> Whenever I copy large files (several hundred MB/GB per file), it happens
> every now and then that some of them have blocks of zeroes in them. If it
> happens, it's always between around 50kB and 150kB.
>
> The problem occurs when I copy files from Debian6 to Samba on RHEL6.5,
> using rsync
why using rsync with a SMB based filesystem instead fire up the rsync 
daemon on the machine hosting it physically or using rsync over ssh?

you don't gain anything with rsync when the filesystem itself is a 
network mount because for building checksums, well, you get the picture....

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20150601/35d13e32/attachment.pgp>

On Mon, Jun 01, 2015 at 12:21:40PM +0200, Peter B.
wrote:
> Hi!
> 
> I'm having a very strange problem, and I'm running out of ideas how
to
> debug it :)
> 
> Whenever I copy large files (several hundred MB/GB per file), it happens
> every now and then that some of them have blocks of zeroes in them. If it
> happens, it's always between around 50kB and 150kB.
> 
> The problem occurs when I copy files from Debian6 to Samba on RHEL6.5,
> using rsync.
> It happens frequently, but not always...
> 
> 
> The system setup is rather complex, since it's a network-based
> multi-server 300 TB mass storage. That's why there's Linux Kernel
software
> RAID, libvirt/KVM, GlusterFS, etc involved.
> 
> I've narrowed it down pretty much to Samba on the RHEL6.5 KVM-Client.
> (Details upon request)
> 
> 
> If anyone has any ideas how to pin this issue down, I'd be very
grateful!
What would be helpful is discovering if this is a client or
server bug. Do you get issues when running rsync on a Windows
client against the same Samba share ?

On 06/01/2015 10:41 PM, Jeremy Allison wrote:
> What would be helpful is discovering if this is a client or
> server bug. Do you get issues when running rsync on a Windows
> client against the same Samba share ?
It's number 1 on my todo-list to find that out.
What I can say so far is, that the Debian 6 client was using the exact
same way/versions/tools for years to write to a Windows server storage
before - without problems.

Now the Windows server was replaced by RHEL+Samba.

It's tricky to triage it, since the zeroblock-issue only appears
"sometimes"...

I've noticed that the Samba logfiles on the RHEL server are mostly empty
(even smbd.log). Where can I configure the logging to see if there's
something happening?


Thanks,
Pb