search for: sysdig

Hi, we have a huge directory tree. * 17M files (number of files) * 2.2TBytes of data. * Only 0.1% changes per day Current pain: rsyncs directory tree traversal needs to long to discover the changed files. Only few files change. I discovered the tool sysdig which could be used to monitor the files which were changed. Then we could feed the list of changed files to rsync and avoid the long directory traversal of rsync. Has someone experience with collecting the changed files with a third party tool which detects which files were changed? Regards,...

> Has someone experience with collecting the changed files > with a third party tool which detects which files were changed? I don't know of sysdig but am the developer of Lsyncd which does exactly that, collect file changes via inotify event mechanism and then calls rsync with a matching filter mask. However, since you say, your directory tree is hugh, the main issue is that for every directory an inotify watch must be created, taking about...

On Thu, 9 Feb 2017 10:55:51 +0100 Axel Kittenberger <axkibe at gmail.com> wrote: > > Has someone experience with collecting the changed files > > with a third party tool which detects which files were changed? > > I don't know of sysdig but am the developer of Lsyncd which does > exactly that, collect file changes via inotify event mechanism and > then calls rsync with a matching filter mask. > > However, since you say, your directory tree is hugh, the main issue > is that for every directory an inotify watch must...

On Fri, 10 Feb 2017 12:38:32 +1300 Henri Shustak <henri.shustak at gmail.com> wrote: > As Ben mentioned, ZFS snapshots is one possible approach. Another > approach is to have a faster storage system. I have seen considerable > speed improvements with rsync on similar data sets by say upgrading > the storage sub system. Another possibility could be to use lvm and lvmcache to

Am 09.02.2017 um 16:21 schrieb Ben RUBSON: > >> On 09 Feb 2017, at 16:10, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >> >> Am 09.02.2017 um 11:05 schrieb Ben RUBSON: >>>> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >>>> >>>> Hi, >>>> >>>> we have a huge

Am 09.02.2017 um 11:05 schrieb Ben RUBSON: >> On 09 Feb 2017, at 10:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote: >> >> Hi, >> >> we have a huge directory tree. >> >> >> * 17M files (number of files) >> * 2.2TBytes of data. >> * Only 0.1% changes per day >> >> Current pain: rsyncs directory tree traversal

On Thu, 9 Feb 2017 14:43:57 +0100 Axel Kittenberger <axkibe at gmail.com> wrote: > > > > Not only that, but inotify is not guaranteed. (At least not on > > 3.16.0. Can't say regards later versions.) So you might miss some > > changes. > > > > Got any info on that? > > I noted that MOVE_FROM and MOVE_TO events are not guaranted to arrive

As Ben mentioned, ZFS snapshots is one possible approach. Another approach is to have a faster storage system. I have seen considerable speed improvements with rsync on similar data sets by say upgrading the storage sub system. -------------------------------------------------------------------- This email is protected by LBackup, an open source backup solution http://www.lbackup.org

...uot;strace -o log -s 100 doveadm mailbox list -u > user at domain" shows. Is it opening the correct mailboxes/ directory? > Maybe the path is just wrong for some reason (some typo added somewhere)? Nope it lstats the correct directories, but does not show them. I send you the strace / sysdig output per private mail, as it contains private information of that user. Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Gesch?ftsf?hrer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0 --------------...

...w in the last few days mails would pile up in the Postfix Queue, waiting to be delivered using the lmtp transport into dovecot. So dovecot was being slow, but why? After all, nothing changed. After reading some articles on stackoverflow I found a way of finding out which file gets the most IO: % sysdig -c topfiles_bytes; This command quickly pointed to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp That file was written excessively. I then put ~/mdbox/mailboxes/INBOX/dbox-Mails/ into tmpfs and alas, the queue would drain quickly. But why is that? Why would the index file be updated so o...

...up in the Postfix Queue, waiting to be delivered using the > lmtp transport into dovecot. > > So dovecot was being slow, but why? After all, nothing changed. > > After reading some articles on stackoverflow I found a way of finding > out which file gets the most IO: > > % sysdig -c topfiles_bytes; > > This command quickly pointed to ~/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.tmp > That file was written excessively. Was it one user's dovecot.index.tmp or for a lot of users? This means that dovecot.index is being rewritten, which should happen only once...

...mailbox list -u >> user at domain" shows. Is it opening the correct mailboxes/ directory? >> Maybe the path is just wrong for some reason (some typo added somewhere)? > > > Nope it lstats the correct directories, but does not show them. > > I send you the strace / sysdig output per private mail, as it contains > private information of that user. Looks like you have some dovecot-acl and dovecot-acl-list files, so it has to be because Dovecot thinks the ACLs are preventing access to the user. Try deleting dovecot-acl-list to see if the problem is with that. If no...

...omain" shows. Is it opening the correct mailboxes/ directory? >>> Maybe the path is just wrong for some reason (some typo added >>> somewhere)? >> >> >> Nope it lstats the correct directories, but does not show them. >> >> I send you the strace / sysdig output per private mail, as it contains >> private information of that user. > > Looks like you have some dovecot-acl and dovecot-acl-list files, so it > has to be because Dovecot thinks the ACLs are preventing access to the > user. Try deleting dovecot-acl-list to see if the prob...

Am 14.09.17 um 01:07 schrieb Timo Sirainen: > On 7 Sep 2017, at 17.42, Ralf Becker <rb at egroupware.org> wrote: >> Dovecot 2.2.31 with mailboxes in mdbox format. >> >> Since a couple of days some mailboxes have the problem, that sieve rules >> moving mails to folders stop working and .sieve.log in mailbox shows: >> >> sieve: info: started log at Sep 07