Ray Van Dolson
2005-Mar-23 23:25 UTC
pauses sync'ing between tmpfs and disk on Linux 2.4.x
I've set up a 1GB tmpfs filesystem on a system with a single IDE disk and 2GB's of memory. I'm storing a large amount of RRD files (~300MB) on the tmpfs filesystem to make their generation a bit speedier... this part works great. However, I want to rsync these files over from time to time to a directory on the local filesystem (same physical server). I'm using rsync 2.6.4pre3 and am hoping to understand a bit better what is happening. When I run rsync -av /path/to/tmpfs /path/to/diskdir things move along pretty fast for while, then there's a big pause... sometimes for up to 30+ seconds where nothing seems to be happening, but all IO on the system ceases (can't do anything in another xterm). Then rsync starts moving along again for a while, and then pauses again... after about three such pauses it finishes the entire rsync process. I'm wondering what I can do to speed things up... perhaps whatever processes that write to the tmpfs filesystem are fighting with rsync... but doesn't rsync just need read access? I've also tried with -W (copy whole file) and used a smaller -B value. Anyways, I'll explain a bit about the curren scenario: 1. Daemon on system receives data from remote devices. 2. Daemon calls rrdtool to write a .rrd file to the tmpfs filesystem. 3. rsync runs periodically (every 5 minutes) to sync up the tmpfs filesystem with a directory on the local filesystem. The tmpfs filesystem is a directory with about 100 directories inside of it, each containing rrd files. I'm considering using a script to just rsync one of these subdirectories at a time over a period of time to "distribute" the load. The main issue is that while rsync is running and enters these random 30 second "pauses", no IO can happen and things get really backed up on the system and it ends up being faster for me to just dump the tmpfs completely and go back to writing directly to disk as slow as that was... Any suggestions? -- Ray Van Dolson Linux/Unix Systems Administrator DigitalPath Networks
Steve Traugott
2005-Mar-23 23:43 UTC
pauses sync'ing between tmpfs and disk on Linux 2.4.x
Sounds like you're swapping -- what does free memory look like? Steve On Wed, Mar 23, 2005 at 03:24:51PM -0800, Ray Van Dolson wrote:> I've set up a 1GB tmpfs filesystem on a system with a single IDE disk and > 2GB's of memory. I'm storing a large amount of RRD files (~300MB) on the > tmpfs filesystem to make their generation a bit speedier... this part works > great. > > However, I want to rsync these files over from time to time to a directory on > the local filesystem (same physical server). I'm using rsync 2.6.4pre3 and > am hoping to understand a bit better what is happening. > > When I run rsync -av /path/to/tmpfs /path/to/diskdir things move along pretty > fast for while, then there's a big pause... sometimes for up to 30+ seconds > where nothing seems to be happening, but all IO on the system ceases (can't do > anything in another xterm). Then rsync starts moving along again for a while, > and then pauses again... after about three such pauses it finishes the entire > rsync process. > > I'm wondering what I can do to speed things up... perhaps whatever processes > that write to the tmpfs filesystem are fighting with rsync... but doesn't > rsync just need read access? > > I've also tried with -W (copy whole file) and used a smaller -B value. > Anyways, I'll explain a bit about the curren scenario: > > 1. Daemon on system receives data from remote devices. > 2. Daemon calls rrdtool to write a .rrd file to the tmpfs filesystem. > 3. rsync runs periodically (every 5 minutes) to sync up the tmpfs filesystem > with a directory on the local filesystem. > > The tmpfs filesystem is a directory with about 100 directories inside of it, > each containing rrd files. I'm considering using a script to just rsync one > of these subdirectories at a time over a period of time to "distribute" the > load. > > The main issue is that while rsync is running and enters these random 30 > second "pauses", no IO can happen and things get really backed up on the > system and it ends up being faster for me to just dump the tmpfs completely > and go back to writing directly to disk as slow as that was... > > Any suggestions? > > -- > Ray Van Dolson > Linux/Unix Systems Administrator > DigitalPath Networks > -- > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-- Stephen G. Traugott (KG6HDQ) UNIX/Linux Infrastructure Architect, TerraLuna LLC stevegt@TerraLuna.Org http://www.stevegt.com -- http://Infrastructures.Org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://lists.samba.org/archive/rsync/attachments/20050323/ceee7969/attachment.bin
I use rsync similarly. In addition to he standard /tmp traffic, junkbuster has a 4MB jarfile and 18MB log, ccache has 16 directories and 8000 files of which 3000 are > 10k. Both are heavily used. At boot time /tmp is populated from on disk storage: /bin/nice -19 /usr/bin/rsync -aq --whole-file /usr/TMP /tmp & At shutdown /tmpfs deltas are put back: /usr/bin/rsync -aq --delete --no-whole-file \ --exclude "*-lock" --exclude "*-unix" --exclude "*=:0.0" /tmp /usr/TMP & Here are some representative timings (start == boot, stop == shutdown): [9:26] abit:/tmp > egrep '(stop|start)\) .* tmp' /var/log/messages.3 Mar 19 20:53:55 abit rc.local: (stop) starting tmp /tmp/ /usr/TMP Mar 19 20:53:57 abit rc.local: (stop) finished tmp /tmp/ /usr/TMP Mar 20 09:31:18 abit rc.local: (start) starting tmp /usr/TMP/ /tmp Mar 20 09:31:55 abit rc.local: (start) finished tmp /usr/TMP/ /tmp Mar 20 21:07:37 abit rc.local: (stop) starting tmp /tmp/ /usr/TMP Mar 20 21:07:41 abit rc.local: (stop) finished tmp /tmp/ /usr/TMP Mar 21 20:33:21 abit rc.local: (start) starting tmp /usr/TMP/ /tmp Mar 21 20:34:17 abit rc.local: (start) finished tmp /usr/TMP/ /tmp There is never a hang or pause, in fact the process would be transparent if I didnt have xosview in a corner window. The problem probably isn't the way rsync, /tmpfs or 2.4 work, so you may want to look elsewhere. kernel: 2.4.30-rc1 cpu/mem: athlon XP 2800, 1GB PC2700 rsync: 2.6.3 disk: 3xRAID-5 (old 20GB PATA drives, about 50MB/s throughput) [9:07] abit:/tmp > df /tmp Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 460800 159492 301308 35% /tmp [9:18] abit:/tmp > ls -la total 7729 drwxrwxrwt 11 root root 480 Mar 26 08:57 . drwxr-xr-x 26 root root 1024 Mar 26 08:51 .. -r--r--r-- 1 root root 11 Mar 26 08:52 .X0-lock drwxrwxrwt 2 root root 60 Mar 26 08:52 .X11-unix drwxrwxr-x 18 tim tim 163900 Mar 23 23:34 .ccache drwxrwxrwt 2 xfs xfs 60 Mar 26 08:52 .font-unix drwx------ 3 tim tim 60 Jan 11 23:35 .wine-tim drwx------ 2 root root 40 Feb 6 00:44 .xf86config1053 drwx------ 2 root root 40 Feb 6 00:46 .xf86config1077 -rw------- 1 tim tim 0 Feb 9 22:24 0vws1uvh.zip -rw------- 1 tim tim 0 Feb 9 22:22 56gtsq4w.zip -rw-rw-r-- 1 tim tim 178108 Mar 15 21:35 SATA_HotPlug.pdf -rw-rw-r-- 1 tim tim 234529 Mar 15 21:33 SATA_PCI_CardBusHost.pdf -rw-rw-r-- 1 tim tim 175906 Mar 15 21:36 SATA_illus_guide.pdf srwx------ 1 tim tim 0 Mar 26 08:52 afterstep-500.DISPLAY=:0.0 -rw-r--r-- 1 root root 16686 Jan 30 17:18 bonnie.log -rw-r--r-- 1 root root 35644 Jan 8 18:08 build.log drwxr-xr-x 2 tim tim 40 Mar 7 21:05 hsperfdata_tim drwxr-xr-x 2 nobody nobody 80 Dec 25 23:56 junkbuster drwx------ 2 tim tim 60 Mar 16 22:46 mcop-tim -rw-r--r-- 1 root root 485728 Feb 21 12:18 rsync.log -rw-rw-r-- 1 tim tim 6693164 Mar 15 21:40 serialata10a.pdf -rw-rw-r-- 1 tim tim 27475 Jan 31 17:07 smp.log -rw-r--r-- 1 root root 0 Feb 12 15:19 tmp.dd t. Ray Van Dolson wrote:> I've set up a 1GB tmpfs filesystem on a system with a single IDE disk and > 2GB's of memory. I'm storing a large amount of RRD files (~300MB) on the > tmpfs filesystem to make their generation a bit speedier... this part works > great. > > However, I want to rsync these files over from time to time to a directory on > the local filesystem (same physical server). I'm using rsync 2.6.4pre3 and > am hoping to understand a bit better what is happening. > > When I run rsync -av /path/to/tmpfs /path/to/diskdir things move along pretty > fast for while, then there's a big pause... sometimes for up to 30+ seconds > where nothing seems to be happening, but all IO on the system ceases (can't do > anything in another xterm). Then rsync starts moving along again for a while, > and then pauses again... after about three such pauses it finishes the entire > rsync process. > > I'm wondering what I can do to speed things up... perhaps whatever processes > that write to the tmpfs filesystem are fighting with rsync... but doesn't > rsync just need read access? > > I've also tried with -W (copy whole file) and used a smaller -B value. > Anyways, I'll explain a bit about the curren scenario: > > 1. Daemon on system receives data from remote devices. > 2. Daemon calls rrdtool to write a .rrd file to the tmpfs filesystem. > 3. rsync runs periodically (every 5 minutes) to sync up the tmpfs filesystem > with a directory on the local filesystem. > > The tmpfs filesystem is a directory with about 100 directories inside of it, > each containing rrd files. I'm considering using a script to just rsync one > of these subdirectories at a time over a period of time to "distribute" the > load. > > The main issue is that while rsync is running and enters these random 30 > second "pauses", no IO can happen and things get really backed up on the > system and it ends up being faster for me to just dump the tmpfs completely > and go back to writing directly to disk as slow as that was... > > Any suggestions? >-- | for direct mail add "private_" in front of user name