Well, I just reproduced the scenario on Solaris 8, and the
the metadata updates are eating your lunch. The short
answer is to switch to a logging or log-structured filesystem.
Initially, Justin Marshall was seeing a "copy *" operation
on 10,000 10k files on an SGI octane was using 70% of the cpu,
and **way** to much clock time.
Repeating the same experiment on a Sun Ultra 30 showed a
similar result: a cp operation took 9:16 min:sec and a smbclient
get of the same thing took 8:03 (the cache was already warm for
the get). This is (10,000 * 10240 bytes)/556 sec = 184,172 bytes/sec
for the cp and (10,000 * 10240)/483 = 212,008 bytes/sec.
This time is the sum of readdir, iget, read, write and writedir
times for 10,000 files.
Switching to intent logging on the filesystem cut the get time
to 5:32, for (10,000 * 10240)/332 = 308,433 bytes/sec. This is
about 1.5 times the speed of the initial cps and gets, which
means logging was costing us about 1/3 of the total time.
Copying (with cp) a single file of 102,400,000 bytes takes 41
seconds,
and getting it with smbclient takes 40 seconds, or about
(10,000 * 10240)/40 = 2,560,000 bytes/second. (Yes, my single-cpu,
single-drive Sun is a fair bit slower than the twin-cpu RAIDed SGI!)
The cpu usage is interesting, too: with logging off, the cp and get
looked like:
%usr %sys %wio %idle
cp 0 7 90 4
get 4 6 83 5
The cpu was spending all it's time in io-wait. As it
happens, this isn't bad, as it can do computation during this
period if there was any to do. (The SAR guys distinguished
between io wait, user and system time because lumping them
all into "busy" tends to cause end-users to panic and buy
more CPUs. Solaris continues this tradition (;-))
When I switched the disk to logging, these changed:
%usr %sys %wio %idle
cp 20 38 39 3
get 4 14 35 48
This means that more of the (shorter) time was spend
in the true idle state, meaning more absolute cpu was
available. This is pleasant to see.
To conclude:
- big files are more efficient to transfer
- inode updates are a significant part of the difference
- the rest of the difference is mostly readdir/iget/open
- filesystem improvements, like logging, can help a lot.
--dave
[PS: this system doesn't have the latest ufs performance
updates applied: I saw another 30-odd percent under netbench
with them.
--
David Collier-Brown, | Always do right. This will gratify some people
185 Ellerslie Ave., | and astonish the rest. -- Mark Twain
Willowdale, Ontario | //www.oreilly.com/catalog/samba/author.html
Work: (905) 415-2849 Home: (416) 223-8968 Email: davecb@canada.sun.com