Hello all. I''m doing a course project to evaluate recovery time of RAID-Z. One of my tests is to examine the impact of aging on recovery speed. I''ve used PostMark to stress the file-system but I didn''t observe any noticeable slowdown. Is there a better way to "age" a ZFS file-system? Does ZFS have aging issues at all? Thanx in advance. -- This message posted from opensolaris.org
Typically you want to do something like this: Write 1,000,000 files of varying length. Randomly select and remove 500,000 of those files. Repeat (a) creating files, and (b) randomly removing files, until your file system is full enough for your test, or you run out of time. That''s a pretty simplistic method, and there are lots of useful variations (e.g. fill the file system entirely the first time before removing roughly half of the storage, to ensure there''s not a big contiguous chunk of space that''s never been allocated). For ZFS, you may also want to throw snapshots into the mix, depending on what you''re trying to test. If you care about directory performance, or anything which walks the file tree, it''s also useful to randomly move the files around. If you care about the performance of files which are appended to, you''d also want to append to some random number of files. (If you''re really serious about this, you''d want to record a set of operations on a filesystem over a period of years and either play it back or generate a similar pattern. I don''t know of any publically available traces which are good enough to do aging with, though.) -- This message posted from opensolaris.org
On Mon, Dec 15, 2008 at 7:57 PM, Thanos McAtos <mcatos at ics.forth.gr> wrote:> Hello all. > > I''m doing a course project to evaluate recovery time of RAID-Z. > > One of my tests is to examine the impact of aging on recovery speed. > > I''ve used PostMark to stress the file-system but I didn''t observe any > noticeable slowdown. > > Is there a better way to "age" a ZFS file-system? > > Does ZFS have aging issues at all? > > Thanx in advance. > <http://mail.opensolaris.org/mailman/listinfo/zfs-discuss> >Anton suggested some practical methods for conducting testing. But do follow proper testing procedure: Since you''re doing this as part of a study course you may already know much of this, but I''ve just seen too many invalid, useless test to let this one pass. So firstly, understand that utilization metrics are not performance. At best they can be considered a symptom. However utilization metrics is important because it may provide hints at a) how to improve the system''s performance, and b) errors in the thinking during the test design phase. Equally important: Know what your testing objectives are. Define important concepts, such as "recovery time" and "file system age". Document your test methodology, expected results, and make a list of scenarios, including a "base-line" for comparison, and a description of what will remain the same and what will be different between the test scenarios. For each test, record all utilization metrics so that you can evaluate these to understand what the bottleneck (bound resource) was in each scenario. Record the results for each test scenario. Include the recorded utilization data in an appendix. Make some conclusions. This is where definitions are important. For example saying that "file system age made [no] significant impact on raid-z recovery time" is completely meaningless unless you also defined "file system age" and "recovery time" The big issue is that everybody has got a case of "X performed better than Z, thus X is better than Z" where X and Z are simple products. You really have to accurately describe your test scenarios, especially in terms of what are different and what are the same between them. Try to be as complete as possible. Include the scripts and their parameters as you used them to generate load, if possible (Not possible when you let users generate real load) So some items to list in your scenarios: o System configuration details. o OS version, patches o Software versions (patch revisions, etc) o Configuration details (at least anything which is non-default. In many case you may want to specifically stress some default values) o The exact test procedure, parameters, etc. So, while talking about Raid-Z recovery time, particularly in terms of the File System "age", I imagine some kind of comparison of recovery times. I am sure you will design a series of increasingly "aged" storage pools, and for each perform a number of "recovery test" for which you will record the run time. What would be your baseline? What do you want to keep constant between the tests? Nr of files? File system usage level? Nr of disks in the pool? I am assuming that not changing the system configuration and patch level between the test scenarios are obvious. I am also assuming that system load will be idle for all tests? In terms of evaluating the results: Do you expect the file system age to actually impact on the recovery time? If so, is this based on how file system age impacts on recovery time for other raid technologies? If your test results will be published I''d love to take a look at it. Good luck -- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke My blog: http://initialprogramload.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081216/b3754bf1/attachment.html>
I followed Anton''s idea but didn''t see any difference. My tests were repetitive PostMark runs, and each run was different. For instance, I didn''t let PostMark delete the files once it finished, deleted the odd numbered ones by hand, put the rest in a different directory, re-run PostMark that failed on some operations due to lack of free space etc. All these runs created more than half a TB of writes on a 72 GB volume. I don''t want to make conclusions like "X is better than Y", I just wan''t to verify whether aging substantially impacts recovery, I''m not interested in how much it does. And of course, the test environment is the same for all runs. My problems are 2: 1) I don''t know how to properly age a file-system. As already said, I need traces of a decade''s workload to properly do this, and to the best of my knowledge there is no easy way to do this automatically. 2) I know very little of ZFS. To be honest, I have no idea what to expect. Maybe I''m doing aging the wrong way or ZFS suffers from aging when is has to allocate blocks for writes/updates and not on recovery. Anyway, I''ve talked to my professor and told me not to investigate this any more. I have a 2 page draft of my report, just tell me if you want to have a look at it. Thanx for your help. -- This message posted from opensolaris.org
On Wed, Dec 17, 2008 at 2:46 PM, Thanos McAtos <mcatos at ics.forth.gr> wrote:> > > My problems are 2: > > 1) I don''t know how to properly age a file-system. As already said, I need > traces of a decade''s workload to properly do this, and to the best of my > knowledge there is no easy way to do this automatically. > > 2) I know very little of ZFS. To be honest, I have no idea what to expect. > Maybe I''m doing aging the wrong way or ZFS suffers from aging when is has to > allocate blocks for writes/updates and not on recovery. > > I would expect the fill level of the pool to be a much bigger factor thanthe "age" of the file system. However an old but very empty file system may have its data blocks spread far apart (large gaps in between). So a "new" empty file system may have all its allocated data blocks at the start of a disk, and a "old" empty file system may be scattered all over the disk. However, since we are talking about more space than data, and ZFS only "rebuilds" the blocks which are in use, this is a special case and while the difference my be relatively large, it will likely be small real difference. But I am speculating. The CoW nature of ZFS will probably make it very hard to consistently create a "fragmented" file system!!! -- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke My blog: http://initialprogramload.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081217/07a14d79/attachment.html>
On Wed, Dec 17, 2008 at 9:09 AM, Johan Hartzenberg <jhartzen at gmail.com>wrote:> > On Wed, Dec 17, 2008 at 2:46 PM, Thanos McAtos <mcatos at ics.forth.gr>wrote: > >> >> >> My problems are 2: >> >> 1) I don''t know how to properly age a file-system. As already said, I need >> traces of a decade''s workload to properly do this, and to the best of my >> knowledge there is no easy way to do this automatically. >> >> 2) I know very little of ZFS. To be honest, I have no idea what to expect. >> Maybe I''m doing aging the wrong way or ZFS suffers from aging when is has to >> allocate blocks for writes/updates and not on recovery. >> >> I would expect the fill level of the pool to be a much bigger factor than > the "age" of the file system. However an old but very empty file system may > have its data blocks spread far apart (large gaps in between). So a "new" > empty file system may have all its allocated data blocks at the start of a > disk, and a "old" empty file system may be scattered all over the disk. > However, since we are talking about more space than data, and ZFS only > "rebuilds" the blocks which are in use, this is a special case and while the > difference my be relatively large, it will likely be small real difference. > > But I am speculating. The CoW nature of ZFS will probably make it very > hard to consistently create a "fragmented" file system!!! > > -- > Any sufficiently advanced technology is indistinguishable from magic. > Arthur C. Clarke > > My blog: http://initialprogramload.blogspot.com >I would expect there to be some sort of clean-up process that either runs automatically or that could be scheduled to optimize block layout on a regular basis to avoid just such an issue. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081217/cfa0d6e0/attachment.html>
Thank you all for your interest. I''m attaching my project report which demonstrates the impact of capacity utilization and the number of files on recovery speed. -- This message posted from opensolaris.org -------------- next part -------------- A non-text attachment was scrubbed... Name: RAID-Z_vs_RAID-5_recovery.pdf Type: application/octet-stream Size: 184119 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081217/a04e9867/attachment.obj>