Darren Moffat and I were having lunch and bouncing around ideas, and I came up with the concept of "views" which neither of us are aware of anyone having previously proposed. The rough idea is something like a snapshot-with-selection-filters, perhaps made directly, or perhaps bred from an existing full-fat snapshot in a analogous way to clones. Views would be readonly, and presumably static unless there are ways to make them dynamic without eating CPU. Since you "observe" a "view", I suppose the syntax would be: zfs observe tank/alecm at viewname zfs set filter="owner=USER,name=REGEX,type=TYPE,..." tank/alecm at viewname ...where the contents of the view includes all objects where: owner is USER (eg: alecm, an obvious test) name matches REGEX (eg: "\.mp3$") TYPE is a string of 1/more from "fdlDCBps..." (ie: file, dir, link, device, chardev, blockdev, ...) ...etc, etc, you can guess the extra sorts of tests to make. Go read the manpage for "find" for inspiration. The view would also include any necessary directory structure of any specification necessary to expose the matching objects; thus if a file were owned by "alecm" but buried deep in a directory structure owned by "root", the latter would be exposed irrespective of "owner=alecm" being specified; I''m not trying to recreate MLDs here, and any security beyond the above is meant to be implemented by permissions. Then you could "cd ~/.zfs/view/alecm at mp3s" and see all your music files. Or large files. Or files with nlinks>1. Or whatever. Whaddya think - too crazy? -a
On Wed, 2006-06-07 at 15:38 +0100, Alec Muffett wrote:> The rough idea is something like a snapshot-with-selection-filters, > perhaps made directly, or perhaps bred from an existing full-fat > snapshot in a analogous way to clones.Yep, I''d like ''em too - they would make the snapshot-GUI stuff I was mumbling about a while ago[1] a lot easier to implement. Here we''d just ask for a "zfs view" with a filter specifying the view to contain contents changed between the creation date of <this snapshot> and <that snapshot>. Right now, all I can do is show each snapshot in it''s own window. I was thinking that RFE 6425091 or 6370738 was what I was waiting for, but views would make this even easier to implement. cheers, tim [1] http://blogs.sun.com/roller/page/timf?entry=zfs_on_your_desktop -- Tim Foster, Sun Microsystems Inc, Operating Platforms Group Engineering Operations http://blogs.sun.com/timf
Alec Muffett wrote:> Then you could "cd ~/.zfs/view/alecm at mp3s" and see all your music > files. Or large files. Or files with nlinks>1. Or whatever. >Should it retain the directory structure for each match, e.g. ~/.zfs/view/alecm at mp3s/Music/A/Abba.mp3, or remap everything, e.g. ~/.zfs/view/alecm at mp3s/Abba.mp3? cheers, /Martin -- Martin Englund, Senior Security Engineer, Sun IT Security Office Email: martin.englund at sun.com Time Zone: GMT+2 PGP: 1024D/AA514677 "The question is not if you are paranoid, it is if you are paranoid enough." -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3311 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060607/4de791a2/attachment.bin>
Martin Englund wrote:> Alec Muffett wrote: > > >> Then you could "cd ~/.zfs/view/alecm at mp3s" and see all your music >> files. Or large files. Or files with nlinks>1. Or whatever. >> >> > Should it retain the directory structure for each match, e.g. > ~/.zfs/view/alecm at mp3s/Music/A/Abba.mp3, or remap everything, e.g. > ~/.zfs/view/alecm at mp3s/Abba.mp3? >Insert mental image of Alec with headphones on, iPod in hand, singing "Dancing Queen" ... :-D
>Should it retain the directory structure for each match, e.g.>~/.zfs/view/alecm at mp3s/Music/A/Abba.mp3, or remap everything, e.g. >~/.zfs/view/alecm at mp3s/Abba.mp3? The former; my intent was to compute the subset of filesystem objects which match the patterns specified, and then further include any *extra* directory structure necessary to expose those objects - even if it does not match the rules - so that finally all necessary directory hierarchies are included but unnecessary ones are ignored. Saves on filename/path clashes. -a
>Insert mental image of Alec with headphones on, iPod in hand, singing >"Dancing Queen" ... :-DTough Guys Don''t Dance. More pointedly, I tend to fatally tread on people and/ or destroy things. I need a folk festival, a field, and serious amounts of ale to really get going. 8-) Back to the matter at hand: would/should "views" be static / frozen in time like a snapshot (presumably therefore computationally cheap to create) - or should they be dynamic like a SQL view? -a
Alec Muffett wrote:> Back to the matter at hand: would/should "views" be static / frozen > in time like a snapshot (presumably therefore computationally cheap > to create) - or should they be dynamic like a SQL view?both types please, I can see uses for both of them, though with the music example I only see the need for dynamic. -- Darren J Moffat
On Wed, Jun 07, 2006 at 04:15:04PM +0100, Alec Muffett wrote:> > >Should it retain the directory structure for each match, e.g. > >~/.zfs/view/alecm at mp3s/Music/A/Abba.mp3, or remap everything, e.g. > >~/.zfs/view/alecm at mp3s/Abba.mp3? > > The former; my intent was to compute the subset of filesystem objects > which match the patterns specified, and then further include any > *extra* directory structure necessary to expose those objects - even > if it does not match the rules - so that finally all necessary > directory hierarchies are included but unnecessary ones are ignored.But how could the filesystem know which directories to present without traversing all the way down and marking directories that contain objects relevant to the filter or directories that do? If views are dynamic then this is hard; if they''re allowed to take up persistent storage then you can take a hit creating them (i.e., not O(1)) and get what you want. I think you may want to be able to filter on object types and you''ll want conjunction and alternator (AND, OR) boolean operators, as well as grouping. That way you can filter directories and files separately. Typically you''d want to see all directories but only certain files. Also, why shouldn''t lofs grow similar support? Nico --
Nicolas Williams wrote:> ... > Also, why shouldn''t lofs grow similar support? >aha! This to me sounds much much better. Put all the funky potentially disasterous code, in lofs, not in zfs please :-) plus that way any filesystem will potentially get the "benefit" of views.
On Wed, Jun 07, 2006 at 11:15:43AM -0700, Philip Brown wrote:> >Also, why shouldn''t lofs grow similar support? > > aha! > This to me sounds much much better. Put all the funky potentially > disasterous code, in lofs, not in zfs please :-) plus that way any > filesystem will potentially get the "benefit" of views.Hmmm, no, I think we''ve been rejecting the idea of stacking filesystems because there are lots and lots of gotchas in stackable filesystems. So if we have "views" then ZFS should implement them natively. But lofs should probably get this feature also, if it really is useful. I''m not entirely sure why views are necessary. Use cases? And what''s wrong with find(1)? :-) :-) Nico --
Nicolas Williams wrote:> On Wed, Jun 07, 2006 at 11:15:43AM -0700, Philip Brown wrote: > >>>Also, why shouldn''t lofs grow similar support? >> >>aha! >>This to me sounds much much better. Put all the funky potentially >>disasterous code, in lofs, not in zfs please :-) plus that way any >>filesystem will potentially get the "benefit" of views. > > > Hmmm, no, I think we''ve been rejecting the idea of stacking filesystems > because there are lots and lots of gotchas in stackable filesystems. >but.. but.. you already have a "stackable" filesystem. lofs. you cant reject what you already have, and are going to keep :-) And if lofs is going to get the feature, there''s not much point in adding it into zfs natively.> I''m not entirely sure why views are necessary. Use cases? And what''s > wrong with find(1)? :-) :-)I think someone''s drunk the microsoft/oracle "filesystems are databases" punch...
On Wed, Jun 07, 2006 at 11:28:28AM -0700, Philip Brown wrote:> Nicolas Williams wrote: > >On Wed, Jun 07, 2006 at 11:15:43AM -0700, Philip Brown wrote: > >>This to me sounds much much better. Put all the funky potentially > >>disasterous code, in lofs, not in zfs please :-) plus that way any > >>filesystem will potentially get the "benefit" of views. > > > > > >Hmmm, no, I think we''ve been rejecting the idea of stacking filesystems > >because there are lots and lots of gotchas in stackable filesystems. > > > > but.. but.. you already have a "stackable" filesystem. lofs. you cant > reject what you already have, and are going to keep :-) > And if lofs is going to get the feature, there''s not much point in adding > it into zfs natively.But that''s it. IIRC there was a discussion on opensolaris-discuss about the 4.4.BSD unionfs. Nice idea, but ZFS snapshots and clones are even nicer.> >I''m not entirely sure why views are necessary. Use cases? And what''s > >wrong with find(1)? :-) :-) > > I think someone''s drunk the microsoft/oracle "filesystems are databases" > punch...They aren''t? :) They just aren''t indexed, typically. I''d love for ZFS to have an index feature mapping dnodes to objects referring to them (in the case of directories, {directory dnode, file name}). OR, with a user-land FEM we could have an index service maintaining such indexes generically, including content indexing. Nico --
Alec Muffett wrote:> > ...etc, etc, you can guess the extra sorts of tests to make. Go read > the manpage for "find" for inspiration.How is it different from creating a snapshot and running "find" on it? More convenient? Faster? Something else? Andreas.
On Wed, Jun 07, 2006 at 02:06:23PM -0700, Andreas Sterbenz wrote:> Alec Muffett wrote: > > > >...etc, etc, you can guess the extra sorts of tests to make. Go read > >the manpage for "find" for inspiration. > > How is it different from creating a snapshot and running "find" on it? > More convenient? Faster? Something else?Yeah, create a snapshot, clone it, find and remove all the things you don''t want to see, done. Views would be faster, initially (they could be O(1) to create), and easier to create, perhaps, but also slower to traverse, since each readdir might have to stat all objects named in the directory (oof). Nico --
>How is it different from creating a snapshot and running "find" on it? >More convenient? Faster? Something else?Ever encountered an operating system that allows you to mount and manipulate a ZIP file as a filesystem? Why did anyone bother to do that, when they could just "zip -l" and pipe the output through "more"? - alec
Alec suggested file system "views" and Andreas questioned:>> How is it different from creating a snapshot and running "find" on it? >> More convenient? Faster? Something else?To which Alec responded:> Ever encountered an operating system that allows you > to mount and manipulate a ZIP file as a filesystem? > > Why did anyone bother to do that, when they could just > "zip -l" and pipe the output through "more"?ZFS is being promoted as a very scalable File System. To me that implies that it should not just be able to manage huge amounts of data, it should also provide some tools to handle that data efficiently. "Views" fit with that model. Cheers, Henk -- e "Henk Langeveld" <hlangeveld at mailworks.org>
Henk, I think Alec was being sarcastic. :-P And, Alec - that wasn''t a good example. That was Windows Explorer (under XP and later) which presented ZIP files as browsable folders, not NTFS. We do need to be careful to differentiate between GUI and CLI _viewing_ of the filesystem, and the actual Filesystem Features (cf. my rants on ZFS "undo" :-) That said, a SQL-ish "view" feature as described would seem to be a good idea for inclusion in ZFS itself. I think I asked this before, but is there some place where we can get a "feature set" list for ZFS, and include things such as status (e.g. "finished", "work-in-progress", "under consideration", "pie-in-the-sky"), build targets (e.g. "B39", "aprox B60", etc.)? I''d be more than happy to volunteer to maintain/update such a page (and the associated communication coordination required). -Erik On Thu, 2006-06-08 at 00:32 +0200, Henk Langeveld wrote:> Alec suggested file system "views" and Andreas questioned: > >> How is it different from creating a snapshot and running "find" on it? > >> More convenient? Faster? Something else? > > To which Alec responded: > > Ever encountered an operating system that allows you > > to mount and manipulate a ZIP file as a filesystem? > > > > Why did anyone bother to do that, when they could just > > "zip -l" and pipe the output through "more"? > > ZFS is being promoted as a very scalable File System. To me > that implies that it should not just be able to manage huge > amounts of data, it should also provide some tools to handle > that data efficiently. "Views" fit with that model. > > Cheers, > Henk >-- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
On Wed, 2006-06-07 at 17:31, Nicolas Williams wrote:> Views would be faster, initially (they could be O(1) to create),if you''re not incrementally maintaining indexes, and you want O(1) creation and O(D) readdir (where D is the size of the directory) of directories, I don''t think you''ll be able to prune the directory tree.. space vs. time tradeoffs seem like they''ll be an Interesting problem.
On Wed, Jun 07, 2006 at 06:48:01PM -0400, Bill Sommerfeld wrote:> On Wed, 2006-06-07 at 17:31, Nicolas Williams wrote: > > Views would be faster, initially (they could be O(1) to create), > > if you''re not incrementally maintaining indexes, and you want O(1) > creation and O(D) readdir (where D is the size of the directory) of > directories, I don''t think you''ll be able to prune the directory tree..There''s no incremental index maintenance to do as views would be like snapshots, so once a view is computed for one directory it could be cached, possibly persistently, so that the next time the readdir doesn''t have to stat all the files in the directory.> space vs. time tradeoffs seem like they''ll be an Interesting problem.Yeah. I like it, but it seems like so just much gravy to me :)
On Wed, 2006-06-07 at 19:08, Nicolas Williams wrote:> There''s no incremental index maintenance to do as views would be like > snapshots, so once a view is computed for one directory it could be > cached, possibly persistently, so that the next time the readdir doesn''t > have to stat all the files in the directory.yeah, but doing a full tree walk under the covers in that first readdir will Suck. I''d rather the view creation be a bit slowerer in order that the first access to the view would be responsive.> I like it, but it seems like so just much gravy to me :)Agreed. - Bill
On Wed, Jun 07, 2006 at 07:21:18PM -0400, Bill Sommerfeld wrote:> On Wed, 2006-06-07 at 19:08, Nicolas Williams wrote: > > There''s no incremental index maintenance to do as views would be like > > snapshots, so once a view is computed for one directory it could be > > cached, possibly persistently, so that the next time the readdir doesn''t > > have to stat all the files in the directory. > > yeah, but doing a full tree walk under the covers in that first readdir > will Suck. I''d rather the view creation be a bit slowerer in order that > the first access to the view would be responsive.Pick your poison. O(N) creation or O(N) initial traversal.> > I like it, but it seems like so just much gravy to me :) > > Agreed.Since it''s gravy it no doubt would fall way down the list of RFEs. So the poison selection can be deferred. :-)
>> yeah, but doing a full tree walk under the covers in that first readdir>> will Suck. I''d rather the view creation be a bit slowerer in order that >> the first access to the view would be responsive. > >Pick your poison. O(N) creation or O(N) initial traversal. the low-hanging fruit is a "view" as a window onto a static snapshot. why not just create the view instantly and then filter all operations which scan a directory from that point onwards, try growing it incrementally upon demand? -a
On Thu, 2006-06-08 at 06:06, Alec Muffett wrote:> the low-hanging fruit is a "view" as a window onto a static snapshot. > > why not just create the view instantly and then filter all operations > which scan a directory from that point onwards, try growing it > incrementally upon demand?It seems difficult to bound the time taken by that initial scan, at least for certain queries. This doesn''t fit well with the observation that certain filesystem clients (most notably certain versions of NFS) have hard deadlines (due to a ticking timer on the client) on how long primitive filesystem operations can take before the client thinks the filesystem is broken. - Bill