Would this be possible to implement ontop ZFS? Maybe it is a dumb idea, I dont know. What do you think, and how to improve this? Assume all files are put in the zpool, helter skelter. And then you can create arbitrary different filters that shows you the files you want to see. As of now, you have files in one directory structure. This makes the organization of the files, hardcoded. You have /Movies/Action and that is it. But if you had all movies in one large zpool, and if you could programmatically define different structures that act as filters, you could have different directory structures. Programmatically defined directory structure1, that acts on the zpool: /Movies/Action Programmatically defined directory structure2: /Movies/Actors/AlPacino etc. Maybe this is what MS WinFS was about? Maybe tag the files? Maybe a relational database ontop ZFS? Maybe no directories at all? I dont know, just brain storming. Is this is a dumb idea? Or old idea? -- This message posted from opensolaris.org
This actually sounds a little like what ms is trying to accomplish, in win7, with libraries. They will act as standard folders if you treat them as such. But they are really designed to group different pools of files into one easy place. You just have to configure it to pull from local and remote sources. I have heard it works well with win home server, and win7 networks. Its also similar to what google and the like are doing with their web crawlers. But I think this is something better left to run on top of the file system. Rather than integrated into the file system. A true database and "crawling bot" would seem to be the better method of implementing this. ------Original Message------ From: Orvar Korvar Sender: zfs-discuss-bounces at opensolaris.org To: zfs Discuss Subject: [zfs-discuss] Dumb idea? Sent: Oct 24, 2009 8:12 AM Would this be possible to implement ontop ZFS? Maybe it is a dumb idea, I dont know. What do you think, and how to improve this? Assume all files are put in the zpool, helter skelter. And then you can create arbitrary different filters that shows you the files you want to see. As of now, you have files in one directory structure. This makes the organization of the files, hardcoded. You have /Movies/Action and that is it. But if you had all movies in one large zpool, and if you could programmatically define different structures that act as filters, you could have different directory structures. Programmatically defined directory structure1, that acts on the zpool: /Movies/Action Programmatically defined directory structure2: /Movies/Actors/AlPacino etc. Maybe this is what MS WinFS was about? Maybe tag the files? Maybe a relational database ontop ZFS? Maybe no directories at all? I dont know, just brain storming. Is this is a dumb idea? Or old idea? -- This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss Sent from my BlackBerry? smartphone with SprintSpeed
Or in OS X with smart folders where you define a set of search terms and as write operations occur on the known filesystems the folder contents will be updated to reflect the current state of the attached filesystems.... The structures you defined seemed to be designed around the idea of reductionism (ie - subfolders representing a subset of the parent) which cannot currently be implemented in Libraries or Smart folders since the contents are read-only listings. I don''t know for sure about the Win7 Libraries behaviour though - it might be more permissive in this respect... Erik On 25 oct. 2009, at 20:48, jay at lentecs.com wrote:> This actually sounds a little like what ms is trying to accomplish, > in win7, with libraries. They will act as standard folders if you > treat them as such. But they are really designed to group different > pools of files into one easy place. You just have to configure it > to pull from local and remote sources. I have heard it works well > with win home server, and win7 networks. > > Its also similar to what google and the like are doing with their > web crawlers. > > But I think this is something better left to run on top of the file > system. Rather than integrated into the file system. A true > database and "crawling bot" would seem to be the better method of > implementing this. > > ------Original Message------ > From: Orvar Korvar > Sender: zfs-discuss-bounces at opensolaris.org > To: zfs Discuss > Subject: [zfs-discuss] Dumb idea? > Sent: Oct 24, 2009 8:12 AM > > Would this be possible to implement ontop ZFS? Maybe it is a dumb > idea, I dont know. What do you think, and how to improve this? > > Assume all files are put in the zpool, helter skelter. And then you > can create arbitrary different filters that shows you the files you > want to see. > > As of now, you have files in one directory structure. This makes the > organization of the files, hardcoded. You have /Movies/Action and > that is it. But if you had all movies in one large zpool, and if you > could programmatically define different structures that act as > filters, you could have different directory structures. > > Programmatically defined directory structure1, that acts on the zpool: > /Movies/Action > > Programmatically defined directory structure2: > /Movies/Actors/AlPacino > > etc. > > Maybe this is what MS WinFS was about? Maybe tag the files? Maybe a > relational database ontop ZFS? Maybe no directories at all? I dont > know, just brain storming. Is this is a dumb idea? Or old idea? > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > Sent from my BlackBerry? smartphone with SprintSpeed > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Sun, 25 Oct 2009, jay at lentecs.com wrote:> But I think this is something better left to run on top of the file > system. Rather than integrated into the file system. A true > database and "crawling bot" would seem to be the better method of > implementing this.I think that you will rue the day you unleash any "crawling bot" on your system. As I have explained to others, turning your system''s filesystem into a search engine is both a privacy and security hazard. It seems that Windows 7 does this automatically and Apple''s OS X does something similar. Hopefully the feature can be disabled. It is naive to think that no one else will ever access your system and appreciate what they can find in a second, when otherwise it might have taken hours or days. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
I remember BFS (BeOS) did something very similar, it had extended metadata attributes akin to having a relational DB built-in. Very searchable, it also had tagging with callback notification on filechanges so you don''t have to waste cycles with periodic checking. On 10/26/09 01:22, zfs-discuss-request at opensolaris.org wrote:> ------------------------------ > > Message: 2 > Date: Sun, 25 Oct 2009 19:48:55 +0000 > From: jay at lentecs.com > To: "Orvar Korvar" <knatte_fnatte_tjatte at yahoo.com>, "zfs Discuss" > <zfs-discuss at opensolaris.org> > Subject: Re: [zfs-discuss] Dumb idea? > Message-ID: > <1672618364-1256500137-cardhu_decombobulator_blackberry.rim.net-1333905628- at bda305.bisx.prod.on.blackberry> > > Content-Type: text/plain; charset="Windows-1252" > > This actually sounds a little like what ms is trying to accomplish, in win7, with libraries. They will act as standard folders if you treat them as such. But they are really designed to group different pools of files into one easy place. You just have to configure it to pull from local and remote sources. I have heard it works well with win home server, and win7 networks. > > Its also similar to what google and the like are doing with their web crawlers. > > But I think this is something better left to run on top of the file system. Rather than integrated into the file system. A true database and "crawling bot" would seem to be the better method of implementing this. > > ------Original Message------ > From: Orvar Korvar > Sender: zfs-discuss-bounces at opensolaris.org > To: zfs Discuss > Subject: [zfs-discuss] Dumb idea? > Sent: Oct 24, 2009 8:12 AM > > Would this be possible to implement ontop ZFS? Maybe it is a dumb idea, I dont know. What do you think, and how to improve this? > > Assume all files are put in the zpool, helter skelter. And then you can create arbitrary different filters that shows you the files you want to see. > > As of now, you have files in one directory structure. This makes the organization of the files, hardcoded. You have /Movies/Action and that is it. But if you had all movies in one large zpool, and if you could programmatically define different structures that act as filters, you could have different directory structures. > > Programmatically defined directory structure1, that acts on the zpool: > /Movies/Action > > Programmatically defined directory structure2: > /Movies/Actors/AlPacino > > etc. > > Maybe this is what MS WinFS was about? Maybe tag the files? Maybe a relational database ontop ZFS? Maybe no directories at all? I dont know, just brain storming. Is this is a dumb idea? Or old idea? > > > >
On Sat, Oct 24, 2009 at 12:12 PM, Orvar Korvar <knatte_fnatte_tjatte at yahoo.com> wrote:> Would this be possible to implement ontop ZFS? Maybe it is a dumb idea, I dont know. What do you think, and how to improve this? > > Assume all files are put in the zpool, helter skelter. And then you can create arbitrary different filters that shows you the files you want to see.So why not pipe ls through grep?> As of now, you have files in one directory structure. This makes the organization of the files, hardcoded. You have /Movies/Action and that is it. But if you had all movies in one large zpool, and if you could programmatically define different structures that act as filters, you could have different directory structures. > > Programmatically defined directory structure1, that acts on the zpool: > /Movies/Action > > Programmatically defined directory structure2: > /Movies/Actors/AlPacino > > etc. > > Maybe this is what MS WinFS was about? Maybe tag the files? Maybe a relational database ontop ZFS? Maybe no directories at all? I dont know, just brain storming. Is this is a dumb idea? Or old idea?Old idea - I remember systems that did this or variants of it a really long time ago. However, there you knew ahead of time what applications you were going to run. Does it make sense to fold this sort of intelligence into the filesystem, or is it really an application-level task? (And then remember that the filesystem can be accessed by an almost infinite array of applications, not to mention remotely over NFS or CIFS - how would they make sense of what you might build?) Talking of NFS, you could imagine some sort of user-level nfs server atop the filesystem that presents different views. Or other layered filesystems (mvfs, for example) that present a modified view. That seems a more fruitful approach than trying to build this into ZFS itself. The problems then become: what additional metadata do you need? (You can hide the metadata in extended attributes if you don''t want it obviously visible.) And how do you keep the metadata in sync with the real data in the face of modifications by applications that aren''t aware of your scheme? -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
>>>>> "pt" == Peter Tribble <peter.tribble at gmail.com> writes:pt> Does it make sense to fold this sort of intelligence into the pt> filesystem, or is it really an application-level task? in general it seems all the time app writers want to access hundreds of thousands of files by unique id rather than filename, and the POSIX directory interface is not really up to the task. In theory there are supposed to be all these hashed directories and decent performance with huge directories, but in practice posters keep mentioning stupid gotchya''s like API''s that enforce full scans of the directory even if the filesystem itself doesn''t, or lack of garbage collection so that churning queue directories end up swollen with dead space. The usual workaround for all gotchyas of making two-levels-deep nested subdirs just means a couple extra seeks per subdir level before you reach your inode number, extra seeks just to accomodate the overcomplex filesystem API, so sometimes app developers end up rolling their own filesystem-in-a-file like Hadoop instead just to get away from the presumption they want filenames and nested directories. A uuid-subdir interface, or directoryless uuid-filesystem option for ''zfs create'', might gather a lot of application consumers. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091029/c77781c7/attachment.bin>
Miles Nordin wrote:>>>>>> "pt" == Peter Tribble <peter.tribble at gmail.com> writes: >>>>>> > > pt> Does it make sense to fold this sort of intelligence into the > pt> filesystem, or is it really an application-level task? > > in general it seems all the time app writers want to access hundreds > of thousands of files by unique id rather than filename, and the POSIX > directory interface is not really up to the task.Dear zfs''ers It''s possible to heavily influence the next POSIX/UNIX standard if you''re interested to test or give feedback ping me off list. The Open Group does take feedback before they implement the next version of the standard and now is a good time to participate in that. Best, ./Christopher
"C. Bergstr?m" <codestr0m at osunix.org> wrote:> Miles Nordin wrote: > >>>>>> "pt" == Peter Tribble <peter.tribble at gmail.com> writes: > >>>>>> > > > > pt> Does it make sense to fold this sort of intelligence into the > > pt> filesystem, or is it really an application-level task? > > > > in general it seems all the time app writers want to access hundreds > > of thousands of files by unique id rather than filename, and the POSIX > > directory interface is not really up to the task. > Dear zfs''ers > > It''s possible to heavily influence the next POSIX/UNIX standard if > you''re interested to test or give feedback ping me off list. The Open > Group does take feedback before they implement the next version of the > standard and now is a good time to participate in that.You seem to missunderstand how the POSIX standard committee works. The POSIX standard usually does not give up previous definitions and it only adopts to already existing and well tested implementations in case they fit well into the existing standard. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily