Since ZFS is COW, can I have a read-only pool (on a central file server, or on a DVD, etc) with a separate block-differential pool on my local hard disk to store writes? This way, the pool in use can be read-write, even if the main pool itself is read-only, without having to make a full local copy of that read-only pool in order to be able to write to it, and without having to use messy filesystem-level union filesystem features. This would also be useful for live-system bootable DVDs, for which the writeable block-differential pool could be stored just in system memory in order to allow a fully functional non-persistent read-write pool without having to use the system''s hard disk, or stored on a small flash thumbdrive which the user carries along with the DVD to allow a persistent read-write pool without having to use the system''s hard disk. For yet another feature, this ability to copy newly written blocks to a separate differential pool could be used even if those new blocks are still written back to the main pool as usual; in this case, the differential pool would serve as a real-time differential backup. For example, I could make a full backup of my laptop''s hard disk onto DVDs, and then while in use have a thumbdrive plugged into the laptop. All updates to the hard disk would be copied to the thumbdrive, and when the thumbdrive fills up, it can be copied to a DVD and then erased. If the laptop''s hard disk dies, I can reconstruct the system''s disk state right up to the moment that it died by restoring all the DVDs to a new hard disk and then restoring the current contents of the thumbdrive. This would effectively provide the redundancy benefits of a full mirror of the laptop''s hard disk, but without having to lug along an entire full-size second hard disk, since I only have to carry a thumbdrive big enough to hold the amount of differential data I expect to generate. Finally, using the upcoming hard disk/flash disk combo drives in laptops, using the flash disk as the differential pool for the main hard disk pool (instead of writing the differential data immediately back to the main pool) would allow persistent writes without having to spin up the sleeping hard disk, and the differential pool could be flushed to the main pool sometime later when the hard disk is forced to spin up anyway to service a read. (This feature is independent of the use of an external thumbdrive to mirror differential data, and both features could be used at the same time.) All of these features would be enabled by allowing pool writes to be redirected to another destination (the differential pool) separate from the pool itself, and keeping track of the txg number at which the redirection began so that pool read requests will be sent to the right place. This message posted from opensolaris.org
On Tue, Jul 25, 2006 at 11:02:14PM -0700, Andrew wrote:> Since ZFS is COW, can I have a read-only pool (on a central file > server, or on a DVD, etc) with a separate block-differential pool on > my local hard disk to store writes? > > This way, the pool in use can be read-write, even if the main pool > itself is read-only, without having to make a full local copy of that > read-only pool in order to be able to write to it, and without having > to use messy filesystem-level union filesystem features.These are some interesting use cases. I''ll have to ponder how they could be best implemented in ZFS. Some of these cases can be solved simply by having a read-only device in the pool (eg. live-boot DVDs). However, cases where you want to transfer the data between devices are somewhat nontrivial (eg. hard drive + flash memory), at least until we have 4852783 "reduce pool capacity". I''ve filed RFE 6453741 "want mostly-read-only devices" to remember this request. --matt
Andrew <andrewee2 at y...> wrote:>> Since ZFS is COW, can I have a read-only pool (on a central file >> server, or on a DVD, etc) with a separate block-differential pool on >> my local hard disk to store writes? >> >> This way, the pool in use can be read-write, even if the main pool >> itself is read-only, without having to make a full local copy of that >> read-only pool in order to be able to write to it, and without having >> to use messy filesystem-level union filesystem features.Matthew Ahrens wrote:> These are some interesting use cases. I''ll have to ponder how they > could be best implemented in ZFS. > > Some of these cases can be solved simply by having a read-only device in > the pool (eg. live-boot DVDs). However, cases where you want to > transfer the data between devices are somewhat nontrivial (eg. hard > drive + flash memory), at least until we have 4852783 "reduce pool capacity". > > I''ve filed RFE 6453741 "want mostly-read-only devices" to remember this > request.I can see where this will lead to eventually... I''ve seen several scenarios where 4852783 becomes essential. How to implement such a thing? You first mark the vdev (do I say that correctly) as "evicting". Then you start a full scrub/rewrite of the whole pool, with any "evict"-ing components not available for writing, so the data is forced to be copied elsewhere. Once the scrub finishes, you change the state of the device to "evicted" and remove it from the pool. Odd thing is, with a read-only, or read-mostly device, you cannot put these marks on the original disk in the first place. This is slightly contrary to the concept of zfs storing all of its configuration and meta-data on-disk. Which to me implies, that before you can actually start scrubbing the pool, you first have to copy ALL metadata from the evicted device to the other devices in the pool. A consequence of this whole process is that this results in yet another method of installing the OS: - boot from a r/o zfs source image (dvd) - add sufficient storage to capture changes. - mark session as one of: o transient - discard change-pool on halt/crash/reboot o persistent - retain changes on reboot o permanent - make the pool bootable independantly from the original r/o image. - followed by the parallel - evict the source-image - configure the system, and purge any history. - if desired, reboot... A final implication is that on such a system, you cannot ever use the original source image, unless you tell the system explicitly NOT to import any zfs devices besides the image. The architecture of zfs is fascinating. Cheers, Henk
Is there any way to boot of from zfs disk "work around" ?? regards, Malahat
On Thu, Jul 27, 2006 at 08:17:03PM -0500, Malahat Qureshi wrote:> Is there any way to boot of from zfs disk "work around" ??Yes, see http://blogs.sun.com/roller/page/tabriz?entry=are_you_ready_to_rumble --mat
On Thu, Jul 27, 2006 at 07:25:55PM -0700, Matthew Ahrens wrote:> On Thu, Jul 27, 2006 at 08:17:03PM -0500, Malahat Qureshi wrote: > > Is there any way to boot of from zfs disk "work around" ?? > > Yes, see > http://blogs.sun.com/roller/page/tabriz?entry=are_you_ready_to_rumbleI followed those directions with snv_38 and was unsucessful, I wonder what I did wrong. Sadly it''s my work desktop and I had to stop screwing around with that and actually get work done. :) I think I''ll just wait until you can install directly to ZFS. Any ETA on that, btw? -brian
Brian Hechinger wrote:> On Thu, Jul 27, 2006 at 07:25:55PM -0700, Matthew Ahrens wrote: > >>On Thu, Jul 27, 2006 at 08:17:03PM -0500, Malahat Qureshi wrote: >> >>>Is there any way to boot of from zfs disk "work around" ?? >> >>Yes, see >>http://blogs.sun.com/roller/page/tabriz?entry=are_you_ready_to_rumble > > > I followed those directions with snv_38 and was unsucessful, I wonder > what I did wrong. > > Sadly it''s my work desktop and I had to stop screwing around with that > and actually get work done. :) > > I think I''ll just wait until you can install directly to ZFS. > > Any ETA on that, btw?While the official release of zfs-boot won''t be out until Update 4 at least, we''re working right now on getting enough pieces available through OpenSolaris so that users can put together a boot CD/DVD/image that will directly install a system with a zfs root. I can''t give an exact date, but we''re pretty close. I expect it within weeks, not months. Lori
On Fri, Jul 28, 2006 at 09:47:48AM -0600, Lori Alt wrote:> > While the official release of zfs-boot won''t be out > until Update 4 at least, we''re working right now on > getting enough pieces available through OpenSolaris > so that users can put together a boot CD/DVD/image > that will directly install a system with a zfs > root. I can''t give an exact date, but we''re pretty > close. I expect it within weeks, not months.That''s great Lori!! I could wait months, but I''d rather wait weeks. :) What about Express? I don''t have a problem running express. In fact, the desktop at work (a Dell POS) has snv_38 on it and the work laptop (a Dell POS, see a patern here? *G*) has snv_40 on it. ;) That being said, I''m (hopefuly safely) assuming that if this makes it into Update 4 it will include support for zfs-root/install on SPARC as well as x86? That would make my U80 at home very, very happy. :) -brian
Brian Hechinger wrote:> On Fri, Jul 28, 2006 at 09:47:48AM -0600, Lori Alt wrote: > >>While the official release of zfs-boot won''t be out >>until Update 4 at least, we''re working right now on >>getting enough pieces available through OpenSolaris >>so that users can put together a boot CD/DVD/image >>that will directly install a system with a zfs >>root. I can''t give an exact date, but we''re pretty >>close. I expect it within weeks, not months. > > > That''s great Lori!! I could wait months, but I''d rather wait weeks. :) > > What about Express?Probably not any time soon. If it makes U4, I think that would make it available in Express late this year.> > That being said, I''m (hopefuly safely) assuming that if this makes it > into Update 4 it will include support for zfs-root/install on SPARC as > well as x86? >We expect to release SPARC support at the same time as x86. Lori
On Fri, Jul 28, 2006 at 02:26:24PM -0600, Lori Alt wrote:> >What about Express? > Probably not any time soon. If it makes U4, > I think that would make it available in Express late > this year.Is there a specific Nevada build you are going to target? I''d love to start testing this as soon as possible. I have both SPARC and x86 here to play with.> >That being said, I''m (hopefuly safely) assuming that if this makes it > >into Update 4 it will include support for zfs-root/install on SPARC as > >well as x86? > > We expect to release SPARC support at the same time as x86.Most excellent. -brian
Brian Hechinger wrote:> On Fri, Jul 28, 2006 at 02:26:24PM -0600, Lori Alt wrote: > >>>What about Express? >> >>Probably not any time soon. If it makes U4, >>I think that would make it available in Express late >>this year. > > > Is there a specific Nevada build you are going to target? I''d love to > start testing this as soon as possible. I have both SPARC and x86 here > to play with.You need more than a Nevada build. You also need the installation code. We''re working on an OpenSolaris community web page for zfs-boot. On that web page will be links to files that can be downloaded for putting together a netinstall image or a DVD for installing a system with a zfs root file system. We hope to have that available in the next few weeks. lori> > >>>That being said, I''m (hopefuly safely) assuming that if this makes it >>>into Update 4 it will include support for zfs-root/install on SPARC as >>>well as x86? >> >>We expect to release SPARC support at the same time as x86. > > > Most excellent. > > -brian > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 15/08/06, Lori Alt <Lori.Alt at sun.com> wrote:> Brian Hechinger wrote: > > On Fri, Jul 28, 2006 at 02:26:24PM -0600, Lori Alt wrote: > > > >>>What about Express? > >> > >>Probably not any time soon. If it makes U4, > >>I think that would make it available in Express late > >>this year. > > > > > > Is there a specific Nevada build you are going to target? I''d love to > > start testing this as soon as possible. I have both SPARC and x86 here > > to play with. > > You need more than a Nevada build. You also need the > installation code. We''re working on an OpenSolaris > community web page for zfs-boot. On that web page > will be links to files that can be downloaded for > putting together a netinstall image or a DVD for > installing a system with a zfs root file system. > We hope to have that available in the next few weeks.That''s excellent news Lori, thanks to everyone who''s working on this. Are you planning to use a single pool, or an ''os pool/application pool'' split? As an aside, is there a general method to generate bootable opensolaris DVDs? The only way I know of getting opensolaris on is installing sxcr and then BFUing on top. -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/
"Dick Davies" <rasputnik at gmail.com> wrote:> As an aside, is there a general method to generate bootable > opensolaris DVDs? The only way I know of getting opensolaris on > is installing sxcr and then BFUing on top.A year ago, I did publish a toolkit to create bootable SchilliX CDs/DVDs. Would this help? J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
On 16/08/06, Joerg Schilling <schilling at fokus.fraunhofer.de> wrote:> "Dick Davies" <rasputnik at gmail.com> wrote: > > > As an aside, is there a general method to generate bootable > > opensolaris DVDs? The only way I know of getting opensolaris on > > is installing sxcr and then BFUing on top. > > A year ago, I did publish a toolkit to create bootable SchilliX CDs/DVDs. > Would this help?Definitely - but I''m just curious to be honest. I just wanted to burn the appropriate thing onto the ''i boot opensolaris'' blank dvds I got sent the other day :) -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/
Dick Davies wrote:> On 15/08/06, Lori Alt <Lori.Alt at sun.com> wrote: > >> Brian Hechinger wrote: >> > On Fri, Jul 28, 2006 at 02:26:24PM -0600, Lori Alt wrote: >> > >> >>>What about Express? >> >> >> >>Probably not any time soon. If it makes U4, >> >>I think that would make it available in Express late >> >>this year. >> > >> > >> > Is there a specific Nevada build you are going to target? I''d love to >> > start testing this as soon as possible. I have both SPARC and x86 >> here >> > to play with. >> >> You need more than a Nevada build. You also need the >> installation code. We''re working on an OpenSolaris >> community web page for zfs-boot. On that web page >> will be links to files that can be downloaded for >> putting together a netinstall image or a DVD for >> installing a system with a zfs root file system. >> We hope to have that available in the next few weeks. > > > That''s excellent news Lori, thanks to everyone who''s working > on this. Are you planning to use a single pool, > or an ''os pool/application pool'' split?I''m not quite sure what you mean by an "os pool/application pool" split. I would have thought that the *executables* for an application would normally be installed somewhere in the Solaris namespace (i.e., the name space established by a Solaris installation: root, /usr, /opt, /var, and so ) and would thus be part of the "personality" of a system. Data, on the other hand, is typically NOT part of that namespace. So your databases, etc. would often be installed somewhere else. Thus I think of the most important split as the "os pool/data pool" split. Maybe that''s what you meant. If so, then the answer to your question is: The plan right now is to encourage (though not mandate) separate pools for the Solaris name space and for data. There are three reasons for this: 1. There will be some restrictions on root pools, at least initially and perhaps permanently, that you would probably not want to place on pools used for data. For example, no RAID-Z or concatenation of vdevs. These might be relaxed at some time, but right now, limitations in the boot PROMs cause us to place restrictions on the devices you can place in a root pool. (root mirroring WILL be supported, however). 2. Data is theoretically shareable and transferable among different instruction set architectures. System software is not. 3. There are advantages to separating the "personality" of a machine from its data. If they are separated, one can be modified (i.e., patched, upgraded, moved from one kind of storage to another, etc) without having to affect the other.> > As an aside, is there a general method to generate bootable > opensolaris DVDs? The only way I know of getting opensolaris on > is installing sxcr and then BFUing on top.I actually don''t know much about that right now. Someone else might have to answer that question for you. Lori
On 17/08/06, Lori Alt <Lori.Alt at sun.com> wrote:> Dick Davies wrote:> > That''s excellent news Lori, thanks to everyone who''s working > > on this. Are you planning to use a single pool, > > or an ''os pool/application pool'' split?> Thus I think of the most important split as the "os pool/data pool" > split. Maybe that''s what you meant.That''s it, yes :) I should probably have said service rather than application.> ...... limitations > in the boot PROMs cause us to place restrictions on the devices > you can place in a root pool. (root mirroring WILL be supported, > however).Does boot prom support mean this will be SPARC only? That''s interesting (last time I tried Tabriz'' hack, it was x86 only). Or is x86 zfs root going to need a grub /boot partition on one of the disks? -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/
Dick Davies wrote:> On 17/08/06, Lori Alt <Lori.Alt at sun.com> wrote: > >> Dick Davies wrote: > > >> > That''s excellent news Lori, thanks to everyone who''s working >> > on this. Are you planning to use a single pool, >> > or an ''os pool/application pool'' split? > > >> Thus I think of the most important split as the "os pool/data pool" >> split. Maybe that''s what you meant. > > > That''s it, yes :) > I should probably have said service rather than application. > >> ...... limitations >> in the boot PROMs cause us to place restrictions on the devices >> you can place in a root pool. (root mirroring WILL be supported, >> however). > > > Does boot prom support mean this will be SPARC only? That''s > interesting (last time I tried Tabriz'' hack, it was x86 only).No, zfs boot will be supported on both x86 and sparc. Sparc''s OBP, and various x86 BIOS''s both have restrictions on the devices that can be accessed at boot time, so we need to limit the devices in a root pool on both architectures.> > Or is x86 zfs root going to need a grub /boot partition on one > of the disks?On x86, each disk capable of booting the system (which means each disk in a root pool) will have grub installed on it in a disk slice which occupies the first few blocks of the disk. It''s not the same as the old /boot partition, because all the slice contains is grub. It doesn''t contain a file system. Lori
On 18/08/06, Lori Alt <Lori.Alt at sun.com> wrote:> No, zfs boot will be supported on both x86 and sparc. Sparc''s > OBP, and various x86 BIOS''s both have restrictions on the devices > that can be accessed at boot time, so we need to limit the > devices in a root pool on both architectures.Gotcha. I wasn''t sure if you were proposing requiring a custom BIOS on x86, but I take it (from your next point) you''re just chainloading a ZFS-aware grub> > Or is x86 zfs root going to need a grub /boot partition on one > > of the disks? > > On x86, each disk capable of booting the system (which means each > disk in a root pool) will have grub installed on it in a disk > slice which occupies the first few blocks of the disk. It''s not > the same as the old /boot partition, because all the slice > contains is grub. It doesn''t contain a file system.I think that was really what I was getting at. So long as one of the disks is still alive, and the BIOS can boot of it, then you''d be alright? That sounds perfect - the implementation is really not that important to me, so long as there''s no single point of failure. Thanks for your time, and have a good weekend. -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/
Lori Alt wrote:> > No, zfs boot will be supported on both x86 and sparc. Sparc''s > OBP, and various x86 BIOS''s both have restrictions on the devices > that can be accessed at boot time, so we need to limit the > devices in a root pool on both architectures.Hi Lori. Can you expand a bit on the above? What sort of limitations are you referring too? (Boot time? Topology?)
Torrey McMahon wrote:> Lori Alt wrote: > >> >> No, zfs boot will be supported on both x86 and sparc. Sparc''s >> OBP, and various x86 BIOS''s both have restrictions on the devices >> that can be accessed at boot time, so we need to limit the >> devices in a root pool on both architectures. > > > Hi Lori. > > Can you expand a bit on the above? What sort of limitations are you > referring too? (Boot time? Topology?)The limitation is mainly about the *number* of disks that can be accessed at one time. If we were going to support booting off a set of disks in a RAID-Z configuration, the early boot code would have to read some blocks from one disk, and then some blocks from another disk, and so on. There are difficulties doing that when using the capabilities of OBP or the BIOS to do I/O. (and if you want me to be more specific about what THOSE difficulties are, I''d have to get someone who knows more about BIOS and OBP to answer the question.) But with straight mirroring, there''s no such problem because any disk in the mirror can supply all of the disk blocks needed to boot. lori
Torrey McMahon wrote:> Lori Alt wrote: > >> >> No, zfs boot will be supported on both x86 and sparc. Sparc''s >> OBP, and various x86 BIOS''s both have restrictions on the devices >> that can be accessed at boot time, so we need to limit the >> devices in a root pool on both architectures. > > > Hi Lori. > > Can you expand a bit on the above? What sort of limitations are you > referring too? (Boot time? Topology?)I think what Lori is referring to here is that we need to limit the rootpool to BIOS/OBP visible devices; not all devices are visible from the BIOS/OBP (fibre channel devices, for example). Tabriz> _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Lori said:> The limitation is mainly about the *number* of disks > that can be accessed at one time. > ... > But with straight mirroring, there''s no such problem > because any disk in the mirror can supply all of the > disk blocks needed to boot.Does that mean that these restrictions will go away once replication can be varied on a per dataset (or per file) basis? You could have all your ''essential to boot'' files mirrored across all disks, then raidz2 the rest... Steve.
Bennett, Steve wrote:> Lori said: > >>The limitation is mainly about the *number* of disks >>that can be accessed at one time. >>... >>But with straight mirroring, there''s no such problem >>because any disk in the mirror can supply all of the >>disk blocks needed to boot. > > > Does that mean that these restrictions will go away once replication can > be varied on a per dataset (or per file) basis? You could have all your > ''essential to boot'' files mirrored across all disks, then raidz2 the > rest... >Maybe. It depends on how per-file replication is implemented. I don''t think we''ve made any design decisions at this time that would prevent that from working in the future. Lori
Tabriz Leman wrote:> > Torrey McMahon wrote: > >> Lori Alt wrote: >> >>> >>> No, zfs boot will be supported on both x86 and sparc. Sparc''s >>> OBP, and various x86 BIOS''s both have restrictions on the devices >>> that can be accessed at boot time, so we need to limit the >>> devices in a root pool on both architectures. >> >> >> Hi Lori. >> >> Can you expand a bit on the above? What sort of limitations are you >> referring too? (Boot time? Topology?) > > I think what Lori is referring to here is that we need to limit the > rootpool to BIOS/OBP visible devices; not all devices are visible from > the BIOS/OBP (fibre channel devices, for example).Actually, OBP and BIOS should have access to fabric devices. I know OBP does and I''ve seen docs that mention BIOS does. (Look in the x86 fabric boot docs for example) I think the problem is more around, as Lori mentioned in her email, the ability to traverse a ZFS pool when the ZFS drivers haven''t been loaded.