Luke Schwab
2008-Oct-06 17:46 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
Hi, I am having a problem running zpool imports when we import multiple storage pools at one time. Below are the details of the setup: - We are using a SAN with Sun 6140 storage arrays. - Dual port HBA on each server is Qlogic running the QLC driver with Sun mpxio(SFCSM) running. - We have 400+ luns on the SAN. We can''t split them up because the luns any of the luns may be able to failover between the different servers. We are running rather large farm where all servers need to see all the luns in case of failover. -Running Solaris 10 U5 kernal. Below are the times I see when importing pools from the SAN storage: 1 pool - 30 seconds 2 pools - 1 minute for both to finish 3 pools - 1 min 30 sec for all 3 to finish 4 pools - 2 minutes for all 4 to finish When I run more then one zpool import in parallel, I can see that all zpool imports are queued up. The more pools I try to import, the longer the pools take to import. For example, when importing 2 pools, the first pool takes 1 minute but then just a few seconds later the 2nd pool finishes its import. When I import 3 pools, the first pool takes 1.5 minutes and then the other two pools finish just after that. The problem we are seeing it that we need to failover up to 32 pools at one time on a server and the imports end up timing out after 5-10 minutes because we are trying to import too many pools at one time. Is this a design choice with ZFS coding or a bug? Is there anything I can do to increase my import times? We do have the same setup on one of our SANs with only 10-20 luns instead of 400+ and the imports take only 1-3 seconds. My guess here is that the large number of luns is effecting imports. But our virtual farm design is broken if we can imported at least 30-40 luns in under a minute on a given server. Any thoughts or questions would be great. Thanks, Luke -- This message posted from opensolaris.org
Tomas Ögren
2008-Oct-06 17:59 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
On 06 October, 2008 - Luke Schwab sent me these 2,0K bytes:> Is this a design choice with ZFS coding or a bug? Is there anything I > can do to increase my import times? We do have the same setup on one > of our SANs with only 10-20 luns instead of 400+ and the imports take > only 1-3 seconds. My guess here is that the large number of luns is > effecting imports. But our virtual farm design is broken if we can > imported at least 30-40 luns in under a minute on a given server.I believe it will scan all available devices (LUNs) for pool information.. If you only want to scan a subset, you can for instance make a new directory somewhere and put symlinks there to the real devices, then ''zpool import -d /that/directory'' to only search there for devices to consider. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
Richard Elling
2008-Oct-06 18:44 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
Do you have a lot of snapshots? If so, CR 6612830 could be contributing. Alas, many such fixes are not yet available in S10. -- richard Luke Schwab wrote:> Hi, > I am having a problem running zpool imports when we import multiple storage pools at one time. Below are the details of the setup: > > - We are using a SAN with Sun 6140 storage arrays. > - Dual port HBA on each server is Qlogic running the QLC driver with Sun mpxio(SFCSM) running. > - We have 400+ luns on the SAN. We can''t split them up because the luns any of the luns may be able to failover between the different servers. We are running rather large farm where all servers need to see all the luns in case of failover. > -Running Solaris 10 U5 kernal. > > Below are the times I see when importing pools from the SAN storage: > 1 pool - 30 seconds > 2 pools - 1 minute for both to finish > 3 pools - 1 min 30 sec for all 3 to finish > 4 pools - 2 minutes for all 4 to finish > > When I run more then one zpool import in parallel, I can see that all zpool imports are queued up. The more pools I try to import, the longer the pools take to import. For example, when importing 2 pools, the first pool takes 1 minute but then just a few seconds later the 2nd pool finishes its import. When I import 3 pools, the first pool takes 1.5 minutes and then the other two pools finish just after that. > > The problem we are seeing it that we need to failover up to 32 pools at one time on a server and the imports end up timing out after 5-10 minutes because we are trying to import too many pools at one time. > > Is this a design choice with ZFS coding or a bug? Is there anything I can do to increase my import times? We do have the same setup on one of our SANs with only 10-20 luns instead of 400+ and the imports take only 1-3 seconds. My guess here is that the large number of luns is effecting imports. But our virtual farm design is broken if we can imported at least 30-40 luns in under a minute on a given server. > > Any thoughts or questions would be great. > > Thanks, > Luke > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Hi all In another thread a short while ago.. A cool little movie with some gumballs was all we got to learn about green-bytes. The product launched and maybe some of the people that follow this list have had a chance to take a look at the code/product more closely? Wstuart asked how they were going to handle section 3.1 of the CDDL, but nobody from green-bytes even made an effort to clarify this. I called since I''m consulting with companies who are potential customers, but are any of developers even subscribed to this list? After a call and exchanging a couple emails I''m left with the impression the source will *not* be released publicly or to customers. I''m not the copyright holder, a legal expert, or even a customer, but can someone from Sun or green-bytes make a comment. I apologize for being a bit off topic, but is this really acceptable to the community/Sun in general? Maybe the companies using Solaris and NetApp don''t care about source code, but then the whole point of opening Solaris is just reduced to marketing hype. In the defense of green-bytes.. I think they''ve truly spent some time developing an interesting product and want to protect their ideas and investment. I said this on the phone, but in my very humble opinion nobody is going to "steal" their patches. In a way I''m curious what others think before a good company gets a lot of bad PR over an honest and small oversight. Cheers, Christopher Bergstr?m +1.206.279.5000
zfs-discuss-bounces at opensolaris.org wrote on 10/06/2008 01:57:10 PM:> Hi all > > In another thread a short while ago.. A cool little movie with some > gumballs was all we got to learn about green-bytes. The product > launched and maybe some of the people that follow this list have had a > chance to take a look at the code/product more closely? Wstuart asked > how they were going to handle section 3.1 of the CDDL, but nobody from > green-bytes even made an effort to clarify this. I called since I''m > consulting with companies who are potential customers, but are any of > developers even subscribed to this list? > > After a call and exchanging a couple emails I''m left with the impression > the source will *not* be released publicly or to customers. I''m not the > copyright holder, a legal expert, or even a customer, but can someone > from Sun or green-bytes make a comment. I apologize for being a bit off > topic, but is this really acceptable to the community/Sun in general? > Maybe the companies using Solaris and NetApp don''t care about source > code, but then the whole point of opening Solaris is just reduced to > marketing hype. >Yes, this would be interesting. CDDL requires them to release code for any executable version they ship. Considering they claim to have "...start with ZFS and makes it better" it sounds like they have modified CDDL covered code. Since Sun owns that code they would need to rattle the cage. Sun? Anyone have any talks with these guys yet?> In the defense of green-bytes.. I think they''ve truly spent some time > developing an interesting product and want to protect their ideas and > investment. I said this on the phone, but in my very humble opinion > nobody is going to "steal" their patches. In a way I''m curious what > others think before a good company gets a lot of bad PR over an honest > and small oversight.If they take opensource code and modify it and that code requires release of the derivative code then there is no "stealing" involved. It is kind of like walking into a car dealership that had a sign "Free winter tires with purchase of car" and feeling slighted when you can''t walk out with just the tires for free. -Wade
Scott Williamson
2008-Oct-06 21:19 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
Speaking of this, is there a list anywhere that details what we can expect to see for (zfs) updates in S10U6? On Mon, Oct 6, 2008 at 2:44 PM, Richard Elling <Richard.Elling at sun.com>wrote:> Do you have a lot of snapshots? If so, CR 6612830 could be contributing. > Alas, many such fixes are not yet available in S10. > -- richard > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081006/8177409a/attachment.html>
Richard Elling
2008-Oct-07 00:08 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
Scott Williamson wrote:> Speaking of this, is there a list anywhere that details what we can > expect to see for (zfs) updates in S10U6?The official release name is "Solaris 10 10/08" http://www.sun.com/software/solaris/10 has links to "what''s new" videos. When the release is downloadable, full doc set will be ready. -- richard
Hello Wade, Monday, October 6, 2008, 8:56:12 PM, you wrote: WSfc> zfs-discuss-bounces at opensolaris.org wrote on 10/06/2008 01:57:10 PM:>> Hi all >> >> In another thread a short while ago.. A cool little movie with some >> gumballs was all we got to learn about green-bytes. The product >> launched and maybe some of the people that follow this list have had a >> chance to take a look at the code/product more closely? Wstuart asked >> how they were going to handle section 3.1 of the CDDL, but nobody from >> green-bytes even made an effort to clarify this. I called since I''m >> consulting with companies who are potential customers, but are any of >> developers even subscribed to this list? >> >> After a call and exchanging a couple emails I''m left with the impression >> the source will *not* be released publicly or to customers. I''m not the >> copyright holder, a legal expert, or even a customer, but can someone >> from Sun or green-bytes make a comment. I apologize for being a bit off >> topic, but is this really acceptable to the community/Sun in general? >> Maybe the companies using Solaris and NetApp don''t care about source >> code, but then the whole point of opening Solaris is just reduced to >> marketing hype. >>WSfc> Yes, this would be interesting. CDDL requires them to release code for WSfc> any executable version they ship. Considering they claim to have "...start WSfc> with ZFS and makes it better" it sounds like they have modified CDDL WSfc> covered code. Since Sun owns that code they would need to rattle the WSfc> cage. Sun? Anyone have any talks with these guys yet? Isn''t CDDL file based so they could implement all the new functionality in new files and only added some includes and couple of useless (if provided alone) changes. ? Best regards, Robert mailto:milek at task.gda.pl http://milek.blogspot.com
Some people wrote:> > > covered code. Since Sun owns that code they would need to rattle the > > cage. Sun? Anyone have any talks with these guys yet? > > Isn''t CDDL file based so they could implement all the new functionality in > >Wouldn''t it be great if programmers could just focus on writing code rather than having to worry about getting sued over whether someone else is able or not to make a derivative program from their code? -- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081007/87220456/attachment.html>
>Some people wrote: > >> >> > covered code. Since Sun owns that code they would need to rattle the >> > cage. Sun? Anyone have any talks with these guys yet? >> >> Isn''t CDDL file based so they could implement all the new functionality in >> >> >Wouldn''t it be great if programmers could just focus on writing code rather >than having to worry about getting sued over whether someone else is able or >not to make a derivative program from their code?Yep, but in THIS world it *is* an important consideration. http://www.linux-watch.com/news/NS3761924232.html http://www.linuxdevices.com/news/NS7575957635.html http://www.theinquirer.net/en/inquirer/news/2004/01/06/kiss-technology-accused-of-stealing-free-software If you use someone else''s code, make sure you read the license and follow it; then you should be fine. Casper
zfs-discuss-bounces at opensolaris.org wrote on 10/07/2008 07:15:46 AM:> Hello Wade, > > Monday, October 6, 2008, 8:56:12 PM, you wrote: > > WSfc> zfs-discuss-bounces at opensolaris.org wrote on 10/06/2008 01:57:10PM:> > >> Hi all > >> > >> In another thread a short while ago.. A cool little movie with some > >> gumballs was all we got to learn about green-bytes. The product > >> launched and maybe some of the people that follow this list have had a > >> chance to take a look at the code/product more closely? Wstuart asked > >> how they were going to handle section 3.1 of the CDDL, but nobody from > >> green-bytes even made an effort to clarify this. I called since I''m > >> consulting with companies who are potential customers, but are any ofs > >> developers even subscribed to this list? > >> > >> After a call and exchanging a couple emails I''m left with theimpression> >> the source will *not* be released publicly or to customers. I''m notthe> >> copyright holder, a legal expert, or even a customer, but can someone > >> from Sun or green-bytes make a comment. I apologize for being a bitoff> >> topic, but is this really acceptable to the community/Sun in general? > >> Maybe the companies using Solaris and NetApp don''t care about source > >> code, but then the whole point of opening Solaris is just reduced to > >> marketing hype. > >> > > WSfc> Yes, this would be interesting. CDDL requires them to releasecode for> WSfc> any executable version they ship. Considering they claim to > have "...start > WSfc> with ZFS and makes it better" it sounds like they have modifiedCDDL> WSfc> covered code. Since Sun owns that code they would need to rattlethe> WSfc> cage. Sun? Anyone have any talks with these guys yet? > > Isn''t CDDL file based so they could implement all the new functionalityin> new files and only added some includes and couple of useless (if > provided alone) changes. >Robert, Yes -- file based and derivative code based (copy covered code to a new file and that file is now covered). New code in a new file is not automatically covered and the authors choice. That said, if they have added dedup to zfs they may have taken extraordinary steps to segment their code from covered code. My hunch is they did not. Everything from resilver, zil etc would need to be dedup aware. Either case, release the required code and there is no harm no foul right? If it is stubs, then so be it. I am more interested to see if they implemented it the same way I started to or if it is something new. If it is code complete and all covered even better. -Wade
Wade.Stuart at fallon.com wrote:> Yes -- file based and derivative code based (copy covered code to a > new file and that file is now covered). New code in a new file is not > automatically covered and the authors choice. That said, if they have > added dedup to zfs they may have taken extraordinary steps to segment their > code from covered code. My hunch is they did not. Everything from > resilver, zil etc would need to be dedup aware. Either case, release the > required code and there is no harm no foul right? If it is stubs, then so > be it. I am more interested to see if they implemented it the same way I > started to or if it is something new. If it is code complete and all > covered even better.If the code in a new file is derived from code in a file covered by the CDDL, it may be that you need to provide this code under CDDL too. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
On Tue, 7 Oct 2008, Casper.Dik at Sun.COM wrote:>>> >> Wouldn''t it be great if programmers could just focus on writing >> code rather than having to worry about getting sued over whether >> someone else is able or not to make a derivative program from their >> code? > > Yep, but in THIS world it *is* an important consideration.Definitely. Copyrights and licenses should always be observed and respected. In today''s "MP3" generation where copyright has been reduced by pimply-faced teenagers to less value than toilet paper, many people have taken up a habit of not respecting anyone''s copyrights or licenses. Meanwhile, the legal system still supports copyrights and violating products may be shut down overnight due to court order. If a copyright or license violation is suspected, then the copyright holder should be contacted since the copyright holder is the only one with the legal right to persue violators. There is little value to "guilty until proven innocent" attacks on mailing lists. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
zfs-discuss-bounces at opensolaris.org wrote on 10/07/2008 10:59:06 AM:> On Tue, 7 Oct 2008, Casper.Dik at Sun.COM wrote: > >>> > >> Wouldn''t it be great if programmers could just focus on writing > >> code rather than having to worry about getting sued over whether > >> someone else is able or not to make a derivative program from their > >> code? > > > > Yep, but in THIS world it *is* an important consideration. > > Definitely. Copyrights and licenses should always be observed and > respected. In today''s "MP3" generation where copyright has been > reduced by pimply-faced teenagers to less value than toilet paper, > many people have taken up a habit of not respecting anyone''s > copyrights or licenses. Meanwhile, the legal system still supports > copyrights and violating products may be shut down overnight due to > court order. > > If a copyright or license violation is suspected, then the copyright > holder should be contacted since the copyright holder is the only one > with the legal right to persue violators. There is little value to > "guilty until proven innocent" attacks on mailing lists. >Bob, The mailing list happens to be run by the copyright holder and interested parties (zfs authors) with the ability to act inside the copyright holder are on this list -- it seems to be a valid medium to notify. *shrug* There are no "guilty until proven innocent" attacks here, just a few people that have noted (and even contacted the vendor to verify) that the code is not available as it is expected to be under common views of CDDL. Further, the discussion has expanded into what people believe the CDDL requirements to be. Al of this discussion could be headed off with a simple "we are on it" from one of the parties involved. -Wade
Jens Elkner
2008-Oct-08 02:17 UTC
[zfs-discuss] zpool imports are slow when importing multiple storage pools
On Mon, Oct 06, 2008 at 05:08:13PM -0700, Richard Elling wrote:> Scott Williamson wrote: > > Speaking of this, is there a list anywhere that details what we can > > expect to see for (zfs) updates in S10U6? > > The official release name is "Solaris 10 10/08"Ooops - no beta this time? Regards, jel. -- Otto-von-Guericke University http://www.cs.uni-magdeburg.de/ Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2 39106 Magdeburg, Germany Tel: +49 391 67 12768
On Tue, October 7, 2008 09:19, Johan Hartzenberg wrote:> Wouldn''t it be great if programmers could just focus on writing code > rather > than having to worry about getting sued over whether someone else is able > or not to make a derivative program from their code?If that''s what you want, it''s easy to achieve. Simply place your code explicitly in the public domain. So stop trying to muck up what lots of the rest of us want, which is that developments based on free code *stay* free, okay? -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
"David Dyer-Bennet" <dd-b at dd-b.net> writes:> On Tue, October 7, 2008 09:19, Johan Hartzenberg wrote: > >> Wouldn''t it be great if programmers could just focus on writing code >> rather than having to worry about getting sued over whether someone >> else is able or not to make a derivative program from their code? > > If that''s what you want, it''s easy to achieve. Simply place your code > explicitly in the public domain. > > So stop trying to muck up what lots of the rest of us want, which is > that developments based on free code *stay* free, okay?Alas, it''s not even as simple as that. The author of SQLite, D. Richard Hipp, took this approach for reasons like those above. He''s said[1] that he wouldn''t do it again, since there are problems for users in some jurisdictions that have no concept of "Public Domain". He said he''d probably use a BSD licence, if he did it all again. [1] See, e.g. http://twit.tv/floss26
Boyd Adamson <boyd-adamson at usa.net> wrote:> Alas, it''s not even as simple as that. The author of SQLite, D. Richard > Hipp, took this approach for reasons like those above. He''s said[1] that > he wouldn''t do it again, since there are problems for users in some > jurisdictions that have no concept of "Public Domain"."no concept" of Public Domain is not the right words. The laws in Europe simply forbid to give up certain rights. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
I had to hard power reset the laptop... Now I can''t import my pool.. zpool status -x bucket UNAVAIL 0 0 0 insufficient replicas c6t0d0 UNAVAIL 0 0 0 cannot open ---- cfgadm usb8/4 usb-storage connected configured ok --- I see a ton of these in dmesg Oct 13 17:22:08 fuzzy scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci1028,24a at 1d,7/storage at 4/disk at 0,0 (sd2): Oct 13 17:22:08 fuzzy drive offline --- What''s the best way to proceed? (I don''t have time to kmdb the issue right now, but hopefully that won''t be necessary) Thanks ./Christopher
C. Bergstr?m ?????:> I had to hard power reset the laptop... Now I can''t import my pool.. > > zpool status -x > bucket UNAVAIL 0 0 0 insufficient replicas > c6t0d0 UNAVAIL 0 0 0 cannot open > > ---- > cfgadm > > usb8/4 usb-storage connected configured ok > > --- > I see a ton of these in dmesg > > Oct 13 17:22:08 fuzzy scsi: [ID 107833 kern.warning] WARNING: > /pci at 0,0/pci1028,24a at 1d,7/storage at 4/disk at 0,0 (sd2): > Oct 13 17:22:08 fuzzy drive offline > ---Looks like device has gone bad. Are you able to read anything off it (e.g. with dd)? What does ''zdb -l /dev/dsk/c6t0d0s0'' show? Victor