Hi all,
while setting of our X4140 I have - following suggestions - added two
SSDs as log devices as follows
zpool add tank log c1t6d0 c1t7d0
I currently have
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
logs
c1t6d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
errors: No known data errors
We have performance problems especially with FrontBase (relational
database) running on this ZFS configuration and need to look for
optimizations.
??I would like to remove the two SSDs as log devices from the pool and
instead add them as a separate pool for sole use by the database to
see how this enhences performance. I could certainly do
zpool detach tank c1t7d0
to remove one disk from the log mirror. But how can I get back the
second SSD?
Any experiinces with running database on ZFS pools? What can I do to
tune the performance? Smaller block size may be?
Thanks a lot,
Andreas
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Andreas H?schler > > ? I would like to remove the two SSDs as log devices from the pool and > instead add them as a separate pool for sole use by the database to > see how this enhences performance. I could certainly do > > zpool detach tank c1t7d0 > > to remove one disk from the log mirror. But how can I get back the > second SSD?If you''re running solaris, sorry, you can''t remove the log device. You better keep your log mirrored until you can plan for destroying and recreating the pool. Actually, in your example, you don''t have a mirror of logs. You have two separate logs. This is fine for opensolaris (zpool>=19), but not solaris (presently up to zpool 15). If this is solaris, and*either* one of those SSD''s fails, then you lose your pool. If you''re running opensolaris, "man zpool" and look for "zpool remove" Is the database running locally on the machine? Or at the other end of something like nfs? You should have better performance using your present config than just about any other config ... By enabling the log devices, such as you''ve done, you''re dedicating the SSD''s for sync writes. And that''s what the database is probably doing. This config should be *better* than dedicating the SSD''s as their own pool. Because with the dedicated log device on a stripe of mirrors, you''re allowing the spindle disks to do what they''re good at (sequential blocks) and allowing the SSD''s to do what they''re good at (low latency IOPS). If you''re running zpool 19 or greater (you can check with "zpool update") it should be safe to run with the log device un-mirrored. In which case, you might think about using one SSD as log, and one SSD as cache. That might help. You can verify the behavior of your database, if you run "zpool iostat" and then do some database stuff, you should see the writes increasing on the log SSD''s. If you don''t see that, then your DB is not doing sync writes (I bet it is). You don''t benefit from dedicated log device unless you''re doing sync writes.
Hi Edward, thanks a lot for your detailed response!>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Andreas H?schler >> >> ? I would like to remove the two SSDs as log devices from the pool and >> instead add them as a separate pool for sole use by the database to >> see how this enhences performance. I could certainly do >> >> zpool detach tank c1t7d0 >> >> to remove one disk from the log mirror. But how can I get back the >> second SSD? > > If you''re running solaris, sorry, you can''t remove the log device. You > better keep your log mirrored until you can plan for destroying and > recreating the pool. Actually, in your example, you don''t have a > mirror of > logs. You have two separate logs. This is fine for opensolaris (zpool >> =19), but not solaris (presently up to zpool 15). If this is >> solaris, and > *either* one of those SSD''s fails, then you lose your pool.I run Solaris 10 (not Open Solaris)! You say the log mirror pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 ... logs c1t6d0 ONLINE 0 0 0 c1t7d0 ONLINE 0 0 0 does not do me anything good (redundancy-wise)!? Shouldn''t I dettach the second drive then and try to use it for something else, may be another machine? I understand it is very dangerous to use SSDs for logs then (no redundancy)!?> If you''re running opensolaris, "man zpool" and look for "zpool remove" > > Is the database running locally on the machine?Yes!> Or at the other end of > something like nfs? You should have better performance using your > present > config than just about any other config ... By enabling the log > devices, > such as you''ve done, you''re dedicating the SSD''s for sync writes. And > that''s what the database is probably doing. This config should be > *better* > than dedicating the SSD''s as their own pool. Because with the > dedicated log > device on a stripe of mirrors, you''re allowing the spindle disks to do > what > they''re good at (sequential blocks) and allowing the SSD''s to do what > they''re good at (low latency IOPS).OK! I actually have two machines here, one production machine (X4240 with 16 disks, no SSDs) with performance issues and another development machine X4140 with 6 disks and two SDDs configured as shown in my previous mail. The question for me is how to improve the performance of the production machine and whether buying SSDs for this machine is worth the investment. "zpool iostat" on the development machine with the SSDs gives me capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- rpool 114G 164G 0 4 13.5K 36.0K tank 164G 392G 3 131 444K 10.8M ---------- ----- ----- ----- ----- ----- ----- When I do that on the production machine without SSDs I get pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- rpool 98.3G 37.7G 0 7 32.5K 36.9K tank 480G 336G 16 53 1.69M 2.05M ---------- ----- ----- ----- ----- ----- ----- It is interesting to note that the write bandwidth on the SSD machine is 5 times higher. I take this as an indicaor that the SSDs have some effect. I am still wondering what your "if one SSd fails you loe your pool" means to me. Would you recommend to dettach one of the SSDs in the development machine and add to o the production machine with zpool add tank log c1t15d0 ?? And how save (reliable) is it to use SSDs for this? I mean when do I have to expect the SSD to fail and thus ruin the pool!? Thanks a lot, Andreas -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 4005 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100405/820b1d56/attachment.bin>
Response below... 2010/4/5 Andreas H?schler <ahoesch at smartsoft.de>> Hi Edward, > > thanks a lot for your detailed response! > > > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of Andreas H?schler >>> >>> ? I would like to remove the two SSDs as log devices from the pool and >>> instead add them as a separate pool for sole use by the database to >>> see how this enhences performance. I could certainly do >>> >>> zpool detach tank c1t7d0 >>> >>> to remove one disk from the log mirror. But how can I get back the >>> second SSD? >>> >> >> If you''re running solaris, sorry, you can''t remove the log device. You >> better keep your log mirrored until you can plan for destroying and >> recreating the pool. Actually, in your example, you don''t have a mirror >> of >> logs. You have two separate logs. This is fine for opensolaris (zpool >> >>> =19), but not solaris (presently up to zpool 15). If this is solaris, >>> and >>> >> *either* one of those SSD''s fails, then you lose your pool. >> > > I run Solaris 10 (not Open Solaris)! > > You say the log mirror > > > pool: tank > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > ... > > logs > c1t6d0 ONLINE 0 0 0 > c1t7d0 ONLINE 0 0 0 > > does not do me anything good (redundancy-wise)!? Shouldn''t I dettach the > second drive then and try to use it for something else, may be another > machine? > >No, he did *not* say that a mirrored SLOG has no benefit, redundancy-wise. He said that YOU do *not* have a mirrored SLOG. You have 2 SLOG devices which are striped. And if this machine is running Solaris 10, then you cannot remove a log device because those updates have not made their way into Solaris 10 yet. You need pool version >= 19 to remove log devices, and S10 does not currently have patches to ZFS to get to a pool version >= 19. If your SLOG above were mirrored, you''d have "mirror" under "logs". And you probably would have "log" not "logs" - notice the "s" at the end meaning plural, meaning multiple independent log devices, not a mirrored pair of logs which would effectively look like 1 device. -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100405/72eca311/attachment.html>
Hi Khyron,> No, he did *not* say that a mirrored SLOG has no benefit, > redundancy-wise. > He said that YOU do *not* have a mirrored SLOG.? You have 2 SLOG > devices > which are striped.? And if this machine is running Solaris 10, then > you cannot > remove a log device because those updates have not made their way into > Solaris 10 yet.? You need pool version >= 19 to remove log devices, > and S10 > does not currently have patches to ZFS to get to a pool version >= 19. > > If your SLOG above were mirrored, you''d have "mirror" under "logs".? > And you > probably would have "log" not "logs" - notice the "s" at the end > meaning plural, > meaning multiple independent log devices, not a mirrored pair of logs > which > would effectively look like 1 device.Thanks for the clarification! This is very annoying. My intend was to create a log mirror. I used zpool add tank log c1t6d0 c1t7d0 and this was obviously false. Would zpool add tank mirror log c1t6d0 c1t7d0 have done what I intended to do? If so it seems I have to tear down the tank pool and recreate it from scratc!?. Can I simply use zpool destroy -f tank to do so? Thanks, Andreas
On 04/05/10 11:43, Andreas H?schler wrote:> Hi Khyron, > >> No, he did *not* say that a mirrored SLOG has no benefit, >> redundancy-wise. >> He said that YOU do *not* have a mirrored SLOG. You have 2 SLOG devices >> which are striped. And if this machine is running Solaris 10, then >> you cannot >> remove a log device because those updates have not made their way into >> Solaris 10 yet. You need pool version >= 19 to remove log devices, >> and S10 >> does not currently have patches to ZFS to get to a pool version >= 19. >> >> If your SLOG above were mirrored, you''d have "mirror" under "logs". >> And you >> probably would have "log" not "logs" - notice the "s" at the end >> meaning plural, >> meaning multiple independent log devices, not a mirrored pair of logs >> which >> would effectively look like 1 device. > > Thanks for the clarification! This is very annoying. My intend was to > create a log mirror. I used > > zpool add tank log c1t6d0 c1t7d0 > > and this was obviously false. Would > > zpool add tank mirror log c1t6d0 c1t7d0zpool add tank log mirror c1t6d0 c1t7d0 You can also do it on the create: zpool create tank <pool devs> log mirror c1t6d0 c1t7d0> > have done what I intended to do? If so it seems I have to tear down > the tank pool and recreate it from scratc!?. Can I simply use > > zpool destroy -f tank > > to do so?Shouldn''t need the "-f"> > > Thanks, > > Andreas > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Andreas H?schler > > Thanks for the clarification! This is very annoying. My intend was to > create a log mirror. I used > > zpool add tank log c1t6d0 c1t7d0 > > and this was obviously false. Would > > zpool add tank mirror log c1t6d0 c1t7d0 > > have done what I intended to do? If so it seems I have to tear down the > tank pool and recreate it from scratc!?. Can I simply use > > zpool destroy -f tank > > to do so?Yes. You''re unfortunately in a bad place right now, due to a simple command line error. If you have the ability to destroy and recreate the pool, that''s what you should do. If you can''t afford the downtime, then you better buy a couple more SSD''s, and attach them to the first ones, to mirror them. On a related note, this creates additional work, but I think I''m going to start recommending this now ... If you create a slice on each drive which is slightly smaller than the drive, and use those slices instead of the full device, you might be happy you did some day in the future. I had a mirrored SSD, and one drive failed. The replacement disk is precisely exactly the same, yet for no apparent reason, it appears 0.001Gb smaller than the original, and hence I cannot un-degrade the mirror. Since you can''t "zpool remove" a log device in solaris 10, that''s a fatal problem. The only solution is to destroy and recreate the pool. Or buy a new SSD which is definitely larger... Say, 64G instead of the 32G I already have. But that''s another thousand bucks, or more. Please see the thread http://opensolaris.org/jive/thread.jspa?threadID=127162&tstart=0 One more note. I heard, but I don''t remember where, that the OS will refuse to use more than half of the system RAM on ZIL log device anyway. So if you''ve got 32G of ram, the maximum useful ZIL log device would be 16G. Not sure if that makes any difference ... For performance reasons, suppose you''ve got 32G ram, and 32G SSD, and you create a 16G or 17G slice to use for log device ... For performance reasons, you should leave the rest of that SSD unused anyway.
On Mon, Apr 05, 2010 at 07:43:26AM -0400, Edward Ned Harvey wrote:> Is the database running locally on the machine? Or at the other end of > something like nfs? You should have better performance using your present > config than just about any other config ... By enabling the log devices, > such as you''ve done, you''re dedicating the SSD''s for sync writes. And > that''s what the database is probably doing. This config should be *better* > than dedicating the SSD''s as their own pool. Because with the dedicated log > device on a stripe of mirrors, you''re allowing the spindle disks to do what > they''re good at (sequential blocks) and allowing the SSD''s to do what > they''re good at (low latency IOPS).Others have addressed the rest of the issues well enough, but I thought I''d respond on this point. What you say is fair, if the db is bound by sync write latency. If it is bound by read latency, you will still suffer. You could add more ssd''s as l2arc (and incur more memory overhead), or you could put the whole pool on ssd (and lose its benefit for other pool uses). There are many factors here that will determine the best config, but the current one may well not be the optimal. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100406/bcd6db7f/attachment.bin>