Hello I recently purchased some hardware which I plan on turning into a data server. I purchased the following: 4 gigs of registered ECC ram 667 SuperMicro X7DCA motherboard (found it for really cheap and figured it couldn''t be too bad) http://www.supermicro.com/products/motherboard/Xeon1333/5100/X7DCA-3.cfm An Intel Xeon 2.5 Ghz Quadcore E5420 4 WD 750 gig desktop hard drives Does this setup seem ok for using opensolaris and particularly ZFS? I am aware of the Time Limited Recover on WD drives when you choose desktop models instead of the Raid editions. http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery I plan on changing the desktop models to include this fact thus turning them into the Raid editions. So based off the motherboard and hard drives would this configuration work for ZFS? If so how should I go about setting up ZFS? For instance in Raid configuration I would set all the hard drives to master and hook them up to my Raid controller. Do I set all the hard drives to master here for ZFS? Also do you recommend getting a smaller hard drive to store the OS and merely use the ZFS drives as my data backup? Thank you for your time -- This message posted from opensolaris.org
Brian wrote:> Hello I recently purchased some hardware which I plan on turning into a data server. > > I purchased the following: > > > 4 gigs of registered ECC ram 667 > > SuperMicro X7DCA motherboard (found it for really cheap and figured it couldn''t be too bad) > > http://www.supermicro.com/products/motherboard/Xeon1333/5100/X7DCA-3.cfm > > An Intel Xeon 2.5 Ghz Quadcore E5420 > > 4 WD 750 gig desktop hard drives > > > > > Does this setup seem ok for using opensolaris and particularly ZFS? I am aware of the Time Limited Recover on WD drives when you choose desktop models instead of the Raid editions. > http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery > I plan on changing the desktop models to include this fact thus turning them into the Raid editions. > > So based off the motherboard and hard drives would this configuration work for ZFS? > > > If so how should I go about setting up ZFS? For instance in Raid configuration I would set all the hard drives to master and hook them up to my Raid controller. Do I set all the hard drives to master here for ZFS? Also do you recommend getting a smaller hard drive to store the OS and merely use the ZFS drives as my data backup? > > Thank you for your time >Overall, that MB looks file. The 1068E is a well-supported SAS/SATA controller in OpenSolaris, so you won''t have any problems using it. Likewise, the ICH9R SATA controller. The NICs are supported as well, though I don''t know about the audio chipset (which is less of a concern). You will need to get a video card, as there is no on-board video controller, and the add-on IPMI card for this board is sub-par. The board supports console redirection to COM1, but I''ve never tried it with these boards. You haven''t said what you plan to use the server for, which will drive how you want to configure the drives (i.e. RAIDZ or mirror/striped) A couple of notes: (1) If you have space in your chassis, I''d get two smaller SATA drives and use them as the (mirrored) boot drives. Attach them to the ICHR9 controller (via the black SATA connectors). You can use ZFS to mirror your boot drives, too. Which is good, since ZFS doesn''t support using stripes or RAIDZ for root volumes. (2) I''d connect your data drives to the 1068E controller, via the two multi-lane connectors. You''ll need a break-out cable to use them. The multi-lanes connectors are in the lower left hand corner (the two silver squares pointing forward, not up). (3) make sure all controllers are operating in non-RAID (i.e. JBOD) mode. (4) if you can, spring for more RAM. 4GB is a bit skimpy. 8GB would likely be much better. (also, there are problems with the memory allocation if you only install 4GB - it''s a chipset thing, and it reduces the amount of RAM usable by almost 40%. This /only/ happens when there is 4GB, so don''t install 4GB. See section 2-3 of the MB manual for more info) (5) depending on use, you might want to invest in a SSD (flash hard drive). See a couple of the other threads on which SSD makes the most sense for you. (6) If you are just doing file-serving, a quad-core CPU is likely overkill. I suspect that even with compression turned on, the CPU will be only modestly loaded. (7) For SAS and SATA drives, there is no Master or Slave. They''re all Master. No setting required. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
Yea I have a cheap nvidia video card I found that should work with this. I found this MB at Fry''s for under 100 dollars so I figured Id try it out. Its probably a discontinued line of server motherboards by SuperMicro so I figured it probably would be an OK board. 1.) Why would I put the boot volumes into a mirrored configuration? I figure if the OS blows up Ill just format it and load it on again. Is it really worth it to have the OS mirrored? 2.) What is the benefit of hooking the SATA hard drives up to the SAS port? Is it not wise to put the OS hard drives and the data hard drives in the same port? 3.) Ill try to figure that out, shouldn''t be too hard as presumably its in the BIOS 4.) Ha thats pretty hillarious that it has trouble operating in the RAM configuration I picked. Who would have thought? I guess Ill pick up two 1 Gig sticks to make it 6 Gigs, as I dont really want to spend another 100 dollars on Ram. 5.) Maybe in a few years 6.) Overkill indeed however who doesn''t like power? "You haven''t said what you plan to use the server for, which will drive how you want to configure the drives (i.e. RAIDZ or mirror/striped)" This is going to be used for my parents business (Im merely setting it up for them and then leaving it.) So basically what I want is reliability and redundancy. I want there to be very little chance of data loss as the business they are in requires them to keep all documents. Currently they have them all on a precarious external hard drive so I want this thing to basically be equivalent to Raid 6. I also want to be able to leave it and have it perform without touching it for decent periods of time. Usually I would use Linux as its great for that but I decided to try out ZFS. Now I read that its advisable to scrub the system every week or month, is it possible just to make a script that will do this so I dont have to be there? Also I know ZFS can use blank hard drives that will activate when a disk fails, is this feature well made in ZFS? Meaning is it trustworthy? I guess I''m just used to trusting several hundred dollar Raid cards, seems odd to be back to software. Thank you for your help -- This message posted from opensolaris.org
Im sorry I forgot to ask again if its worth setting to the Time Limited Recovery to its Raid counterpart mode. The reason I ask is because all I can find to do this is a DOS file so Im not sure how I would go about doing it in OpenSolaris. http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery#Western_Digital_Time_Limit_Error_Recovery_Utility_-_WDTLER.EXE All it lists is a .exe file, so is changing these settings something that must be done? I guess I am unclear on how important this is, though I have read that someone lost their data ''possibly'' due to this. -- This message posted from opensolaris.org
On Sun, 26 Jul 2009, Brian wrote:> as its great for that but I decided to try out ZFS. Now I read that > its advisable to scrub the system every week or month, is it > possible just to make a script that will do this so I dont have to > be there? Also I know ZFS can use blank hard drives that willThis is trivially easy via entries in crontab: # crontab -l | grep scrub 20 4 * * 1 /usr/sbin/zpool scrub Sun_2540 15 2 * * 0 /usr/sbin/zpool scrub USB_Pool It is useful to check for faults and send an email to someone in case there is a problem. I use this script which is also executed via crontab: #!/bin/sh REPORT=/tmp/faultreport.txt SYSTEM=$1 rm -f $REPORT /usr/sbin/fmadm faulty 2>&1 > $REPORT if test -s $REPORT then /usr/ucb/Mail -s "$SYSTEM Fault Alert" root < $REPORT fi rm -f $REPORT Since I have multiple systems sending email to the same address, I supply the identification of the system via an script argument. The name could be obtained from `uname -n` or `hostname` instead. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Brian wrote:> Im sorry I forgot to ask again if its worth setting to the Time Limited Recovery to its Raid counterpart mode. The reason I ask is because all I can find to do this is a DOS file so Im not sure how I would go about doing it in OpenSolaris. > > http://en.wikipedia.org/wiki/Time-Limited_Error_Recovery#Western_Digital_Time_Limit_Error_Recovery_Utility_-_WDTLER.EXE > > All it lists is a .exe file, so is changing these settings something that must be done? I guess I am unclear on how important this is, though I have read that someone lost their data ''possibly'' due to this. >You should set TLER. You''ll have to boot to DOS (via a floppy or CDROM image) - look at the FreeDOS.org website for details about getting a free bootable image.> Yea I have a cheap nvidia video card I found that should work with this. I found this MB at Fry''s for under 100 dollars so I figured Id try it out. Its probably a discontinued line of server motherboards by SuperMicro so I figured it probably would be an OK board. > > > 1.) Why would I put the boot volumes into a mirrored configuration? I figure if the OS blows up Ill just format it and load it on again. Is it really worth it to have the OS mirrored? > >You /should/ mirror your OS, especially if you''re just leaving at another location and don''t want to mess with it very often. You gets lots of benefits from the redundancy it offers (including all those nifty ZFS checksum-based autorecovery ones). I see 100GB 2.5" SATA notebook drives for $50 at my local store all the time.> 2.) What is the benefit of hooking the SATA hard drives up to the SAS port? Is it not wise to put the OS hard drives and the data hard drives in the same port? > >The 1068E (and most other modern SAS controller chips) are really SAS/SATA controllers. They autodetect the drive type attached to them, and react accordingly. I''d use the SAS ports since the 1068E is really a better controller than the ICH9R in terms of performance. I suggested putting the OS drives on the SATA ports for simplicity''s sake, since most motherboards make it easy to boot from the SATA drives, and it requires a BIOS reconfig to boot from the SAS ports. Not difficult to do, jut another step.> 3.) Ill try to figure that out, shouldn''t be too hard as presumably its in the BIOS >Yes. It should be in the BIOS. The SATA config will be in the motherboard BIOS, while the SAS controller config will be separate (push CTRL-L or something similar during BIOS init).> 4.) Ha thats pretty hillarious that it has trouble operating in the RAM configuration I picked. Who would have thought? I guess Ill pick up two 1 Gig sticks to make it 6 Gigs, as I dont really want to spend another 100 dollars on Ram. > > 5.) Maybe in a few years > > 6.) Overkill indeed however who doesn''t like power? > > > > "You haven''t said what you plan to use the server for, which will drive > how you want to configure the drives (i.e. RAIDZ or mirror/striped)" > > > This is going to be used for my parents business (Im merely setting it up for them and then leaving it.) So basically what I want is reliability and redundancy. I want there to be very little chance of data loss as the business they are in requires them to keep all documents. Currently they have them all on a precarious external hard drive so I want this thing to basically be equivalent to Raid 6. I also want to be able to leave it and have it perform without touching it for decent periods of time. Usually I would use Linux as its great for that but I decided to try out ZFS. Now I read that its advisable to scrub the system every week or month, is it possible just to make a script that will do this so I dont have to be there? Also I know ZFS can use blank hard drives that will activate when a disk fails, is this feature well made in ZFS? Meaning is it trustworthy? I guess I''m just used to trusting several hundred dollar Raid cards, seems odd to be back to software. > > Thank you for your help >ZFS is great for what you describe. For maximum redundancy, you''ll want to use RAIDZ2 (the analogue of RAID-6). To set it up (assuming your drives are on what the OS thinks is controller c2): zpool create tank -m <mountpoint> raidz2 c2d0 c2d1 c2d2 c2d3 This will give you 2 drives worth of data space, and 2 redundant drives. Bob already gave you the scrub and monitoring scripts. Personally, I''d look at turning on the Time Slider feature to enable automatic snapshots (probably weekly or so in your case). You also are going to need some form of backup strategy, since you indicated that the data is important to your parent''s data. RAID isn''t enough - that just helps against disk failure. You need something to protect against SERVER failure, so look into a cheap tape drive or consider the external USB drive. In either case, your parents will need to backup the machine nightly, and take the tape/USB drive home with them at night (and bring it back in the morning). -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
> This is going to be used for my parents business (Im > merely setting it up for them and then leaving it.) > So basically what I want is reliability and > redundancy. I want there to be very little chance > of data loss as the business they are in requires > them to keep all documents.Ok, ZFS is good, but what you really need here is a proper backup strategy. If need be, skimp on the server so that you can create a good backup system. Never, ever, keep all your eggs in one basket. If their data is that important, you need to get a copy off-site, and you need some kind of automated process to do that - people don''t realise how important backups are, and if you leave it to a manual system it won''t get done or checked. I''d be very tempted to use zfs send/receive to send the data to another machine, even if it''s just a virtualbox server you run at home. PS. You''re also going to need some kind of remote monitoring of that server - sure, raidz2 will keep your data going when a disk fails, but unless you know that the disk needs replacing, what''s going to happen? What''s going to happen to that server in a couple of years time when you''ve forgotten all about it and suddenly get a call from your parents to say it''s stopped working? If I were you, I''d write a script to run "zpool status -x", and email you if there are any errors. PPS. Yes, you can and should scrub regularly, running that once a week is as easy as adding a line to crontab. -- This message posted from opensolaris.org
Yes Ive thought about some off-site strategy. My parents are used to loading their data onto an external hard drive, however this always struck me as a bad strategy. A tape backup system is unlikely due to the cost, however I could get them to continue also loading the data onto an external hard drive. In the ideal case the following would happen: They drag their windows files onto the ZFS server to store them. This then triggers an automatic load of the files onto an external hard drive. To implement this I guess I could have the external hard drive always plugged into the ZFS system and write a scrip to load the files onto the external also. Im not sure if its possible to do this on the windows side, i.e writing some script to load the data onto the external when its plugged into the windows machine. That said I have read about software that will automatically backup data. Presumably I could configure the software to load the data onto the server, and at the same time load it onto the external which would be attached to their windows machine, and have it do this every night. I haven''t looked too hard into this yet as I''m presuming there will be complications in having a windows backup program deal with loading data onto an OpenSolaris server. Though it might be straightforward, I just haven''t looked at it yet. The ZFS send/receive command can presumably only send the filesystem to another OpenSolaris OS right? Is there anyone way to send it to a normal Linux distribution (ext3)? I have one more question about the hardware setup. You said to get a breakout cable for the SAS plugs. Is it advisable to put 4 hard drives onto one SAS plug? I was thinking of just getting 4 SATA to SAS cables and hooking each drive up to a different SAS plug (as I believe there are 8.) I am also going to look into setting up some system for accessing the server over the internet. FTP would be the easiest choice, however it might be fun to set up an Apache server. -- This message posted from opensolaris.org
Thank you, Ill definitely implement a script to scrub the system, and have the system email me if there is a problem. -- This message posted from opensolaris.org
Brian wrote:> Yes Ive thought about some off-site strategy. My parents are used to loading their data onto an external hard drive, however this always struck me as a bad strategy. A tape backup system is unlikely due to the cost, however I could get them to continue also loading the data onto an external hard drive. In the ideal case the following would happen: They drag their windows files onto the ZFS server to store them. This then triggers an automatic load of the files onto an external hard drive. To implement this I guess I could have the external hard drive always plugged into the ZFS system and write a scrip to load the files onto the external also. Im not sure if its possible to do this on the windows side, i.e writing some script to load the data onto the external when its plugged into the windows machine. > > That said I have read about software that will automatically backup data. Presumably I could configure the software to load the data onto the server, and at the same time load it onto the external which would be attached to their windows machine, and have it do this every night. I haven''t looked too hard into this yet as I''m presuming there will be complications in having a windows backup program deal with loading data onto an OpenSolaris server. Though it might be straightforward, I just haven''t looked at it yet. > > The ZFS send/receive command can presumably only send the filesystem to another OpenSolaris OS right? Is there anyone way to send it to a normal Linux distribution (ext3)? >Send/receive is ZFS-only. rsync is a common way of moving data these days.> I have one more question about the hardware setup. You said to get a breakout cable for the SAS plugs. Is it advisable to put 4 hard drives onto one SAS plug? I was thinking of just getting 4 SATA to SAS cables and hooking each drive up to a different SAS plug (as I believe there are 8.) > >The two plugs that I indicated are multi-lane SAS ports, which /require/ using a breakout cable; don''t worry - that the design for them. "multi-lane" means exactly that - several actual SAS connections in a single plug. The other 6 ports next to them (in black) are SATA ports connected to the ICH9R. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA
The zfs send command generates a differential file between the two selected snapshots so you can send that to anything you''d like. The catch of course is that then you have a collection of files on your Linux box that are pretty much useless since your can''t mount them or read the contents in any meaningful way. If you''re running a Linux server as the destination the easiest solution is to create a virtual machine running the same revision of OpenSolaris as the server and use that as a destination. It doesn''t necessarily need a publicly exposed IP address - you can get the source to send the differential file to the Linux box and then have the VM "import" the file using a recv command to integrate the contents into a local ZFS filesystem. I think that VirtualBox lets you access shared folders so you could write a script to check for new files and then use the recv command to process them. The trick as always for this kind of thing is determining that the file is complete before attempting to import it. There''s some good examples in the ZFS Administration Guide (p187) for handling remote transfers. zfs send tank/cindy at today | ssh newsys zfs recv sandbox/restfs at today For a staged approach you could pipe the output to a compressed file and send that over to the Linux box. Combined with a key exchange between the two systems you don''t need to keep passwords in your scripts either. Cheers, Erik On 27 juil. 09, at 11:15, Brian wrote:> The ZFS send/receive command can presumably only send the filesystem > to another OpenSolaris OS right? Is there anyone way to send it to > a normal Linux distribution (ext3)?
On 27-Jul-09, at 5:46 AM, erik.ableson wrote:> The zfs send command generates a differential file between the two > selected snapshots so you can send that to anything you''d like. > The catch of course is that then you have a collection of files on > your Linux box that are pretty much useless since your can''t mount > them or read the contents in any meaningful way. If you''re running > a Linux server as the destination the easiest solution is to create > a virtual machine running the same revision of OpenSolaris as the > server and use that as a destination. > > It doesn''t necessarily need a publicly exposed IP address - you can > get the source to send the differential file to the Linux box and > then have the VM "import" the file using a recv command to > integrate the contents into a local ZFS filesystem. I think that > VirtualBox lets you access shared folders so you could write a > script to check for new files and then use the recv command to > process them.VirtualBox can forward a host port to a guest, so one can ssh from outside and process the stream directly. Also note Erik''s public key idea below. --Toby> The trick as always for this kind of thing is determining that the > file is complete before attempting to import it. > > There''s some good examples in the ZFS Administration Guide (p187) > for handling remote transfers. > zfs send tank/cindy at today | ssh newsys zfs recv sandbox/restfs at today > > For a staged approach you could pipe the output to a compressed > file and send that over to the Linux box. > > Combined with a key exchange between the two systems you don''t need > to keep passwords in your scripts either. > > Cheers, > > Erik > > On 27 juil. 09, at 11:15, Brian wrote: > >> The ZFS send/receive command can presumably only send the >> filesystem to another OpenSolaris OS right? Is there anyone way >> to send it to a normal Linux distribution (ext3)? > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090728/26d4aa54/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: graphics1 Type: image/gif Size: 3422 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090728/26d4aa54/attachment.gif>
> > > The two plugs that I indicated are multi-lane SAS > ports, which /require/ > using a breakout cable; don''t worry - that the > design for them. > "multi-lane" means exactly that - several actual SAS > connections in a > single plug. The other 6 ports next to them (in > black) are SATA ports > connected to the ICH9R.Just a quick question before I address everyone else. I bought this connector http://www.newegg.com/Product/Product.aspx?Item=N82E16812198020 However its pretty clear to me now (after Ive ordered it) that it won''t at all fit in the SAS connector on the board. What kind of cable do I need for this? -- This message posted from opensolaris.org
On Tue, Jul 28, 2009 at 03:04, Brian<no-reply at opensolaris.org> wrote:> Just a quick question before I address everyone else. > I bought this connector > http://www.newegg.com/Product/Product.aspx?Item=N82E16812198020 > > However its pretty clear to me now (after Ive ordered it) that it won''t at all fit in the SAS connector on the board. What kind of cable do I need for this?Search "8087 forward" on provantage.com. They''re about $15, unless you want attached power connectors (which would be necessary for SAS drives, unless some kind of backplane were in play), in which case they''re $30. Will