Is it possible to implement file balancing with zfs ? I am using a pair of disk to record video streaming at home, 24 hours a day, 365 days a year. I am recording video from 2 cameras in files of 1 minute, I would like to set this pair of disk as a zpool, and would like to spread one file to one disk, and the other file to the other disk. Is it possible to configure the zpool, with both disks, and set the file balancing? I want this configuration to keep some files if one of the disks is distroyed (for some reason). Thanks in advance.
On 30 October, 2008 - Sergio Arroutbi sent me these 0,7K bytes:> Is it possible to implement file balancing with zfs ? > > I am using a pair of disk to record video streaming at home, 24 hours > a day, 365 days a year. I am recording video from 2 cameras in files > of 1 minute, > > I would like to set this pair of disk as a zpool, and would like to > spread one file to one disk, and the other file to the other disk. > > Is it possible to configure the zpool, with both disks, and set the > file balancing? > > I want this configuration to keep some files if one of the disks is > distroyed (for some reason).Create two pools is one way.. Or if the "keep (some) files" is the important bit, do mirroring.. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
My point is that I want to use the same directory in the recorder program. I get the streaming, and start writing the file in /mnt/streamingDirectory So I would like to record in the same way, just configuring (via zfs if possible), that one file should go to /dev/sda and the other file to the /dev/sdb disk (I am using this in Linux via fuse). Obviously, I could set a mirror, but do not want mirroring, as this could imply having a half of storage data ... Thanks a lot --- El jue, 30/10/08, Tomas ?gren <stric at acc.umu.se> escribi?:> De: Tomas ?gren <stric at acc.umu.se> > Asunto: Re: [zfs-discuss] Enabling load balance with zfs > Para: zfs-discuss at opensolaris.org > Fecha: jueves, 30 octubre, 2008 11:01 > On 30 October, 2008 - Sergio Arroutbi sent me these 0,7K > bytes: > > > Is it possible to implement file balancing with zfs ? > > > > I am using a pair of disk to record video streaming at > home, 24 hours > > a day, 365 days a year. I am recording video from 2 > cameras in files > > of 1 minute, > > > > I would like to set this pair of disk as a zpool, and > would like to > > spread one file to one disk, and the other file to the > other disk. > > > > Is it possible to configure the zpool, with both > disks, and set the > > file balancing? > > > > I want this configuration to keep some files if one of > the disks is > > distroyed (for some reason). > > Create two pools is one way.. Or if the "keep (some) > files" is the > important bit, do mirroring.. > > /Tomas > -- > Tomas ?gren, stric at acc.umu.se, > http://www.acc.umu.se/~stric/ > |- Student at Computing Science, University of Ume? > `- Sysadmin at {cs,acc}.umu.se > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Sergio Arroutbi wrote:> My point is that I want to use the same directory in the recorder program. > > I get the streaming, and start writing the file in /mnt/streamingDirectoryThey it has to be a single dataset in a single pool.> So I would like to record in the same way, just configuring (via zfs if possible), that one file should go to /dev/sda and the other file to the /dev/sdb disk (I am using this in Linux via fuse). > > Obviously, I could set a mirror, but do not want mirroring, as this could imply having a half of storage data ...Not could but will. Basically you are looking for the features of mirroing without wanting the expense. Given you have only two drives raidz or raidz2 isn''t practical either. They best you can do here is put both disks into the pool as a stripe and then set "copies=2 (or even copies=3)" on the filesystem. This doesn''t guarantee that the data will be spread around the disks but ZFS will try. Note however that it does so at the "block" level not file level. Mirroring is really the best answer though. -- Darren J Moffat
On Thu, Oct 30, 2008 at 12:13 PM, Sergio Arroutbi <sarroutbi at yahoo.es>wrote:> My point is that I want to use the same directory in the recorder program. > > I get the streaming, and start writing the file in /mnt/streamingDirectory > > So I would like to record in the same way, just configuring (via zfs if > possible), that one file should go to /dev/sda and the other file to the > /dev/sdb disk (I am using this in Linux via fuse). > >This is interesting to me! What fuse file system allows you to spread a single directory (file system) across two disks in a non-redundant manner but not loose access to the file system if one of the disks fail? My suggestion: Create two ZFS pools, and mount them on different directories, for example /mnt/storage_a /mnt/storage_b Then write a script which will do the following: Start up periodically. If new files exist in /mnt/streamingDirectory, copy them alternatingly (is that a word) to /mnt/storage_a and /mnt/storage_b Something like: #!/bin/ksh # use mkdir as a lock/test since the kernel will give us automatic semaphore, thus we can have mutual-exclusion mkdir /tmp/task_is_running || exit # continue where we left off NEXT_TARGET="$(cat -s /etc/last_target)" find /mnt/streamingDirectory -type f | while read FILENAME do [ "$NEXT_TARGET" = "/mnt/storage_a/" ] && NEXT_TARGET=/mnt/storage_b/ || NEXT_TARGET=/mnt/storage_a/ cp $FILENAME ${NEXT_TARGET}/ || exit 1 rm $FILENAME done echo $NEXT_TARGET > /etc/last_target # On exit without errors, remove the lock. rmdir /tmp/task_is_running Sorry I did not test this, it is off the cuff, typoes may exist. Or logic errors. Also note the "find ... | read" will not port well to other shells. _hartz -- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke My blog: http://initialprogramload.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081030/c62b0586/attachment.html>
On Thu, Oct 30, 2008 at 02:18:34PM +0200, Johan Hartzenberg wrote:> On Thu, Oct 30, 2008 at 12:13 PM, Sergio Arroutbi <sarroutbi at yahoo.es>wrote: > > This is interesting to me! What fuse file system allows you to spread a > single directory (file system) across two disks in a non-redundant manner > but not loose access to the file system if one of the disks fail? > > My suggestion:My suggestion is to setup a mirror since that''s all you''re doing anyway except that there is a chance that data can be lost with a script that just copies stuff. If what the OP is looking for is redundant but not nessesarily exact copies (he only wants the last X days on the backup disk, for example) he may want to consider looking into SAM. -brian -- "Coding in C is like sending a 3 year old to do groceries. You gotta tell them exactly what you want or you''ll end up with a cupboard full of pop tarts and pancake mix." -- IRC User (http://www.bash.org/?841435)
>>>>> "jh" == Johan Hartzenberg <jhartzen at gmail.com> writes:jh> What fuse file system allows you to spread a single directory jh> (file system) across two disks in a non-redundant manner but jh> not loose access to the file system if one of the disks fail? I think GlusterFS does things like this, but haven''t tried it out yet. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081030/1d272e2d/attachment.bin>
On Thu, Oct 30, 2008 at 4:42 PM, Brian Hechinger <wonko at 4amlunch.net> wrote:> > If what the OP is looking for is redundant but not nessesarily exact copies > (he > only wants the last X days on the backup disk, for example) he may want to > consider > looking into SAM. >The OP described a solution where if one disk fails, only SOME of the files are lost, thus a Non-Redundant solution. I Quote: I would like to set this pair of disk as a zpool, and would like to spread>> one file to one disk, and the other file to the other disk. >> >> Is it possible to configure the zpool, with both disks, and set the file >> balancing? >> >> I want this configuration to keep some files if one of the disks is >> distroyed (for some reason). >> >-- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke My blog: http://initialprogramload.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081031/74407afc/attachment.html>
I think the better solution is to have two pools, and write a script to change the recording destination time to time, or move the files after it. Like the prototype by Hartz. ps.: Is this a reality show? ;-) -- This message posted from opensolaris.org