datahoarder
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
view the rest of the comments
You want to pool 2 drives managed by software RAID with 4 other drives in an external enclosure managed by hardware RAID, connected via USB?
Yes, this is a terrible idea, as you already suspect. If you even get it to work, it will be unstable and the USB bus will be a nasty bottleneck for data transfer.
For less than the price of a good 4-bay NAS you could get a used Dell PowerEdge server. If that is outside your price range, consider one of the PowerEdge towers.
This would be a significantly more stable and reliable setup, and offers room for future expansion. In addition, if data hoarding is the goal, you would be able to use an OS like TrueNAS which would give you better control over your drive pool(s).
Hey - thanks for the input and suggestions. Part of my idea is because of my desire to not rebuild my whole stack but I see that I likely will need to. Part of the fun in this hobby I suppose ๐
Really appreciate it and thanks again.
You don't need to. I like mergerfs a lot, especially paired with snapraid. It all depends what you plan on using the storage for. Does it need to be blazing fast?
I definitely wouldn't mix hardware and software raid though. You can always load your data onto a new mergerfs pool on the new drives, of there's no other way, then add your old drives to the pool. I imagine it's not necessary, but i've only ever started with an empty mergerfs array and added data to it, so I wouldn't know how to tell you to do it.
I see this:
But also this:
ref
So... what happens when the USB cable gets bumped mid-write, and the drives in the external enclosure suddenly go offline? Because the cable will get bumped at some point, probably at the worst possible time.
Genuine question, I don't have experience with MergerFS. OP's planned setup seems fault-prone to me, rather than fault-resistant.
I've not had it happen, but I imagine it'd be the same as if a sata drive failed. There's no fault tolerance, as you pointed out. My understanding is that each drive has the same directories, and the pool shows all the files from all drives. If a drive goes offline those files should disappear.
I use snapraid to add fault tolerance. Really basically it takes a snapshot of your files and you can recover back to that snapshot if one drive fails. You might think you'd run into a problem if a drive failed, because it might just think you'd deleted a bunch of files, but I believe the default behavior is that it throws an error and notifies you if you have deleted more than a certain threshold of files. That might not be built into snapraid. It might be part of snapraid runner, which I would recommend you use anyway to make it easier to deal with.
So basically, you'd notice your files disappeared, or a cron job would notice, or snapraid would notice, then you'd go plug the drives back in.
I get the concern, but if you're that concerned about reliability then you should probably use some commercial product that won't require much know how or intervention.
I'm loving the flexibility of mergerfs, snapraid, and a diy nas. When I run out of physical space I'll likely just add a few drives in a USB enclosure, so I definitely wouldn't try to persuade you not to.