I'm way over-analyzing at this point, so I'd love any suggestions or advice on approaching this new setup.
Today I run OpenMediaVault 7 on an i5 NUC with a cheap USB enclosure with 4x8TB in RAID5 (Hardware RAID, which I greatly regret).
Upgrading to a Minisforum N5 NAS Pro with 5x22TB and 3x4TB NVMEs.
My primary use is media which is the vast bulk of storage. I also point some Time Machine backups at it and use it to archive "what if I need this someday" stuff from old external drives we've used over the years. But all the critical stuff is also sent to Backblaze, so this is not primary backup per se, more for the local convenience.
I have decided against Proxmox, so this will be OMV (or maybe Unraid) bare metal. I've also ruled out TrueNAS. Proxmox and TrueNAS both just add too many new "pro" layers I don't really want to deal with.
I'm considering:
Setup 1:
- 3 of the drives in mergerfs for media storage (all the stuff that annoying at most if lost), maybe SnapRAID.
- 2 drives in RAID 1 for all the more important stuff like documents, user shares (which nobody in the house uses today except me), and the backups
- 1 SSD as bcache for each of the above?
Setup 2:
- RAID5 the whole thing again.
- No idea what to do with the SSDs in this case, bcache again? Can you mirror + bcache SSDs?
Setup 3:
- Take the ZFS plunge - My only real concern is the overhead (plus having zero experience with it). This machine will handle it fine (96GB RAM) BUT I was hoping to be able to leverage most of that RAM to do some Local LLM stuff. Nothing crazy, but I worry about ZFS reducing my ability to do that.
- ZFS has built in tools to incorporate the SSDs as cache right?
Setup 4:
- Switch to Unraid. I like OMV, but I can dig the "simplification" of Unraid.
- 5 Drives in array (1 parity) plus nvmes for cache.
The caching stuff I clearly don't understand but I'm very interested in. I'm thinking about it mostly in "download and consume immediately" situations. Today I have a huge bottleneck in unpacking and moving. I've got 1gb fiber and can saturate it, getting a complete iso in just a few minutes, but then it's another 30min plus waiting for that to actually be usable.
Again, I've completely paralyzed myself with all the options, so slap me out of it with whatever you've got.
you shouldn't worry about ZFS overhead if you are planning to use mergerfs.
you can tune the memory usage of ZFS significantly. by default it targets using half of your RAM, but on a home setup that's wasting resources, you should be able to limit the arc cache to 1-2 GB, maybe somewhat more depending on how you want to use it. It's done with sysctl or kernel parameters.