I download lots of media files. So far I have been storing these files after I am done with them on a 2TB hard disk. I have been copying over the files with rsync. This has so far worked fairly well. However the hard disk I am using is starting to get close to full. Now I am needing to find a solution so I can span my files over multiple disks. If I were to continue to do it as I do now, I would end up copying over files that would already be on the other disk. Does the datahoading community have any solutions to this?
For more information, my system is using Linux. The 2TB drive is formatted with ext4. When I make the backup to the drive I use ’rsync -rutp’. I don’t use multiple disks at the same time due to having only one usb sata enclosure for 3 1/2 inch disks. I don’t keep the drive connected all the time due to not needing it all the time. I keep local copies until I am done with the files (and they are backed up).
This sounds like exactly one of the use cases described by git annex
You will need a way of connecting both your current 2TB disk and a new one at the same time. A USB hub (if you don’t have free USB ports) and second enclosure, or 2-bay disk dock (much cheaper than a NAS device and no networking required) will do.
You can then combine their storage with mergerfs (available for most distros). Both disks will still work independently, and you can use indexing software like gwhere, cdcat or gcstar to scan each drive so you can tell where a particular file ends up.
You might also be able to buy yourself some more space by using jdupes or rdfind to hardlink duplicate files.
Can you not just modify the paths to sync a dir to one and a dir to the other? Maybe there’s a sorting system, such as a plex library that you don’t want to break up, but I think you can point a system like that at a higher dir and have it sort things out and understand? Sounds like you need to silo your data.