Hello everyone, I’m in the process of setting up long term storage of a large amount of media files for my family. Since this is stuff that’s rarely accessed, I don’t need it to be always fully available, and I want to limit the actual transfer of files to a specific schedule.
Since knowing what’s already in storage is kinda non-trivial, are there any tools out there that allow me to browse which files are in cold storage, without the actual drives being attached / mounted? So only info like file location, name, metadata, perhaps icon, etc.
The transfer part I can manage to automate, as I can think of a few ways of allowing my users to “stage” pulling and pushing info from and to cold storage, and having the transfers be done on a fixed schedule like every two weeks or something.
Obviously self-hosted, open source stuff that’s not vibecoded is preferred.
Thanks in advance!
Just out of curiosit, what do you gain from unmounting the fs? Depending on your reasoning, it might he an option to use a zfs filesystem, set it up so automatic jobs rarely run and then set a somewhat agressive disk spin down policy. That way fs info would be handled by the arc, not the disks and they would only spin up when an actual file is accessed.
At least I assume it would work this way. Look-Ahead reads or your explorer generating previews, etc. might well make thta option fail.
This is exactly what git-annex was built for:
https://git-annex.branchable.com/
use case: The Archivist
Bob has many drives to archive his data, most of them kept offline, in a safe place.
With git-annex, Bob has a single directory tree that includes all his files, even if their content is being stored offline. He can reorganize his files using that tree, committing new versions to git, without worry about accidentally deleting anything.
When Bob needs access to some files, git-annex can tell him which drive(s) they’re on, and easily make them available. Indeed, every drive knows what is on every other drive.
I have no need for it, but what a neat piece oft Software.
Before unmounting:
$ tree -afx /PATH/TO/VOLUME > file_listing.txtLater, when you want to search:
$ grep PATTERN file_listing.txtMaybe zip that up and then zgrep it?
I like the cdfinder suite of products, neofinder on Mac, abemedia on windows and Linux. It keeps in depth catalogs and thumbnails of media for long term archives. You can also save scans from beyond compare so you can do a sync comparison of files against an pre scanned offline media.
Not a GUI/webapp, but I like to use ncdu snapshots (“exports”). ncdu is an interactive file browser / disk usage analyzer with a terminal UI, which allow for convenient browsing but also, y’know, figuring out what to delete in case of storage cleanups.
creating a snapshot (“exporting”):
ncdu /pathyouwanttoscan -2 --one-file-system --compress -o snapshot_ncdu.json.zstd(
-2enables progress bar and error messages during the scan)reading (“importing”):
ncdu -f snapshot_ncdu.json.zstdLove ncdu, didn’t know it had this capability. Cool!
Pipe tree to a text file?
What kind of media files? For a typical media collection something like Emby or Jellyfin is quite nice. Then you have your collection and it would just refuse to play media if the file in question isn’t mounted.
High res image sadly, otherwise I would go the Jellyfin route
I heavily use Immich for images and am very happy. Since you can specify custom storage locations it might be worth a shot adding your external storage, let it index everything and then test how it behaves when the external storage is offline. Maybe it keeps thumbnails separate too then you could even browse the library visually and you would only run into errors once you try to open an image.
I had an issue setting up immich for my parents, whre the fs would stall too long for it to successfully get files. Missing media was just getting displayed as a grey box with a “missing media” text on them.
Haven’t used it yet, but I think git annex is built for this. Plan to do the same eventually. It lets you keep the index on hot storage.
If generating of a local index isnt a problem …
Something like snap2html which makes the index browsable.
Ooh this is really nice. I’ll definitely be considering this!
Lots of good ideas here. For other reasons I use:
find /path/to/mount -xdev -printf '%p\t%y\t%s\t%T@\n' | zstd > /path/to/output_fs.tsv.zston a regular basis. For convenience I also munge the output into json so that it can be piped into
gdu -f -for a very convenient tui interface; but you could also just pass-oto gdu and save the resulting json. Very convenient.Like,
find /your/mount -print |sort > files.list?And then the same for the new dir and using any diff viewer for differences or
grep -ffor automation.You mean like that?
Something like Katalog? https://stephanecouturier.github.io/Katalog/
I’d probably write a bash script that recursively goes through the disk and creates any folders it sees on another path, and when it finds a file it creates a corresponding empty file with the same name. If you want more info you could script some metadata tools that extract the info and create a dummy media file with the same metadata.
rsync
need it to be a hierarchical thing? otherwise just
cp -vur src/ dst/ >> files.txt && sort -bdufo files.txt files.txtand look in the file.
