Caution
Since July 12 2024, JellyGrail could not work properly anymore due to Real Debrid API changes impacting rclone_rd app. This is now fixed in my fork, and with improvements but looking at the rclone_rd code I realized that:
- 1/ You should not not change or remove the rclone.tpl.sh
--tpslimit 4argument. Otherwise you'll get 429 http errors from RD service. it seems to be the no.1 reason Real Debrid had issues with all API endpoints beeing overloaded because of bad rclone_rd implementations. Jellygrail always had this argument set to 4. - 2/ you should absolutely let a reasonable value for
--dir-cache-timeargument, such as10s. If reduced rclone root refresh triggers /torrents endpoint too much -> it seems to be a potential 2nd reason Real Debrid had issues with /torrents API endpoint beeing overloaded because of bad rclone_rd implementations. Jellygrail always had this argument set to 10s. - 3/ re-starting every rclone instance (jellygrail restarts overnight) is not optimal: -> FIXED with regular dump to file for
/downloadsand/torrent/infodata. Only/torrentsis fetched regularly. - 4/ rclone_rd did not not know how to unrestrict links on the fly (or to fix bad unrestricted links). -> FIXED
- And it will unrestrict only on very first listing and then keep the old link untill the user really opens the file. Huge difference from original rclone_rd. It decreases unrestricting calls a lot. Combined with jellygrail cache for RAR, ISO and ffprobe data, API endpoints and remote assets are rarely requested.
- 5/ Log file
/jellygrail/log/remote_realdebridis still very verbose to track any issue or abnormal API calls. - 6/ Note that my rclone_rd fork has a tuned cache system for random access on RAR and ISO file structures, thus avoiding multiple repetitive parallel HTTP sessions. Other solution to avoid these rate-limiting related issues would be to unrestrict a new link but it is surely not a fair-use practice. So Jellygrail won't do that.
These Real Debrid related quirks are now -> FIXED.
The only remaining issue seems to be that accumulated unrestricted links (accumulation is on purpose) are deduplicated but not aligned upon refreshed torrents list, so this array grows a little bit too much over time, but nothing to worry about in terms of execution speed and RAM. This will be fixed way before it becomes a problem.
JellyGrail is an experimental modified Jellyfin* docker image to manage all your video storages (local and cloud/remote) in one merged virtual folder that you can organize as if it were a real one. It's optimized for Real-Debrid service and provides on-the-fly RAR extraction.
*Jellyfin is an opensource alternative to Plex.
-
Access remote and Real-Debrid files as if they were local (like https://github.com/itsToggle/rclone_RD and Zurg).
-
✨✨ RAR archives extracted on the fly (https://github.com/hasse69/rar2fs):
- No more hassle to extract your local RAR downloads.
- No more hassle downloading and extracting Real-Debrid RAR torrents, now you can just stream and extract on-the-fly.
- It provides an optimized cache to strongly mitigate Real-Debrid rate-limiting issues that can happen with ISO and RAR files (with my rclone_rd fork : https://github.com/philamp/rclone_jelly)
RAR on-the-fly extract only works with "archive" mode (= no compression actually used). Other modes are very rarely used in this context anyway.
-
✨ Auto-organized TV shows and movies in a virtual folder:
- Subtitle files renaming following standards as most as possible.
- Detects extras and put them in the movie's "extras" subfolder.
- You can manage the virtual folder as if it were a real one (rename and move files the way you want).
- Smart deletion of actual assets behind virtual files (including rclone cache files).
-
✨ Almost fully automatized Jellyfin configuration (except login/password) and scan triggering:
- New items detection for Real-Debrid and local files (with rd_api_py and pyinotify), triggering Jellyfin or PLEX library refresh. (Jellyfin can also be disabled if another or no media center used).
-
✨ Can be used without any media center while keeping some practicality:
- "scrapper-less/offline-mode" filename cleaner for movies (https://github.com/platelminto/parse-torrent-title - accurate 99,8%). This improves filesystem browsing.
- Movie variants merged into common folder when possible (with https://github.com/seatgeek/thefuzz).
- Virtual folder can be shared on your local network through any protocol since it's like a regular file-system (+ WebDAV nginx server included on port 8085).
- Every storage is merged into this unique virtual folder (with my BindFS fork: https://github.com/philamp/bindfs_jelly)
-
Real-Debrid magnet hashes management:
- Automatic backup of all Real-Debrid torrents hashes + a service to restore them if RD account emptied by mistake.
- RD torrent-hashes sync from another instance of JellyGrail (although no secured proxy or VPN is provided in this container).
Caution
- I'm not responsible of any data loss / I'm not responsible of any illegal use / Use at your own risks.
- This solution does not include any torrent indexer search.
- Do not open ports 8085 and 6502 to the internet.
⚠️ File Deletion in the virtual folder actually deletes corresponding files of underlying file-system(s).- There can be some rare cases where nginx/jellyfin hangs on readdir or readfile request. Workaround is a docker restart. See details below in
Known issuessection.
| Functionnalities/Solutions | Plex w/ rclone_rd | Jellyfin w/ rclone_rd + Kodi | File-System share + Kodi | Streamio | JellyGrail + Kodi |
|---|---|---|---|---|---|
| Play on request | 🟠 | 🟠 | ❌ | ✔️ | 🟠within few minutes |
| Video variants grouping | ✔️ | ❌ | ❌ | ✔️ | ✔️ |
| On-the-fly RAR extract. | ❌ | ❌ | ❌ | N/A | ✔️ |
| File-System share fallback | ❌Media library / DLNA access only | ❌Media library / DLNA access only | N/A | ❌ | ✔️ |
| Subtitle management | ✔️ | ✔️ | ✔️w/ kodi add-on or Bazarr | 🟠 | ✔️with jellyfin add-on or kodi add-on |
| Mobile streaming transcoding | ✔️ | ✔️Including DoVi profile 5* | ❌ | N/A | ✔️Including DoVi profile 5* |
| Remote&local storages merging | ✔️ | ✔️ | ❌ | N/A | ✔️Also in WebDAV share fallback |
| Open Source | ❌ | ✔️ | ✔️ | ❌ | ✔️ |
| Plays nearly every formats including BDMV & DVD ISOs | ❌ | 🟠 | ✔️ | ❌ | ✔️WebDAV share fallback |
| Own curated library with unlimited storage | ✔️ | ✔️ | ❌ | ❌ | ✔️ |
| High-Quality audio passthrough to Soundbars etc. | 🟠 | ✔️** | ✔️** | ❌ | ✔️** |
🟠 = "more or less"
* See requirements here: https://jellyfin.org/docs/general/administration/hardware-acceleration/#hardware-accelerated-tone-mapping
** At least AC3+ (Atmos) and DTS with most Kodi platform versions and up to TrueHD DTS-MA with CoreELEC Kodi
Follow sections 1/ to 7/
- Linux system 🐧 with Bash shell.
- Tested on x86 system, should build on ARM and should run on a Raspberry 4, but not tested yet.
- Docker 🐳.
- Git client to clone this repo (TODO: provide a prebuilt image).
- Having a Real-Debrid account is better.
Find a conveniant directory on your system, beware this folder will store jellygrail subfolder and the rclone cache (0.5%~ of your real-debrid storage size). The jellygrail subfolder created by git clone ... is represented by a dot . in this page.
git clone https://github.com/philamp/jellygrail.git
cd jellygrail/docker
sudo docker build -t philamp/jellygrail .
If you upgrade, replace the
git clone ...command by agit pullcommand inside the.folder
Grab your Real-Debrid API key : https://real-debrid.com/apitoken.
Make sure you're back in the root . folder where PREPARE.SH is located and run:
sudo chmod u+x PREPARE.SH
sudo ./PREPARE.SH
Make sure you're back in the root . folder where PREPARE.SH is located and run:
sudo chmod u+x PREPARE.SH
sudo ./PREPARE.SH upgrade
This creates settings files and also prepares "rshared" mounted folder ./Video_Library/ (so its content reflects the magic ✨ happening inside the docker container and is available to the host system, not only inside the container)
Learn more about "rshared" here : https://forums.docker.com/t/make-mount-point-accesible-from-container-to-host-rshared-not-working/108759
Take a notepad and progressively paste portions of code in sub-sections 4.1 to 4.3 below:
don't forget the "\" before a new line and ignore "..." lines
Example with common transcoding device access mounted and running in host mode (TODO: provide ports forwarding version)
The first time you launch this command, you can run with "run -it" instead of "run -d" if you want, so that you can see the output, once first tasks are finished it stops and restarts in deamonized mode anyway.
sudo docker run -d --privileged --security-opt apparmor=unconfined \
--cap-add MKNOD \
--cap-add SYS_ADMIN \
--device /dev/fuse \
--device /dev/dri/renderD128 \
--device /dev/dri/card0 \
--network host \
-v ${PWD}/jellygrail:/jellygrail \
-v ${PWD}/Video_Library:/Video_Library:rshared \
-v ${PWD}/mounts/remote_realdebrid:/mounts/remote_realdebrid \
-v ${PWD}/fallbackdata:/mounts/fallback \
...
⚠ Not yet tested without "--privileged --security-opt apparmor=unconfined", so I let it and thus it's unsecure. Remember its experimental stuff.
Example with 2 local folders
...
-v /volume1/video:/mounts/local_drive1 \
-v /volumeUSB1/usbshare/video:/mounts/local_drive2 \
...
⚠ Your local folders must be mounted inside
/mountswith local_ prefix and they must contain at least a movies/ folder or a shows/ folder (it follows the same naming convention as rclone_rd )⚠ local-storage movies/ folders also supports video files that would be directly inside this folder. But shows must always be in a subfolder (ex : video/shows/scrubs/video.mkv)
...
--restart unless-stopped \
--name jellygrail \
philamp/jellygrail:latest
- Verify that
./jellygrail/config/settings.envis populated with proper values. - Verify that
./mounts/remote_realdebrid/rclone.confis populated with proper values. - Verify that your working directory is
.(the folder containing PREPARE.SH file). - Paste your docker command in your bash prompt and hit enter !
On http://your_system_ip:6502 an http server is provided to respond to these path calls below.
With recent commits, only
/backupshould be called manually or via crontab.
Not mandatory to be set as cron as rd_progress potentially calls it every 2 minutes.
Should be triggered to scan your folders in order to fill the ./Video_Library/virtual/ folder and refresh Jellyfin Library.
Should be triggered frequently to backup your RD torrents (dump file stored in ./jellygrail/data/backup).
Simple web page to choose the backup file to restore from
Not mandatory to be set as cron since it's triggered internally every 7 minutes (if remote endpoint is configured).
to trigger the pull of new hashes from another JellyGrail instance (if configured in ./jellygrail/config/settings.env)
Tip
/remotescan is the local trigger that will call a remote service (which is actually /getrdincrement) on the other JellyGrail instance (but no secured proxy or VPN is provied here, so be careful).
Caution
You should absolutely not open the python service to internet (do not open port 6502).
Basically you won't use this trigger unless you want to synchronize your RD torrents with another instance of this app (aka friend remote instance).
Not mandatory to be set as cron since it's triggered internally every 2 minutes.
This is a service to check if there are changes worth calling /scan subsequently.
- Verify that you have some torrents in your RD account (JellyGrail does not provide any torrent indexer search or RD downloader).
- Wait for the
./Video_Library/virtual/folder to be filled (The first library scan is called within 2 minutes if there are torrents in your RD account)- or trigger it with
/scan(See 📡 Tasks triggering section above).
- or trigger it with
- Access the content in
./Video_Library/virtual/(in the folder you ran the docker command). - Jellyfin is ready to run and preconfigured with corresponding libraries on http://your_system_ip:8096.
- Initialize the user and language and don't do anoything else (don't add librairies)
- You can also disable Jellyfin at config time and point your plex Libraries to the
./Video_Library/virtual/movies/and./Video_Library/virtual/shows/folders.- If you don't need the filesystem fallback functionnality and use Plex, you can as well point your Plex libraries to folders inside
./Video_Library/actual/rar2fs_*/.
- If you don't need the filesystem fallback functionnality and use Plex, you can as well point your Plex libraries to folders inside
- For TV/Projector usage : it's recommended to use Kodi + Jellyfin add-on on an Android TV device (or LibreELEC/Coreelec on specific devices).
- On Mobile device, you can install Jellyfin app and switch to native included player in its settings (in other words: avoid the webview player because it leads Jellyfin to do unnecessary transcoding)
- Beware to have a paid RD account:
- configure
/backupcron (See 📡 Tasks triggering section above). - if you forgot a payment or deleted torrents by mistake, you can find your RD hashes backup in ./jellygrail/data/backup/ and use the /restore service (See 📡 Tasks triggering section above).
- configure
- You can re-arrange your virtual/shows and virtual/movies folders the way you like as if it were a normal file-system. Future calls to /scan service won't mess-up with your changes. Don't forget to refresh Jellyfin library after your changes.
- JellyGrail being experimental, it restarts by itself at 6.30am 🕡 every day to improve reliability
Tip
If you restart your NAS frequently, add STOP.SH script to your shutdown tasks and START.SH script to your startup tasks so that shared mount points are still accessible (alternatively, you can use fstab)
Note
./fallbackdata/ folder contains files added by you or any process that tries to write a file in virtual folder and its subfolders.
./Video_Library/virtual_dv/ is a dynamically filtered folder containing only Dolby Vision MP4/MKV files.
./Video_Library/virtual_bdmv/ is a dynamically filtered folder containing only DVDs and Blu-rays data.
Caution
./Video_Library/virtual/ folder, as it will also delete corresponding files in the underlying file-systems. Just delete the ./jellygrail/.bindfs_jelly.db file, restart the docker container and trigger a new /scan
You can check it's running with following commands:
sudo docker ps
tail -f ./jellygrail/log/jelly_update.log
sudo docker logs --follow jellygrail
curl http://localhost:6502/test
Open http://your_system_ip:8096 to launch Jellyfin web interface.
- Check 🚀 First and daily Usage section above.
- m2ts/ts files not inside a BDMV structure are ignored.
- there can be some rare cases (bad .MKV, .TS, .ISO file or big complex .RAR file) where bindfs hangs (being mono-threaded) because of rclone hanged (due to lot of seeks and read in those bad files, causing somewhat undefined behavior in my rclone_rd fork) it causes nginx and jellyfin to possibily hang as well. Current workaround is a full restart of the docker.
⚠️ If you've restarted your system, the docker container was maybe restarted but the rshared mount of folder./Video_Library/was not made so you have to run./STOPSTART.SHto fix it.- JELLYFIN_FFmpeg__analyzeduration reduced to 4 seconds to be light on Real-Debrid requests and rclone cache. On some video files ffprobe report might be uncomplete. TODO: reconsider an increase of JELLYFIN_FFmpeg__analyzeduration.
- Additional Remote mounts points : You can add other rclone remote mount points (with your favorite cloud provider) by following the same structure as the provided example used for real_debrid in
./mounts/folder provided but follow this convention:- name your rclone config title (in between [ ] ) the same as the parent folder containing this rclone config file.
- and name the file "rclone.conf".
- the cloud mount source is not configurable (yet)
- video files can't be directly located within the root of the mount (/mounts/remote_mycloud_provider/video.mkv will not be scanned it should rather be /mounts/remote_mycloud_provider/movies/Title/Title.mkv)
- Underlying files deletion:
- REMOTE : follows rclone RD fork system : Inside folders containing multiple video files, only 1 file will be deleted (TODO: fix this issue to improve other cloud provider support). In other words it means that underlying files deletion are sometimes uncomplete in this case.
- LOCAL : Underlying files are deleted but not folders (TODO:fix)
- RD Torrents that becomes unavailable (despite rclone fork trying to re-download them) are not fully detected by JellyGrail: corresponding virtual files are not displayed and Jellyfin will thus remove them from library but corresponding parent folders will stay (TODO: trying to fix that in a next version)
- 3 Jellyfin plugins are pre-installed:
SubBuzz:not enabled on library scan but can be used on induvidual items. You can enable it on library scan if you want but beware it will cause additional download requests to Real-Debrid.Merge Versions:Movies not merged by initial scan can be merged thanks to this Jellyfin plugin. Shows episodes are not set to be merged because in this case it causes troubles (like whole season merged into one media item).Kodi Sync Queue:to improve the experience with Jellyfin kodi add-on
- rclone_jelly is an experimental fork of https://github.com/itsToggle/rclone_RD to change the normal behavior of rclone's vfs_cache and thus it's not a "cache" anymore: it stores RAR/ISO file structure data to improve access reliability especially when using Real-Debrid service.
- This cache will have a size equal to 0.5%~ of your real-debrid storage size, using it on an SSD is better (but not mandatory).
- bindfs_jelly is a fork of https://github.com/mpartel/bindfs that brings virtual folders and virtual renaming.
- Its sqlite DB is initialized through inluded Python service that scans mounted local and remote folders (upon first start the virtual folder is empty).
⚠️ You can manage your assets only through the virtual folder (rename, delete, move) otherwise if you do it directly on the underlying filesystems, linkage will be lost between virtual tree and actual trees. TODO: autofix when linkage is dead between bindFS and underlying filesystems- You can use a Real-Debrid download manager like rdt-client and disable downloading files to host since you don't need to have these files stored locally anymore. Thus you also have to stop using rename-and-organize feature of Radarr and Sonarr (basically you have to stop radarr/sonarr handling of finished downloads).
- if the Video_Library folder is then accessed through a SMB protocol in windows, renaming does not seem to work (an error pops up) but it's actually working, just refresh the content of the folder and you'll see the renaming is effective. (TODO: fix that in bindfs_jelly if possible).
- When detected as extras, videos are moved into extras subfolder but without their corresponding subtitles if any
- Nvidia Shield: https://www.kodinerds.net/thread/69428-maven-s-kodi-builds-f%C3%BCr-android/ -> Nexus release (arm64-v8a))
- Chromecast with Google TV: https://www.kodinerds.net/thread/69428-maven-s-kodi-builds-f%C3%BCr-android/ -> Nexus release (armeabi-v7a) (to be completed...)
- CoreElec compatible box
-
Jellyfin add-on
*- with 'add-on' paths, not 'native' paths, otherwise you loose the functionnality to choose the video variant upon play.
-
Jellycon add-on
- works very well too and works without hacking the Kodi main db. Although last time I checked it only show variants as a merged item when they're merged in filesystem, not when dynamically merged with "merge versions" plugin
-
Artic Horizon 2 skin
*- Allow third party default dependencies in add-on settings before instlaling the skin. (repository.jurialmunkey-3.4.zip)
-
a4k subtitles add-on
* -
Up Next (optionnal)
-
Keymap editor add-on (optionnal)
*Kodi repo included (with "install from zip") in HTTP WebDAV server provided on port 8085 in./Video_Library/actual/kodi/software/