Retrieve metadata from online sources

[gotson]

Mimicking Plex, Komga could manage metadata for series and books, and retrieve metadata from online providers.

See also #48 for manual metadata edition.

Potential providers:

• for Comics:
  • ComicVine which offers a public API
  • League Of Comic Geeks has no API
• for French BD:
  • AppBubble which has a nice (private) API, but cannot redistribute its data as its coming from a third-party provider (ORB)
  • Bedetheque has no API, but a scrapper exists
• for Manga:
  • Kitsu
  • MyAnimeList

In addition Komga should be able to:

• Manually override metadata that was retrieved from online sources

[bayang]

I use https://leagueofcomicgeeks.com/ for series information and also for tracking comics (like trakt for comics) but unfortunately, no API provided 😞

[gotson]

I use https://leagueofcomicgeeks.com/ for series information and also for tracking comics (like trakt for comics) but unfortunately, no API provided 😞

Indeed, the ajax methods return html directly :(

It also doesn't have the completeness information about a series (whether a series is ongoing, finished, abandoned, or in hiatus), which is one of the most important metadata i am looking after!

[The7thSage]

Oh, Just read the last Checkbox, You can disregaurd the rest of this then...
then "maually override" is the same as "editing" metadata, Same result.

This is some Pie-in-The sky stuff, brace yourself....
What about local metadata? As long as it is read from an editable file (not a database?) at some point, you don't have to worry too bad about sources.

As long as you give us a template and location to write the data. That would let the user leverage any data-scraping (or elbow-grease copy-pasting) to write metadata not covered by direct functionality (and hopefully mildly future-proofing the feature).

Another example of a pre-set metadata format would be ComicRack's comicinfo.xml file within .cbz. I am only suggesting this one because "I" Use it alongside a pickier .json (both provided by Hdoujin Down-loader)

Just throwing out some ideas, no pressure.
Not in any hurry, I am using a separate database/reader for tagged manga, and reading in

If I had any idea how making a plugin woks and coding plugins, I'd have made a plugin already (probably leveraging HPX, my other database I'm running alongside this)

[gotson]

Hi all, I would be interested to know a bit more about the metadata you are after, and how you are using it, so i get a better idea of what to implement and how to implement it.

Could you give some insights about:

• what kind of metadata fields are you interested in (Author, Description...)
• is that metadata for individual books, or for a series
• how you use (or plan to use) this metadata, and how it impacts your workflow or reading experience

I'll throw in some that are of particular interest for me:

• I would like to have the completeness status of a Series, which is whether it is complete (all books published), ongoing, or in hiatus/abandoned, so that I can filter my library and start reading completed series only.
• I would like to have the list of all books in a Series, and match this with the list of books I have in my library, so that i can now which books I am missing.

[bayang]

For me, if we follow the plex analogy, given a filename/folder hierarchy komga should be able to retrieve automatically series informations and book informations in series.
At least :
For series :
Authors, tile, description/summary, list of all books, completeness status (also like pull lists for comics), and a picture (cover/thumbnail) that a client can display
In series, for each book :
authors, title, description/summary, number in series and a picture (cover/thumbnail) that a client can display

Ideally komga should provide everything a client would need to satisfy at least basic needs.
The client would be in charge of the reading part only.

Considering tracking (like trakt for videos or anilist and kitsu for manga/anime), I'm not sure if it is the responsibility of komga or of the client.

Actually I use yacreader to manage comics library and reading -> but no information about books or series, no metadata, no tracking
And on mobile I use tachiyomi for manga (which also serves my tracking purpose) but a desktop client is missing (I'm currently having a look at https://github.com/xgi/houdoku, and I think komga could be implemented as a plugin for it, I'm currently having a look).

So if I could use komga on desktop AND mobile (with the tachiyomi plugin), for comics AND manga, that would be nice.

[gotson]

Thanks for the detailed answer!

For the thumbnail, at the moment Komga generates one from the first page of each book. For a series, it's the thumbnail of the first book in the series. Do you think there would be a need to have a thumbnail coming from external sources ?

Considering tracking (like trakt for videos or anilist and kitsu for manga/anime), I'm not sure if it is the responsibility of komga or of the client.

I will add tracking (read status) in Komga at some point. It might be manual to start with, because the implementation in the clients is not in my hands. For example in Tachiyomi it is not managed as an extension, but in the main app. There is also some questions on how to track, should the tracking be done on the matched series/books, so with recognized IDs like ComicVine, or using the internal Komga ID (but those can change if you move your files on disk for example).

[bayang]

No the thumbnails seem fine.
And tracking is indeed hard to get correctly, I don't have much ideas for now.

[WillowMist]

I'm not sure where you're currently pulling information from, but there are two fairly common sources to check for, which I would suggest honoring before trying to pull from ComicVine (which would require each installation to get a CV API key, and should be throttled so you don't try to pull metadata for over 200 books in an hour):

ComicInfo.xml may exist inside an archive, especially if Mylar or ComicRack have been involved in the process of curating the books.

Additionally, ComicBookLover tags may exist in the zipfile comments (for CBZ only)

[frameset]

Mylar plus a viewer such as Ubooquity or one of the ComicStreamer forks is a common usage scenario for many of us, so our collections come with metadata embedded in the file.

I'd love to replace Ubooquity with Komga as Komga is open source and has a seemingly much friendlier developer. 😉

I've got both running side by side for now, and I'm excited to see Komga get even better.

[WillowMist]

Yeah, if we get some control over how the OPDS is presented to the client (like with custom filters, or reading lists, etc) then this will be the perfect complement to Mylar. :)

[gotson]

Yeah, if we get some control over how the OPDS is presented to the client (like with custom filters, or reading lists, etc) then this will be the perfect complement to Mylar. :)

Opds is quite flexible, and I plan to add reading lists to it later 😊

[MI3Guy]

Personally, I would really like to see ComicInfo.xml support. I prefer having metadata embedded in the files.

[gotson]

Personally, I would really like to see ComicInfo.xml support. I prefer having metadata embedded in the files.

Planned in #54

[GlassedSilver]

Personally, I would really like to see ComicInfo.xml support. I prefer having metadata embedded in the files.

Planned in #54

Excellent, most importantly, I would really like to have some staging going on.

What I mean is: ComicInfo.xml should always override the online source and maybe the finding of the scanned source should be written into ComicInfo.xml (if there is none) into the folder/zip/etc... Why? Because it would be really important to be able to move that meta data out of Komga easily. This is one of my biggest gripes with Plex.

That way should a source ever return false information your earlier scrapes are always safe.

I usually like to manually verify the metadata, knowing that the information that is shown will not be altered unless I explicitly force override would be nice. With Plex I'm always a bit skeptical.

Another feat: even if I have to rebuild the entire library with a new database, the previous scrapes' metadata will be transferred.

BONUS: another key metadata source is doujinshi.org for all the doujinshi collectors out there. :) (this would also mean another media "kind". Doujinshi typically aren't published by companies, but "circles" and that's pretty much an important nomenclature. Also very important: usually doujinshis are released at conventions like Comiket for example (most famous example) and not only is there a certain numbering system derived from that (C50 for example would be an identifier usually found in the beginning of a doujinshi's digital file name) but also the importance to have a field for which convention it was released at.

If you look at doujinshi.org at sample entries (NSFW warning, it's very mixed content there and there is definitely no setting to view the site in SFW mode :D) you'll find which metadata is important to include.

[Gin-no-kami]

Just wanted to put forward a "better" metadata source for manga, MangaUpdates. It has more up to date and detatiled information about manga than Kitsu or MyAnimeList.

[gotson]

Would this feature in embed the data into the cbz like comic tagger

Already requested here: #82

[Inervo]

Hi. As Komga is getting better and better with each update, with a nice metadata feature, I'm curious if this feature is still being considered (I hope 🤞 ).
What are the prerequisite necessary you wish to implement/have before having this feature?
Can we help somehow? :)

Thank you for this marvelous software

[tomandocubatas]

Hello,
Any progress with this functionality?
I think it would be a giant step in the functionality of the application.
Being able to fill in the metadata based on certain online websites is very, very interesting.

In any case, the work you are doing seems incredible to me. Thanks a lot!!!

[Pfuenzle]

As there is no integration in Komga for an Anime Metadata provider, I made my own using the metadata from Anisearch.
https://github.com/Pfuenzle/AnisearchKomga. It supports all languages that are available on Anisearch and pushes the metadata directly to Komga.
If someone wants to help me to port it to Java to implement it in Komga it would be great

[Inervo]

As this is no integration in Komga for Metadata provider, I made my own using the metadata from Anisearch. https://github.com/Pfuenzle/AnisearchKomga. It supports all languages that are available on Anisearch and pushes the metadata directly to Komga. If someone wants to help me to port it to Java to implement it in Komga it would be great

Great work!

If there's the same sort of metadata for comics and BD (french/belgium comics), i would love this!

[gotson]

If someone wants to help me to port it to Java to implement it in Komga it would be great

Komga is in Kotlin 😉

The metadata retrieval is much more than hitting an api and mapping fields though. That bit is probably only 10% of what I envision for metadata retrieval.

[Inervo]

Thanks for your reply gotson :)

Can you share with us the main components or behavior you envision for metadata retrieval?
Maybe some of us can help you a bit for some part of the code ;) And by doing so, speed up the release date of this feature

[Inervo]

Thanks for your reply gotson :)

Can you share with us the main components or behavior you envision for metadata retrieval? Maybe some of us can help you a bit for some part of the code ;) And by doing so, speed up the release date of this feature

Hi @gotson. Happy new year !! :)
If that's okay with you, could you share with us your vision regarding the main components for metadata retrieval?
Maybe some of us can help you a bit for some part of the code ;) And by doing so, speed up the release date of this incredible feature and contribute to the great software you created :)

[chu-shen]

Bangumi metadata scraper for Komga👉https://github.com/chu-shen/BangumiKomga

Inspired by https://github.com/Pfuenzle/AnisearchKomga Thanks❤️

[Inervo]

In the meantime, for our french friends who wish to refresh their BD metadata from Bedetheque, here is a small metadata scrapper i've written 👉 https://github.com/Inervo/BedethequeKomga

Inspired from chu-shen/BangumiKomga and aubustou/bedetheque_scraper. Thanks a lot ❤️

NB: it's been ages since i've written some code, so it's far from perfect. Don't hesitate to raise any issue or to contribute :)

[knguyen1]

If someone wants to help me to port it to Java to implement it in Komga it would be great

Komga is in Kotlin 😉

The metadata retrieval is much more than hitting an api and mapping fields though. That bit is probably only 10% of what I envision for metadata retrieval.

Don't let "perfect" be the enemy of "good". ;) I'm sure if you start something others will contribute.

[Lreaper]

I consider this the most crucial feature still missing in Komga. Besides the obvious benefits of metadata scraping this would also greatly assist in tracking the current status of a series.

[Blazeflack]

Having this feature built-in would be very nice. I currently use the Komf server and userscript to give me the possibility to identify and import metadata for a series. It can also auto-identify an entire library, but I am much too scared to use that functionality, so I prefer the manual single-identify personally :)

Edit: It would be nice if this feature also has the possibility to merge info from multiple sources when importing metadata. No metadata provider has all wanted information when it comes to manga. While a preferred provider has the best descriptions, it may not present tags for that series, while other providers do. So merging in information like tags and authors/artists from other providers is really helpful. This is something Komf supports right now, and is something Komga would need to be able to do if it wants to shine in this area too :)

[aakc]

I think being able to edit/merge scraped data would be extremely useful. Scraping using komf sometimes you'll get data that's formatted differently (e.g. Eiichiro Oda/Oda Eiichiro/ EIICHIRO Oda). Komga treats the different formatting as different authors, so even when you click on the author's name you won't see works by the same author because of how the name is formatted.