Skip to content

Latest commit

 

History

History
799 lines (676 loc) · 36.6 KB

TODO.md

File metadata and controls

799 lines (676 loc) · 36.6 KB

Add: https://www.manga.club/ https://www.lezhin.com/en http://helveticascans.com/r/directory/ https://lnwdoujin.com/ ?

To Check: 80 : (ScrapePlugins.M.MangaMadokami.Run, hours(4)), 81 : (ScrapePlugins.M.BooksMadokami.Run, hours(4)),

0   : (ScrapePlugins.M.BtBaseManager.Run,                   hours( 1)),
2   : (ScrapePlugins.M.BuMonitor.Run,                       hours( 1)),

12  : (ScrapePlugins.M.IrcGrabber.IrcEnqueueRun,            hours(12)),  # Queue up new items from IRC bots.
15  : (ScrapePlugins.M.IrcGrabber.BotRunner,                hours( 1)),  # Irc bot never returns. It runs while the app is live. Rerun interval doesn't matter, as a result.

FIX: 24 : (ScrapePlugins.M.MangaBox.Run, hours(12)), 23 : (ScrapePlugins.M.ZenonLoader.Run, hours(24)),

62 : (ScrapePlugins.M.FoolSlide.Modules.ChibiMangaRun,      hours(12)),  No longer foolslide?
75 : (ScrapePlugins.M.FoolSlide.Modules.MangazukiRun,       hours(12)),



# FoolSlide modules
	73 : (ScrapePlugins.M.FoolSlide.Modules.TwistedHelRun,      hours(12)),

Broken: tsumino DoujinMoe Sura's Place

Gone: WebtoonsReader

http://www.tsumino.com/

http://cafeconirst.com/ https://www.spottoon.com/

02:30:50 Danzaiver | There's mangago.me for yaoi people │ gl77bug 02:32:33 Danzaiver | and mangahere.co perhaps │ Gordo 02:34:52 Danzaiver | and mangatown.com and that's pretty much it. │ Hal9k

refetch items with 3 or fewer images.

Broken volume number: Vol.1 part 2 chapter 8: level 2 I think it's converting Vol.1 to Vol, 0.1, and then casting 0.1 to integer, and getting 0

invert hash table to find potential dups

Todo: Replace messy walking system for madokami with the stupidapi call dir listing thing.

Add spottoon scraper.

http://www.nekyou.net/ https://www.spottoon.com/

http://mirage-trans.dreamwidth.org/ http://mirage-trans.dreamwidth.org/profile http://tasha-poisonous.livejournal.com http://animemangadaisuki.tumblr.com/post/64006063205/kuroko-no-basuke-replace-novel-compilation-not http://animemangadaisuki.tumblr.com/kurobasu-replace-3-chapter-2-english-translation http://mocopersonal.tumblr.com/post/23932157908 http://mocopersonal.tumblr.com/post/25799016522 http://mocopersonal.tumblr.com/tagged/kuroko-no-basuke-novel http://forums.mangafox.me/threads/379264-Kuroko-no-Basuke-Novel-Replace-I-and-II

09:09 < keane> to who ever has the scrapper http://bato.to/comic/_/comics/henjo-hen-na-joshi-kousei-amaguri-senko-r15331 is not being updated https://manga.madokami.com/Manga/G%20-%20M/Henjo%20-%20Hen%20na%20Joshi%20Kousei%20Amaguri%20Chiko 09:12 -!- thefunkydealer [~[email protected]] has joined #madokami 09:23 -!- AcidWeb [~[email protected]] has quit [Remote host closed the connection] 09:23 -!- OmegaGlory [~[email protected]] has joined #madokami 09:30 -!- AcidWeb [~[email protected]] has joined #madokami 09:35 <@Fake-Name> @keane - check if it's being updated under another name 09:36 < keane> other title is https://manga.madokami.com/Manga/_Autouploads/AutoUploaded%20from%20Assorted%20Sources/Henjo%20-%20Hen%20na%20Joshi%20Kousei%20Amaguri%20Chiko but the one from 2 days ago wasn't updated 09:37 <@Fake-Name> alright, I'll look at it this evening 09:37 < keane> also thanks for the scrapper 09:38 < keane> some of the new uploads also have the word "test-" in the beginning of the file 09:38 <@Lord-Simon> Fake-Name, go create a temple under a fake name. So they can worship you. 09:38 < keane> I just noticed when downloading 09:38 -!- Coppola [~[email protected]] has joined #madokami 09:39 < keane> like https://manga.madokami.com/Manga/G%20-%20M/Helck 09:39 < keane> or basically https://manga.madokami.com/recent 09:40 <@Fake-Name> that's probably one of the source sites doing 09:40 <@Fake-Name> is it consistently one source? 09:40 < keane> >/G - M/Livingstone/test-Livingstone Vol.002 Ch.009 Word Animal [KissManga].zip 09:40 <@Fake-Name> do they all have [kissmanga.com] in the title? 09:41 < keane> >/N - Z/Super Danganronpa 2 - Chou Koukoukyuu no Kouun to Kibou to Zetsubou/test-Super Danganronpa 2 - Chou Koukoukyuu no Kouun to Kibou to Zetsubou - v2 c8 [batoto].zip 09:41 <@Fake-Name> huh 09:41 < keane> seems like batoto and kissmanga 09:42 < Blueshirt> if Fake-Name took a fake name would he be Real-Name 09:43 < cranon> http://i.imgur.com/XWOuu8G.gif 09:46 -!- Bromura [~[email protected]] has joined #madokami

Webcomics module? http://sinnergate.com/

Handle fractional parts (e.g. 1/6, 2/3, etc... ) of chapters.

https://www.wlnupdates.com/series-id/2333/

undecentlnt.wordpress.com source parser http://10.1.1.8:8081/books/render?url=https%3A//pummels.wordpress.com/2015/06/18/konjiki-no-word-master-chapter-128-how-to-deal-with-zombies http://10.1.1.8:8081/books/render?url=http%3A//www.wuxiaworld.com/atg-index/atg-chapter-77/# http://10.1.1.8:8081/books/book-item?dbid=1963 http://10.1.1.8:8081/books/render?url=https%3A//drive.google.com/open%3Fid%3D0B_mXfd95yvDfQWQ1ajNWZTJFRkk not extracting drive

http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fdrive.google.com%2Ffolderview%3Fid%3D0B7PIRwu9hTEFODllMFFRemljMGc links broken http://pacem.wikia.com/wiki/Pacem_Community_Translations_Wiki

http://hotchocolatescans.mokkori.fr/

search for foolslide sites http://demonicscans.com/FoOlSlide/read/shen_yin_wang_zuo/en/0/13/page/2

https://sites.google.com/site/englishfantranslations/home http://www.spcnet.tv/forums/showthread.php/37833-Stellar-Transformations-NO-SPOILERS-ALLOWED#.VSNS9pUtBdl

https://docs.google.com/document/d/1xeivM_JKYlpxN7gZrjBkk6_Z79AZ9PFzZbLvoeSWoV4/preview?pli=1 not scraped

fix last crawl time

hide size for directories

pad inside book content div

not yet retrieved link to sauce

internal links in http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1DL6dpctSl_DME-4cnnZ8mkfCIb9nIMlu8P8JhDU-it8%2Fpub broken table on http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1NjsYhS3_PQvA3xSjOV0IubpNmyJk98eBu19ak2NMaIQ%2Fpub table on http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fsites.google.com%2Fsite%2Fsekaigameoredake%2Fchapter-5%3Fmobile%3Dtrue

Fetch covers from LNDB/MangaUpdates. Hook them into the web-content system.

Covers in manga pages too!

sources from http://bato.to/forums/topic/19625-where-can-i-find-recommendations/

Western:

LNDB / MU covers

delay kiss manga scrape dl 12 hours book search link all parameter

autocleaner thing?

pg pool!

Not consolidating:

http://10.1.1.8:8081/books/book-item?dbid=172 http://10.1.1.8:8081/books/book-item?dbid=1488

basename in book changes not workimg

Hachinantte, Sore wa Naideshou!

TO ADD:

=======

http://hotchocolatescans.mokkori.fr (Not LN?) http://ckmscans.halofight.com/ http://heaven.neo-romance.net/manga/ch00.php http://www.surasplace.com/index.php/projects.html

Livejournal: http://toshosen-tsl.livejournal.com/

Blogger: http://www.pegasusfarts.com/

Tumblr: http://pokkoo-shuu.tumblr.com/thelunacyofdukevenomania http://sayasamax3.tumblr.com/post/56013645417/high-speed-translations-and-summaries

General Resource: http://englishlightnovels.com/

Use http://lndb.info/ as BU-like?

Also: http://aho-updates.com/

Non-Catalog Novels:

http://10.1.1.8:8081/search-b/b?q=kansustoppu

http://10.1.1.8:8081/search-b/b?q=Maou+no+Hajimekata

LN TODO: Add transparent scraping into GDoc content Scrape external links that look like images Make link converter aware of all the other LN scrapers, so inter-site links work.

Check LN Scrapers after changes:

GDoc driven:
	~~HaruParty http://lasolistia.com/haruparty/ ~~
	~~ReTranslations~~


## SolitaryTranslation - Password bullshit

Should I start scraping VN translation patches?:

~~http://tlwiki.org/~~
http://novelnews.net/

everyone at http://www.mangaupdates.com/releases.html?search=116177&stype=series

http://deusexscans.blogspot.com/ http://deusexscans.blogspot.com/2014/08/one-turn-kill-of-dark-partisan-vol-1_29.html

H source: https://teamkoinaka.wordpress.com/

  • Scrape "Type" entry from mangaupdates

  • tagging system

  • revisit gdoc extractor. the extra spaces are annoying

  • lcs diff system instead of levenshtein for change measurement.

  • filter rating view by read and crosslink

  • test for bad del

  • consolidate existence checks

New LN Trans groups:

Todo:

  • chapter and volume columns

  • volume/chapter tracking, rather then just c

  • request priority sorting

  • custom book synonyms

  • sync reading progress both directions

  • store manga progress locally

  • new books view still fucked

  • search also include full text search

  • feed link to trieview

  • dupscan just kiss

  • Add update time tracking to textscrape.

  • lower request priority for trigram searches

  • mu archive release info

  • mangaeden api

  • custom list page add search for existing item

  • serve compressed HTML

  • tadanohito shutting down scrape everything

  • use jdownloader api to scrape other h blogs

  • search in rss tags too

  • filter rating view by crosslink status

  • h rating system

  • deduper expose get matching archives for spiral

  • info table position when multiple sources

  • volumeifacator

  • Proper to-download series system

  • autotrim empty dirs

  • Trigger full series download if a series is seen by a scraper, and the local directory is both found, and rated above a threshold (Done for Batoto, needs per-plugin work. Add facilities to pluginBase?)

  • Trigger series download if on any BU list as well (partial)

  • base nt off muid system Axed, MU isn't as encyclopedic as I thought at one point.

  • ability to browse dirs by mu list cross-link

  • select 201 to determine next page

  • add half star rating options

  • h tag collation system

  • consolidation system for h tags

  • out of row colours (Fixed by some low-effort HSV transforms)

  • linebreaks in long filenames in reader popup need work

  • zoom mode indicator for smart mode in reader

  • ability to specify MU id in directory name? [Lnnn] or sommat?

  • way to search for non linked directories - maybe then do levenshtein search for match?

  • import all existing files somehow

  • do some clever set shit to check for misplaced items in directories

  • color code mangaupdates status in reader

  • scan times to deduper for rescanning. Also filesizes

  • Group the smaller scanlators into a single colour-code?

  • artist and author in filebrowser if i have it

  • bu page opens in new window

  • IRC grabber needs a transfer stall timeout.

  • Modularize the side-bar in the manga browser, so the plugins can each provide their own lookup interface if they present the correct API (should be automatically discovered, ideally).

  • Potential race-condition in deduper when two things are scanned by separate threads simultaneously. Add a global "deletion" lock to prevent accidental removal of all copies of file

  • Prevent full base dir refresh on directory rename.

  • prioritize downloads by rating

  • tagging in web interface

  • properly show if things are one shot

  • fork daiz numbering

  • steal code from free manga downloader?

  • most common image browse and filtering system 'SELECT COUNT(itemhash), itemhash FROM dedupitems GROUP BY itemhash;'

Deduper R2:

  • deduper spiral out to significant intersections on new scan (Depends on new cleaned-up deduper system) Too much overhead.
    • Filter spiral aggressively by manga directory?
  • phash filter by resolution for deletion decision (Done)
  • Image entropy is not useful for quantifying image ancestry

Long Term:

  • cover images in file browser?
  • Ability to disable bulk-downloading.
  • Get Watermark-Stripping system running.

Add Scrapers for

Reader

  • fit width only if oversize?
  • make zoom mode pop up last longer
  • fancy fade out when toolbars hidden?
  • Add ability to rename directories to reader (res, name)
  • Add current page position bar when popup menus are visible.
  • Trigger directory cache update if a non-existent directory access is attempted
  • smart zoom mode in overlay
  • Make zoom mode a bit more intelligent (e.g. look at aspect ratio to guess zoom mode).
  • show current image info
  • Chapter read to from BU in item sidebar.

File System Organization

  • Coerce directory structure to match MangaUpdates naming.
  • Scrape ALL MangaUpdates metadata, and use that to group series when different sources use different naming schemes. (WIP)
  • Automatically organize and sort directories so each series only has one directory. Aggregate multiple directories so they're named in accord with MangaUpdates naming approach. Note this makes MangaUpdates the final authority on what to refer to series as. Deal with it

Nametools Issues

  • Getsurin ni Kiri Saku
  • Imasugu Onii-chan ni Imouto da tte Iitai!

Bad MangaUpdates Links:

  • I think these are fixed by the latest NameTools patches.
    • Dungeon ni Deai wo Motomeru no wa Machigatteiru Darou ka
    • neko ane
    • rescue me
    • maken-ki!
    • murcielago
    • Keyman - The Hand of Judgement
    • Okusama ga Seito Kaichou! [+++][EE] npt linking
    • Log Horizon - Nishikaze no Ryodan not linking properly
    • 'Hero Co., Ltd.' link only working one way
  • Add testing to nametool system (Done, wasn't the problem source)
  • Everything is getting sorted into '[zion]' Fixed, it was a scraper bug. Derp

Things to search for:

Partial series: - 15:14 -!- Irssi: Starting query in Rizon with Sola - 15:14 hi hello ok - 15:14 Kamisama no Iutoori Ni is missing 50-74 and 76 - 15:15 soul cartel 127, 134, 135 - 15:15 tiara 31-34, 37-41 - 15:15 Hayate no Gotoku! c454-460 and c473 - 15:16 Grappler Baki c94 - 15:29 Himouto! Umaru-chan 41, 42, 44-65, 70, - 15:32 Maken-Ki! 66-68 - 15:38 is all for today - 15:42 is pasting the list for duplicates so far, yes? - 15:43 I realize Crunchyroll is a special case but... whatever. I listed it too - 15:43 13-nin no Short Suspense and Horror - KissManga, Batoto, MangaCow, IIII - 15:43 Akame ga Kill - PSY, BatotoIIII - 15:43 Ansatsu Kyoushitsu - Batoto, MangaJoyIIII - 15:43 Ao Haru Ride - MangaJoy, BatotoIIII - 15:43 Ao no Exorcist - S2S, KissManga, MangaJoyIIII - 15:43 Bartender - Batoto, MangaJoy, KissMangaIIII - 15:43 Black Haze - MangaJoy, Batoto, KissManga, MangaCowIIII - 15:43 Blind Faith Descent - MangaCow, KissMangaIIII - 15:43 Cavalier of the Abyss - VISCANS, MangaCowIIII - 15:43 Fairy Tail - MangaStream, Crunchyroll, KissMangaIIII - 15:43 Grappler Baki - KissManga, MangaJoy, BatotoIIII - 15:43 Himouto! Umaru-chan - Batoto, KissMangaIII - 15:43 Hirunaka no Ryuusei - MangaJoy, BatotoIIII - 15:43 Karate Shoukoushi Kohinata Minoru - Batoto, KissMangaIIII - 15:43 Kero Kero Chime - Batoto, KissMangaIIII - 15:43 Magical Exam Student - Webtoons.com, KissManga, MangaJoyIIII - 15:43 Maken-Ki! - Batoto, KissMangaIIII - 15:43 Metronome - Batoto, KissMangaIIII - 15:43 Nanatsu no Taizai - Crunchyroll, Batoto, KissMangaIIII - 15:43 Noah - Batoto, MangaJoyII - 15:43 Noragami - MangaJoy, KissMangaIIII - 15:43 Okitenemuru - Crunchyroll, KissMangaIIII - 15:43 Ore ga Ojou-sama - MangaJoy, BatotoIIII - 15:43 Pure-Mari - Batoto, KissManga, MangaJoyIIII - 15:43 Sailor Fuku, Tokidoki Apron - MangaBox, MangaJoyIIII - 15:43 Senyuu. - Batoto, KissManga, MangaJoyIIII - 15:43 Soul Cartel - KissManga, MangaCowIII - 15:43 Tonari no Kashiwagi-san Batoto, MangaJoyIIII - 15:43 Tonari no Seki-kun - Batoto, MangaJoyIIII - 15:43 Yamada-kun to 7-nin no Majo - Crunchyroll, MangaJoy, KissMangaIIII

Complete:

  • duplicate h tags of different case
  • rss separate from ln scrape
  • wp/blogspot use labels for posts
  • still missing sites from aho
  • rss+timeouts for change monitoring http://aho-updates.com/groups
  • avail chapters color code in book list
  • rework books page, defer loading moar
  • pastebin stuff override title
  • generalized title injection would be useful, actually
  • separate plugin status page
  • pron ordering fucked up?
  • netloc in book search table
  • sort trigram search results by similarity
  • also expose similarity value
  • new spirit migration chapters missing from search? (Just hadn't updated?)
  • rss item - book series search
  • home swap bold and red
  • Move scheduling system to a persistent jobstore.
  • http://reader.s2smanga.com/ fooslide based add scraper
  • http://10.1.1.8:8081/books/render?url=https%3A//yoraikun.wordpress.com/2015/02/09/konjiki-no-wordmaster-chapter-32/ prev not relinked Fixt
  • Name synonyms in bu table for books.
  • book availability not updating (different components)
  • sync read to as well
  • http://10.1.1.8:8081/books/render?url=http%3A//www.baka-tsuki.org/project/index.php%3Ftitle%3DGekka_no_Utahime_to_Magi_no_Ou Missing items. Rescraping (Fetched, had to add scrape source)
  • book-item load search ajax
  • Description from BU is missing internal HTML.
  • Block comment crap on FictionPress
  • Add functional text scrapers to scheduler
  • include altnames in search set (Ajax load ALL THE THIGNS)
  • http://10.1.1.8:8081/books/render?url=http%3A//japtem.com/arifureta-volume-9-chapter-4/ missing (Not anymore?)
  • mu link works in list table, fails in item: http://10.1.1.8:8081/books/book-item?dbid=24 (Naming issue)
  • http://10.1.1.8:8081/books/render?url=https%3A//docs.google.com/document/d/1M2VpqmXNb31RwrRIeQ2FsKEbd8xfIdKTmHtR4K9ga8U/edit should relink as http://10.1.1.8:8081/books/render?url=https%3A//docs.google.com/document/d/1M2VpqmXNb31RwrRIeQ2FsKEbd8xfIdKTmHtR4K9ga8U container http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1xInAD8v06AIX_urMZRRXHBocDsqBEePMoU1EOTfGRZQ%2Fpub fixed gdoc relink cleaning issue
  • bs is scraping wp comtent again also bt (Fixed, was class variable issue.)
  • missing image: http://10.1.1.8:8081/books/render?url=https%3A//shintranslations.wordpress.com/vol-1-chapter-1/ (Seems to have it now)
  • MISSING NEW MU TAGS
  • change book links to point to book page rather then search
  • http://10.1.1.8:8081/books/render?url=http://skythewood.blogspot.com/2014/10/O12.html Run-on text mess Source is wierd. Can't do much.
  • content in iframe: https://bluesilvertranslations.wordpress.com/2015/01/19/000-prologue-tang-third-young-master-crossing-over-v2/ wat (Now pulling links from iframes)
  • http://10.1.1.8:8081/books/render?url=http%3A%2F%2Fskythewood.blogspot.ca%2Fp%2Fknights-and-magic-author-amazake-no.html funky relinking
  • book ratings!
  • Write tool for flushing/aggregating book resource content items. (Done)
  • http://10.1.1.8:8081/books/render?url=https%3A//docs.google.com/document/d/16v8Q9Kice7pinXq4_aoEBr6Viu7CuRWPyv_TyE7wXOg formatting oddness
  • http://10.1.1.8:8081/books/render?url=https%3A%2F%2Fhokagetranslations.wordpress.com%2Ftag%2Fmonster-musume-harem-o-tsukurou-make-a-monster-girl-harem%2F header not cleared
  • http://10.1.1.8:8081/books/render?url=https%3A//drive.google.com/folderview%3Fid%3D0B8UYgI2TD_nmMjE2ZnFodjZ1Y3c%26usp%3Ddrive_web
  • http://10.1.1.8:8081/books/render?url=https%3A//docs.google.com/document/d/10_3RfB9ZSNhNrlsF1YB53NJqjQ1Nu3F0b6d01bfPHxU formatting wut
  • http://10.1.1.8:8081/books/render?url=https%3A//docs.google.com/document/d/1eJWjpwP04A-CANLD22dZTujLBN1FIyLyjH6XQvjeNYc/pub strip container div
  • http://10.1.1.8:8081/books/render?url=https%3A//oniichanyamete.wordpress.com/2015/01/29/burikko-chapter-10/ header text
  • dangling /preview in gdocs
  • Extract the titles from gdoc?
  • missing page link to original
  • add jump to source to end of page
  • book search natsort
  • mu view only book items
  • none in tag list broken
  • add filter to mu page for only items with found dirs
  • heretic trans missing clockwork planet
  • rtd scraper broken
  • strip (novel) from search text
  • Oniichanyamete all Yomigaeri no Maou (Novel) text missing (Fixed, unicode in URL issue)
  • readability performance The problem is entirely levenshtein distance calculation. Huh. Wound up just using string length difference. Eh...
  • mask empty lists List generation is procedural, I don't know if the list is empty until I've finished generating it.
  • mu page cahp number comparrisons are messed up
  • also handle read to > latest translated as noted on mu
  • h search to each dir page It's probably going to be 95% false positives, but what the hell
  • unlimitednovelfailures
  • http://yoraikun.wordpress.com/
  • delay loading h searches until within viewport
  • h search in bu list page Hopefully the trigram search load won't cause issues.
  • Damaged Archive remover Implemented.
  • del json response undefined, AFICT, this is just IE11's alert() behaviour being bizarre on ARM.
  • Low-Num highlight in mangaonly view
  • gin on h tags (Already done)
  • Tadanohito cl sync cookies before run
  • Delete items via web interface!
  • sort all by rating Had already done it. Huh.
  • highlight chapters < 10
  • Deduper not blocking queries while tree is rebuilt
  • special case bolding so it only applies in aggregate views
  • crawl all batoto
  • change startup. start webserver first
  • change db table generation to use found item paths for tables in directory view.
  • strip trailing hyphens
  • mixed case tag issues
  • books are broken again Dicked about in the scheduler. Hopefully, it's fixed?
  • positive tags in green
  • include h in mu tag view
  • homepage table do not include deduped
  • negative h keywords red
  • Deduper - enable the ability to check for duplicates using phash as well. (Partial - Needs ability to search by hamming distance to work properly)
  • bu page and rating page columns are backwards
  • bu tag browser already
  • mechanism for highlighting chosen tags in table (specifically deduped in J)
  • Add failed item introspection table.
  • ability to browse dirs by rating
  • ability to browse by mu tags
  • synonyms without exclamation points
  • Madokami is fucked again. Fuck you HTTPS simple auth.
  • chap regex prefer ch or c prefix
  • string difference system for books
  • tsuki has issues?
  • unlinked autouploads broken
  • Deduper - Move database interface into scanner, one interface per thread. Make each archive scan a transaction.
  • mu tag browser - GiN/GiST index?
  • ex filter by category too
  • random h already updated tag It's actually from the source. Not much I can do.
  • fix djmoe
  • rescan series on batoto
  • booktrie nodes decrease in size (Whoops, CSS Stupid)
  • rating changing is broken
  • "None"s in btSeries markup Stupid context issue
  • kissmanga phash dedup
  • move non matching dirs to another folder
  • add ability to sort directory by rating. (Added in MangaUpdates stuff, not sure if I want it elsewhere)
  • filesize in browser
  • different tag for phash desuplication
  • fakku broken.
  • hbrowse missing artists amd title truncated
  • Batoto doesn't list every file in the recent updates page. Scan into series pages Doing more thorough search
  • Move to python-sql for dynamic sql generation
  • reset download button in mangatable for specific key view
  • filtered h isn't being properly skipped
  • fix pururin page ordering already It was a sorting issue in the session system? Fuuuuuuuuuuck.
  • Migrate to new queries from tests.test-Queries Superceeded by procedural queries using server-side cursors.
  • tag/flag for when items are mirrored to mk? DlState=3 means uploaded
  • fakku is broken? Fixed
  • Load tables asynchronously from base page
  • dlstate 3 not rendering right
  • have vol and chap in separate columns
  • key not found error resulting in HTTP 500 for bad path after rating change
  • colons in seriesnames Should already be removed. Not sure what's going on
  • sorter not properly handling items with only volume number in filename (generally prepended by "volume {xxx}").
  • mk uploader needs to add uploaded files to the mk downloader list so they dont get re transferred Should be done, I think?
  • 7z support in archtool Added as a result of wanting to add it to the deduper. Shared code FTW.
  • figure out why bad dir-lookup matches are all defaulting to {'dirKey': 'it takes a wizard'} Someone had put "None" in the alternative names for the "It takes a Wizard" manga. Whoops?
  • Make user-agent randomize Should have something like ~32K possible configurations now.
  • import archived djm stuff?
  • tie mk uploader in properly
  • proxy for name lookups.
  • fix lo colums?
  • automover patch path in db for moved items Added fix-dl-paths to cleanDb.
  • mangajoy sometimes only fetches one image Added some delay, hopefully it'll fix things.
  • look into pu sorting issues. Stupid logic error in download delay mechanism.
  • check that new items in bu are updating properly
  • djm retagger no longer running?
  • Murcielago. again Hopefully fixed by forcing NFKD unicode normalization. Fuck unicode. Arrrgh.
  • mu not in sidebar Fucked up the flags at some point. Fixed.
  • tags by number for h
  • Try to do something clever with sorting items in the directory viewer. Preprocess to extract vol/chapter inteligently? Simple regex implemented. I'll have to see how it pans out
  • sort bu lists contents alphabetically
  • aggregation query is fucked. somehow. Fixed with procedural filtering system.
  • scan downloads, retry missing not deduped Functin added to utilities.cleanDb
  • irc defer dir search to actual download (mk too)
  • push dir updating into separate thread The issue wasn't dir-updating, it was the DB loading it's cache from disk. Fixed by postgre
  • Better mechanism for chosing colours for rows. Use a calculating system, rather then requiring manual choice
  • Add file existence check to tooltip in manga table I'll have to see if 200 file existence checks is a problem for page-rendering time.
  • Logger output coloring system
  • mangacow missing last page Fucking off-by-one error
  • UNIQUE constraint on buId for mangaseries table
  • Add parent-thread info to logger path for webUtilities.
  • Recreate triggers to update counts on insert/delete
  • make tags case-insensitive (Switch to CITEXT should do this, added .lower() to query generator anyways)
  • mu cross references all broken
  • switch relevant columns to CITEXT
  • Distinct filter not working!
  • strip metainfo from links in h (artist-, scanlators-, etc) (Also added a configurable tag highlighter)
  • Defer dir updating to after page-load to prevent occational 20 second page-loads. I think this is actually the DB loading the indexes from disk. Not sure. (Hopefully fixed by move to Postgre
  • IRC scraper is broken for filenames with spaces.... Yeah....
  • proper transaction system for DB (or just go to postgres) (went to postgre)
  • Fix Madokami scraper
  • batoto cross verify number of images Never mind, it's not a blind exploration, it's actually using the image navigator dropdown to generate image urls.
  • Implement dir moving system already!
  • total chapters not always known. Handle sanely.
  • find or create only choosing dirs in picked. (Fixed a while ago)
  • irc scrapinator
  • auto upload to madokami
  • automate color choices for reader; fukkit just do a naive implementstion of rotation in a hsv colour space
  • split item color generation into hemtai and manga to provide better control
  • ~~remove k scale from filesize readout. ~~
  • reader directory page also includes database items for series
  • show reader some general luv
  • fakku scraper barfs on unicode (I think it's fixed?)
  • itemsManga page isn't using activePlugins.mako
  • Also the itemsPron page.
  • scrape mangacow
  • Non-distinct manga view is borked
  • Queue whole of any new series on batoto when a rating is found that's >= "++"
  • Deduper - Check that local duplicate of file found via DB still exists before deleting new downloads.
  • Scrape Fakku
  • optimise name cleaning. Spent some time profiling. Not worth the effort (not much room for improvement).
  • optimize optimize optimize! 1 second for home rendering. (~0.5 seconds! Woot!)
  • mangafox if they dont resize. Never mind. they took down all their Manga because licensing reasons, apparently?
  • clean ! from matching system. (Was already done)
  • split porn/nonporn again?
  • Fix BU Watcher login issues. Cookies are the fucking bane of my existence.
  • Add planned routes to look into the various tables (can I share code across the various query mechanisms?) (Mostly complete)(I'm calling this complete, since I only have two table-generator calls ATM)
  • Scrape download.japanzai.com
  • Fix rating change facility being broken by the new reader
  • Finish reader redesign
  • Fix presetting of item rating field. (Accidentally fixed, I think? Not sure how, but it's now working.)
  • reader shits itself on unicode urls.
  • Allow arbitrarily nested folders in reader. (added in new reader)
  • Prefferentially rescan MangaUpdates series that got releases today (e.g. scan https://www.mangaupdates.com/releases.html).
  • also pururin.com
  • pagechange buttons for porn broken in some instances.
  • MangaUpdates name lookup passthrouth in nametools.
  • fukkit, scrape batoto.
  • Add legend key for what row colours mean (easy).
  • Add better time-stamp granularity to Starkana Scraper. (I think?)
  • MangaBaby.com scraper
  • Flatten any found duplicate directories, when they span more then one of the manga-folders.
  • FIX NATURAL SORTING (Fixed upstream in the natsort package)
  • Make series monitoring tool for MT update periodically
  • Automated tag update mechanism!
  • Commit hooks to track the number of items in the mangaTable, without the massive overhead SELECT COUNT(*) has on SQLite (this should be fun and educational in terms of SQL).
  • Generalize the image-cleaner to remove all annoying batoto/starkana/whatever images from downloaded archives. Possibly make it possible to run in batch mode? It should have a local directory of "bad" images that are scanned on start, and compare using hashes (or full on bitwise?).
  • Scrape perveden.com Fuck them, they watermark their shit. Never mind.
  • automover

LN:

https://ichisora.wordpress.com (Songs)