v1.0.81

What's Changed

• Implement XML parsing using stdlib by @HarshNarayanJha in #155
• Better xml parsing by @yzqzss in #157
• Add --hq-batch-concurrency by @CorentinB in #158

Full Changelog: v1.0.80...v1.0.81

v1.0.80

What's Changed

• Update to latest version of gocrawlhq by @NGTmeaty in #156

Full Changelog: v1.0.79...v1.0.80

v1.0.79

What's Changed

• Add --disable-ipv4 & --disable-ipv6 by @CorentinB in #149
• URL fixes by @NGTmeaty in #148
• Fix Reddit support by @CorentinB in #150
• Add support for ina.fr videos by @CorentinB in #152
• Add Amazon S3 extractor by @CorentinB in #153

Full Changelog: v1.0.78...v1.0.79

v1.0.78

What's Changed

• Fix: cookie named incorrectly. by @NGTmeaty in #145
• Add proper YouTube archiving via YT-DLP by @CorentinB in #126

Full Changelog: v1.0.77...v1.0.78

v1.0.77

What's Changed

• Add some live logging of the HQ consumer behavior by @CorentinB in #143
• fix: HQSeencheckURL() incorrectly distinguishes between new and old URL by @yzqzss in #142
• fix: incorrect crawl speed unpausing behavior by @NGTmeaty in #144

Full Changelog: v1.0.76...v1.0.77

v1.0.76

remove: print in outlinks' ref extraction

v1.0.75

What's Changed

• Fix seencheck re-implementation with current queue by @CorentinB in #139

Full Changelog: v1.0.74...v1.0.75

v1.0.74

What's Changed

• Add job and WARC prefix to pyroscope tags for better search by @NGTmeaty in #133
• fix: recursion present in Telegram URLs by @NGTmeaty in #134
• Add custom code for Reddit archiving by @CorentinB in #137
• Add XML extractor tests by @CorentinB in #138

Full Changelog: v1.0.73...v1.0.74

v1.0.73

Update build.yml

v1.0.72

fixing go version in go.mod