Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Dec 13, 2023

• Route tuples to the status updater bolt based on URLs,fixes #65

  Show description for efd0d24
  jnioche

  committed
  efd0d24

  Copy full SHA for efd0d24

• Have as many WARCBolt instances as there are workers, fix #64

  Show description for 63dafc1
  jnioche

  committed
  63dafc1

  Copy full SHA for 63dafc1

Commits on Nov 15, 2023

• Temporary workaround for #42: use http/1.1

  Show description for 0bfccf5
  jnioche

  committed
  0bfccf5

  Copy full SHA for 0bfccf5

Commits on Nov 14, 2023

• Added Maven workflow for GitHub

  Show description for f30c764
  jnioche

  committed
  f30c764

  Copy full SHA for f30c764

• Nutch-compatible implementation of FastURLFilter + use it in PreFilterBolt, fix #59. Storm 1.2.4

  Show description for 2c462d7
  jnioche

  committed
  2c462d7

  Copy full SHA for 2c462d7

Commits on Jul 25, 2023

• Add Java 8 as requirement

  sebastian-nagel

  committed
  b0474ac

  Copy full SHA for b0474ac

Commits on Mar 31, 2022

• Improve command (command-line bin/es_status) to reset URL status

  sebastian-nagel

  committed
  4194f9c

  Copy full SHA for 4194f9c

Commits on Oct 29, 2021

• Sync config files with Stormcrawler 1.18

  sebastian-nagel

  committed
  6229cf0

  Copy full SHA for 6229cf0

• Upgrade to Stormcrawler 1.18

  sebastian-nagel

  committed
  f70a4b8

  Copy full SHA for f70a4b8

• CrawlTopology: make seeds argument optional

  Show description for cd453a4
  sebastian-nagel

  committed
  cd453a4

  Copy full SHA for cd453a4

• Crawl config: increase fetch delay (5 -> 6 sec.)

  sebastian-nagel

  committed
  c8194a4

  Copy full SHA for c8194a4

• Fix duplicate colon in warcinfo header

  sebastian-nagel

  committed
  fb1c64d

  Copy full SHA for fb1c64d

Commits on Jan 28, 2021

• Upgrade dependency-check plugin

  sebastian-nagel

  committed
  2c62b3c

  Copy full SHA for 2c62b3c

• Allow to compile on Java 9 and upwards, fixes #43

  Show description for a45e551
  sebastian-nagel

  committed
  a45e551

  Copy full SHA for a45e551

Commits on Sep 21, 2020

• Update README:

  Show description for 766ec23
  sebastian-nagel

  committed
  766ec23

  Copy full SHA for 766ec23

Commits on Jul 28, 2020

• Upgrade to StormCrawler 1.17

  sebastian-nagel

  committed
  7b4238a

  Copy full SHA for 7b4238a

• Update crawler configuration

  Show description for cd0f81e
  sebastian-nagel

  committed
  cd0f81e

  Copy full SHA for cd0f81e

• typo

  sebastian-nagel

  committed
  a5d43d0

  Copy full SHA for a5d43d0

• Update es_status script:

  Show description for 45ec767
  sebastian-nagel

  committed
  45ec767

  Copy full SHA for 45ec767

• Bootstrap topology: add file spout for seed injection

  Show description for bde92a5
  sebastian-nagel

  committed
  bde92a5

  Copy full SHA for bde92a5

Commits on Jan 30, 2020

• Remove Injector topology from crawler run script

  Show description for 7281e85
  sebastian-nagel

  committed
  7281e85

  Copy full SHA for 7281e85

• Merge crawler configuration updates/changes from storm-crawler

  sebastian-nagel

  committed
  f976a67

  Copy full SHA for f976a67

• Add project URL and license to pom.xml

  sebastian-nagel

  committed
  5265ede

  Copy full SHA for 5265ede

• Upgrade to StormCrawler 1.16 and Elasticsearch 7.5.0

  Show description for a24934e
  sebastian-nagel

  committed
  a24934e

  Copy full SHA for a24934e

Commits on Dec 12, 2019

• Downgrade Jackson version to 2.9.10.1 to avoid conflicts with Storm dependencies

  sebastian-nagel

  committed
  d21f5e0

  Copy full SHA for d21f5e0

• Docker build: skip chown on kibana directory

  Show description for a29215b
  sebastian-nagel

  committed
  a29215b

  Copy full SHA for a29215b

• Elasticsearch metrics index initialization, merge changes from StormCrawler:

  Show description for eafe3e2
  sebastian-nagel

  committed
  eafe3e2

  Copy full SHA for eafe3e2

• Dockerfile: hold news crawler version in variable

  sebastian-nagel

  committed
  44a3a7b

  Copy full SHA for 44a3a7b

Commits on Nov 25, 2019

• Make configuration more polite: fetcher.server.delay.force: true

  Show description for f02bbe8
  sebastian-nagel

  committed
  f02bbe8

  Copy full SHA for f02bbe8

• Upgrade jackson dependency

  sebastian-nagel

  committed
  17a6c5e

  Copy full SHA for 17a6c5e

Commits on Sep 26, 2019

• Update example seed list (news paper feeds and sitemaps)

  sebastian-nagel

  committed
  d8f85ce

  Copy full SHA for d8f85ce

• Move StormCrawler dependency up, to solve version conflicts

  Show description for 98ca6f4
  sebastian-nagel

  committed
  98ca6f4

  Copy full SHA for 98ca6f4

Commits on Sep 23, 2019

• Upgrade to StormCrawler 1.15, Elasticsearch 7.3 and Storm 1.2.3

  Show description for 13ad009
  sebastian-nagel

  committed
  13ad009

  Copy full SHA for 13ad009

Commits on May 24, 2019

• README: add link to news dataset announcement

  sebastian-nagel

  committed
  af9a3af

  Copy full SHA for af9a3af

Commits on May 22, 2019

• Update and improve crawler configuration

  sebastian-nagel

  committed
  fc91ef7

  Copy full SHA for fc91ef7

Pagination

Previous

Next