Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle revisit records in the same way as cdxj-indexer and/or Heritrix #4

Open
anjackson opened this issue Nov 14, 2023 · 1 comment

Comments

@anjackson
Copy link
Contributor

The crawl-time index is not working for playback because of how revisit records are indexed. See

# FIXME! This pushes REVISIT records to the CDX Server with e.g. the original Content Type, not matching

This should be updated to change revisit record handling so they get CDX-indexed in the same way as cdxj-indexer.

@anjackson anjackson changed the title Handle revisit records in the same way as cdxj-indexer Handle revisit records in the same way as cdxj-indexer and/or Heritrix Nov 14, 2023
@anjackson
Copy link
Contributor Author

I note that Heritrix3 seems to rely on modifying the content type:

https://github.com/ukwa/ukwa-heritrix/blob/0c21b2756c823697839013254a66f06f80cfea3b/src/main/java/uk/bl/wap/util/OutbackCDXClient.java#L532-L534

We should presumably at least match that, and ideally make sure H3 and warcprox do the same as cdxj-indexer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant