incorporate ct-fetch from https://github.com/jcjones/ct-mapreduce (#156)

* S3 Population code

* Initial python code

* Move to the latest github.com/google/certificate-transparency/go

* Add metadata storage for disk database

* Convert diskdatabase to write single-files, but it isn't handling threading correctly yet

* Complete single-threaded conversion

* Update dependencies for the move of Google's CT go lib to its own repo

* Fix indentation

* Fix #4, #5, by tolerating cert errors

* Fix #6

* Fix #2 to increase throughput

Thanks to @damz for this fix.

* Change away from iniflags

* Remove S3 support

* Add a least-recently-used cache to speed up concurrent disk access.

* Fix #7 - Make DiskDatabase threadsafe

* Move to glog logging

* Add a cache size config option

* Fix a cache timing error

* Put log URL instead of ID into pem files

* Nicer percentage complete info

* Refactor log sync code to one object per URL

* Rename the LogDownloader to LogSyncEngine

* Reorder

* Change progress bars to github.com/vbauerster/mpb

* Improve styling and accuracy of progress bars

* Smooth out EWMA

* Add jitter to polling delay

* Update imports

* Update to mpb 3.3.4 using contexts

* Fix #13 - Add a output_refresh_ms CLI option

* Fix pre-commit errors

* WIP - rework diskdatabase

* Rename storage adapters

DiskDatabase -> FilesystemDatabase
DiskBackend -> LocalDiskBackend

* Restructure the Backend interface to use explicit methods

Mostly breaks the LocalDiskBackend. I might fix it or remove it later.

* Add firestore config

* Move to int64 since uint64 is forbidden for firestore

* Refactor log state storage

* Issuer ID should be urlsafeb64(sha256(issuer.rawSPKI))

Fixes #29

Make the issuer value across the datastore be the SHA256 digest of the
issuer SPKI as a urlsafebase64-encoded string.

* Catch setting up both local disk and firestore

* Keep track of log entry IDs in the PEM headers

* Disable local disk store

* Abstract out the storageDB construction to an engine package

* Fix output speed flag to expect lowercase

* Use standard SIGINT for signal handling

* Convert to using redis cache for known certificates

* Add a savePeriod

* Add mechanism for metric gathering

* Add useful timing metrics

* Fix config processing. Use a normally-distributed jitter mean for runForever.

* Fix readme

* Add more diagnostic metrics

* emit startup stats

* Improve metrics management

* Intelligently choose whether to call Exists

* Skip CAs and provide more context when warning about missing issuers

* Fix #17 - DownloadCTRangeToChannel should not drop entries on contention

* Print log position on erroneous certs

* Fix off-by-one error in log prints

* Deepen the work queue size for more CPUs

* Add new StoreKnownCertificateList method

* Make CTConfig init an explicit step

Works around golang/go issue #31859

* Make a method to initialize telemetry

* Flush glog at closure

* Move context construction to util setup

* Fix issue #14 - Use mpb v4

* Fix issue #19 - Close progress bars when the downloader errors out

* Fix #22 - Back-off on HTTP 429

* Fix logging for retries

* Fix #12 - Make time estimates real

* Adjust backoff

* Metrics for 429 backoffs

* Make metrics more uniform

* Rename ct-fetch/main.go to ct-fetch.go

* Fix log message

* Fix stackdriver errors on URL-scheming

* Remove some metrics

* Move to google/certificate-transparency-go v1.1 and set a user agent

* Move to mpb v5

* More details for exiting with improper usage

* Fix #40: Add a health check (#41)

* Add the infrastructure to do health checks

* Use the update time instead of entry time for the health check

* Use a configuration value and more explicit error reporting

* Wire up the server address configuration to the health server

* Revert "Fix #40: Add a health check (#41)" (#44)

This reverts commit 2b0680e.

* Provide a higher-performance health check (#42) (#45)

This changes the health check from #41 from a database-check to a local
timestamp updated by all the CT downloader threads.

* Add a -nobars option to quiet log output

* update references to jcjones/ct-mapreduce, run go fmt

Co-authored-by: J.C. Jones <[email protected]>
Co-authored-by: J.C. Jones <[email protected]>

README.md

@@ -52,10 +52,10 @@ The keys used into the CRLite data structure consist of the SHA256 digest of the
 
 ## Local Installation
 
-It's possible to run the tools locally, though you will need local instances of Redis and Firestore. First, install the tools and their dependnecnies
+It's possible to run the tools locally, though you will need local instances of Redis and Firestore. First, install the tools and their dependencies
 
 ```sh
-go install -u github.com/jcjones/ct-mapreduce/cmd/ct-fetch
+go install -u github.com/mozilla/crlite/go/cmd/ct-fetch
 go install -u github.com/mozilla/crlite/go/cmd/aggregate-crls
 go install -u github.com/mozilla/crlite/go/cmd/aggregate-known
 

containers/Dockerfile

@@ -10,8 +10,7 @@ ARG ct_mapreduce_ver=v1.0.10
 # Get the current checked-out versions of the CRLite tools
 RUN go build -o bin/aggregate-crls /build/cmd/aggregate-crls
 RUN go build -o bin/aggregate-known /build/cmd/aggregate-known
-RUN go get github.com/jcjones/ct-mapreduce/cmd/ct-fetch@$ct_mapreduce_ver
-RUN go get github.com/jcjones/ct-mapreduce/cmd/storage-statistics@$ct_mapreduce_ver
+RUN go build -o bin/ct-fetch /build/cmd/ct-fetch
 
 FROM python:3.8-buster
 RUN apt update && apt install -y ca-certificates && \
@@ -31,7 +30,6 @@ RUN mkdir /processing && chown app /processing && chmod 777 /processing
 VOLUME /var/log /processing /persistent
 
 COPY --from=builder /build/bin /app/
-COPY --from=builder /build/gopath/bin /app/
 
 COPY moz_crlite_lib /app/moz_crlite_lib
 COPY create_filter_cascade /app/create_filter_cascade