Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to fix IOException/GOAWAY happening within parseResults() #818

Merged
merged 5 commits into from
Feb 16, 2024
Merged

Attempt to fix IOException/GOAWAY happening within parseResults() #818

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1b0d552
attempt to fix IOException/GOAWAY happening within parseResults()
phanecak-maptiler Feb 15, 2024
8377e61
wait 5s berfore retrying WikiData request after GOAWAY, configurable …
phanecak-maptiler Feb 16, 2024
f8679b2
use Duration() for http_retry_wait for nicer handling of units
phanecak-maptiler Feb 16, 2024
01dec42
do a sleep on all IOException (not just GOAWAY) + no need to catch In…
phanecak-maptiler Feb 16, 2024
7b78813
clean-up: description of http_retry_wait adjusted
phanecak-maptiler Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions planetiler-core/src/main/java/com/onthegomap/planetiler/config/PlanetilerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ public record PlanetilerConfig(
String httpUserAgent,
Duration httpTimeout,
int httpRetries,
Duration httpRetryWait,
long downloadChunkSizeMB,
int downloadThreads,
double downloadMaxBandwidth,
Expand Down Expand Up @@ -166,6 +167,7 @@ public static PlanetilerConfig from(Arguments arguments) {
"Planetiler downloader (https://github.com/onthegomap/planetiler)"),
arguments.getDuration("http_timeout", "Timeout to use when downloading files over HTTP", "30s"),
arguments.getInteger("http_retries", "Retries to use when downloading files over HTTP", 1),
arguments.getDuration("http_retry_wait", "How long to wait before retrying HTTP request", "5s"),
arguments.getLong("download_chunk_size_mb", "Size of file chunks to download in parallel in megabytes", 100),
arguments.getInteger("download_threads", "Number of parallel threads to use when downloading each file", 1),
Parse.bandwidth(arguments.getString("download_max_bandwidth",
Expand Down
18 changes: 11 additions & 7 deletions planetiler-core/src/main/java/com/onthegomap/planetiler/util/Wikidata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,10 +300,15 @@ private LongObjectMap<Map<String, String>> queryWikidata(List<Long> qidsToFetch)
.POST(HttpRequest.BodyPublishers.ofString(query, StandardCharsets.UTF_8))
.build();

InputStream response = null;
for (int i = 0; i <= config.httpRetries() && response == null; i++) {
LongObjectMap<Map<String, String>> result = null;
for (int i = 0; i <= config.httpRetries() && result == null; i++) {
try {
response = client.send(request);
var response = client.send(request);
if (response != null) {
try (var bis = new BufferedInputStream(response)) {
result = parseResults(bis);
phanecak-maptiler marked this conversation as resolved.
Show resolved Hide resolved
}
}
} catch (IOException e) {
boolean lastTry = i == config.httpRetries();
if (!lastTry) {
phanecak-maptiler marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -312,13 +317,12 @@ private LongObjectMap<Map<String, String>> queryWikidata(List<Long> qidsToFetch)
LOGGER.error("sparql query failed, exhausted retries: " + e);
throw e;
}
Thread.sleep(config.httpRetryWait());
}
}

if (response != null) {
try (var bis = new BufferedInputStream(response)) {
return parseResults(bis);
}
if (result != null) {
return result;
} else {
throw new IllegalStateException("No response or exception"); // should never happen
}
Expand Down
Loading