Skip to content

Commit

Permalink
Perform assessments serially to avoid artificially slow snatches (cro…
Browse files Browse the repository at this point in the history
…ss-seed#571)

Prowlarr pauses multiple requests to a given upstream that are opened in
parallel. While this is convenient for most consumers because it means
they don't have to retry, for cross-seed it's a bit of a different story
because we both search and snatch heavily. In this case, when all the
requests are coming from us, the queueing behavior doesn't really help
us. What we can do is to just move the queueing behavior into
cross-seed, by running assessments in a loop instead of `Promise.all`ing
them, which will make `snatchTimeout` work better.

closes cross-seed#560 

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Refactor**
- Improved the efficiency of the candidate assessment process in the
search functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
  • Loading branch information
mmgoodnow authored Dec 29, 2023
2 parents a34e3e9 + b174754 commit 7dd224a
Showing 1 changed file with 23 additions and 11 deletions.
34 changes: 23 additions & 11 deletions src/pipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,17 +61,27 @@ interface FoundOnOtherSites {
matches: number;
}

async function assessCandidates(
candidates: Candidate[],
searchee: Searchee,
hashesToExclude: string[]
): Promise<AssessmentWithTracker[]> {
const assessments: AssessmentWithTracker[] = [];
for (const result of candidates) {
const assessment = await assessCandidate(
result,
searchee,
hashesToExclude
);
assessments.push({ assessment, tracker: result.tracker });
}
return assessments;
}

async function findOnOtherSites(
searchee: Searchee,
hashesToExclude: string[]
): Promise<FoundOnOtherSites> {
const assessEach = async (
result: Candidate
): Promise<AssessmentWithTracker> => ({
assessment: await assessCandidate(result, searchee, hashesToExclude),
tracker: result.tracker,
});

// make sure searchee is in database
await db("searchee")
.insert({ name: searchee.name })
Expand All @@ -94,11 +104,13 @@ async function findOnOtherSites(
}))
);

const assessed = await Promise.all<AssessmentWithTracker>(
results.map(assessEach)
const assessments = await assessCandidates(
results,
searchee,
hashesToExclude
);

const { rateLimited, notRateLimited } = assessed.reduce(
const { rateLimited, notRateLimited } = assessments.reduce(
(acc, cur, idx) => {
const candidate = results[idx];
if (cur.assessment.decision === Decision.RATE_LIMITED) {
Expand All @@ -113,7 +125,7 @@ async function findOnOtherSites(
}
);

const matches = assessed.filter(
const matches = assessments.filter(
(e) =>
e.assessment.decision === Decision.MATCH ||
e.assessment.decision === Decision.MATCH_SIZE_ONLY
Expand Down

0 comments on commit 7dd224a

Please sign in to comment.