Engine Enhancement: Segment integrity validation during downloads

[AminBhst]

Some servers might show unexpected behaviors when handling range requests. For example, they might respond with the incorrect byte-range or have missing or extra bytes in the response. While this is entirely the server's fault, a segment integrity validation must be run for all connections during the download. This ensures that the engine can fall back to a healthy state before it's too late to recover. The segment integrity validation must be implemented based on the following specification:

• Check for Content-Range header in server response:
  When each connection receives the response headers, the returned value for Content-Range header must be checked against the requested byte range. If the requested byte range doesn't match the returned one (in the header response), then the range response is faulty and the download should not continue based on this segmentation. Instead, we should move up towards the parent node and invalidate the refresh segment of the running connection belonging to the faulty segments' sibling node in the segment tree. i.e. The lower tree nodes, one of which belonging to a faulty segment, are discarded and the left node (the connection belonging to the left node) falls back to its parent's byte range and disregards the previously set refresh segment.

• Check for total downloaded segment size:
  When a requested segment is finished downloading, the total size of the segment must be calculated and matched against the original requested byte range length. If they don't match, then the download segment is corrupted and should be discarded immediately and the engine should fallback to a healthy state.

• UX: Warn the user of the server's unexpected behaviors. To avoid a download constantly being recursively reverted to a healthy state and failing again, wasting bandwidth and data, the download should be paused automatically and the user should be warned with an alert dialog explaining the situation and asking for confirmation to proceed with downloading this file. Also, if a download was started from a queue, the current faulty download should be skipped and the queue must move on to the next item.

This is pretty complex to test during development and would probably require some random forced corruption in the MockHttpClientProxy to simulate a faulty server. Also, since Brisk's download engine implementation isn't consistent with its end-byte inclusive/exclusiveness in the temp files, the temp file naming implementation might also need fixing to work with this faulty segment detection mechanism.