Silent Data Corruption Still Occurs Occasionally

[DavidBerdik]

Apparently, there is still an issue with data corruption occurring intermittently when using InfiniDrive. Further investigation in to this is necessary.

It was previously thought that this issue had been resolved, however, further testing has revealed additional occurrences of such corruption.

[DavidBerdik]

The following 6 Word documents were manually located and downloaded from a 50GB InfiniDrive upload that was known to be corrupted with 3 duplicates. Both versions of the 3 occurring duplicates are attached. Note that the first two duplicates appear to be merely copies of the same data, but the second file in the third pair of duplicates appears to be corrupted.

Duplicate Set 1

• 2681 - Version 1.docx
• 2681 - Version 2.docx

Duplicate Set 2

• 3638 - Version 1.docx
• 3638 - Version 2.docx

Duplicate Set 3

• 4848 - Version 1.docx
• 4848 - Version 2.docx

[DavidBerdik]

b0275cc is expected to resolve this issue. This patch is currently undergoing testing.

[DavidBerdik]

b0275cc did not resolve the issue. cdc632c is a redesign of the corruption detection and repair system that is currently in testing.

[DavidBerdik]

Testing of uploads using the cdc632c redesign suggests that the issue has been properly addressed. To ensure this, download tests are currently being performed. If the resulting files are properly restored, this issue will be marked as closed and the fix will be released as v1.0.16.

[DavidBerdik]

When performing download testing, an additional edge case was discovered. File corruption occasionally occurs in the form of a 1x1 image. The following is a sample fragment which demonstrates this.

162.docx

[DavidBerdik]

5d47d87 attempts to patch the aforementioned edge case. This patch is currently being tested.

[DavidBerdik]

The introduction of CRC32 hashing to verify fragments successfully resolved this problem. As of v1.0.16, there are no known ways for corruption to occur and remain undetected.