Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tuples #1440

michaelsembwever · 2024-11-26T13:01:02Z

What is the issue

DSP-24600 Unreadable SSTables upgrading from DSE 5.1.x to DSE 6.8.36 and up

All the SSTables for the table that has the tuple (with something freezable inside it) is unreadable with newer versions. While it may report these SSTables as corrupt, they are not – as they have not been altered in any way. A rollback to the previous version will make them readable again. In an online upgrade we expect this problem to be identified when the first node has been upgraded, and definitely before the cluster is finished upgraded, and rolling back an in-progress upgrade is normal (though not ideal) operation.

What does this PR fix and why was it fixed

Adds unit test to reproduce the issue.
Fix is @roxananeo's commit, formats me and earlier are marked as having implicitly frozen tuples.

Checklist before you submit for review

Make sure there is a PR in the CNDB project updating the Converged Cassandra version
Use NoSpamLogger for log lines that may appear frequently in the logs
Verify test results on Butler
Test coverage for new/modified code is > 80%
Proper code formatting
Proper title for each commit staring with the project-issue number, like CNDB-1234
Each commit has a meaningful description
Each commit is not very long and contains related changes
Renames, moves and reformatting are in distinct commits

…ples

When upgrading from 5.1 to HCD we noticed that UDTs are not correctly serialized/deserialized. This is fixable by allowing the fix() function from the SerializationHeader class to execut which in turn will happen if the `hasImplicitFrozenTuples` returns true. As this function was returning always false the fix() was never attempted.

…exist of LegacySSTableTest

test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java

…en variants

src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java

michaelsembwever · 2024-12-02T20:58:57Z

test failures do not look related: https://jenkins-stargazer.aws.dsinternal.org/job/ds-cassandra-pr-gate/view/change-requests/job/PR-1440/

( and butler does not appear to be working: http://butler-stargazer.aws.dsinternal.org/#/ci/upstream/compare/ds-cassandra-pr-gate/mck/DSP-24600/main )

…icitlyFrozenTuples in TrieIndexFormat

jacek-lewandowski · 2024-12-05T07:10:20Z

src/java/org/apache/cassandra/io/sstable/format/trieindex/TrieIndexFormat.java

@@ -348,7 +348,7 @@ static class TrieIndexVersion extends Version
            hasOriginatingHostId = version.matches("(a[d-z])|(b[b-z])") || version.compareTo("ca") >= 0;
            hasMaxColumnValueLengths = version.matches("b[a-z]"); // DSE only field
            correspondingMessagingVersion = version.compareTo("ca") >= 0 ? MessagingService.VERSION_SG_10 : MessagingService.VERSION_3014;
-            hasExplicitlyFrozenTuples = version.compareTo("cc") < 0 || version.compareTo("da") >= 0; // we don't know if what DA is going to be eventually, but it is almost certain it will not include explicitly frozen tuples
+            hasImplicitlyFrozenTuples = version.compareTo("cc") < 0 || version.compareTo("da") >= 0; // we don't know if what DA is going to be eventually, but it is almost certain it will not include implicitly frozen tuples


also the comment is probably off, hmm...

that is - "DA ... will include implicitly frozen tuples" (because the patch was not applied on OSS... yet)

will adjust

michaelsembwever · 2024-12-05T09:10:46Z

src/java/org/apache/cassandra/io/sstable/SSTableHeaderFix.java

@@ -84,7 +84,7 @@ public static void fixNonFrozenUDTIfUpgradeFrom30()
        CassandraVersion previousVersion = new CassandraVersion(previousVersionString);
        if (previousVersion.major != 3 || previousVersion.minor > 0)


todo, this also applies to 3.1 to 3.5
rather than support that, we should probably just add an assertion here that we're not upgrading from those versions

The safest thing we could do is to reject ma sstables as we would never know whether they are fixed or not. For the safety, the user needs to convert such tables manually with its knowledge of the history

Update:

based on agreement to drop support for big sstables before me (which makes sense as HCD-1.1 only needs to support upgrades from 3.0.25+, 3.11.11+ and DSE 5.1)

BigFormat.earliest_supported_version raised to me

need to investigate if scrub honours this too, and that means SSTableHeaderFix can be deleted altogether,

SSTableHeaderFix is no longer called from CassandraDaemon.

add startup checks that we're not upgrading from <3.0.25 or <3.11.11

djatnieks · 2024-12-06T18:14:44Z

Will porting this to main-5.0 require special consideration?

michaelsembwever · 2024-12-06T22:09:53Z

Will porting this to main-5.0 require special consideration?

No, main-5.0 only further raises online compatibility to big format na
And for HCD, we're not initially planning to support upgrades from C* 3.x or DSE 5.1

Furthermore SSTableHeaderFix has been removed altogether there, so we're converging.
I need to investigate if stuff in the tests warrant upstreaming… (I would say we should be using LegacySSTableTest a lot more! given how efficient it is at catching upgrade sstable issues)

michaelsembwever · 2024-12-06T22:10:44Z

There's a few test failures here, so still wip…

…nly from versions >=3.0.25, >=3.11.11, and dse-5.1 this removes the need for, and all bugs related to, SSTableHeaderFix startup with fail hard if detecting a previous version <3.0.25 or <3.11.11 remove all test data for now unsupported sstable formats

…rted_version applies to both online and offline

sonarqubecloud · 2024-12-07T08:05:07Z

Quality Gate failed

Failed conditions
41.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

michaelsembwever · 2024-12-07T13:52:36Z

There's a few test failures here, so still wip…

most of the failures (maybe all) are fixed…

roxananeo

LGTM

src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java

…java Co-authored-by: mck <[email protected]>

sonarqubecloud · 2025-01-09T02:06:27Z

Quality Gate failed

Failed conditions
41.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

cassci-bot · 2025-01-09T02:12:56Z

❌ Build ds-cassandra-pr-gate/PR-1440 rejected by Butler

6 new test failure(s) in 11 builds
See build details here

Found 6 new test failures

Test	Explanation	Branch history	Upstream history
...estMaxSSTablesToCompact[useDiskBoundaries true]	regression	🔴🔴🔴🔴🔴🔴🔴	🔵🔵🔵🔵🔵🔵🔵
...tHybridSearchHoleInClusteringColumnOrdering[dc]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...t,wide=false,scenario=MEMTABLE_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...ii,ascii>>,wide=false,scenario=COMPACTED_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...p<int,int>>>,wide=true,scenario=MEMTABLE_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵
...t,varint>,wide=false,scenario=POST_BUILD_QUERY]	regression	🔴🔵🔵🔵🔵🔵🔵	🔵🔵🔵🔵🔵🔵🔵

Found 57 known test failures

Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tu…

1bd9c6a

…ples

michaelsembwever force-pushed the mck/DSP-24600/main branch from d62af39 to 10a1e64 Compare November 26, 2024 14:37

SQUASH – test data

c3870f0

michaelsembwever force-pushed the mck/DSP-24600/main branch from 10a1e64 to c3870f0 Compare November 26, 2024 20:34

roxananeo and others added 5 commits November 28, 2024 10:29

SQUASH – add .keep files that ensure all required format directories …

66220d3

…exist of LegacySSTableTest

SQUASH – fix handling of intentionally empty test data directories

861aadc

SQUASH – add nb test sstable data

5cbbdff

SQUASH – added more test data

5d1afb8

michaelsembwever force-pushed the mck/DSP-24600/main branch from 9729425 to 5d1afb8 Compare November 28, 2024 09:30

michaelsembwever added 3 commits November 28, 2024 10:50

SQUASH – all formats up to "me" have implicitly frozen tuples

221a350

SQUASH – remove unnecessary .keep files

ad001b0

SQUASH – add test for when tuple column has been dropped

44565a9

michaelsembwever commented Nov 28, 2024

View reviewed changes

test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java Outdated Show resolved Hide resolved

michaelsembwever added 2 commits November 30, 2024 14:05

SQUASH – add non-frozen tuple, frozen and non-frozen UDTs to the schema

8423eb2

SQUASH – extend mb & me tuple test data to include UDTs, and non-froz…

923f281

…en variants

michaelsembwever commented Nov 30, 2024

View reviewed changes

src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java Outdated Show resolved Hide resolved

michaelsembwever marked this pull request as ready for review November 30, 2024 14:11

SQUASH – fix language around hasExplicitlyFrozenTuples versus hasImpl…

32bb558

…icitlyFrozenTuples in TrieIndexFormat

jacek-lewandowski self-requested a review December 5, 2024 07:05

jacek-lewandowski reviewed Dec 5, 2024

View reviewed changes

michaelsembwever commented Dec 5, 2024

View reviewed changes

michaelsembwever force-pushed the mck/DSP-24600/main branch from 406d782 to 93bd9ff Compare December 7, 2024 01:30

michaelsembwever added 2 commits December 7, 2024 10:51

SQUASH – delete SSTableHeaderFix altogether, BigFormat.earliest_suppo…

8f8b28e

…rted_version applies to both online and offline

michaelsembwever force-pushed the mck/DSP-24600/main branch from 93bd9ff to 8f8b28e Compare December 7, 2024 06:52

roxananeo approved these changes Dec 19, 2024

View reviewed changes

michaelsembwever commented Jan 8, 2025

View reviewed changes

src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java Outdated Show resolved Hide resolved

Update src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.…

9b9b18b

…java Co-authored-by: mck <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tuples #1440

Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tuples #1440

michaelsembwever commented Nov 26, 2024 •

edited

Loading

michaelsembwever commented Dec 2, 2024

jacek-lewandowski Dec 5, 2024

jacek-lewandowski Dec 5, 2024

michaelsembwever Dec 5, 2024

michaelsembwever Dec 5, 2024

jacek-lewandowski Dec 5, 2024

michaelsembwever Dec 6, 2024

djatnieks commented Dec 6, 2024

michaelsembwever commented Dec 6, 2024

michaelsembwever commented Dec 6, 2024

sonarqubecloud bot commented Dec 7, 2024

michaelsembwever commented Dec 7, 2024

roxananeo left a comment

sonarqubecloud bot commented Jan 9, 2025

cassci-bot commented Jan 9, 2025

		@@ -84,7 +84,7 @@ public static void fixNonFrozenUDTIfUpgradeFrom30()
		CassandraVersion previousVersion = new CassandraVersion(previousVersionString);
		if (previousVersion.major != 3 \|\| previousVersion.minor > 0)

Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tuples #1440

Are you sure you want to change the base?

Add LegacySSTableTest method testVerifyOldTupleSSTables for frozen tuples #1440

Conversation

michaelsembwever commented Nov 26, 2024 • edited Loading

What is the issue

What does this PR fix and why was it fixed

Checklist before you submit for review

michaelsembwever commented Dec 2, 2024

jacek-lewandowski Dec 5, 2024

Choose a reason for hiding this comment

jacek-lewandowski Dec 5, 2024

Choose a reason for hiding this comment

michaelsembwever Dec 5, 2024

Choose a reason for hiding this comment

michaelsembwever Dec 5, 2024

Choose a reason for hiding this comment

jacek-lewandowski Dec 5, 2024

Choose a reason for hiding this comment

michaelsembwever Dec 6, 2024

Choose a reason for hiding this comment

djatnieks commented Dec 6, 2024

michaelsembwever commented Dec 6, 2024

michaelsembwever commented Dec 6, 2024

sonarqubecloud bot commented Dec 7, 2024

Quality Gate failed

michaelsembwever commented Dec 7, 2024

roxananeo left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 9, 2025

Quality Gate failed

cassci-bot commented Jan 9, 2025

❌ Build ds-cassandra-pr-gate/PR-1440 rejected by Butler

Found 6 new test failures

Found 57 known test failures

michaelsembwever commented Nov 26, 2024 •

edited

Loading