Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature request: UDT Support (from struct) #60

Open
hartmut-co-uk opened this issue Jan 1, 2022 · 0 comments
Open

Feature request: UDT Support (from struct) #60

hartmut-co-uk opened this issue Jan 1, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@hartmut-co-uk
Copy link

Description

To support "non-flat" value schema, implementing struct support to write -> Scylla as UDT should be possible.

Limitations

  • scylladb.table.manage.enabled might not be possible, so it would require the scylla schema to be created and maintained manually upfront
  • (?) UDT fields must be frozen

References
@avelanarius avelanarius mentioned this issue May 9, 2022
Open
Bouncheck added a commit to Bouncheck/kafka-connect-scylladb that referenced this issue May 17, 2022
@Bouncheck
Add complex types support and tests.
c9965a8 
Adds support for all Collections (List, Set, Map), UDT and Tuples.

Adds tests that attempt inserting into columns of complex type.
Those tests check if implemented codecs kick in when needed.

Kafka Struct fields get translated into UDTs. Connector will try to infer correct
UserType from table column. If table does not exist yet or is
being altered the connector will try to find UserType with the same
name as struct's schema. In all cases the UserType needs to already
be created in targeted keyspace.

Kafka Maps get translated into Scylla maps, some can be inserted into UDTs.
Kafka Arrays can be inserted into Lists, Sets and Tuples.
Kafka Structs can be inserted into UDTs and Tuples.

Updates README with basic information about supported types on Scylla side.

All added types should work with table managed by Scylla.
UDTs, Maps, Lists, Sets should work with table managed by connector.

Fixes scylladb#60
Bouncheck added a commit to Bouncheck/kafka-connect-scylladb that referenced this issue May 30, 2022
@Bouncheck
Add complex types support and tests.
d2e1d93 
Adds support for all Collections (List, Set, Map), UDT and Tuples.

Adds tests that attempt inserting into columns of complex type.
Those tests check if implemented codecs kick in when needed.

Kafka Struct fields get translated into UDTs. Connector will try to infer correct
UserType from table column. If table does not exist yet or is
being altered the connector will try to find UserType with the same
name as struct's schema. In all cases the UserType needs to already
be created in targeted keyspace.

Kafka Maps get translated into Scylla maps, some can be inserted into UDTs.
Kafka Arrays can be inserted into Lists, Sets and Tuples.
Kafka Structs can be inserted into UDTs and Tuples.

Updates README with basic information about supported types on Scylla side.

All added types should work with table managed by Scylla.
UDTs, Maps, Lists, Sets should work with table managed by connector.

Fixes scylladb#60
@Bouncheck Bouncheck added the enhancement New feature or request label Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hartmut-co-uk @Bouncheck