Add clickhouse connector #969
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pankaj-peerdb
changed the title
serprex
reviewed
Jan 3, 2024
serprex
reviewed
Jan 3, 2024
pankaj-peerdb
changed the title
pankaj-peerdb
force-pushed
the
clickhouse-add-connector
branch
from
701e2cd to
a07b74d
Compare
|
add |
|
Tested this locally. Promising. Blockers for landing:
|
iskakaushik
reviewed
Jan 8, 2024
| } | ||
|
|
||
| func (c *ClickhouseConnector) ConsolidateQRepPartitions(config *protos.QRepConfig) error { | ||
| c.logger.Info("Consolidating partitions noop") |
There was a problem hiding this comment.
support UPSERT as well.
There was a problem hiding this comment.
Ignore as first target for CDC only needs append.
cleanup cleanup cleanup cleanup cleanup remove metadataSchema cleanup cleanup cleanup
pankaj-peerdb
force-pushed
the
clickhouse-add-connector
branch
from
127a009 to
0845521
Compare
iskakaushik
reviewed
Jan 16, 2024
flow/cmd/handler.go
Outdated
| @@ -663,6 +663,16 @@ func (h *FlowRequestHandler) CreatePeer( | |||
| } | |||
| s3Config := s3ConfigObject.S3Config | |||
| encodedConfig, encodingErr = proto.Marshal(s3Config) | |||
| case protos.DBType_CLICKHOUSE: | |||
|
|
|||
iskakaushik
reviewed
Jan 16, 2024
| _ "github.com/ClickHouse/clickhouse-go/v2" | ||
| _ "github.com/ClickHouse/clickhouse-go/v2/lib/driver" | ||
|
|
||
| //_ "github.com/ClickHouse/clickhouse-go" |
iskakaushik
reviewed
Jan 16, 2024
| logger slog.Logger | ||
| } | ||
|
|
||
| // creating this to capture array results from clicknhouse. |
There was a problem hiding this comment.
you mentioned array support not being ready, can this be added as a part of array support MR?
iskakaushik
reviewed
Jan 16, 2024
| return &ClickhouseConnector{ | ||
| ctx: ctx, | ||
| database: database, | ||
| //database: conn, |
iskakaushik
reviewed
Jan 16, 2024
iskakaushik
reviewed
Jan 16, 2024
| dsn := fmt.Sprintf("tcp://%s:%d?username=%s&password=%s", //&database=%s" | ||
| config.Host, config.Port, config.User, config.Password) //, config.Database | ||
|
|
||
| fmt.Println("connecting...", dsn) |
There was a problem hiding this comment.
remove printlns from this file, pass a logger to this, see snowflake connector as an example.
iskakaushik
reviewed
Jan 16, 2024
| } | ||
|
|
||
| // parseTableName parses a table name into schema and table name. | ||
| func parseTableName(tableName string) (*tableNameComponents, error) { |
There was a problem hiding this comment.
see: ParseSchemaTable method.
iskakaushik
reviewed
Jan 16, 2024
iskakaushik
reviewed
Jan 16, 2024
flow/connectors/clickhouse/qrep.go
Outdated
|
|
||
| func (c *ClickhouseConnector) getTableSchema(tableName string) ([]*sql.ColumnType, error) { | ||
| //nolint:gosec | ||
| queryString := fmt.Sprintf(` |
There was a problem hiding this comment.
this query can be in one line, reads better
iskakaushik
reviewed
Jan 16, 2024
flow/connectors/clickhouse/qrep.go
Outdated
| return false, fmt.Errorf("failed to execute query: %w", err) | ||
| } | ||
|
|
||
| //return count.Int64 > 0, nil |
iskakaushik
reviewed
Jan 16, 2024
flow/connectors/clickhouse/qrep.go
Outdated
| } | ||
|
|
||
| func (c *ClickhouseConnector) SetupQRepMetadataTables(config *protos.QRepConfig) error { | ||
|
|
There was a problem hiding this comment.
emplty new line - remove
iskakaushik
reviewed
Jan 16, 2024
| "github.com/PeerDB-io/peer-flow/model" | ||
| "github.com/PeerDB-io/peer-flow/model/qvalue" | ||
| "github.com/PeerDB-io/peer-flow/shared" | ||
| _ "github.com/snowflakedb/gosnowflake" |
iskakaushik
approved these changes
Jan 17, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.