Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/googleps04 71 support set datatype #18

Draft
wants to merge 23 commits into
base: new_init_process
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 28 additions & 88 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,18 @@ DynamoDB Adapter currently supports the following DynamoDB data types
| `BOOL` (boolean) | `BOOL` |
| `B` (binary type) | `BYTES(MAX)` |
| `S` (string and data values) | `STRING(MAX)` |
| `SS` (string set) | `ARRAY<STRING(MAX)>`
| `NS` (number set) | `ARRAY<FLOAT64>`
| `BS` (binary set) | `ARRAY<BYTES(MAX)>`

## Configuration

### config.yaml
This file defines the necessary settings for the adapter. A sample configuration might look like this:

This file defines the necessary settings for the adapter.
A sample configuration might look like this:

spanner:
spanner:
project_id: "my-project-id"
instance_id: "my-instance-id"
database_name: "my-database-name"
Expand All @@ -87,125 +91,58 @@ special characters in column names while Cloud Spanner only supports
underscores(_). For more: [Spanner Naming Conventions](https://cloud.google.com/spanner/docs/data-definition-language#naming_conventions)

### Initialization Modes

DynamoDB Adapter supports two modes of initialization:

#### Dry Run Mode

This mode generates the Spanner queries required to:

Create the dynamodb_adapter_table_ddl table in Spanner.
Insert metadata for all DynamoDB tables into dynamodb_adapter_table_ddl.
These queries are printed to the console without executing them on Spanner, allowing you to review them before making changes.
These queries are printed to the console without executing them on Spanner,
allowing you to review them before making changes.

```sh
go run config-files/init.go --dry_run
```

#### Execution Mode
This mode executes the Spanner queries generated during the dry run on the Spanner instance. It will:

This mode executes the Spanner queries generated
during the dry run on the Spanner instance. It will:

Create the dynamodb_adapter_table_ddl table in Spanner if it does not exist.
Insert metadata for all DynamoDB tables into the dynamodb_adapter_table_ddl table.

```sh

go run config-files/init.go

```

### Prerequisites for Initialization

AWS CLI:
Configure AWS credentials:

```sh

aws configure set aws_access_key_id YOUR_ACCESS_KEY
aws configure set aws_secret_access_key YOUR_SECRET_KEY
aws configure set default.region YOUR_REGION
aws configure set aws_session_token YOUR_SESSION_TOKEN

```

Google Cloud CLI:
Authenticate and set up your environment:
```sh
gcloud auth application-default login
gcloud confi g set project [MY_PROJECT_NAME]
### config-files/{env}/config.json

`config.json` contains the basic settings for DynamoDB Adapter; GCP Project,
Cloud Spanner Database and query record limit.

| Key | Description |
| ----------------- | ----------- |
| GoogleProjectID | Your Google Project ID |
| SpannerDb | Your Spanner Database Name |
| QueryLimit | Default limit for the number of records returned in query |

For example:

```json
{
"GoogleProjectID" : "first-project",
"SpannerDb" : "test-db",
"QueryLimit" : 5000
}
```

### config-files/{env}/spanner.json

`spanner.json` is a key/value mapping file for table names with a Cloud Spanner
instance ids. This enables the adapter to query data for a particular table on
different Cloud Spanner instances.

For example:
```sh

```json
{
"dynamodb_adapter_table_ddl": "spanner-2 ",
"dynamodb_adapter_config_manager": "spanner-2",
"tableName1": "spanner-1",
"tableName2": "spanner-1"
...
...
}
```
gcloud auth application-default login
gcloud config set project [MY_PROJECT_NAME]

### config-files/{env}/tables.json

`tables.json` contains the description of the tables as they appear in
DynamoDB. This includes all table's primary key, columns and index information.
This file supports the update and query operations by providing the primary
key, sort key and any other indexes present.

| Key | Description |
| ----------------- | ----------- |
| tableName | Name of the table in DynamoDB |
| partitionKey | Primary key of the table in DynamoDB |
| sortKey | Sorting key of the table in DynamoDB |
| attributeTypes | Key/Value list of column names and type |
| indices | Collection of index objects that represent the indexes present in the DynamoDB table |

For example:

```json
{
"tableName": {
"partitionKey": "primary key or Partition key",
"sortKey": "sorting key of dynamoDB adapter",
"attributeTypes": {
"column_a": "N",
"column_b": "S",
"column_of_bytes": "B",
"my_boolean_column": "BOOL"
},
"indices": {
"indexName1": {
"sortKey": "sort key for indexName1",
"partitionKey": "partition key for indexName1"
},
"another_index": {
"sortKey": "sort key for another_index",
"partitionKey": "partition key for another_index"
}
}
},
.....
.....
}
```

## Starting DynamoDB Adapter
Expand All @@ -219,22 +156,25 @@ credentials to use the Cloud Spanner API.
In particular, ensure that you run

```sh

gcloud auth application-default login

```

to set up your local development environment with authentication credentials.

Set the GCLOUD_PROJECT environment variable to your Google Cloud project ID:

```sh

gcloud config set project [MY_PROJECT NAME]

```

```sh
go run main.go
```


```

```

Expand Down
Loading