Skip to content

Commit

Permalink
Add mirror gateway definitions and Implementations (#2262)
Browse files Browse the repository at this point in the history
* add mirror definition and update proto dependencies

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway component (#1949)

* add mirror component

Signed-off-by: hlts2 <[email protected]>

* fix mirror dockerfile path

Signed-off-by: hlts2 <[email protected]>

* fix deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix insert rpc handler

Signed-off-by: hlts2 <[email protected]>

* deleted unused file

Signed-off-by: hlts2 <[email protected]>

* fix api resource name

Signed-off-by: hlts2 <[email protected]>

* add mirror service and fix handler logic

Signed-off-by: hlts2 <[email protected]>

* fix values and helm template for new configuration

Signed-off-by: hlts2 <[email protected]>

* Format code with gofumpt and prettier

* fix template rendering bug of mirror configmap

Signed-off-by: hlts2 <[email protected]>

* fix nil pointer error

Signed-off-by: hlts2 <[email protected]>

* bugfix to advertize different address

Signed-off-by: hlts2 <[email protected]>

* returns error when there is no other mirror gateways

Signed-off-by: hlts2 <[email protected]>

* add info log

Signed-off-by: hlts2 <[email protected]>

* extract only mirror addresses

Signed-off-by: hlts2 <[email protected]>

* fix log variable

Signed-off-by: hlts2 <[email protected]>

* add logging

Signed-off-by: hlts2 <[email protected]>

* bugfix mirror connection logic

Signed-off-by: hlts2 <[email protected]>

* add remove api proxy implementation

Signed-off-by: hlts2 <[email protected]>

* add error handling for register API

Signed-off-by: hlts2 <[email protected]>

* fix debug message

Signed-off-by: hlts2 <[email protected]>

* refactor discover logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add remove api

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* add remove api and remove api rollback logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add update rpc handler

Signed-off-by: hlts2 <[email protected]>

* refactor context variable for grpc method

Signed-off-by: hlts2 <[email protected]>

* fix trace span name

Signed-off-by: hlts2 <[email protected]>

* add getObjects method

Signed-off-by: hlts2 <[email protected]>

* refactor update and remove handler

Signed-off-by: hlts2 <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add multi handler implementation

Signed-off-by: hlts2 <[email protected]>

* fix mirror client trace span

Signed-off-by: hlts2 <[email protected]>

* rename discover service to mirror service

Signed-off-by: hlts2 <[email protected]>

* fix build error of bidirectional stream

Signed-off-by: hlts2 <[email protected]>

* add more error handling to search handler

Signed-off-by: hlts2 <[email protected]>

* add debug comment

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* fix named return bug

Signed-off-by: hlts2 <[email protected]>

* add debug log

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add force status wrap

Signed-off-by: hlts2 <[email protected]>

* execute upsert method to local cluster's lb

Signed-off-by: hlts2 <[email protected]>

* add target debug log

Signed-off-by: hlts2 <[email protected]>

* fix delete unwrap of circuitbreaker

Signed-off-by: hlts2 <[email protected]>

* refactor error handling for new status parse function

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* refactor error handling

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix error handling for broadcast operation of getobject

Signed-off-by: hlts2 <[email protected]>

* fix error join bug of getobject

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix status handling and add address to resource name

Signed-off-by: hlts2 <[email protected]>

* use vald client for update operation

Signed-off-by: hlts2 <[email protected]>

* add rollback test for mirror crud

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix variable name

Signed-off-by: hlts2 <[email protected]>

* refactor mirror servie

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix trace attribute and refactor variable name

Signed-off-by: hlts2 <[email protected]>

* add error handling when crud rpc fails

Signed-off-by: hlts2 <[email protected]>

* deleted unused value

Signed-off-by: hlts2 <[email protected]>

* make format

Signed-off-by: hlts2 <[email protected]>

* Revert "make format"

This reverts commit f605563.

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* apply suggestion

Signed-off-by: hlts2 <[email protected]>

* bugfix and refactor mirror client

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Rename ip to host of proto payload field for mirror gateway (#2031)

* fix field name and yaml indent

Signed-off-by: hlts2 <[email protected]>

* make proto/add format

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* make proto/add & make format

Signed-off-by: hlts2 <[email protected]>

* fix docker build for mirror

Signed-off-by: hlts2 <[email protected]>

* Revert "fix docker build for mirror"

This reverts commit 53ab94f.

* fix docker build error for mirror gateway

Signed-off-by: hlts2 <[email protected]>

* Add Custom Resource for mirror target (#2023)

* add mirror definition and update proto dependencies

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway component (#1949)

* add mirror component

Signed-off-by: hlts2 <[email protected]>

* fix mirror dockerfile path

Signed-off-by: hlts2 <[email protected]>

* fix deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix insert rpc handler

Signed-off-by: hlts2 <[email protected]>

* deleted unused file

Signed-off-by: hlts2 <[email protected]>

* fix api resource name

Signed-off-by: hlts2 <[email protected]>

* add mirror service and fix handler logic

Signed-off-by: hlts2 <[email protected]>

* fix values and helm template for new configuration

Signed-off-by: hlts2 <[email protected]>

* Format code with gofumpt and prettier

* fix template rendering bug of mirror configmap

Signed-off-by: hlts2 <[email protected]>

* fix nil pointer error

Signed-off-by: hlts2 <[email protected]>

* bugfix to advertize different address

Signed-off-by: hlts2 <[email protected]>

* returns error when there is no other mirror gateways

Signed-off-by: hlts2 <[email protected]>

* add info log

Signed-off-by: hlts2 <[email protected]>

* extract only mirror addresses

Signed-off-by: hlts2 <[email protected]>

* fix log variable

Signed-off-by: hlts2 <[email protected]>

* add logging

Signed-off-by: hlts2 <[email protected]>

* bugfix mirror connection logic

Signed-off-by: hlts2 <[email protected]>

* add remove api proxy implementation

Signed-off-by: hlts2 <[email protected]>

* add error handling for register API

Signed-off-by: hlts2 <[email protected]>

* fix debug message

Signed-off-by: hlts2 <[email protected]>

* refactor discover logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add remove api

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* add remove api and remove api rollback logic

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* add update rpc handler

Signed-off-by: hlts2 <[email protected]>

* refactor context variable for grpc method

Signed-off-by: hlts2 <[email protected]>

* fix trace span name

Signed-off-by: hlts2 <[email protected]>

* add getObjects method

Signed-off-by: hlts2 <[email protected]>

* refactor update and remove handler

Signed-off-by: hlts2 <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add multi handler implementation

Signed-off-by: hlts2 <[email protected]>

* fix mirror client trace span

Signed-off-by: hlts2 <[email protected]>

* rename discover service to mirror service

Signed-off-by: hlts2 <[email protected]>

* fix build error of bidirectional stream

Signed-off-by: hlts2 <[email protected]>

* add more error handling to search handler

Signed-off-by: hlts2 <[email protected]>

* add debug comment

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* add status code log

Signed-off-by: hlts2 <[email protected]>

* fix named return bug

Signed-off-by: hlts2 <[email protected]>

* add debug log

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add debug command

Signed-off-by: hlts2 <[email protected]>

* add force status wrap

Signed-off-by: hlts2 <[email protected]>

* execute upsert method to local cluster's lb

Signed-off-by: hlts2 <[email protected]>

* add target debug log

Signed-off-by: hlts2 <[email protected]>

* fix delete unwrap of circuitbreaker

Signed-off-by: hlts2 <[email protected]>

* refactor error handling for new status parse function

Signed-off-by: hlts2 <[email protected]>

* Format code with prettier and gofumpt

* refactor error handling

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix error handling for broadcast operation of getobject

Signed-off-by: hlts2 <[email protected]>

* fix error join bug of getobject

Signed-off-by: hlts2 <[email protected]>

* refactor

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix status handling and add address to resource name

Signed-off-by: hlts2 <[email protected]>

* use vald client for update operation

Signed-off-by: hlts2 <[email protected]>

* add rollback test for mirror crud

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix variable name

Signed-off-by: hlts2 <[email protected]>

* refactor mirror servie

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix trace attribute and refactor variable name

Signed-off-by: hlts2 <[email protected]>

* add error handling when crud rpc fails

Signed-off-by: hlts2 <[email protected]>

* deleted unused value

Signed-off-by: hlts2 <[email protected]>

* make format

Signed-off-by: hlts2 <[email protected]>

* Revert "make format"

This reverts commit f605563.

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* apply suggestion

Signed-off-by: hlts2 <[email protected]>

* bugfix and refactor mirror client

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* add discovery logic for other mirror target

Signed-off-by: hlts2 <[email protected]>

* add k8s template file

Signed-off-by: hlts2 <[email protected]>

* add crd for mirror target resource

Signed-off-by: hlts2 <[email protected]>

* add new values for mirror discovery logic

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* fix schema register error and fix manifest to multi vald cluster

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* bugfix: adapt the service account and add early return

Signed-off-by: hlts2 <[email protected]>

* bugfix: usage of errors join

Signed-off-by: hlts2 <[email protected]>

* bugfix: type conversion bug

Signed-off-by: hlts2 <[email protected]>

* fix metadata hash logic and add mirror group function

Signed-off-by: hlts2 <[email protected]>

* add status handling for network connection

Signed-off-by: hlts2 <[email protected]>

* refactoring

Signed-off-by: hlts2 <[email protected]>

* bugfix: status update handling

Signed-off-by: hlts2 <[email protected]>

* bugfix: update status using status writer

Signed-off-by: hlts2 <[email protected]>

* add status timestamp

Signed-off-by: hlts2 <[email protected]>

* fix yaml field format

Signed-off-by: hlts2 <[email protected]>

* improve status changes handling

Signed-off-by: hlts2 <[email protected]>

* reduced number of status updates

Signed-off-by: hlts2 <[email protected]>

* added status check handling before changing status

Signed-off-by: hlts2 <[email protected]>

* update vald mirror target definition and add new cluster role

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* deploy multiple vald clusters and mirror target resource to connect with each other

Signed-off-by: hlts2 <[email protected]>

* fix e2d test to mirror-gateway and crd view field

Signed-off-by: hlts2 <[email protected]>

* define mirror app name

Signed-off-by: hlts2 <[email protected]>

* deleted unused code

Signed-off-by: hlts2 <[email protected]>

* fix golangci warning

Signed-off-by: hlts2 <[email protected]>

* crd generate command for mirror target using mirror-target-values

Signed-off-by: hlts2 <[email protected]>

* add string field validation

Signed-off-by: hlts2 <[email protected]>

* fix build error

Signed-off-by: hlts2 <[email protected]>

* add configuration comment

Signed-off-by: hlts2 <[email protected]>

* style: Format code with gofumpt and prettier

* add image tag for debug

Signed-off-by: hlts2 <[email protected]>

* fix rebase error

Signed-off-by: hlts2 <[email protected]>

* fix wait timeout option

Signed-off-by: hlts2 <[email protected]>

* add new option to set mirror address

Signed-off-by: hlts2 <[email protected]>

* add new comment

Signed-off-by: hlts2 <[email protected]>

* bugfix for ingress configuration

Signed-off-by: hlts2 <[email protected]>

* not create lb ingress when mirror is enabled

Signed-off-by: hlts2 <[email protected]>

* not create filter ingress when mirror is enabled

Signed-off-by: hlts2 <[email protected]>

* add error handling for unimplemented codes error

Signed-off-by: hlts2 <[email protected]>

* bugfix register addr process

Signed-off-by: hlts2 <[email protected]>

* fix build error

Signed-off-by: hlts2 <[email protected]>

* Revert "update docker file for build error"

This reverts commit f40986d.

* add new package for docker build error

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway example (#2082)

* add example code

Signed-off-by: hlts2 <[email protected]>

* execute search and getobject request to all vald clusters

Signed-off-by: hlts2 <[email protected]>

* fixed to send data to each cluster based on the dataset

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* add mirror target metrics

Signed-off-by: hlts2 <[email protected]>

* fix example ngt dimension

Signed-off-by: hlts2 <[email protected]>

* make proto/all & make format

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>

* fix mirror image tag

Signed-off-by: hlts2 <[email protected]>

* Add mirror gateway document (#2105)

* add mirror document

Signed-off-by: hlts2 <[email protected]>

* fix svg to png

Signed-off-by: hlts2 <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Update docs/api/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* style: Format code with prettier and gofumpt

* Update docs/overview/component/mirror-gateway.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* style: Format code with prettier and gofumpt

* Apply suggestions from code review

* Update docs/overview/component/mirror-gateway.md

* fix expand bug for mirror tutorial

Signed-off-by: hlts2 <[email protected]>

* style: Format code with prettier and gofumpt

* fix filename typo

Signed-off-by: hlts2 <[email protected]>

* add new link

Signed-off-by: hlts2 <[email protected]>

* add cluster role settings for mirror gateway

Signed-off-by: hlts2 <[email protected]>

* add hdf5 installation section

Signed-off-by: hlts2 <[email protected]>

* Update docs/user-guides/mirroring-configuration.md

Co-authored-by: Kiichiro YUKAWA <[email protected]>

* update link

Signed-off-by: hlts2 <[email protected]>

* delete quote

Signed-off-by: hlts2 <[email protected]>

* update mirror component overview image

Signed-off-by: hlts2 <[email protected]>

* update mirror component overview image

Signed-off-by: hlts2 <[email protected]>

* Update docs/overview/component/mirror-gateway.md

Co-authored-by: Kevin Diu <[email protected]>

* Update docs/overview/component/mirror-gateway.md

* Update docs/api/mirror-gateway.md

Co-authored-by: Yusuke Kadowaki <[email protected]>

* Update docs/api/mirror-gateway.md

* Update docs/user-guides/mirroring-configuration.md

Co-authored-by: Kevin Diu <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kevin Diu <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>

* merge ingress configuration for lb and filter

Signed-off-by: hlts2 <[email protected]>

* fix duplicate definition error

Signed-off-by: hlts2 <[email protected]>

* make proto/all & make schema update & make format

Signed-off-by: hlts2 <[email protected]>

* fix test execution error for mirror handler

Signed-off-by: hlts2 <[email protected]>

* make proto/all format and update schema and fails build error

Signed-off-by: hlts2 <[email protected]>

* use resuable workflow to build mirror image

Signed-off-by: hlts2 <[email protected]>

* use internal sync package and refactor

Signed-off-by: hlts2 <[email protected]>

* fix error handling

Signed-off-by: hlts2 <[email protected]>

* add network policy settings and update deploy command

Signed-off-by: hlts2 <[email protected]>

* add default setting logic

Signed-off-by: hlts2 <[email protected]>

* add new fileld to crd schema and update crd

Signed-off-by: hlts2 <[email protected]>

* fix make command to deploy vald with mirror components

Signed-off-by: hlts2 <[email protected]>

* add e2e test for mirror

Signed-off-by: hlts2 <[email protected]>

* add debug command to get pod status

Signed-off-by: hlts2 <[email protected]>

* bugfix command

Signed-off-by: hlts2 <[email protected]>

* disable sidecar e2e and add mirror e2e command

Signed-off-by: hlts2 <[email protected]>

* update cluster role

Signed-off-by: hlts2 <[email protected]>

* add cluster-info dump command

Signed-off-by: hlts2 <[email protected]>

* Revert "add cluster-info dump command"

This reverts commit 7ece16f.

* debug: add logs command

Signed-off-by: hlts2 <[email protected]>

* Revert "debug: add logs command"

This reverts commit 454eaf3.

* add RemoveByTimestamp for mirror

Signed-off-by: hlts2 <[email protected]>

* fix: update sleep time for operator deploy

Signed-off-by: hlts2 <[email protected]>

* update wait time for vmt

Signed-off-by: hlts2 <[email protected]>

* Returns all connection information in the registration rpc (#2199)

* fix: returns all connection information in the registration rpc

Signed-off-by: hlts2 <[email protected]>

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* refactor: deleted unnecessary parameter

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Add handler logic to continue processing (#2204)

* feat: add new crud logic

Signed-off-by: hlts2 <[email protected]>

* refactor code

Signed-off-by: hlts2 <[email protected]>

* refactor: error handling logic

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix error handling for rpc

Signed-off-by: hlts2 <[email protected]>

* feat: add test for RemoveByTimestamp

Signed-off-by: hlts2 <[email protected]>

* fix: fails error test

Signed-off-by: hlts2 <[email protected]>

* refactor: test logic

Signed-off-by: hlts2 <[email protected]>

* fix: add new condition and add test case

Signed-off-by: hlts2 <[email protected]>

* refactor code

Signed-off-by: hlts2 <[email protected]>

* fix: error handling and comment

Signed-off-by: hlts2 <[email protected]>

* fix: test case name and test case order

Signed-off-by: hlts2 <[email protected]>

* fix: execute format

Signed-off-by: hlts2 <[email protected]>

* fix: add new comment

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>

* Update Mirror Gateway document (#2207)

* fix: update mirror document

Signed-off-by: hlts2 <[email protected]>

* fix: format

Signed-off-by: hlts2 <[email protected]>

* feat: add removeByTimestamp section

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning and refactor document

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* feat: add mirror gateway troubleshooting docs

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 2c492f7 according to the output
from Gofumpt and Prettier.

Details: #2207

* feat: add contents of troubleshooting document

Signed-off-by: hlts2 <[email protected]>

* fix: grammar warning

Signed-off-by: hlts2 <[email protected]>

* fix: grammar

Signed-off-by: hlts2 <[email protected]>

* fix: link path

Signed-off-by: hlts2 <[email protected]>

* fix: invalid link path

Signed-off-by: hlts2 <[email protected]>

* fix: refactor docs

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary contents

Signed-off-by: hlts2 <[email protected]>

* fix: status handling logic document

Signed-off-by: hlts2 <[email protected]>

* fix: document refactor

Signed-off-by: hlts2 <[email protected]>

* fix: bugfix status handling document

Signed-off-by: hlts2 <[email protected]>

* fix: refactor sentence

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>

* feat: schema and protobuf update

Signed-off-by: hlts2 <[email protected]>

* fix: reviewdog warning and add comment

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 7d7a690 according to the output
from Gofumpt and Prettier.

Details: #2237

* fix: reviewdog warning

Signed-off-by: hlts2 <[email protected]>

* fix: reviewdog warning

Signed-off-by: hlts2 <[email protected]>

* fix: move mirror client mock definition to internal package

Signed-off-by: hlts2 <[email protected]>

* feat: add new handler base logic

Signed-off-by: hlts2 <[email protected]>

* fix: refactor stream list object rpc

Signed-off-by: hlts2 <[email protected]>

* fix: divide logic to other function

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in e62a657 according to the output
from Gofumpt and Prettier.

Details: #2237

* make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: return grpc error

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* feat: add mirror service test

Signed-off-by: hlts2 <[email protected]>

* fix: add mock definition code

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in d6580ec according to the output
from Gofumpt and Prettier.

Details: #2237

* fix: use internal method

Signed-off-by: hlts2 <[email protected]>

* fix: delete Exist method because it is not needed

Signed-off-by: hlts2 <[email protected]>

* fix: use address cache insted of connectedAddr method

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* feat: improve connection management and add new test

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* fix: refactor error handling

Signed-off-by: hlts2 <[email protected]>

* feat: add test for syncronizee based on address and small refactor

Signed-off-by: hlts2 <[email protected]>

* fix: improve error handling and example image tag

Signed-off-by: hlts2 <[email protected]>

* fix: schema update and format

Signed-off-by: hlts2 <[email protected]>

* Update charts/vald/values/multi-vald/dev-vald-with-mirror.yaml

* fix: lint warnning

Signed-off-by: hlts2 <[email protected]>

* style: format code with Gofumpt and Prettier

This commit fixes the style issues introduced in 4c11568 according to the output
from Gofumpt and Prettier.

Details: #2262

* fix: reviewdog warning and deepsource warning

Signed-off-by: hlts2 <[email protected]>

* feat: use new command to docker build

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: update schema and format

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary RPCs due to failure of previous rebasing

Signed-off-by: hlts2 <[email protected]>

* fix: proto and execute make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: revert unnecessary changes

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: deleted unnecessary code

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: deepsource warning

Signed-off-by: hlts2 <[email protected]>

* fix: make proto/all format

Signed-off-by: hlts2 <[email protected]>

* fix: apply feedback for comment

Signed-off-by: hlts2 <[email protected]>

* feat: add internal hash package to generate hash value

Signed-off-by: hlts2 <[email protected]>

* fix: ignore deepsource warning

Signed-off-by: hlts2 <[email protected]>

* feat: add function and method comment

Signed-off-by: hlts2 <[email protected]>

* Update pkg/gateway/mirror/handler/grpc/handler.go

* fix: schema and proto update and format

Signed-off-by: hlts2 <[email protected]>

* feat: add ingress rule for lb and filter

Signed-off-by: hlts2 <[email protected]>

* fix: deleted wildcard path

Signed-off-by: hlts2 <[email protected]>

* fix: ingress unified into one manifest

Signed-off-by: hlts2 <[email protected]>

* fix: make format

Signed-off-by: hlts2 <[email protected]>

* fix: lb ingress port settings and small refactor

Signed-off-by: hlts2 <[email protected]>

* feat: add cluster wide ingress resource name

Signed-off-by: hlts2 <[email protected]>

* fix: add deploy ingress resource using make command

Signed-off-by: hlts2 <[email protected]>

---------

Signed-off-by: hlts2 <[email protected]>
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
Co-authored-by: Kiichiro YUKAWA <[email protected]>
Co-authored-by: Kevin Diu <[email protected]>
Co-authored-by: Yusuke Kadowaki <[email protected]>
  • Loading branch information
5 people committed Jan 21, 2024
1 parent 05ba05c commit ee80948
Show file tree
Hide file tree
Showing 141 changed files with 22,838 additions and 688 deletions.
1 change: 1 addition & 0 deletions .github/actions/detect-docker-image-tags/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ runs:
["vdaas/vald-agent-sidecar"]="agent.sidecar.image.tag"
["vdaas/vald-discoverer-k8s"]="discoverer.image.tag"
["vdaas/vald-lb-gateway"]="gateway.lb.image.tag"
["vdaas/vald-mirror-gateway"]="gateway.mirror.image.tag"
["vdaas/vald-manager-index"]="manager.index.image.tag"
["vdaas/vald-index-creation"]="manager.index.creator.image.tag"
["vdaas/vald-index-save"]="manager.index.saver.image.tag"
Expand Down
4 changes: 2 additions & 2 deletions .github/actions/e2e-deploy-vald-helm-operator/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ runs:
${HELM_EXTRA_OPTIONS} \
charts/vald-helm-operator/.
sleep 3
sleep 6
env:
DEFAULT_IMAGE_TAG: ${{ inputs.default_image_tag }}
HELM_EXTRA_OPTIONS: ${{ inputs.helm_extra_options }}
Expand All @@ -97,7 +97,7 @@ runs:
run: |
kubectl apply -f ${VALDRELEASE}
sleep 3
sleep 6
kubectl wait --for=condition=ready pod -l ${WAIT_FOR_SELECTOR} --timeout=${WAIT_FOR_TIMEOUT}
Expand Down
128 changes: 128 additions & 0 deletions .github/actions/e2e-deploy-vald-readreplica/action.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
name: "Deploy Vald Read Replica for E2E test"
description: "A action to deploy vald read replica for E2E test"

inputs:
require_minio:
description: "If Minio is required, set this to true."
required: false
default: "false"
helm_extra_options:
description: "Extra options that passed to Helm command."
required: false
default: ""
values:
description: "Path to the values.yaml that passed to Helm command."
required: false
default: "false"
wait_for_selector:
description: "Label selector used for specifying a pod waited for"
required: false
default: "app=vald-lb-gateway"
wait_for_timeout:
description: "Timeout used for waiting for pods"
required: false
default: "600s"
use_local_charts:
description: "If you want to use local charts, set this to true."
required: false
default: "true"
default_image_tag:
description: "Default image tag. e.g) nightly, vx.x, vx.x.x"
required: true
default: "nightly"
outputs:
POD_NAME:
description: "A pod name that waited for"
value: ${{ steps.get_real_pod_name.outputs.POD_NAME }}

runs:
using: "composite"
steps:
- name: Deploy Minio
id: deploy_minio
shell: bash
if: ${{ inputs.require_minio == 'true' }}
run: |
make K8S_SLEEP_DURATION_FOR_WAIT_COMMAND=10 k8s/external/minio/deploy
- name: Dump Helm values
shell: bash
run: |
cat ${{ inputs.values }}
- name: Deploy vald read replica from remote charts
shell: bash
id: deploy_vald_readreplica_remote
if: ${{ inputs.use_local_charts == 'false' }}
run: |
helm install \
--values ${VALUES} \
--set defaults.image.tag=${DEFAULT_IMAGE_TAG} \
${HELM_EXTRA_OPTIONS} \
--generate-name charts/vald-readreplica
sleep 3
kubectl wait --for=condition=ready pod -l ${WAIT_FOR_SELECTOR} --timeout=${WAIT_FOR_TIMEOUT}
kubectl get pods
podname=`kubectl get pods --selector=${WAIT_FOR_SELECTOR} | tail -1 | awk '{print $1}'`
echo "POD_NAME=${podname}" >> $GITHUB_OUTPUT
env:
DEFAULT_IMAGE_TAG: ${{ inputs.default_image_tag }}
VALUES: ${{ inputs.values }}
HELM_EXTRA_OPTIONS: ${{ inputs.helm_extra_options }}
WAIT_FOR_SELECTOR: ${{ inputs.wait_for_selector }}
WAIT_FOR_TIMEOUT: ${{ inputs.wait_for_timeout }}

- name: Deploy vald read replica from local charts
shell: bash
id: deploy_vald_readreplica_local
if: ${{ inputs.use_local_charts == 'true' }}
run: |
make k8s/vald-readreplica/deploy VERSION=${DEFAULT_IMAGE_TAG} HELM_VALUES=${VALUES} HELM_EXTRA_OPTIONS="${HELM_EXTRA_OPTIONS}"
sleep 3
kubectl wait --for=condition=ready pod -l ${WAIT_FOR_SELECTOR} --timeout=${WAIT_FOR_TIMEOUT}
kubectl get pods
podname=`kubectl get pods --selector=${WAIT_FOR_SELECTOR} | tail -1 | awk '{print $1}'`
echo "POD_NAME=${podname}" >> $GITHUB_OUTPUT
env:
DEFAULT_IMAGE_TAG: ${{ inputs.default_image_tag }}
VALUES: ${{ inputs.values }}
HELM_EXTRA_OPTIONS: ${{ inputs.helm_extra_options }}
WAIT_FOR_SELECTOR: ${{ inputs.wait_for_selector }}
WAIT_FOR_TIMEOUT: ${{ inputs.wait_for_timeout }}

- name: Get real pod name
shell: bash
id: get_real_pod_name
env:
PODNAME_LOCAL_DEPLOY: ${{ steps.deploy_vald_readreplica_local.outputs.POD_NAME }}
PODNAME_REMOTE_DEPLOY: ${{ steps.deploy_vald_readreplica_remote.outputs.POD_NAME }}
# Set GITHUB_OUTPUT to the not empty one, PODNAME_LOCAL_DEPLOY or PODNAME_REMOTE_DEPLOY
run: |
if [[ -n "${PODNAME_LOCAL_DEPLOY}" ]]; then
echo "POD_NAME=${PODNAME_LOCAL_DEPLOY}" >> $GITHUB_OUTPUT
else
echo "POD_NAME=${PODNAME_REMOTE_DEPLOY}" >> $GITHUB_OUTPUT
fi
11 changes: 11 additions & 0 deletions .github/actions/e2e-deploy-vald/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ inputs:
description: "Path to the values.yaml that passed to Helm command."
required: false
default: "false"
namespace:
description: "Namespace to deploy."
required: false
wait_for_selector:
description: "Label selector used for specifying a pod waited for"
required: false
Expand Down Expand Up @@ -65,6 +68,14 @@ runs:
run: |
cat ${{ inputs.values }}
- name: Change namespace
if: ${{ inputs.namespace != '' }}
shell: bash
run: |
kubectl create ns ${NAMESPACE} ; kubectl config set-context --current --namespace=${NAMESPACE}
env:
NAMESPACE: ${{ inputs.namespace }}

- name: Deploy vald from remote charts
shell: bash
id: deploy_vald_remote
Expand Down
14 changes: 14 additions & 0 deletions .github/actions/setup-e2e/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ inputs:
description: "If k3d is not required, set this to false"
required: false
default: "true"
require_minikube:
description: "If minikube is not required, set this to true and set require_k3d to false"
required: false
default: "false"
ingress_port:
description: 'If it is not "0", ingress will be exposed to the specified port'
required: false
Expand Down Expand Up @@ -94,6 +98,16 @@ runs:
agents: 3
ingress_port: ${{ inputs.ingress_port }}

# TODO: minikubeでもagentsやingress_portのようなオプションは必要なのか
# TODO: PVCはオプショナルなのでそれを追加する必要がある
- name: Setup Minikube environment
if: ${{ inputs.require_minikube == 'true' }}
# uses: medyagh/setup-minikube@master
shell: bash
run: |
make minikube/install
make minikube/start
- name: Check Kubernetes cluster
shell: bash
run: |
Expand Down
24 changes: 24 additions & 0 deletions .github/helm/values/vald-mirror-target.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
apiVersion: vald.vdaas.org/v1
kind: ValdMirrorTarget
metadata:
name: mirror-target-02
spec:
colocation: dc1
target:
host: vald-mirror-gateway.vald-02.svc.cluster.local
port: 8081
81 changes: 81 additions & 0 deletions .github/helm/values/values-mirror-01.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#
# Copyright (C) 2019-2024 vdaas.org vald team <[email protected]>
#
# Licensed under the Apache License, Version 2.0 (the "License");
# You may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

defaults:
logging:
level: info
networkPolicy:
enabled: false

gateway:
mirror:
enabled: true
clusterRoleBinding:
name: vald-mirror-01
serviceAccount:
name: vald-mirror-01
lb:
enabled: true
minReplicas: 1
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi
gateway_config:
index_replica: 3

agent:
minReplicas: 3
maxReplicas: 10
podManagementPolicy: Parallel
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi
ngt:
auto_index_duration_limit: 2m
auto_index_check_duration: 30s
auto_index_length: 1000
dimension: 784

discoverer:
minReplicas: 1
hpa:
enabled: false
resources:
requests:
cpu: 100m
memory: 50Mi
clusterRoleBinding:
name: vald-01
serviceAccount:
name: vald-01

manager:
index:
replicas: 1
resources:
requests:
cpu: 100m
memory: 30Mi
indexer:
auto_index_duration_limit: 2m
auto_index_check_duration: 30s
auto_index_length: 1000
Loading

0 comments on commit ee80948

Please sign in to comment.