Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IWF-186: Upgrade Temporal SDK #442

Merged
merged 17 commits into from
Oct 11, 2024
Merged

IWF-186: Upgrade Temporal SDK #442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a749e4c
IWF-186: Upgrade Temporal SDK
lwolczynski Oct 8, 2024
7bd5fcc
IWF-186: Fix unit tests
lwolczynski Oct 8, 2024
93cc34e
IWF-186: Fix flaky integration test
lwolczynski Oct 8, 2024
4ded334
IWF-186: Add required WaitForStage param
lwolczynski Oct 8, 2024
a94e80f
IWF-186: Bump go.temporal.io/api version
lwolczynski Oct 8, 2024
9262c2c
IWF-186: Bump Temporal server version
lwolczynski Oct 9, 2024
456682c
IWF-186: Bump sleep timer
lwolczynski Oct 9, 2024
96df06f
IWF-186: Test increasing the sleep time
lwolczynski Oct 9, 2024
596a1c3
IWF-186: Add TODO
lwolczynski Oct 9, 2024
7d030dc
IWF-186: Fix continueAsNew bug
lwolczynski Oct 9, 2024
f44cff1
IWF-186: Bump test sleep
lwolczynski Oct 10, 2024
0399451
IWF-186: Add nolint to a test
lwolczynski Oct 10, 2024
66a48a3
IWF-186: Remove TODO
lwolczynski Oct 10, 2024
0e7ca98
IWF-186: Rename method
lwolczynski Oct 10, 2024
36c8d58
IWF-186: Bump all tests timeout
lwolczynski Oct 10, 2024
c23f402
IWF-186: Refactor
lwolczynski Oct 10, 2024
88b06c2
IWF-186: Fix CI failure
lwolczynski Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,9 @@ Or maybe both just for testing to ensure the code works for both Cadence and Tem

### Option 1: Run with our docker-compose file (Recommended)

Simply run `docker compose -f docker-compose/integ-dependencies.yml up` will:
Start with shutting down already running dependencies: `docker compose -f docker-compose/integ-dependencies.yml down`

Then simply run `docker compose -f docker-compose/integ-dependencies.yml up`. It will:

* Start both Cadence & Temporal as dependencies
* Set up required system search attributes
Expand All @@ -86,20 +88,20 @@ Assuming you are using `default` namespace:
1. Make sure you have registered system search attributes required by iWF server

```shell
temporal operator search-attribute create -name IwfWorkflowType -type Keyword
temporal operator search-attribute create -name IwfGlobalWorkflowVersion -type Int
temporal operator search-attribute create -name IwfExecutingStateIds -type KeywordList
temporal operator search-attribute create --name IwfWorkflowType --type Keyword
temporal operator search-attribute create --name IwfGlobalWorkflowVersion --type Int
temporal operator search-attribute create --name IwfExecutingStateIds --type KeywordList
```

2. For `persistence_test.go` integTests, you need to register below custom search attributes.

```shell
temporal operator search-attribute create -name CustomKeywordField -type Keyword
temporal operator search-attribute create -name CustomIntField -type Int
temporal operator search-attribute create -name CustomBoolField -type Bool
temporal operator search-attribute create -name CustomDoubleField -type Double
temporal operator search-attribute create -name CustomDatetimeField -type Datetime
temporal operator search-attribute create -name CustomStringField -type Text
temporal operator search-attribute create --name CustomKeywordField --type Keyword
temporal operator search-attribute create --name CustomIntField --type Int
temporal operator search-attribute create --name CustomBoolField --type Bool
temporal operator search-attribute create --name CustomDoubleField --type Double
temporal operator search-attribute create --name CustomDatetimeField --type Datetime
temporal operator search-attribute create --name CustomStringField --type Text
```

3. If you run into any issues with Search Attributes registration, use the below command to check the existing Search
Expand Down
8 changes: 5 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ integTestsTemporalWithCover: # for local debugging
$Q go tool cover -func coverage.out -o coverage.out

integTests:
$Q go test -v ./integ
$Q go test -v ./integ -timeout 15m

temporalIntegTests:
$Q go test -v ./integ -cadence=false
Expand All @@ -183,10 +183,12 @@ ci-temporal-integ-test:
$Q go test -v ./integ -cover -coverprofile coverage.out -coverpkg ./service/... -search=false -cadence=false -dependencyWaitSeconds=60

ci-all-tests:
$Q go test -v ./... -cover -coverprofile coverage.out -coverpkg ./service/...
# Fails CI when used with -coverprofile flag due to tests that panic; see https://go.dev/doc/build-cover#panicprof
# $Q go test -v ./... -timeout 15m -cover -coverprofile coverage.out -coverpkg ./service/...
$Q go test -v ./... -timeout 15m
Comment on lines +186 to +188
Copy link
Contributor Author

@lwolczynski lwolczynski Oct 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@longquanzheng CI issue is caused by Go upgrade from 1.19 to 1.22

Beginning in Go 1.20, Go supports collection of coverage profiles from applications and from integration tests, larger and more complex tests for Go programs.

If my program panics, will coverage data be written?
Programs built with go build -cover will only write out complete profile data at the end of execution if the program invokes os.Exit() or returns normally from main.main. If a program terminates in an unrecovered panic, or if the program hits a fatal exception (such as a segmentation violation, divide by zero, etc), profile data from statements executed during the run will be lost.

When tests panic, the task returns "Error 1" which fails the Github Action. We should fix it by refactoring tests, but I removed the coverage options to unblock for now

https://go.dev/doc/build-cover#panicprof

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good finding! Thanks!!!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#446


integTestsNoSearch:
$Q go test -v ./integ -search=false
$Q go test -v ./integ -search=false -timeout 15m

stressTestsWithSearch:
$Q go test -v ./integ -repeat=10 -intervalMs=100 -searchWaitMs=100 | tee test.log # TODO https://github.com/indeedeng/iwf/issues/134
Expand Down
4 changes: 2 additions & 2 deletions docker-compose/.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@ CASSANDRA_VERSION=3.11.9
ELASTICSEARCH_VERSION=7.16.2
MYSQL_VERSION=8
POSTGRESQL_VERSION=13
TEMPORAL_VERSION=1.22.4
TEMPORAL_UI_VERSION=2.22.3
TEMPORAL_VERSION=1.25
TEMPORAL_UI_VERSION=2.31.2
2 changes: 1 addition & 1 deletion docker-compose/ci-cadence-temporal-dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ services:
- postgresql
- elasticsearch
environment:
- DB=postgresql
- DB=postgres12
- DB_PORT=5432
- POSTGRES_USER=temporal
- POSTGRES_PWD=temporal
Expand Down
2 changes: 1 addition & 1 deletion docker-compose/ci-temporal-dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ services:
- postgresql
- elasticsearch
environment:
- DB=postgresql
- DB=postgres12
- DB_PORT=5432
- POSTGRES_USER=temporal
- POSTGRES_PWD=temporal
Expand Down
2 changes: 1 addition & 1 deletion docker-compose/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ services:
- postgresql
- elasticsearch
environment:
- DB=postgresql
- DB=postgres12
- DB_PORT=5432
- POSTGRES_USER=temporal
- POSTGRES_PWD=temporal
Expand Down
18 changes: 9 additions & 9 deletions docker-compose/init-ci-temporal.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,23 @@ fi
echo "now trying to register iWF system search attributes..."
for run in {1..120}; do
sleep 1
temporal operator search-attribute create -name IwfWorkflowType -type Keyword
temporal operator search-attribute create --name IwfWorkflowType --type Keyword
sleep 0.1
temporal operator search-attribute create -name IwfGlobalWorkflowVersion -type Int
temporal operator search-attribute create --name IwfGlobalWorkflowVersion --type Int
sleep 0.1
temporal operator search-attribute create -name IwfExecutingStateIds -type KeywordList
temporal operator search-attribute create --name IwfExecutingStateIds --type KeywordList
sleep 0.1
temporal operator search-attribute create -name CustomKeywordField -type Keyword
temporal operator search-attribute create --name CustomKeywordField --type Keyword
sleep 0.1
temporal operator search-attribute create -name CustomIntField -type Int
temporal operator search-attribute create --name CustomIntField --type Int
sleep 0.1
temporal operator search-attribute create -name CustomBoolField -type Bool
temporal operator search-attribute create --name CustomBoolField --type Bool
sleep 0.1
temporal operator search-attribute create -name CustomDoubleField -type Double
temporal operator search-attribute create --name CustomDoubleField --type Double
sleep 0.1
temporal operator search-attribute create -name CustomDatetimeField -type Datetime
temporal operator search-attribute create --name CustomDatetimeField --type Datetime
sleep 0.1
temporal operator search-attribute create -name CustomStringField -type Text
temporal operator search-attribute create --name CustomStringField --type Text
sleep 0.1

if checkExists "IwfWorkflowType" ] && checkExists "IwfGlobalWorkflowVersion" && checkExists "IwfExecutingStateIds" && checkExists "CustomKeywordField" && checkExists "CustomIntField" && checkExists "CustomBoolField" && checkExists "CustomDoubleField" && checkExists "CustomDatetimeField" && checkExists "CustomStringField" ] ; then
Expand Down
6 changes: 3 additions & 3 deletions docker-compose/init-compose.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ echo "now trying to register iWF system search attributes..."

for run in {1..60}; do
sleep 1
temporal operator search-attribute create -name IwfWorkflowType -type Keyword
temporal operator search-attribute create --name IwfWorkflowType --type Keyword
sleep 0.1
temporal operator search-attribute create -name IwfGlobalWorkflowVersion -type Int
temporal operator search-attribute create --name IwfGlobalWorkflowVersion --type Int
sleep 0.1
temporal operator search-attribute create -name IwfExecutingStateIds -type KeywordList
temporal operator search-attribute create --name IwfExecutingStateIds --type KeywordList
sleep 0.1
if checkExists "IwfWorkflowType" ] && checkExists "IwfGlobalWorkflowVersion" && checkExists "IwfExecutingStateIds" ] ; then
echo "All search attributes are registered"
Expand Down
2 changes: 1 addition & 1 deletion docker-compose/integ-dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ services:
- postgresql
- elasticsearch
environment:
- DB=postgresql
- DB=postgres12
- DB_PORT=5432
- POSTGRES_USER=temporal
- POSTGRES_PWD=temporal
Expand Down
47 changes: 24 additions & 23 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
module github.com/indeedeng/iwf

go 1.19
go 1.22

require (
github.com/gin-gonic/gin v1.9.1
github.com/google/uuid v1.3.0
github.com/google/uuid v1.6.0
github.com/prometheus/client_golang v1.12.1
github.com/stretchr/testify v1.8.4
github.com/stretchr/testify v1.9.0
github.com/uber-go/tally/v4 v4.1.1
github.com/uber/cadence-idl v0.0.0-20220713235846-fda89e95df1e
github.com/urfave/cli v1.22.5
go.temporal.io/sdk v1.25.1
go.temporal.io/sdk v1.29.1
go.temporal.io/sdk/contrib/tally v0.1.0
go.temporal.io/sdk/contrib/tools/workflowcheck v0.0.0-20220331154559-fd0d1eb548eb
go.uber.org/cadence v0.17.1-0.20230105221902-f50f452a8eae // pin to pick GetUnhandledSignalNames API
Expand All @@ -21,15 +21,17 @@ require (

require (
github.com/pkg/errors v0.9.1
go.temporal.io/api v1.24.0
go.temporal.io/api v1.39.0
)

require (
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
golang.org/x/crypto v0.19.0 // indirect
golang.org/x/net v0.21.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230815205213-6bfd019c3878 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230815205213-6bfd019c3878 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/nexus-rpc/sdk-go v0.0.10 // indirect
golang.org/x/crypto v0.26.0 // indirect
golang.org/x/net v0.28.0 // indirect
golang.org/x/sync v0.8.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240827150818-7e3bb234dfed // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240827150818-7e3bb234dfed // indirect
)

require (
Expand All @@ -39,7 +41,7 @@ require (
github.com/benbjohnson/clock v1.1.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bytedance/sonic v1.9.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.1 // indirect
github.com/cristalhq/jwt/v3 v3.1.0 // indirect
Expand All @@ -55,9 +57,9 @@ require (
github.com/gogo/googleapis v1.4.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/gogo/status v1.1.1 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
github.com/golang/mock v1.6.0
github.com/golang/protobuf v1.5.4 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/jessevdk/go-flags v1.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kisielk/errcheck v1.5.0 // indirect
Expand All @@ -77,7 +79,7 @@ require (
github.com/prometheus/procfs v0.7.3 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/stretchr/objx v0.5.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/twmb/murmur3 v1.1.5 // indirect
github.com/uber-go/mapdecode v1.0.0 // indirect
Expand All @@ -91,17 +93,16 @@ require (
go.uber.org/net/metrics v1.3.0 // indirect
go.uber.org/thriftrw v1.29.2 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/exp v0.0.0-20230905200255-921286631fa9
golang.org/x/exp v0.0.0-20231127185646-65229373498e
golang.org/x/exp/typeparams v0.0.0-20220218215828-6cf2b201936e // indirect
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/sys v0.17.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/mod v0.17.0 // indirect
golang.org/x/sys v0.24.0 // indirect
golang.org/x/text v0.17.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.13.0 // indirect
google.golang.org/genproto v0.0.0-20230815205213-6bfd019c3878 // indirect
google.golang.org/grpc v1.57.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
google.golang.org/grpc v1.66.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
honnef.co/go/tools v0.3.2 // indirect
)
Loading
Loading