Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added automatically generated acceptance tests for CLI commands #2501

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .codegen.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
{
"mode": "cli_legacy",
"api_changelog": true,
"formatter": "go run golang.org/x/tools/cmd/goimports@latest -w $FILENAMES && go fmt ./...",
"formatter": "go run golang.org/x/tools/cmd/goimports@latest -w cmd && go fmt ./...",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated, but we can also remove go fmt ./...

ref: https://arc.net/l/quote/widxburh

"services": {
".codegen/service.go.tmpl": "cmd/{{if .IsAccounts}}account{{else}}workspace{{end}}/{{(.TrimPrefix \"account\").KebabName}}/{{(.TrimPrefix \"account\").KebabName}}.go"
".codegen/service.go.tmpl": "cmd/{{if .IsAccounts}}account{{else}}workspace{{end}}/{{(.TrimPrefix \"account\").KebabName}}/{{(.TrimPrefix \"account\").KebabName}}.go",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated, but seems like .KebabName no longer includes an account or a workspace prefix. Can we simply this?

".codegen/tests/cmd/workspace/script.tmpl": "acceptance/cmd/workspace/{{.KebabName}}/script",
".codegen/tests/cmd/workspace/test.toml.tmpl": "acceptance/cmd/workspace/{{.KebabName}}/test.toml"
},
"batch": {
".codegen/cmds-workspace.go.tmpl": "cmd/workspace/cmd.go",
Expand All @@ -14,6 +16,8 @@
"go"
],
"post_generate": [
"go test ./acceptance -v -update -run TestAccept/cmd/workspace || true",
"go test ./acceptance -v -run TestAccept/cmd/workspace",
"[ ! -f tagging.py ] || mv tagging.py internal/genkit/tagging.py",
"go test -timeout 240s -run TestConsistentDatabricksSdkVersion github.com/databricks/cli/internal/build",
"make schema",
Expand Down
48 changes: 48 additions & 0 deletions .codegen/tests/cmd/workspace/script.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Code generated from OpenAPI specs by Databricks SDK Generator. DO NOT EDIT.
{{- $excludes :=
list
"command-execution"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation, spaces vs tabs.

"statement-execution"
"dbfs"
"dbsql-permissions"
"account-access-control-proxy"
"files"
"dashboards"
"queries-legacy"
}}

{{- $cmdGrp := .KebabName }}
{{- if or .HasParent (or .IsAccounts (in $excludes .KebabName)) }}
{{skipThisFile}}
{{- end}}
{{- $generated := false -}}
{{- $supportedMethods := list "get" "list" "list-pipelines" }}
{{- range .Methods}}
{{- if in $supportedMethods .KebabName}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can inline skipThisFile and remove $generated by using a negative predicate:

Suggested change
{{- if in $supportedMethods .KebabName}}
{{- if not (in $supportedMethods .KebabName) }}

{{- $generated = true }}
trace errcode $CLI {{$cmdGrp}} {{.KebabName}} {{ template "required-input" . }}
{{- end}}
{{- end}}

{{- if not $generated }}
{{skipThisFile}}
{{- end}}

{{- define "required-input" -}}
{{- range .RequiredPositionalArguments }}{{ template "param-value" . }} {{ end }}
{{- end -}}

{{- define "param-value" -}}
{{- if .Entity.IsString -}}
"input-{{.SnakeName}}"
{{- end}}
{{- if .Entity.IsNumber -}}
42
{{- end -}}
{{- if .Entity.IsEnum -}}
ENUM
{{- end -}}
{{- if .Entity.IsBool -}}
false
{{- end -}}
{{- end -}}
100 changes: 100 additions & 0 deletions .codegen/tests/cmd/workspace/test.toml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Code generated from OpenAPI specs by Databricks SDK Generator. DO NOT EDIT.
{{- $excludes :=
list
"command-execution"
"statement-execution"
"dbfs"
"dbsql-permissions"
"account-access-control-proxy"
"files"
"dashboards"
"queries-legacy"
}}

{{- $seen := list "" }}
{{- $cmdGrp := .KebabName }}
{{- if or .HasParent (or .IsAccounts (in $excludes .KebabName)) }}
{{skipThisFile}}
{{- end}}
{{- $generated := false -}}
{{- $supportedMethods := list "get" "list" "list-pipelines" }}
{{- range .Methods}}
{{- if in $supportedMethods .KebabName}}
{{- $generated = true }}

[[Server]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of defining a server definition for every command, should we generate a FakeWorkspace, extending the idea in libs/testserver/fake_workspace.go to support stubs for all API calls?

The advantage of fake workspace is eventually it'll be possible to do more CRUD-like autogenerated tests as well. It's also a more general interface, which can be useful in other contexts since it's basically a local simulation of a Databricks workspace.

Writing the server in Go also allows us handle all sorts of API paths and support CREATE, DELETE etc since generating the server stubs in Go is a lot more powerful that generating them in toml.

Pattern = "{{.Verb}} {{replaceAll "2.2" "2.1" .Path}}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this replacement?

Response.Body = '''
{{- if .Response.ArrayValue }}
{{ template "response-body-array" dict "SnakeName" .Response.SnakeName "Response" .Response "Seen" $seen }}
{{- else}}
{{ template "response-body" dict "Response" .Response "Seen" $seen }}
{{- end}}
'''
{{- end}}
{{- end}}

{{- if not $generated }}
{{skipThisFile}}
{{- end}}

{{- define "response-body" -}}
{{- $seen := .Seen -}}
{
{{- $fields := .Response.Fields}}
{{- range $i, $f := $fields}}
{{- if (not (eq .SnakeName "next_page_token")) }}
"{{.SnakeName}}": {{ template "value" dict "Key" .SnakeName "Entity" .Entity "Seen" $seen "Indent" 0}}{{if notLast $i $fields}},{{end}}
{{- else}}
"{{.SnakeName}}": ""{{if notLast $i $fields}},{{end}}
{{- end}}
{{- end}}
}
{{- end -}}

{{- define "response-body-array" -}}
{{ template "value" dict "Key" .SnakeName "Entity" .Response "Seen" .Seen "Indent" 0}}
{{- end -}}

{{- define "value" -}}
{{- $seen := .Seen }}
{{- $indent := add .Indent 4 }}
{{- $key := .Key }}
{{- if .Entity.IsString -}}
"{{ $key }}"
{{- else if .Entity.IsNumber -}}
42
{{- else if .Entity.IsEnum -}}
"ENUM"
{{- else if .Entity.IsBool -}}
false
{{- else if .Entity.IsMap -}}
{
{{- $fields := .Entity.Fields }}
{{- range $i, $f := $fields}}
{{ indent $indent}}"{{.Name}}": {{template "value" dict "Key" .Name "Entity" .Entity.MapValue "Seen" $seen "Indent" $indent }}{{if notLast $i $fields}},{{end}}
{{- end }}
{{ indent $indent}}}
{{- else if .Entity.IsObject -}}
{
{{- $objName := .Entity.KebabName }}
{{- if not (in $seen $objName) }}
{{- range .Entity.Package.Types}}
{{- if eq .KebabName $objName}}
{{- $seen := append $seen .KebabName }}
{{- $fields := .Fields }}
{{- range $i, $f := $fields}}
{{ indent $indent}}"{{.Name}}": {{ template "value" dict "Key" .Name "Entity" .Entity "Seen" $seen "Indent" $indent }}{{if notLast $i $fields}},{{end}}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
{{ indent $indent}}}
{{- else if .Entity.IsArray -}}
[
{{ indent $indent}}{{ template "value" dict "Key" .Entity.SnakeName "Entity" .Entity.ArrayValue "Seen" $seen "Indent" $indent }}
{{ indent $indent}}]
{{- else -}}
null
{{- end }}
{{- end -}}
129 changes: 126 additions & 3 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,129 @@
acceptance/cmd/workspace/alerts-legacy/script linguist-generated=true
acceptance/cmd/workspace/alerts-legacy/test.toml linguist-generated=true
acceptance/cmd/workspace/alerts/script linguist-generated=true
acceptance/cmd/workspace/alerts/test.toml linguist-generated=true
acceptance/cmd/workspace/apps/script linguist-generated=true
acceptance/cmd/workspace/apps/test.toml linguist-generated=true
acceptance/cmd/workspace/artifact-allowlists/script linguist-generated=true
acceptance/cmd/workspace/artifact-allowlists/test.toml linguist-generated=true
acceptance/cmd/workspace/catalogs/script linguist-generated=true
acceptance/cmd/workspace/catalogs/test.toml linguist-generated=true
acceptance/cmd/workspace/clean-room-assets/script linguist-generated=true
acceptance/cmd/workspace/clean-room-assets/test.toml linguist-generated=true
acceptance/cmd/workspace/clean-room-task-runs/script linguist-generated=true
acceptance/cmd/workspace/clean-room-task-runs/test.toml linguist-generated=true
acceptance/cmd/workspace/clean-rooms/script linguist-generated=true
acceptance/cmd/workspace/clean-rooms/test.toml linguist-generated=true
acceptance/cmd/workspace/cluster-policies/script linguist-generated=true
acceptance/cmd/workspace/cluster-policies/test.toml linguist-generated=true
acceptance/cmd/workspace/clusters/script linguist-generated=true
acceptance/cmd/workspace/clusters/test.toml linguist-generated=true
acceptance/cmd/workspace/connections/script linguist-generated=true
acceptance/cmd/workspace/connections/test.toml linguist-generated=true
acceptance/cmd/workspace/consumer-fulfillments/script linguist-generated=true
acceptance/cmd/workspace/consumer-fulfillments/test.toml linguist-generated=true
acceptance/cmd/workspace/consumer-installations/script linguist-generated=true
acceptance/cmd/workspace/consumer-installations/test.toml linguist-generated=true
acceptance/cmd/workspace/consumer-listings/script linguist-generated=true
acceptance/cmd/workspace/consumer-listings/test.toml linguist-generated=true
acceptance/cmd/workspace/consumer-personalization-requests/script linguist-generated=true
acceptance/cmd/workspace/consumer-personalization-requests/test.toml linguist-generated=true
acceptance/cmd/workspace/consumer-providers/script linguist-generated=true
acceptance/cmd/workspace/consumer-providers/test.toml linguist-generated=true
acceptance/cmd/workspace/data-sources/script linguist-generated=true
acceptance/cmd/workspace/data-sources/test.toml linguist-generated=true
acceptance/cmd/workspace/external-locations/script linguist-generated=true
acceptance/cmd/workspace/external-locations/test.toml linguist-generated=true
acceptance/cmd/workspace/functions/script linguist-generated=true
acceptance/cmd/workspace/functions/test.toml linguist-generated=true
acceptance/cmd/workspace/git-credentials/script linguist-generated=true
acceptance/cmd/workspace/git-credentials/test.toml linguist-generated=true
acceptance/cmd/workspace/global-init-scripts/script linguist-generated=true
acceptance/cmd/workspace/global-init-scripts/test.toml linguist-generated=true
acceptance/cmd/workspace/grants/script linguist-generated=true
acceptance/cmd/workspace/grants/test.toml linguist-generated=true
acceptance/cmd/workspace/groups/script linguist-generated=true
acceptance/cmd/workspace/groups/test.toml linguist-generated=true
acceptance/cmd/workspace/instance-pools/script linguist-generated=true
acceptance/cmd/workspace/instance-pools/test.toml linguist-generated=true
acceptance/cmd/workspace/instance-profiles/script linguist-generated=true
acceptance/cmd/workspace/instance-profiles/test.toml linguist-generated=true
acceptance/cmd/workspace/ip-access-lists/script linguist-generated=true
acceptance/cmd/workspace/ip-access-lists/test.toml linguist-generated=true
acceptance/cmd/workspace/jobs/script linguist-generated=true
acceptance/cmd/workspace/jobs/test.toml linguist-generated=true
acceptance/cmd/workspace/lakeview/script linguist-generated=true
acceptance/cmd/workspace/lakeview/test.toml linguist-generated=true
acceptance/cmd/workspace/metastores/script linguist-generated=true
acceptance/cmd/workspace/metastores/test.toml linguist-generated=true
acceptance/cmd/workspace/model-versions/script linguist-generated=true
acceptance/cmd/workspace/model-versions/test.toml linguist-generated=true
acceptance/cmd/workspace/notification-destinations/script linguist-generated=true
acceptance/cmd/workspace/notification-destinations/test.toml linguist-generated=true
acceptance/cmd/workspace/online-tables/script linguist-generated=true
acceptance/cmd/workspace/online-tables/test.toml linguist-generated=true
acceptance/cmd/workspace/permissions/script linguist-generated=true
acceptance/cmd/workspace/permissions/test.toml linguist-generated=true
acceptance/cmd/workspace/pipelines/script linguist-generated=true
acceptance/cmd/workspace/pipelines/test.toml linguist-generated=true
acceptance/cmd/workspace/policy-families/script linguist-generated=true
acceptance/cmd/workspace/policy-families/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-exchange-filters/script linguist-generated=true
acceptance/cmd/workspace/provider-exchange-filters/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-exchanges/script linguist-generated=true
acceptance/cmd/workspace/provider-exchanges/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-files/script linguist-generated=true
acceptance/cmd/workspace/provider-files/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-listings/script linguist-generated=true
acceptance/cmd/workspace/provider-listings/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-personalization-requests/script linguist-generated=true
acceptance/cmd/workspace/provider-personalization-requests/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-provider-analytics-dashboards/script linguist-generated=true
acceptance/cmd/workspace/provider-provider-analytics-dashboards/test.toml linguist-generated=true
acceptance/cmd/workspace/provider-providers/script linguist-generated=true
acceptance/cmd/workspace/provider-providers/test.toml linguist-generated=true
acceptance/cmd/workspace/providers/script linguist-generated=true
acceptance/cmd/workspace/providers/test.toml linguist-generated=true
acceptance/cmd/workspace/quality-monitors/script linguist-generated=true
acceptance/cmd/workspace/quality-monitors/test.toml linguist-generated=true
acceptance/cmd/workspace/queries/script linguist-generated=true
acceptance/cmd/workspace/queries/test.toml linguist-generated=true
acceptance/cmd/workspace/query-history/script linguist-generated=true
acceptance/cmd/workspace/query-history/test.toml linguist-generated=true
acceptance/cmd/workspace/recipients/script linguist-generated=true
acceptance/cmd/workspace/recipients/test.toml linguist-generated=true
acceptance/cmd/workspace/registered-models/script linguist-generated=true
acceptance/cmd/workspace/registered-models/test.toml linguist-generated=true
acceptance/cmd/workspace/repos/script linguist-generated=true
acceptance/cmd/workspace/repos/test.toml linguist-generated=true
acceptance/cmd/workspace/schemas/script linguist-generated=true
acceptance/cmd/workspace/schemas/test.toml linguist-generated=true
acceptance/cmd/workspace/service-principals/script linguist-generated=true
acceptance/cmd/workspace/service-principals/test.toml linguist-generated=true
acceptance/cmd/workspace/serving-endpoints/script linguist-generated=true
acceptance/cmd/workspace/serving-endpoints/test.toml linguist-generated=true
acceptance/cmd/workspace/shares/script linguist-generated=true
acceptance/cmd/workspace/shares/test.toml linguist-generated=true
acceptance/cmd/workspace/storage-credentials/script linguist-generated=true
acceptance/cmd/workspace/storage-credentials/test.toml linguist-generated=true
acceptance/cmd/workspace/system-schemas/script linguist-generated=true
acceptance/cmd/workspace/system-schemas/test.toml linguist-generated=true
acceptance/cmd/workspace/tables/script linguist-generated=true
acceptance/cmd/workspace/tables/test.toml linguist-generated=true
acceptance/cmd/workspace/token-management/script linguist-generated=true
acceptance/cmd/workspace/token-management/test.toml linguist-generated=true
acceptance/cmd/workspace/tokens/script linguist-generated=true
acceptance/cmd/workspace/tokens/test.toml linguist-generated=true
acceptance/cmd/workspace/users/script linguist-generated=true
acceptance/cmd/workspace/users/test.toml linguist-generated=true
acceptance/cmd/workspace/volumes/script linguist-generated=true
acceptance/cmd/workspace/volumes/test.toml linguist-generated=true
acceptance/cmd/workspace/warehouses/script linguist-generated=true
acceptance/cmd/workspace/warehouses/test.toml linguist-generated=true
acceptance/cmd/workspace/workspace-bindings/script linguist-generated=true
acceptance/cmd/workspace/workspace-bindings/test.toml linguist-generated=true
acceptance/cmd/workspace/workspace/script linguist-generated=true
acceptance/cmd/workspace/workspace/test.toml linguist-generated=true
cmd/account/access-control/access-control.go linguist-generated=true
cmd/account/billable-usage/billable-usage.go linguist-generated=true
cmd/account/budget-policy/budget-policy.go linguist-generated=true
Expand Down Expand Up @@ -134,6 +260,3 @@ cmd/workspace/warehouses/warehouses.go linguist-generated=true
cmd/workspace/workspace-bindings/workspace-bindings.go linguist-generated=true
cmd/workspace/workspace-conf/workspace-conf.go linguist-generated=true
cmd/workspace/workspace/workspace.go linguist-generated=true
bundle/internal/tf/schema/\*.go linguist-generated=true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not go away. Seems unintended.

go.sum linguist-generated=true
bundle/schema/jsonschema.json linguist-generated=true
Loading
Loading