Skip to content

Commit

Permalink
📝 update docs (#3181)
Browse files Browse the repository at this point in the history
* update generated code

Signed-off-by: Ivan Milchev <[email protected]>

* update dev docs

Signed-off-by: Ivan Milchev <[email protected]>

* add go dictionary

Signed-off-by: Ivan Milchev <[email protected]>

* change

Signed-off-by: Ivan Milchev <[email protected]>

* fix comments

Update docs/development.md

Co-authored-by: Letha <[email protected]>

Update docs/development.md

Co-authored-by: Letha <[email protected]>

Update docs/development.md

Co-authored-by: Letha <[email protected]>

Update docs/development.md

Co-authored-by: Letha <[email protected]>

Update docs/development.md

Co-authored-by: Letha <[email protected]>

Update docs/development.md

Co-authored-by: Letha <[email protected]>

---------

Signed-off-by: Ivan Milchev <[email protected]>
Co-authored-by: Letha <[email protected]>
  • Loading branch information
imilchev and misterpantz authored Feb 2, 2024
1 parent 42080fb commit 7d127df
Show file tree
Hide file tree
Showing 19 changed files with 131 additions and 17 deletions.
1 change: 1 addition & 0 deletions .github/workflows/spell-check.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
cspell:software-terms/src/software-tools.txt
cspell:companies/src/companies.txt
mondoo:mondoo_dictionary.txt
cspell:golang/dict/go.txt

comment:
name: Report
Expand Down
113 changes: 113 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ go 1.21
use (
./cnquery
./cnquery/providers/atlassian
./cnquery/providers/arista
./cnquery/providers/aws
./cnquery/providers/azure
Expand All @@ -200,6 +201,118 @@ use (
)
```

## Providers development best practices

The more time we spend building providers, the more learn how to do better in the future. Here we describe learnings that will help you get started with providers development.

### Referencing MQL resources
Often we have a top-level MQL resource, which we want to reference in another top-level resource.

For example, GCP networks can be retrieved for a project. That is a top-level resource:
```
// GCP Compute Engine
private gcp.project.computeService {
// Google Compute Engine VPC network in a project
networks() []gcp.project.computeService.network
}
```

However, we have a reference to a GCP network in a GCP Compute address. This allows us to quickly navigate to the network in which an address is created:
```
private gcp.project.computeService.address {
// Static IP address
address string
// Network in which to reserve the address
network() gcp.project.computeService.network
}
```

The simple way to implement the reference would be to call the GCP API every time `gcp.project.computeService.address.network` is executed. However, this would generate an excessive amount of API calls when scanning large GCP projects. If we have 10 addresses, this would mean 10 separate API calls to get the network, one for each of them.

MQL has powerful caching capabilities that let us achieve the same end result with a single (or fewer) API calls.

First, create an init function for `gcp.project.computeService.network`, which is the resource we are cross-referencing:
```go
func initGcpProjectComputeServiceNetwork(runtime *plugin.Runtime, args map[string]*llx.RawData) (map[string]*llx.RawData, plugin.Resource, error) {
// Here we check that the resource isn't fully initialized yet
if len(args) > 2 {
return args, nil, nil
}

// If no args are set, try reading them from the platform ID
if len(args) == 0 {
if ids := getAssetIdentifier(runtime); ids != nil {
args["name"] = llx.StringData(ids.name)
args["projectId"] = llx.StringData(ids.project)
} else {
return nil, nil, errors.New("no asset identifier found")
}
}

// We create a gcp.project.computeService resource which would allow us to retrieve networks via MQL
obj, err := CreateResource(runtime, "gcp.project.computeService", map[string]*llx.RawData{
"projectId": args["projectId"],
})
if err != nil {
return nil, nil, err
}

// Cast the resource to the appropriate type
computeSvc := obj.(*mqlGcpProjectComputeService)
// List the networks: equivalent to gcp.project.computeService.networks MQL query. This retrieves all networks in the project and caches them in the MQL cache. Consecutive calls to this retrieve the data from the cache and do not execute any API calls.
networks := computeSvc.GetNetworks()
if networks.Error != nil {
return nil, nil, networks.Error
}

// Filter the networks in memory by comparing them with the input arguments
for _, n := range networks.Data {
network := n.(*mqlGcpProjectComputeServiceNetwork)
name := network.GetName()
if name.Error != nil {
return nil, nil, name.Error
}
projectId := network.GetProjectId()
if projectId.Error != nil {
return nil, nil, projectId.Error
}

// return the resource if found
if name.Data == args["name"].Value && projectId.Data == args["projectId"].Value {
return args, network, nil
}
}
return nil, nil, fmt.Errorf("not found")
}
```

Then, we implement the function for retrieving the network for a GCP compute address:
```go
func (g *mqlGcpProjectComputeServiceAddress) network() (*mqlGcpProjectComputeServiceNetwork, error) {
if g.NetworkUrl.Error != nil {
return nil, g.NetworkUrl.Error
}
networkUrl := g.NetworkUrl.Data

// Format is https://www.googleapis.com/compute/v1/projects/project1/global/networks/net-1
params := strings.TrimPrefix(networkUrl, "https://www.googleapis.com/compute/v1/")
parts := strings.Split(params, "/")
resId := resourceId{Project: parts[1], Region: parts[2], Name: parts[4]}

// Use the init function for the resource to find the one that we need
res, err := CreateResource(g.MqlRuntime, "gcp.project.computeService.network", map[string]*llx.RawData{
"name": llx.StringData(resId.Name),
"projectId": llx.StringData(resId.Project),
})
if err != nil {
return nil, err
}
return res.(*mqlGcpProjectComputeServiceNetwork), nil
}

```

## Contribute changes

### Mark PRs with emojis
Expand Down
2 changes: 1 addition & 1 deletion explorer/cnquery_explorer.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion explorer/scan/cnquery_explorer_scan.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion llx/llx.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/inventory/inventory.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/plugin/plugin.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/plugin/plugin_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/resources/resources.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/upstream/health/errors.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/upstream/health/health.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/upstream/mvd/cvss/cvss.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/upstream/mvd/mvd.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/upstream/upstream.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers-sdk/v1/vault/vault.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion providers/aws/resources/aws.lr.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion sbom/sbom.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion shared/proto/cnquery.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion shared/proto/cnquery_grpc.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 7d127df

Please sign in to comment.