Skip to content

Latest commit

 

History

History
84 lines (53 loc) · 5.14 KB

CONTRIBUTING.md

File metadata and controls

84 lines (53 loc) · 5.14 KB
Raw

GCP Source Plugin Contribution Guide

Thanks for contributing to CloudQuery! You are awesome. This document serves as a guide for adding new services and resources to the GCP source plugin.

There are two main steps to adding a new GCP resource:

  1. Add a code generation recipe
  2. Writing the resolver function to fetch the resource using the AWS SDK

1. Add a Code Generation Recipe

Every supported GCP service has a recipe file under codegen/recipes.

In the following examples, we will use the fictional MyService GCP service with MyResource resource as an example. We recommend taking a look at a few examples in codegen/recipes first, as these steps will make more sense with some examples to reference.

If you are adding a service that needs a new recipe, see Add a New Recipe File. Otherwise, if the GCP service is already supported but is missing resource(s), you may skip to Add a Resource to a Recipe.

Add a New Recipe File

The process to follow for adding a new recipe is:

  1. Add a new file under codegen/recipes called myservice.go.
  2. Inside the new file, add a function called MyServiceResources() that returns []*Resource.
  3. Call the function from codegen/main.go by adding resources = append(resources, recipes.MyServiceResources()...)
  4. Define the list of resources to be generated and return it inside this function. See Add a Resource to a Recipe for more details.

Add a Resource to a Recipe

MyServiceResources() should return a slice of *Resource instances. Each resource should, at a minimum, have the following fields defined:

  1. Service: This will become the table prefix, and will usually be the same as the filename you chose for the recipe.
  2. SubService: This will be the final part of the table name, e.g. gcp_myservice_subservice
  3. Struct: This should be a pointer to the struct that will be synced to the destination. CloudQuery's plugin-sdk code generation will read the fields of this struct and convert it to a Table instance with appropriate column types.

All Available Resource Fields

All available Resource fields can be seen in base.go. See the documentation for each field for an explanation of what it does.

Common Fields

If all the resources share the same value for a field (as is often the case for Service and Multiplex), our convention is to reduce boilerplate by setting these properties in a loop after defining the resources slice, e.g.

// set default values
for _, r := range resources {
    r.Service = "myservice"
}

Run Code Generation

With the recipe file added and some resources defined, you are ready to run codegen. Inside the codegen directory, run:

go run main.go

This will update all resources and generate a new directory for your service under resources/services.

2. Setting up the resource

By following the steps outlined above, you should now have generated a myservice directory under resources/services, containing a file called myresource.go (these names are examples, your actual filenames will differ). We will now set up the resource. This involves two steps: refining the codegen recipe, and writing one or more resolver functions.

  1. Open the generated myservice/myresource.go and inspect the schema.Table that is being returned. Does it contain the appropriate columns for the resource? Does it have a primary key? If something looks off, return to the recipe for this resource (under codegen/recipes) and make adjustments. Then re-run code generation as described in Run Code Generation. Repeat this process until the Table looks right.
  2. Your generated Table will reference a Resolver function that needs to be implemented. Sometimes this resolver can also be generated, and sometimes it will need to be written by hand.
  3. Finally, implement a mock test in myresource_mock_test.go. This can also often be generated, but not always.

We recommend looking at other resources similar to yours to get an idea of what needs to be done in this step.

Implementing Resolver Functions

A few important things to note when adding functions that call the GCP API:

  • If possible, always use an API call that allows you to fetch many resources at once
  • Take pagination into account. Ensure you fetch all the resources.
  • Columns may also have their own resolver functions (not covered in this guide). This may be used for simple transformations or when additional calls can help add further context to the table.

General Tips

  • Keep transformations to a minimum. As far as possible, we aim to deliver an accurate reflection of what the API provides.
  • We generally only unroll structs one level deep. Nested structs should be transformed into JSON columns.
  • If you get stuck or need help, feel free to reach out on Discord. We are a friendly community and would love to help!