From eac1433cc067a419982dd97a937bf2b65c2b18d2 Mon Sep 17 00:00:00 2001 From: John Smyth Date: Fri, 8 Sep 2023 16:12:32 -0500 Subject: [PATCH 1/9] add guide to client-side rate limiting --- docs/guides/limiter.md | 340 ++++++++++++++++++++++++++++++++++++++++ docs/guides/overview.md | 3 +- docs/sidebar.json | 3 +- 3 files changed, 344 insertions(+), 2 deletions(-) create mode 100644 docs/guides/limiter.md diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md new file mode 100644 index 0000000..c96b7b5 --- /dev/null +++ b/docs/guides/limiter.md @@ -0,0 +1,340 @@ +--- +title: Users Guide to Client-Side Rate Limiting with `limiter` +sidebar_label: Client-Side Rate Limiting +--- + +# Client-side rate limiting with `limiter` + +Steampipe is designed to be fast - it provides parallel execution at multiple layers: +- It runs controls in parallel +- It runs queries in parallel +- For a given query it runs [List, Get, and Hydrate functions](/docs/develop/writing-plugins#hydrate-functions) in parallel + +This high degree of concurrency results in low latency and high throughput, but may at times overwhelm the underlying service or API. Features like exponential back-off & retry and [caching](/docs/guides/caching) markedly improve the situation, but at large scale you may still run out of local or remote resources. The steampipe `limiter` was created to help solve these types of problems. Limiters provide a simple, flexible interface to implement client-site rate limiting and concurrency thresholds at compile time or run time. You can use limiters to: +- Smooth the request rate from steampipe to reduce load on the remote API or service +- Limit the number of parallel request to reduce contention for client and network resources +- Avoid hitting server limits and throttling + +## Defining limiters + +Limiters may be defined in Go code and compiled into a plugin, or they may be defined in HCL in `.spc` configuration files. In either case, the possible settings are the same. + +Each limiter must have a name. In the case of an HCL definition, the label on the `limiter` block is used as the rate limiter name. For a limiter defined in Go, you must include a `Name`. + +A limiter may specify a `max_concurrency` which sets a ceiling on the number of [List, Get, and Hydrate functions](/docs/develop/writing-plugins#hydrate-functions) that can run in parallel. + +```hcl +# run up to 250 hydrate/list/get functions concurrently +plugin "aws" { + limiter "aws_global_concurrency" { + max_concurrency = 250 + } +} +``` + +A limiter may also specify a `bucket_size` and `fill_rate` to limit the rate at which List, Get, and Hydrate functions may run. The rate limiter uses a token-bucket algorithm, where the `bucket_size` specifies the maximum number of tokens that may accrue (the burst size) and the `fill_rate` specifies how many tokens are refilled each second. + +```hcl +# run up to 1000 hydrate/list/get functions per second +plugin "aws" { + limiter "aws_global_rate_limit" { + bucket_size = 1000 + fill_rate = 1000 + } +} +``` + +Every limiter has a **scope**. The scope defines the context for the limit - which resources are subject to / counted against the limit. There are built-in scopes for `connection`, `table`, and any matrix qualifiers that the plugin may include. A plugin author may also add [hydrate function tags](#defining-tags) that can also be used as scopes. + +If no scope is specified, then the limiter applies to all functions in the plugin. If you specify a list of scopes, then *a limiter instance is created for each unique combination of scope values* - it acts much like `group by` in a sql statement. + +```hcl +# run up to 1000 hydrate/list/get functions per second in each region of each connection +plugin "aws" { + limiter "aws_regional_rate_limit" { + bucket_size = 1000 + fill_rate = 1000 + scope = ["connection", "region"] + } +} +``` + +You can use a `where` clause to further filter the scopes to specific values. + +```hcl +# run up to 1000 hydrate/list/get functions per second in us-east-1 for each connection +plugin "aws" { + limiter "aws_rate_limit_us_east_1" { + bucket_size = 1000 + fill_rate = 1000 + scope = ["connection", "region"] + where = "region = 'us-east-1'" + } +} +``` + + +You can define multiple limiters. If a function is included in the scope of multiple rate limiters, they will all apply - the function will wait until every rate limiter that applies to it has available bucket tokens and is below its max concurrency. + +## Defining Tags + +Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation. + +Tags can be added to a ListConfig, GetConfig, or HydrateConfig. + +```go +//// TABLE DEFINITION +func tableAwsSnsTopic(_ context.Context) *plugin.Table { + return &plugin.Table{ + Name: "aws_sns_topic", + Description: "AWS SNS Topic", + Get: &plugin.GetConfig{ + KeyColumns: plugin.SingleColumn("topic_arn"), + IgnoreConfig: &plugin.IgnoreConfig{ + ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"NotFound", "InvalidParameter"}), + }, + Hydrate: getTopicAttributes, + Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, + }, + List: &plugin.ListConfig{ + Hydrate: listAwsSnsTopics, + Tags: map[string]string{"service": "sns", "action": "ListTopics"}, + }, + + HydrateConfig: []plugin.HydrateConfig{ + { + Func: listTagsForSnsTopic, + Tags: map[string]string{"service": "sns", "action": "ListTagsForResource"}, + }, + { + Func: getTopicAttributes, + Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, + }, + }, + ... +``` + +Once the tags are added to the plugin, you can use them in the `scope` and `where` arguments for your rate limiter. + +```hcl +plugin "aws" { + limiter "sns_get_topic_attributes_us_east_1" { + bucket_size = 3000 + fill_rate = 3000 + + scope = ["connection", "region", "service", "action"] + where = "action = 'GetTopicAttributes' and service = 'sns' and region = 'us-east-1' " + } +} +``` + +## Accounting for Paged List calls +The Steampipe plugin SDK transparently handles the most of the details around waiting for limiters. List calls, however, usually iterate through pages of results, and each call to fetch a page must wait for any limiters that are defined. The SDK provides a hook, `WaitForListRateLimit`, which should be called before paging to apply rate limiting to the list call: + +```go +// List call +for paginator.HasMorePages() { + + // apply rate limiting + d.WaitForListRateLimit(ctx) + + output, err := paginator.NextPage(ctx) + if err != nil { + plugin.Logger(ctx).Error("List error", "api_error", err) + return nil, err + } + for _, items := range output.Items { + d.StreamListItem(ctx, items) + + // Context can be cancelled due to manual cancellation or the limit has been hit + if d.RowsRemaining(ctx) == 0 { + return nil, nil + } + } +} +``` + +## Viewing and Overriding Limiters +Steampipe includes the `steampipe_rate_limiter` table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL. + +```sql +> select name,plugin,source,status,bucket_size,fill_rate,max_concurrency from steampipe_rate_limiter ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | exec | plugin | active | | | 15 | +| aws_global_concurrency | aws | config | active | | | 200 | +| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | +| sns_read_900 | aws | config | active | 810 | 810 | | +| sns_read_150 | aws | config | active | 135 | 135 | | +| sns_read_30 | aws | config | active | 27 | 27 | | ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +``` + +You can override a limiter that is compiled into a plugin by creating an HCL limiter with the same name. In the previous example, we can see that the `exec` plugin includes a default limiter named `exec_max_concurrency_limiter` that sets the max_concurrency to 15. We can override this value at run time by creating an HCL `limiter` for this plugin with the same name. The `limiter` block must be contained in a `plugin` block. Like `connection`, Steampipe will load all `plugin` blocks that it finds in any `.spc` file in the `~/.steampipe/config` directory. For example, we can add the following snippet to the `~/.steampipe/config/exec.spc` file: + +```hcl +plugin "exec" { + limiter "exec_max_concurrency_limiter" { + max_concurrency = 20 + } +} +``` + +Querying the `steampipe_rate_limiter` table again, we can see that there are now 2 rate limiters for the `exec` plugin named `exec_max_concurrency_limiter`, but the one from the plugin is overridden by the one in the config file. + +```sql ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | exec | plugin | overridden | | | 15 | +| exec_max_concurrency_limiter | exec | config | active | | | 20 | +| aws_global_concurrency | aws | config | active | | | 200 | +| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | +| sns_read_900 | aws | config | active | 810 | 810 | | +| sns_read_150 | aws | config | active | 135 | 135 | | +| sns_read_30 | aws | config | active | 27 | 27 | | ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +``` + + +## Exploring & Troubleshooting with Diagnostics Mode + +To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostics Mode. To enable Diagnostics Mode, set the `STEAMPIPE_DIAGNOSTICS_LEVEL` environment variable to `ALL` when you start the Steampipe DB: +```bash +STEAMPIPE_DIAGNOSTICS_LEVEL=ALL steampipe service start +``` + +With diagnostics enabled, the `_ctx` column will contain information about what functions were called to fetch the row, the scope values (including any [tags](#defining-tags)) for the function, the limiters that were in effect and the amount of time the request was delayed by the `limiters`. This diagnostic information can help you discover what scopes are available to use in limiters as well as to see the effect and impact of limiters that you have defined. + +```sql +> select jsonb_pretty(_ctx),topic_arn,display_name from aws_sns_topic + + ++---------------------------------------------------------+---------------------- +| jsonb_pretty | topic_arn ++---------------------------------------------------------+---------------------- +| { | arn:aws:sns:us-east-2 +| "connection": "aws_dev_01" , | +| "diagnostics": { | +| "calls": [ | +| { | +| "type": "list", | +| "scope_values": { | +| "table": "aws_sns_topic", | +| "action": "ListTopics", | +| "region": "us-east-2", | +| "service": "sns", | +| "connection": "aws_dev_01" | +| }, | +| "function_name": "listAwsSnsTopics", | +| "rate_limiters": [ | +| "aws_global_concurrency", | +| "sns_read_150" | +| ] | +| }, | +| { | +| "type": "hydrate", | +| "scope_values": { | +| "table": "aws_sns_topic", | +| "action": "GetTopicAttributes", | +| "region": "us-east-2", | +| "service": "sns", | +| "connection": "aws_dev_01" | +| }, | +| "function_name": "getTopicAttributes", | +| "rate_limiters": [ | +| "aws_global_concurrency", | +| "sns_read_150" | +| ] | +| } | +| ] | +| } | +| } | +| { | arn:aws:sns:us-east-1 +| "connection": "aws_dev_01" , | +| "diagnostics": { | +| "calls": [ | +| { | +| "type": "list", | +| "scope_values": { | +| "table": "aws_sns_topic", | +| "action": "ListTopics", | +| "region": "us-east-1", | +| "service": "sns", | +| "connection": "aws_dev_01" | +| }, | +| "function_name": "listAwsSnsTopics", | +| "rate_limiters": [ | +| "sns_read_us_east_1", | +| "aws_global_concurrency" | +| ] | +| }, | +| { | +| "type": "hydrate", | +| "scope_values": { | +| "table": "aws_sns_topic", | +| "action": "GetTopicAttributes", | +| "region": "us-east-1", | +| "service": "sns", | +| "connection": "aws_dev_01" | +| }, | +| "function_name": "getTopicAttributes", | +| "rate_limiters": [ | +| "aws_global_concurrency", | +| "sns_read_us_east_1" | +| ], | +| "rate_limiter_delay_ms": 169 | +| } | +| ] | +| } | +| } | +``` + +Note that fields with null values will not appear in the diagnostics - If there are no rate limiters that are in scope for a function then `rate_limiters` will not appear for the function, if there is no delay due to a limiter then `rate_limiter_delay_ms` will not appear, etc. + + +## Hints, Tips, & Best practices + +- You can use ANY scope in the `where`, even if it does not appear in the `scope` for the limiter. Remember that the `scope` defines the grouping; it acts similar to `group by` in SQL. Consider the following rate limiter: + + ```hcl + plugin "aws" { + limiter "aws_sns_read_rate_limit" { + bucket_size = 2500 + fill_rate = 2500 + scope = ["connection", "region", "service", "action"] + where = "service = 'sns' and (action like 'Get%' or action like 'List%') " + } + } + ``` + + This will create a separate rate limiter instance for every action in the `sns` service in every region of every account - You can do 2500 `GetTopicAttributes` requests/sec in each account/region, and also 2500 `ListTagsForResource` requests/sec in each account/region, and also 2500 `ListTopics` requests/sec in each account/region. + + If we remove `action` from the `scope`, there will be *one* rate limiter instance for *all* actions in the `sns` service in each region/account - You can do 2500 total `GetTopicAttributes` or `ListTagsForResource` or `ListTopics` requests per second in each account/region. + + ```hcl + plugin "aws" { + limiter "aws_sns_read_rate_limit" { + bucket_size = 2500 + fill_rate = 2500 + scope = ["connection", "region", "service"] + where = "service = 'sns' and (action like 'Get%' or action like 'List%') " + } + } + ``` + +- Setting `max_concurrency` at the plugin level can help prevent running out of local resources like network bandwidth, ports, file handles, etc. + ```hcl + plugin "aws" { + max_concurrency = 250 + } + ``` + +- Optimizing rate limiters requires knowledge of how the API is implemented. If the API publishes information about what the rate limits are, and how they are applied it provides a good starting place for setting your `bucket_size` and `fill_rate` values. Getting the `limiter` values right usually involves some trial & error though, and simply setting `max_concurrency` is often good enough to get past a problem. + +- Use the plugin logs (`~/.steampipe/logs/plugin*.log`) to verify that the rate limiters are reducing the throttling and other errors from the API as you would expect. + +- Use the `steampipe_rate_limiter` table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use `STEAMPIPE_DIAGNOSTICS_LEVEL=ALL` to enable extra diagnostic info in the `_ctx` to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the `STEAMPIPE_DIAGNOSTICS_LEVEL` variable must be set in the database service process - if you run steampipe as a service, it must be set when you run `steampipe service start` + +- Throttling errors from the server, such as `429 Too Many Requests` are not *inherently* bad. Most cloud SDKs actually account for retrying such errors and expect that it will sometimes occur. Steampipe plugins generally implement an exponential back-off & retry to account for such cases. You can use client side limiters to help avoid resource contention and to reduce throttling from the server, but completely avoiding server-side throttling is probably not necessary in most cases. diff --git a/docs/guides/overview.md b/docs/guides/overview.md index b74a579..d9bc1da 100644 --- a/docs/guides/overview.md +++ b/docs/guides/overview.md @@ -9,4 +9,5 @@ Guides provide expanded explanations for common cases. - **[Use the search_path to target specific connections (or aggregators) →](guides/search-path)** - **[Use the Steampipe CLI with AWS Organizations →](guides/aws-orgs)** -- **[Understand and Configure the Query Cache →](guides/caching)** \ No newline at end of file +- **[Understand and Configure the Query Cache →](guides/caching)** +- **[Implement client-side rate limiting with `limiter` →](guides/limiter)** \ No newline at end of file diff --git a/docs/sidebar.json b/docs/sidebar.json index 942602e..17f308d 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -81,7 +81,8 @@ "items": [ "guides/search-path", "guides/aws-orgs", - "guides/caching" + "guides/caching", + "guides/limiter" ] }, From a5c8bc0df7f96c1fdf589a82d0c7ca15e369aca5 Mon Sep 17 00:00:00 2001 From: John Smyth Date: Wed, 13 Sep 2023 07:12:41 -0500 Subject: [PATCH 2/9] add table of limiter args to client-side rate limiting guide --- docs/guides/limiter.md | 173 ++++++++++++++++++++++------------------- 1 file changed, 91 insertions(+), 82 deletions(-) diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md index c96b7b5..dd188a8 100644 --- a/docs/guides/limiter.md +++ b/docs/guides/limiter.md @@ -208,90 +208,99 @@ STEAMPIPE_DIAGNOSTICS_LEVEL=ALL steampipe service start With diagnostics enabled, the `_ctx` column will contain information about what functions were called to fetch the row, the scope values (including any [tags](#defining-tags)) for the function, the limiters that were in effect and the amount of time the request was delayed by the `limiters`. This diagnostic information can help you discover what scopes are available to use in limiters as well as to see the effect and impact of limiters that you have defined. ```sql -> select jsonb_pretty(_ctx),topic_arn,display_name from aws_sns_topic - - -+---------------------------------------------------------+---------------------- -| jsonb_pretty | topic_arn -+---------------------------------------------------------+---------------------- -| { | arn:aws:sns:us-east-2 -| "connection": "aws_dev_01" , | -| "diagnostics": { | -| "calls": [ | -| { | -| "type": "list", | -| "scope_values": { | -| "table": "aws_sns_topic", | -| "action": "ListTopics", | -| "region": "us-east-2", | -| "service": "sns", | -| "connection": "aws_dev_01" | -| }, | -| "function_name": "listAwsSnsTopics", | -| "rate_limiters": [ | -| "aws_global_concurrency", | -| "sns_read_150" | -| ] | -| }, | -| { | -| "type": "hydrate", | -| "scope_values": { | -| "table": "aws_sns_topic", | -| "action": "GetTopicAttributes", | -| "region": "us-east-2", | -| "service": "sns", | -| "connection": "aws_dev_01" | -| }, | -| "function_name": "getTopicAttributes", | -| "rate_limiters": [ | -| "aws_global_concurrency", | -| "sns_read_150" | -| ] | -| } | -| ] | -| } | -| } | -| { | arn:aws:sns:us-east-1 -| "connection": "aws_dev_01" , | -| "diagnostics": { | -| "calls": [ | -| { | -| "type": "list", | -| "scope_values": { | -| "table": "aws_sns_topic", | -| "action": "ListTopics", | -| "region": "us-east-1", | -| "service": "sns", | -| "connection": "aws_dev_01" | -| }, | -| "function_name": "listAwsSnsTopics", | -| "rate_limiters": [ | -| "sns_read_us_east_1", | -| "aws_global_concurrency" | -| ] | -| }, | -| { | -| "type": "hydrate", | -| "scope_values": { | -| "table": "aws_sns_topic", | -| "action": "GetTopicAttributes", | -| "region": "us-east-1", | -| "service": "sns", | -| "connection": "aws_dev_01" | -| }, | -| "function_name": "getTopicAttributes", | -| "rate_limiters": [ | -| "aws_global_concurrency", | -| "sns_read_us_east_1" | -| ], | -| "rate_limiter_delay_ms": 169 | -| } | -| ] | -| } | -| } | +> select jsonb_pretty(_ctx),display_name from aws_sns_topic + + ++-----------------------------------------------------------+--------------+ +| jsonb_pretty | display_name | ++-----------------------------------------------------------+--------------+ +| { | | +| "connection": "aws_dev_01", | | +| "diagnostics": { | | +| "calls": [ | | +| { | | +| "type": "list", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "ListTopics", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "listAwsSnsTopics", | | +| "rate_limiters": [ | | +| "sns_list_topics", | | +| "aws_global_concurrency" | | +| ] | | +| }, | | +| { | | +| "type": "hydrate", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "GetTopicAttributes", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "getTopicAttributes", | | +| "rate_limiters": [ | | +| "sns_get_topic_attributes_us_east_1", | | +| "aws_global_concurrency" | | +| ], | | +| "rate_limiter_delay_ms": 107 | | +| } | | +| ] | | +| } | | +| } | | +| { | | +| "connection": "aws_dev_01", | | +| "diagnostics": { | | +| "calls": [ | | +| { | | +| "type": "list", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "ListTopics", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "listAwsSnsTopics", | | +| "rate_limiters": [ | | +| "sns_list_topics", | | +| "aws_global_concurrency" | | +| ] | | +| }, | | +| { | | +| "type": "hydrate", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "GetTopicAttributes", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "getTopicAttributes", | | +| "rate_limiters": [ | | +| "sns_get_topic_attributes_us_east_1", | | +| "aws_global_concurrency" | | +| ], | | +| "rate_limiter_delay_ms": 119 | | +| } | | +| ] | | +| } | | +| } | ``` -Note that fields with null values will not appear in the diagnostics - If there are no rate limiters that are in scope for a function then `rate_limiters` will not appear for the function, if there is no delay due to a limiter then `rate_limiter_delay_ms` will not appear, etc. +The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including: + +| Key | Description +|-------------------------|---------------------- +| `type` | The type of function (`list`, `get`, or `hydrate`). +| `function_name` | The name of the function. +| `scope_values` | A map of scope names to values. This includes the built-in scopes as well as any matrix qualifier scopes and function tags. +| `rate_limiters` | A list of the rate limiters that are scoped to the function. +| `rate_limiter_delay_ms` | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (`limiter`) rate limiting. ## Hints, Tips, & Best practices From 06db0d205fa47ee7eba840eefa32ae86e407acfd Mon Sep 17 00:00:00 2001 From: John Smyth Date: Fri, 22 Sep 2023 16:32:03 -0500 Subject: [PATCH 3/9] add docs for plugin config option, options plugin, memory_max env vars and options, diagnostics, and limiters --- docs/guides/limiter.md | 6 +- docs/reference/config-files/connection.md | 32 +++- docs/reference/config-files/options.md | 21 +++ docs/reference/config-files/overview.md | 2 +- docs/reference/config-files/plugin.md | 170 ++++++++++++++++++ docs/reference/env-vars/overview.md | 3 + .../env-vars/steampipe_diagnostic_level.md | 74 ++++++++ .../env-vars/steampipe_memory_max_mb.md | 21 +++ .../steampipe_plugin_memory_max_mb.md | 27 +++ docs/sidebar.json | 4 + 10 files changed, 354 insertions(+), 6 deletions(-) create mode 100644 docs/reference/config-files/plugin.md create mode 100644 docs/reference/env-vars/steampipe_diagnostic_level.md create mode 100644 docs/reference/env-vars/steampipe_memory_max_mb.md create mode 100644 docs/reference/env-vars/steampipe_plugin_memory_max_mb.md diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md index dd188a8..bbfb0f0 100644 --- a/docs/guides/limiter.md +++ b/docs/guides/limiter.md @@ -200,9 +200,9 @@ Querying the `steampipe_rate_limiter` table again, we can see that there are now ## Exploring & Troubleshooting with Diagnostics Mode -To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostics Mode. To enable Diagnostics Mode, set the `STEAMPIPE_DIAGNOSTICS_LEVEL` environment variable to `ALL` when you start the Steampipe DB: +To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostics Mode. To enable Diagnostics Mode, set the `STEAMPIPE_DIAGNOSTIC_LEVEL` environment variable to `ALL` when you start the Steampipe DB: ```bash -STEAMPIPE_DIAGNOSTICS_LEVEL=ALL steampipe service start +STEAMPIPE_DIAGNOSTIC_LEVEL=ALL steampipe service start ``` With diagnostics enabled, the `_ctx` column will contain information about what functions were called to fetch the row, the scope values (including any [tags](#defining-tags)) for the function, the limiters that were in effect and the amount of time the request was delayed by the `limiters`. This diagnostic information can help you discover what scopes are available to use in limiters as well as to see the effect and impact of limiters that you have defined. @@ -344,6 +344,6 @@ The diagnostics information includes information about each Get, List, and Hydra - Use the plugin logs (`~/.steampipe/logs/plugin*.log`) to verify that the rate limiters are reducing the throttling and other errors from the API as you would expect. -- Use the `steampipe_rate_limiter` table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use `STEAMPIPE_DIAGNOSTICS_LEVEL=ALL` to enable extra diagnostic info in the `_ctx` to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the `STEAMPIPE_DIAGNOSTICS_LEVEL` variable must be set in the database service process - if you run steampipe as a service, it must be set when you run `steampipe service start` +- Use the `steampipe_rate_limiter` table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use `STEAMPIPE_DIAGNOSTIC_LEVEL=ALL` to enable extra diagnostic info in the `_ctx` to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the `STEAMPIPE_DIAGNOSTIC_LEVEL` variable must be set in the database service process - if you run steampipe as a service, it must be set when you run `steampipe service start` - Throttling errors from the server, such as `429 Too Many Requests` are not *inherently* bad. Most cloud SDKs actually account for retrying such errors and expect that it will sometimes occur. Steampipe plugins generally implement an exponential back-off & retry to account for such cases. You can use client side limiters to help avoid resource contention and to reduce throttling from the server, but completely avoiding server-side throttling is probably not necessary in most cases. diff --git a/docs/reference/config-files/connection.md b/docs/reference/config-files/connection.md index e15ad9e..2a37a8f 100644 --- a/docs/reference/config-files/connection.md +++ b/docs/reference/config-files/connection.md @@ -13,14 +13,13 @@ Most `connection` arguments are plugin-specific, and they are used to specify cr | Argument | Default | Values | Description |-|-|-|- | `import_schema` | `enabled` | `enabled`, `disabled` | Enable or disable the creation of a Postgres schema for this connection. When `import_schema` is disabled, Steampipe will not create a schema for the connection (and will delete it if it exists), but the connection will still be queryable from any aggregator that includes it. For installations with a large number of connections, setting `import_schema` to `disabled` can decrease startup time and increase performance. -| `plugin` | none | [plugin version string](#plugin-version-strings) | The plugin version that this connection uses. This must refer to an [installed plugin version](/docs/managing/plugins#installing-plugins). +| `plugin` | none | [plugin version string](#plugin-version-strings) or [plugin reference](/docs/reference/config-files/plugin) | The plugin version / instance that this connection uses. This must refer to an [installed plugin version](/docs/managing/plugins#installing-plugins). | `type` | `plugin` | `plugin`, `aggregator` | The type of connection - [plugin connection](/docs/managing/plugins#installing-plugins) or [aggregator](/docs/managing/connections#using-aggregators). | `{plugin argument}`| varies | varies| Additional options are defined in each plugin - refer to the documentation for your plugin on the [Steampipe Hub](https://hub.steampipe.io/plugins). ### Plugin Version Strings - Steampipe plugin versions are in the format: ``` [{organization}/]{plugin name}[@release stream] @@ -65,6 +64,9 @@ connection "myplugin" { ``` ## Examples + +Connections using [plugin version strings](#plugin-version-strings): + ```hcl connection "aws_all" { type = "aggregator" @@ -92,4 +94,30 @@ connection "aws_03" { regions = ["us-east-1", "us-west-2"] } +``` + +Connections using [plugin reference](/docs/reference/config-files/plugin): +```hcl +plugin "aws" { + memory_max_mb = 2048 +} + +connection "aws_all" { + type = "aggregator" + plugin = plugin.aws + connections = ["aws_*"] +} + +connection "aws_01" { + plugin = plugin.aws + profile = "aws_01" + regions = ["*"] +} + +connection "aws_02" { + plugin = plugin.aws + import_schema = "disabled" + profile = "aws_02" + regions = ["us-*", "eu-*"] +} ``` \ No newline at end of file diff --git a/docs/reference/config-files/options.md b/docs/reference/config-files/options.md index ce47537..605066f 100644 --- a/docs/reference/config-files/options.md +++ b/docs/reference/config-files/options.md @@ -21,6 +21,7 @@ The following `options` are currently supported: | [database](#database-options) | Database options. | [dashboard](#dashboard-options) | Dashboard options. | [general](#general-options) | General CLI options, such as auto-update options. +| [plugin](#plugin-options) | Plugin options. @@ -97,6 +98,7 @@ options "dashboard" { | Argument | Default | Values | Description |-|-|-|- | `log_level` | `warn` | `trace`, `debug`, `info`, `warn`, `error` | Sets the output logging level. Standard log levels are supported. This can also be set via the [STEAMPIPE_LOG_LEVEL](reference/env-vars/steampipe_log) environment variable. +| `memory_max_mb` | `1024` | Set a memory soft limit for the `steampipe` process. Set to `0` to disable the memory limit. This can also be set via the [STEAMPIPE_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_memory_max_mb) environment variable. | `telemetry` | `none` | `none`, `info` | Set the telemetry level in Steampipe. This can also be set via the [STEAMPIPE_TELEMETRY](reference/env-vars/steampipe_telemetry) environment variable. See also: [Telemetry](https://steampipe.io/blog/release-0-15-0#telemetry). | `update_check` | `true` | `true`, `false` | Enable or disable automatic update checking. This can also be set via the [STEAMPIPE_UPDATE_CHECK](reference/env-vars/steampipe_update_check) environment variable. @@ -112,12 +114,31 @@ options "dashboard" { ```hcl options "general" { log_level = "warn" # trace, debug, info, warn, error + memory_max_mb = 512 # megabytes telemetry = "info" # info, none update_check = true # true, false } ``` --- +## Plugin Options + +**Plugin** options are used to set plugin default options, such as memory soft limits. + +### Supported options +| Argument | Default | Values | Description +|-|-|-|- +| `memory_max_mb` | `1024` | Set a default memory soft limit for each plugin process. Note that each plugin can have its own `memory_max_mb` set in [a `plugin` definition](/docs/reference/config-files/plugin), and that value would override this default setting. Set to `0` to disable the memory limit. This can also be set via the [STEAMPIPE_PLUGIN_MEMORY_MAX_MB](/docs/reference/env-vars/steampipe_plugin_memory_max_mb) environment variable. + + +### Example: Plugin Options + +```hcl +options "plugin" { + memory_max_mb = 2048 # megabytes +} +``` + diff --git a/docs/reference/config-files/overview.md b/docs/reference/config-files/overview.md index 50f5577..b193aec 100644 --- a/docs/reference/config-files/overview.md +++ b/docs/reference/config-files/overview.md @@ -10,5 +10,5 @@ Configuration file resource are defined using HCL in one or more Steampipe confi Typically, config files are laid out as follows: - Steampipe creates a `~/.steampipe/config/default.spc` file for setting [options](reference/config-files/options). -- Each plugin creates a `~/.steampipe/config/{plugin name}.spc` (e.g. `aws.spc`, `github.spc`, `net.spc`, etc). Define your [connections](reference/config-files/connection) in these files. +- Each plugin creates a `~/.steampipe/config/{plugin name}.spc` (e.g. `aws.spc`, `github.spc`, `net.spc`, etc). Define your [connections](reference/config-files/connection) and [plugins](reference/config-files/plugin) in these files. - Define your [workspaces](reference/config-files/workspace) in `~/.steampipe/config/workspaces.spc`. \ No newline at end of file diff --git a/docs/reference/config-files/plugin.md b/docs/reference/config-files/plugin.md new file mode 100644 index 0000000..0ea5061 --- /dev/null +++ b/docs/reference/config-files/plugin.md @@ -0,0 +1,170 @@ +--- +title: plugin +sidebar_label: plugin +--- + +# plugin + +The `plugin` block allows you to set plugin-level options like soft memory limits and rate limiters. You can then associate connections with the the plugin. + +```hcl +plugin "aws" { + memory_max_mb = 2048 + + limiter "aws_global" { + max_concurrency = 200 + } +} +``` + + +## Supported options +| Argument | Default | Description +|-|-|-|- +| `source` | none | A [plugin version string](#plugin-version-strings) the specifies which plugin this configuration applies to. If not specified, the plugin block label is assumed to be the plugin source. +| `memory_max_mb` | `1024` | The soft memory limit for the plugin, in MB. +| `limiter` | none | Optional [limiter](#limiter) blocks used to set concurrency and/or rate limits + + + +## Plugins and Connections + +You may optionally define a `plugin` in a `.spc` file to set plugin-level options like soft memory limits and rate limiters. + +```hcl +plugin "aws" { + memory_max_mb = 2048 +} +``` + +The block label is assumed to be the plugin short name unless the `source` argument is present. The label may only contain alphanumeric characters, dashes, or underscores. The `source` argument, however, accepts any [plugin version string](/docs/reference/config-files/connection#plugin-version-strings) allowing you to refer to any version. + +```hcl +plugin "my_aws" { + source = "aws@0.41.0" + memory_max_mb = 2048 +} +``` + +In a `connection` you may continue to use the current syntax for `plugin` argument - Steampipe will resolve the `connection` to the `plugin` as long as they resolve to the same plugin version: + +```hcl +connection { + plugin = "aws" +} + +plugin "aws" { + memory_max_mb = 2048 +} +``` + +Note that if a `connection` specifies a plugin version string that resolves to more than 1 plugin instance, `steampipe` will not be able to load the connection, as it cannot assume which plugin instance to resolve to. For example, this configuration will cause a warning and the connection will be in error: + +```hcl +connection { + plugin = "aws" +} + +plugin "aws" { + memory_max_mb = 2048 +} + +plugin "aws2" { + source = "aws" + memory_max_mb = 512 +} +``` + + +You may instead specify a reference to a `plugin` block in your `connection` to disambiguate: +```hcl +connection { + plugin = plugin.aws +} + +plugin "aws" { + memory_max_mb = 2048 +} + +plugin "aws2" { + source = "aws" + memory_max_mb = 512 +} +``` + +Steampipe will create a separate plugin process for each `plugin` defined that has connections associated to it. This allows you to run multiple versions side by side, but also to create multiple processes with the SAME version to allow you to create QOS groups. In this example, steampipe will create 2 plugins process. + - one process as a 2000 MB memory soft limit and no limiters, and will contain the `prod_1`, `prod_2`, and `prod_3` connections + - one process as a 500 MB memory soft limit and the `all_requests` limiter, and will contain the `dev_1` and `dev_2` connections + + +```hcl +plugin "aws_high" { + memory_max_mb = 2000 + +} + +plugin "aws_low" { + memory_max_mb = 500 + + limiter "all_requests" { + plugin = "aws" + bucket_size = 100 + fill_rate = 100 + max_concurrency = 50 + } +} + + +connection "prod_1" { + plugin = plugin.aws_high + profile = "prod1" +} + +connection "prod_2" { + plugin = plugin.aws_high + profile = "prod2" +} + +connection "prod_3" { + plugin = plugin.aws_high + profile = "prod3" +} + +connection "dev_1" { + plugin = plugin.aws_low + profile = "dev1" +} +connection "dev_2" { + plugin = plugin.aws_low + profile = "dev2" +} + +``` + + +## limiter + +Limiters provide a simple, flexible interface to implement client-site rate limiting and concurrency thresholds. You can use limiters to: +- Smooth the request rate from steampipe to reduce load on the remote API or service +- Limit the number of parallel request to reduce contention for client and network resources +- Avoid hitting server limits and throttling + +[link to guide here] + +### Supported options +| Argument | Default | Description +|-------------------|-----------|-------------------- +| `bucket_size` | unlimited | The maximum number of requests that may be made per second (the burst size). Used in combination with `fill_rate` to implement a token-bucket rate limit. +| `fill_rate` | unlimited | The number of requests that are added back to refill the bucket each second. Used in combination with `fill_rate` to implement a token-bucket rate limit. +| `max_concurrency` | The maximum number of [List, Get, and Hydrate functions](/docs/develop/writing-plugins#hydrate-functions) that can run in parallel. +| `scope` | `[]` | The context for the limit - which resources are subject to / counted against the limit. If no scope is specified, then the limiter applies to all functions in the plugin. If you specify a list of scopes, then *a limiter instance is created for each unique combination of scope values* - it acts much like `group by` in a sql statement. +| `where` | none | A `where` clause to further filter the scopes to specific values. + + +## Examples +```hcl + +``` + + + diff --git a/docs/reference/env-vars/overview.md b/docs/reference/env-vars/overview.md index 196990f..e492d88 100644 --- a/docs/reference/env-vars/overview.md +++ b/docs/reference/env-vars/overview.md @@ -25,13 +25,16 @@ Note that plugins may also support environment variables, but these are plugin-s | [STEAMPIPE_CLOUD_TOKEN](reference/env-vars/steampipe_cloud_token) | | Set the Turbot Pipes authentication token for connecting to Turbot Pipes workspace. | [STEAMPIPE_DATABASE_PASSWORD](reference/env-vars/steampipe_database_password)| randomly generated | Set the steampipe database password for this session. This variable must be set when the steampipe service starts. | [STEAMPIPE_DATABASE_START_TIMEOUT](reference/env-vars/steampipe_database_start_timeout)| `30` | Set the maximum time (in seconds) to wait for the Postgres process to start accepting queries after it has been started. +| [STEAMPIPE_DIAGNOSTIC_LEVEL](reference/env-vars/steampipe_diagnostic_level)| `NONE` | Sets the diagnostic level. Supported levels are `ALL`, `NONE`. | [STEAMPIPE_INSTALL_DIR](reference/env-vars/steampipe_install_dir)| `~/.steampipe` | The directory in which the Steampipe database, plugins, and supporting files can be found. | [STEAMPIPE_INTROSPECTION](reference/env-vars/steampipe_introspection) | `none` | Enable introspection tables that allow you to query the mod resources in the workspace. | [STEAMPIPE_LOG](reference/env-vars/steampipe_log) | `warn` | Set the logging output level [DEPRECATED - use STEAMPIPE_LOG_LEVEL]. | [STEAMPIPE_LOG_LEVEL](reference/env-vars/steampipe_log) | `warn` | Set the logging output level. | [STEAMPIPE_MAX_PARALLEL](reference/env-vars/steampipe_max_parallel) | `10` | Set the maximum number of parallel executions. +| [STEAMPIPE_MEMORY_MAX_MB](reference/env-vars/steampipe_memory_max_mb)| `1024` | Set a soft memory limit for the `steampipe` process. | [STEAMPIPE_MOD_LOCATION](reference/env-vars/steampipe_mod_location) | current working directory | Set the workspace working directory. | [STEAMPIPE_OTEL_LEVEL](reference/env-vars/steampipe_otel_level) | `NONE` | Specify which [OpenTelemetry](https://opentelemetry.io/) data to send via OTLP. +| [STEAMPIPE_PLUGIN_MEMORY_MAX_MB](reference/env-vars/steampipe_pligin_memory_max_mb)| `1024` | Set a default memory soft limit for each plugin process. | [STEAMPIPE_QUERY_TIMEOUT](reference/env-vars/steampipe_query_timeout) | `240` for controls, unlimited in all other cases. | Set the amount of time to wait for a query to complete before timing out, in seconds. | [STEAMPIPE_SNAPSHOT_LOCATION](/docs/reference/env-vars/steampipe_snapshot_location) | The Turbot Pipes user's personal workspace | Set the Turbot Pipes workspace or filesystem path for writing snapshots. | [STEAMPIPE_TELEMETRY](reference/env-vars/steampipe_telemetry) | `info` | Set the level of telemetry data to collect and send. diff --git a/docs/reference/env-vars/steampipe_diagnostic_level.md b/docs/reference/env-vars/steampipe_diagnostic_level.md new file mode 100644 index 0000000..c1ac068 --- /dev/null +++ b/docs/reference/env-vars/steampipe_diagnostic_level.md @@ -0,0 +1,74 @@ +--- +title: STEAMPIPE_DIAGNOSTIC_LEVEL +sidebar_label: STEAMPIPE_DIAGNOSTIC_LEVEL +--- +# STEAMPIPE_DIAGNOSTIC_LEVEL + +Sets the diagnostic level. Supported levels are `ALL`, `NONE`. By default, the diagnostic level is `NONE`. + +## Usage +```bash +export STEAMPIPE_DIAGNOSTIC_LEVEL=ALL +``` + +When enabled, diagnostics information will appear in the `_ctx` column for all tables: + + +```sql +> select jsonb_pretty(_ctx),display_name from aws_sns_topic limit 1 + + ++-----------------------------------------------------------+--------------+ +| jsonb_pretty | display_name | ++-----------------------------------------------------------+--------------+ +| { | | +| "connection": "aws_dev_01", | | +| "diagnostics": { | | +| "calls": [ | | +| { | | +| "type": "list", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "ListTopics", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "listAwsSnsTopics", | | +| "rate_limiters": [ | | +| "sns_list_topics", | | +| "aws_global_concurrency" | | +| ], | | +| "rate_limiter_delay_ms": 0 | | +| }, | | +| { | | +| "type": "hydrate", | | +| "scope_values": { | | +| "table": "aws_sns_topic", | | +| "action": "GetTopicAttributes", | | +| "region": "us-east-1", | | +| "service": "sns", | | +| "connection": "aws_dev_01" | | +| }, | | +| "function_name": "getTopicAttributes", | | +| "rate_limiters": [ | | +| "sns_get_topic_attributes_us_east_1", | | +| "aws_global_concurrency" | | +| ], | | +| "rate_limiter_delay_ms": 107 | | +| } | | +| ] | | +| } | | +| } | | + +``` + +The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including: + +| Key | Description +|-------------------------|---------------------- +| `type` | The type of function (`list`, `get`, or `hydrate`). +| `function_name` | The name of the function. +| `scope_values` | A map of scope names to values. This includes the built-in scopes as well as any matrix qualifier scopes and function tags. +| `rate_limiters` | A list of the rate limiters that are scoped to the function. +| `rate_limiter_delay_ms` | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (`limiter`) rate limiting. diff --git a/docs/reference/env-vars/steampipe_memory_max_mb.md b/docs/reference/env-vars/steampipe_memory_max_mb.md new file mode 100644 index 0000000..0bb5f95 --- /dev/null +++ b/docs/reference/env-vars/steampipe_memory_max_mb.md @@ -0,0 +1,21 @@ +--- +title: STEAMPIPE_MEMORY_MAX_MB +sidebar_label: STEAMPIPE_MEMORY_MAX_MB +--- +# STEAMPIPE_MEMORY_MAX_MB + +Set a soft memory limit for the `steampipe` process. + +Set the `STEAMPIPE_MEMORY_MAX_MB` to `0` disable the soft memory limit. + +## Usage + +Set the memory soft limit to 2GB: +```bash +export STEAMPIPE_MEMORY_MAX_MB=2048 +``` + +Disable the memory soft limit: +```bash +export STEAMPIPE_MEMORY_MAX_MB=0 +``` \ No newline at end of file diff --git a/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md b/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md new file mode 100644 index 0000000..70aa2fd --- /dev/null +++ b/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md @@ -0,0 +1,27 @@ +--- +title: STEAMPIPE_PLUGIN_MEMORY_MAX_MB +sidebar_label: STEAMPIPE_PLUGIN_MEMORY_MAX_MB +--- + +# STEAMPIPE_PLUGIN_MEMORY_MAX_MB + +Set a default memory soft limit for each plugin process. + +Note that each plugin can have its own `memory_max_mb` set in [a `plugin` definition](/docs/reference/config-files/plugin), and that value would override this default setting. + +Set the `STEAMPIPE_PLUGIN_MEMORY_MAX_MB` to `0` disable the default soft memory limit. + + +## Usage + +Set the default plugin memory soft limit to 2GB: + +```bash +export STEAMPIPE_PLUGIN_MEMORY_MAX_MB=2048 +``` + +Disable the default plugin memory soft limit: + +```bash +export STEAMPIPE_PLUGIN_MEMORY_MAX_MB=0 +``` \ No newline at end of file diff --git a/docs/sidebar.json b/docs/sidebar.json index 17f308d..d0c0c08 100644 --- a/docs/sidebar.json +++ b/docs/sidebar.json @@ -194,12 +194,15 @@ "reference/env-vars/steampipe_cloud_token", "reference/env-vars/steampipe_database_password", "reference/env-vars/steampipe_database_start_timeout", + "reference/env-vars/steampipe_diagnostic_level", "reference/env-vars/steampipe_install_dir", "reference/env-vars/steampipe_introspection", "reference/env-vars/steampipe_log", "reference/env-vars/steampipe_max_parallel", + "reference/env-vars/steampipe_memory_max_mb", "reference/env-vars/steampipe_mod_location", "reference/env-vars/steampipe_otel_level", + "reference/env-vars/steampipe_plugin_memory_max_mb", "reference/env-vars/steampipe_query_timeout", "reference/env-vars/steampipe_snapshot_location", "reference/env-vars/steampipe_telemetry", @@ -218,6 +221,7 @@ "items": [ "reference/config-files/connection", "reference/config-files/options", + "reference/config-files/plugin", "reference/config-files/workspace" ] }, From 30db59d21a05252b5bde0a39ca1fd0317732a811 Mon Sep 17 00:00:00 2001 From: John Smyth Date: Mon, 25 Sep 2023 14:21:44 -0500 Subject: [PATCH 4/9] update plugin reference docs, rate limimter guide --- docs/guides/limiter.md | 59 ++++++++++++++++--- docs/reference/config-files/plugin.md | 82 ++++++++++++++++++++++----- 2 files changed, 119 insertions(+), 22 deletions(-) diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md index bbfb0f0..fc0508c 100644 --- a/docs/guides/limiter.md +++ b/docs/guides/limiter.md @@ -1,9 +1,9 @@ --- -title: Users Guide to Client-Side Rate Limiting with `limiter` -sidebar_label: Client-Side Rate Limiting +title: Users Guide to Concurrency & Rate Limiting with `limiter` +sidebar_label: Concurrency & Rate Limiting --- -# Client-side rate limiting with `limiter` +# Concurrency & Rate Limiting with `limiter` Steampipe is designed to be fast - it provides parallel execution at multiple layers: - It runs controls in parallel @@ -35,8 +35,8 @@ plugin "aws" { A limiter may also specify a `bucket_size` and `fill_rate` to limit the rate at which List, Get, and Hydrate functions may run. The rate limiter uses a token-bucket algorithm, where the `bucket_size` specifies the maximum number of tokens that may accrue (the burst size) and the `fill_rate` specifies how many tokens are refilled each second. ```hcl -# run up to 1000 hydrate/list/get functions per second plugin "aws" { + # run up to 1000 hydrate/list/get functions per second limiter "aws_global_rate_limit" { bucket_size = 1000 fill_rate = 1000 @@ -46,11 +46,24 @@ plugin "aws" { Every limiter has a **scope**. The scope defines the context for the limit - which resources are subject to / counted against the limit. There are built-in scopes for `connection`, `table`, and any matrix qualifiers that the plugin may include. A plugin author may also add [hydrate function tags](#defining-tags) that can also be used as scopes. -If no scope is specified, then the limiter applies to all functions in the plugin. If you specify a list of scopes, then *a limiter instance is created for each unique combination of scope values* - it acts much like `group by` in a sql statement. +If no scope is specified, then the limiter applies to all functions in the plugin. For instance, this limiter will allow 1000 hydrate/list/get functions per second *across all connections*: +```hcl +plugin "aws" { + # run up to 1000 hydrate/list/get functions per second across all aws connections + limiter "aws_regional_rate_limit" { + bucket_size = 1000 + fill_rate = 1000 + } +} +``` +If you specify a list of scopes, then *a limiter instance is created for each unique combination of scope values* - it acts much like `group by` in a sql statement. + +For example, to limit to 1000 hydrate/list/get functions per second in *each region of each connection*: ```hcl -# run up to 1000 hydrate/list/get functions per second in each region of each connection plugin "aws" { + + # run up to 1000 hydrate/list/get functions per second in each region of each connection limiter "aws_regional_rate_limit" { bucket_size = 1000 fill_rate = 1000 @@ -59,11 +72,12 @@ plugin "aws" { } ``` -You can use a `where` clause to further filter the scopes to specific values. +You can use a `where` clause to further filter the scopes to specific values. For example, we can restrict the limiter so that it only applies to a specific region: ```hcl -# run up to 1000 hydrate/list/get functions per second in us-east-1 for each connection + plugin "aws" { + # run up to 1000 hydrate/list/get functions per second in us-east-1 for each connection limiter "aws_rate_limit_us_east_1" { bucket_size = 1000 fill_rate = 1000 @@ -76,6 +90,35 @@ plugin "aws" { You can define multiple limiters. If a function is included in the scope of multiple rate limiters, they will all apply - the function will wait until every rate limiter that applies to it has available bucket tokens and is below its max concurrency. + +```hcl +plugin "aws" { + + # run up to 250 functions concurrently across all connections + limiter "aws_global_concurrency" { + max_concurrency = 250 + } + + # run up to 1000 functions per second in us-east-1 for each connection + limiter "aws_rate_limit_us_east_1" { + bucket_size = 1000 + fill_rate = 1000 + scope = ["connection", "region"] + where = "region = 'us-east-1'" + } + + # run up to 200 functions per second in regions OTHER than us-east-1 + # for each connection + limiter "aws_rate_limit_non_us_east_1" { + bucket_size = 200 + fill_rate = 200 + scope = ["connection", "region"] + where = "region <> 'us-east-1'" + } +} +``` + + ## Defining Tags Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation. diff --git a/docs/reference/config-files/plugin.md b/docs/reference/config-files/plugin.md index 0ea5061..f48ee4d 100644 --- a/docs/reference/config-files/plugin.md +++ b/docs/reference/config-files/plugin.md @@ -49,7 +49,7 @@ plugin "my_aws" { In a `connection` you may continue to use the current syntax for `plugin` argument - Steampipe will resolve the `connection` to the `plugin` as long as they resolve to the same plugin version: ```hcl -connection { +connection "aws" { plugin = "aws" } @@ -61,7 +61,7 @@ plugin "aws" { Note that if a `connection` specifies a plugin version string that resolves to more than 1 plugin instance, `steampipe` will not be able to load the connection, as it cannot assume which plugin instance to resolve to. For example, this configuration will cause a warning and the connection will be in error: ```hcl -connection { +connection "aws" { plugin = "aws" } @@ -69,7 +69,7 @@ plugin "aws" { memory_max_mb = 2048 } -plugin "aws2" { +plugin "aws_low_mem" { source = "aws" memory_max_mb = 512 } @@ -78,7 +78,7 @@ plugin "aws2" { You may instead specify a reference to a `plugin` block in your `connection` to disambiguate: ```hcl -connection { +connection "aws" { plugin = plugin.aws } @@ -86,21 +86,22 @@ plugin "aws" { memory_max_mb = 2048 } -plugin "aws2" { +plugin "aws_low_mem" { source = "aws" memory_max_mb = 512 } ``` -Steampipe will create a separate plugin process for each `plugin` defined that has connections associated to it. This allows you to run multiple versions side by side, but also to create multiple processes with the SAME version to allow you to create QOS groups. In this example, steampipe will create 2 plugins process. - - one process as a 2000 MB memory soft limit and no limiters, and will contain the `prod_1`, `prod_2`, and `prod_3` connections - - one process as a 500 MB memory soft limit and the `all_requests` limiter, and will contain the `dev_1` and `dev_2` connections +
+ +Steampipe will create a separate plugin process for each `plugin` defined that has connections associated to it. This allows you to run multiple versions side by side, but also to create multiple processes with the SAME version to allow you to create QOS groups. In the following example, Steampipe will create 2 plugin processes: +- One process has a 2000 MB memory soft limit and no limiters, and contains the `aws_prod_1`, `aws_prod_2`, and `aws_prod_3` connections. +- One process has a 500 MB memory soft limit and the `all_requests` limiter, and contains the `aws_dev_1` and `aws_dev_2` connections. ```hcl plugin "aws_high" { memory_max_mb = 2000 - } plugin "aws_low" { @@ -115,28 +116,81 @@ plugin "aws_low" { } -connection "prod_1" { +connection "aws_prod_1" { plugin = plugin.aws_high profile = "prod1" + regions = ["*"] } -connection "prod_2" { +connection "aws_prod_2" { plugin = plugin.aws_high profile = "prod2" + regions = ["*"] } -connection "prod_3" { +connection "aws_prod_3" { plugin = plugin.aws_high profile = "prod3" + regions = ["*"] } -connection "dev_1" { +connection "aws_dev_1" { plugin = plugin.aws_low profile = "dev1" + regions = ["*"] } -connection "dev_2" { + +connection "aws_dev_2" { plugin = plugin.aws_low profile = "dev2" + regions = ["*"] +} + +``` + + +Note that the aggregators can only aggregate connections from the single plugin instance for which they are configured. Extending the previous example: + +```hcl + +connection "aws_prod" { + plugin = plugin.aws_high + type = "aggregator" + connections = ["*"] +} + +connection "aws_dev" { + plugin = plugin.aws_low + type = "aggregator" + connections = ["*"]} +``` + +- The `aws_prod` aggregator will include the `aws_prod_1`, `aws_prod_2`, and `aws_prod_3` connections +- The `aws_dev` aggregator will include the `aws_dev_1` and `aws_dev_2` connections + + +This capability also allows you to run multiple plugin versions side-by-side: + +```hcl +plugin "aws_latest" { + source = "aws" +} + +plugin "aws_0_117_0" { + source = "aws@0.117" +} + + +connection "aws_prod_1" { + plugin = plugin.aws_latest + profile = "prod1" + regions = ["*"] +} + +connection "aws_prod_2" { + plugin = plugin.aws_0_117_0 + profile = "prod2" + regions = ["*"] } ``` From b97e5a4f1fbba59767759adf0d9e93645f425fcc Mon Sep 17 00:00:00 2001 From: John Smyth Date: Mon, 25 Sep 2023 15:45:22 -0500 Subject: [PATCH 5/9] move Go code from limiter guide to developer section, with references each way. --- docs/develop/writing-plugins.md | 84 +++++++++ docs/guides/limiter.md | 261 +++++++++++++------------- docs/reference/config-files/plugin.md | 34 +++- 3 files changed, 242 insertions(+), 137 deletions(-) diff --git a/docs/develop/writing-plugins.md b/docs/develop/writing-plugins.md index 4807ed0..b3a586c 100644 --- a/docs/develop/writing-plugins.md +++ b/docs/develop/writing-plugins.md @@ -486,6 +486,89 @@ Currently supported data types are: --- + +## Client-Side Rate Limiting + +The Steampipe Plugin SDK supports a [client-side rate limiting implementation](/docs/guides/limiter) to allow users to define [plugin `limiter` blocks](/docs/reference/config-files/plugin#limiter) to control concurrency and rate limiting. Support for limiters is build in to the SDK and basic functionality requires no changes to the plugin code; Just indluding the SDK will enable users to create limiters for your plugin using the built in `connection`, `table`, and `function_name` scopes. You can add additional flexibility by adding [function tags](#function-tags) and by [accounting for pageing in List calls](#accounting-for-paged-list-calls). +## Function Tags + +Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation. + +Tags can be added to a ListConfig, GetConfig, or HydrateConfig. + +```go +//// TABLE DEFINITION +func tableAwsSnsTopic(_ context.Context) *plugin.Table { + return &plugin.Table{ + Name: "aws_sns_topic", + Description: "AWS SNS Topic", + Get: &plugin.GetConfig{ + KeyColumns: plugin.SingleColumn("topic_arn"), + IgnoreConfig: &plugin.IgnoreConfig{ + ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"NotFound", "InvalidParameter"}), + }, + Hydrate: getTopicAttributes, + Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, + }, + List: &plugin.ListConfig{ + Hydrate: listAwsSnsTopics, + Tags: map[string]string{"service": "sns", "action": "ListTopics"}, + }, + + HydrateConfig: []plugin.HydrateConfig{ + { + Func: listTagsForSnsTopic, + Tags: map[string]string{"service": "sns", "action": "ListTagsForResource"}, + }, + { + Func: getTopicAttributes, + Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, + }, + }, + ... +``` + +Once the tags are added to the plugin, you can use them in the `scope` and `where` arguments for your rate limiter. + +```hcl +plugin "aws" { + limiter "sns_get_topic_attributes_us_east_1" { + bucket_size = 3000 + fill_rate = 3000 + + scope = ["connection", "region", "service", "action"] + where = "action = 'GetTopicAttributes' and service = 'sns' and region = 'us-east-1' " + } +} +``` + +## Accounting for Paged List calls +The Steampipe plugin SDK transparently handles the most of the details around waiting for limiters. List calls, however, usually iterate through pages of results, and each call to fetch a page must wait for any limiters that are defined. The SDK provides a hook, `WaitForListRateLimit`, which should be called before paging to apply rate limiting to the list call: + +```go +// List call +for paginator.HasMorePages() { + + // apply rate limiting + d.WaitForListRateLimit(ctx) + + output, err := paginator.NextPage(ctx) + if err != nil { + plugin.Logger(ctx).Error("List error", "api_error", err) + return nil, err + } + for _, items := range output.Items { + d.StreamListItem(ctx, items) + + // Context can be cancelled due to manual cancellation or the limit has been hit + if d.RowsRemaining(ctx) == 0 { + return nil, nil + } + } +} +``` + +-- ## Logging A logger is passed to the plugin via the context. You can use the logger to write messages to the log at standard log levels: @@ -521,3 +604,4 @@ For example, consider a `myplugin` plugin that you have developed. To install i ``` - Your connection will be loaded the next time Steampipe runs. If Steampipe is running service mode, you must restart it to load the connection. +--- diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md index fc0508c..4973320 100644 --- a/docs/guides/limiter.md +++ b/docs/guides/limiter.md @@ -44,7 +44,7 @@ plugin "aws" { } ``` -Every limiter has a **scope**. The scope defines the context for the limit - which resources are subject to / counted against the limit. There are built-in scopes for `connection`, `table`, and any matrix qualifiers that the plugin may include. A plugin author may also add [hydrate function tags](#defining-tags) that can also be used as scopes. +Every limiter has a **scope**. The scope defines the context for the limit - which resources are subject to / counted against the limit. There are built-in scopes for `connection`, `table`, `function_name`, and any matrix qualifiers that the plugin may include. A plugin author may also add [function tags](#function-tags) that can also be used as scopes. If no scope is specified, then the limiter applies to all functions in the plugin. For instance, this limiter will allow 1000 hydrate/list/get functions per second *across all connections*: ```hcl @@ -119,45 +119,12 @@ plugin "aws" { ``` -## Defining Tags - -Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation. - -Tags can be added to a ListConfig, GetConfig, or HydrateConfig. - -```go -//// TABLE DEFINITION -func tableAwsSnsTopic(_ context.Context) *plugin.Table { - return &plugin.Table{ - Name: "aws_sns_topic", - Description: "AWS SNS Topic", - Get: &plugin.GetConfig{ - KeyColumns: plugin.SingleColumn("topic_arn"), - IgnoreConfig: &plugin.IgnoreConfig{ - ShouldIgnoreErrorFunc: shouldIgnoreErrors([]string{"NotFound", "InvalidParameter"}), - }, - Hydrate: getTopicAttributes, - Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, - }, - List: &plugin.ListConfig{ - Hydrate: listAwsSnsTopics, - Tags: map[string]string{"service": "sns", "action": "ListTopics"}, - }, - - HydrateConfig: []plugin.HydrateConfig{ - { - Func: listTagsForSnsTopic, - Tags: map[string]string{"service": "sns", "action": "ListTagsForResource"}, - }, - { - Func: getTopicAttributes, - Tags: map[string]string{"service": "sns", "action": "GetTopicAttributes"}, - }, - }, - ... -``` -Once the tags are added to the plugin, you can use them in the `scope` and `where` arguments for your rate limiter. +## Function Tags + +Hydrate function tags provide useful diagnostic metadata, and they can also be used as scopes in rate limiters. Rate limiting requirements vary by plugin because the underlying APIs that they access implement rate limiting differently. Tags provide a way for a plugin author to scope rate limiters in a way that aligns with the API implementation. + +Function tags must be [added in the plugin code by the plugin author](/docs/develop/writing-plugins#function-tags). Once the tags are added to the plugin, you can use them in the `scope` and `where` arguments for your rate limiter. ```hcl plugin "aws" { @@ -171,79 +138,59 @@ plugin "aws" { } ``` -## Accounting for Paged List calls -The Steampipe plugin SDK transparently handles the most of the details around waiting for limiters. List calls, however, usually iterate through pages of results, and each call to fetch a page must wait for any limiters that are defined. The SDK provides a hook, `WaitForListRateLimit`, which should be called before paging to apply rate limiting to the list call: - -```go -// List call -for paginator.HasMorePages() { - - // apply rate limiting - d.WaitForListRateLimit(ctx) - - output, err := paginator.NextPage(ctx) - if err != nil { - plugin.Logger(ctx).Error("List error", "api_error", err) - return nil, err - } - for _, items := range output.Items { - d.StreamListItem(ctx, items) - - // Context can be cancelled due to manual cancellation or the limit has been hit - if d.RowsRemaining(ctx) == 0 { - return nil, nil - } - } -} -``` - -## Viewing and Overriding Limiters -Steampipe includes the `steampipe_rate_limiter` table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL. +You can view the available tags in the `scope_values` when in [diagnostic mode](#exploring--troubleshooting-with-diagnostic-mode). For example, to see the tags in the `aws_sns_topic` table: ```sql -> select name,plugin,source,status,bucket_size,fill_rate,max_concurrency from steampipe_rate_limiter -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ -| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ -| exec_max_concurrency_limiter | exec | plugin | active | | | 15 | -| aws_global_concurrency | aws | config | active | | | 200 | -| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | -| sns_read_900 | aws | config | active | 810 | 810 | | -| sns_read_150 | aws | config | active | 135 | 135 | | -| sns_read_30 | aws | config | active | 27 | 27 | | -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ -``` - -You can override a limiter that is compiled into a plugin by creating an HCL limiter with the same name. In the previous example, we can see that the `exec` plugin includes a default limiter named `exec_max_concurrency_limiter` that sets the max_concurrency to 15. We can override this value at run time by creating an HCL `limiter` for this plugin with the same name. The `limiter` block must be contained in a `plugin` block. Like `connection`, Steampipe will load all `plugin` blocks that it finds in any `.spc` file in the `~/.steampipe/config` directory. For example, we can add the following snippet to the `~/.steampipe/config/exec.spc` file: - -```hcl -plugin "exec" { - limiter "exec_max_concurrency_limiter" { - max_concurrency = 20 - } -} +with one_row as materialized ( + select * from aws_sns_topic limit 1 +) +select + c ->> 'function_name' as function_name, + jsonb_pretty(c -> 'scope_values') as scope_values +from + one_row, + jsonb_array_elements(_ctx -> 'diagnostics' -> 'calls') as c ``` -Querying the `steampipe_rate_limiter` table again, we can see that there are now 2 rate limiters for the `exec` plugin named `exec_max_concurrency_limiter`, but the one from the plugin is overridden by the one in the config file. - ```sql -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ -| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ -| exec_max_concurrency_limiter | exec | plugin | overridden | | | 15 | -| exec_max_concurrency_limiter | exec | config | active | | | 20 | -| aws_global_concurrency | aws | config | active | | | 200 | -| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | -| sns_read_900 | aws | config | active | 810 | 810 | | -| sns_read_150 | aws | config | active | 135 | 135 | | -| sns_read_30 | aws | config | active | 27 | 27 | | -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ ++-------------------------------+--------------------------------------------+ +| function_name | scope_values | ++-------------------------------+--------------------------------------------+ +| listAwsSnsTopics | { | +| | "table": "aws_sns_topic", | +| | "action": "ListTopics", | +| | "region": "us-east-1", | +| | "service": "sns", | +| | "connection": "aws_dmi", | +| | "function_name": "listAwsSnsTopics" | +| | } | +| listTagsForSnsTopic | { | +| | "table": "aws_sns_topic", | +| | "action": "ListTagsForResource", | +| | "region": "us-east-1", | +| | "service": "sns", | +| | "connection": "aws_dmi", | +| | "function_name": "listTagsForSnsTopic" | +| | } | +| listRegionsForServiceUncached | { | +| | "table": "aws_sns_topic", | +| | "region": "us-east-1", | +| | "connection": "aws_dmi" | +| | } | +| getTopicAttributes | { | +| | "table": "aws_sns_topic", | +| | "action": "GetTopicAttributes", | +| | "region": "us-east-1", | +| | "service": "sns", | +| | "connection": "aws_dmi", | +| | "function_name": "" | +| | } | ++-------------------------------+--------------------------------------------+ ``` +## Exploring & Troubleshooting with Diagnostic Mode -## Exploring & Troubleshooting with Diagnostics Mode - -To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostics Mode. To enable Diagnostics Mode, set the `STEAMPIPE_DIAGNOSTIC_LEVEL` environment variable to `ALL` when you start the Steampipe DB: +To assist in troubleshooting your rate limiter setup, Steampipe has introduced Diagnostic Mode. To enable Diagnostic Mode, set the `STEAMPIPE_DIAGNOSTIC_LEVEL` environment variable to `ALL` when you start the Steampipe DB: ```bash STEAMPIPE_DIAGNOSTIC_LEVEL=ALL steampipe service start ``` @@ -251,14 +198,13 @@ STEAMPIPE_DIAGNOSTIC_LEVEL=ALL steampipe service start With diagnostics enabled, the `_ctx` column will contain information about what functions were called to fetch the row, the scope values (including any [tags](#defining-tags)) for the function, the limiters that were in effect and the amount of time the request was delayed by the `limiters`. This diagnostic information can help you discover what scopes are available to use in limiters as well as to see the effect and impact of limiters that you have defined. ```sql -> select jsonb_pretty(_ctx),display_name from aws_sns_topic - - +select jsonb_pretty(_ctx) as _ctx ,display_name from aws_sns_topic limit 2 +``` +```sql +-----------------------------------------------------------+--------------+ -| jsonb_pretty | display_name | +| _ctx | display_name | +-----------------------------------------------------------+--------------+ | { | | -| "connection": "aws_dev_01", | | | "diagnostics": { | | | "calls": [ | | | { | | @@ -266,37 +212,40 @@ With diagnostics enabled, the `_ctx` column will contain information about what | "scope_values": { | | | "table": "aws_sns_topic", | | | "action": "ListTopics", | | -| "region": "us-east-1", | | +| "region": "us-east-2", | | | "service": "sns", | | -| "connection": "aws_dev_01" | | +| "connection": "aws_dmi", | | +| "function_name": "listAwsSnsTopics" | | | }, | | | "function_name": "listAwsSnsTopics", | | | "rate_limiters": [ | | -| "sns_list_topics", | | -| "aws_global_concurrency" | | -| ] | | +| "aws_global", | | +| "sns_list_topics" | | +| ], | | +| "rate_limiter_delay_ms": 0 | | | }, | | | { | | | "type": "hydrate", | | | "scope_values": { | | | "table": "aws_sns_topic", | | | "action": "GetTopicAttributes", | | -| "region": "us-east-1", | | +| "region": "us-east-2", | | | "service": "sns", | | -| "connection": "aws_dev_01" | | +| "connection": "aws_dmi", | | +| "function_name": "" | | | }, | | | "function_name": "getTopicAttributes", | | | "rate_limiters": [ | | -| "sns_get_topic_attributes_us_east_1", | | -| "aws_global_concurrency" | | +| "sns_get_topic_attributes_150", | | +| "aws_global" | | | ], | | -| "rate_limiter_delay_ms": 107 | | +| "rate_limiter_delay_ms": 808 | | | } | | | ] | | -| } | | +| }, | | +| "connection_name": "aws_dmi" | | | } | | | { | | -| "connection": "aws_dev_01", | | | "diagnostics": { | | | "calls": [ | | | { | | @@ -306,13 +255,15 @@ With diagnostics enabled, the `_ctx` column will contain information about what | "action": "ListTopics", | | | "region": "us-east-1", | | | "service": "sns", | | -| "connection": "aws_dev_01" | | +| "connection": "aws_dmi", | | +| "function_name": "listAwsSnsTopics" | | | }, | | | "function_name": "listAwsSnsTopics", | | | "rate_limiters": [ | | -| "sns_list_topics", | | -| "aws_global_concurrency" | | -| ] | | +| "aws_global", | | +| "sns_list_topics" | | +| ], | | +| "rate_limiter_delay_ms": 597 | | | }, | | | { | | | "type": "hydrate", | | @@ -321,20 +272,24 @@ With diagnostics enabled, the `_ctx` column will contain information about what | "action": "GetTopicAttributes", | | | "region": "us-east-1", | | | "service": "sns", | | -| "connection": "aws_dev_01" | | +| "connection": "aws_dmi", | | +| "function_name": "" | | | }, | | | "function_name": "getTopicAttributes", | | | "rate_limiters": [ | | | "sns_get_topic_attributes_us_east_1", | | -| "aws_global_concurrency" | | +| "aws_global" | | | ], | | -| "rate_limiter_delay_ms": 119 | | +| "rate_limiter_delay_ms": 0 | | | } | | | ] | | -| } | | -| } | +| }, | | +| "connection_name": "aws_dmi" | | +| } | | ++-----------------------------------------------------------+--------------+ ``` + The diagnostics information includes information about each Get, List, and Hydrate function that was called to fetch the row, including: | Key | Description @@ -346,6 +301,52 @@ The diagnostics information includes information about each Get, List, and Hydra | `rate_limiter_delay_ms` | The amount of time (in milliseconds) that Steampipe waited before calling this function due to client-side (`limiter`) rate limiting. +## Viewing and Overriding Limiters +Steampipe includes the `steampipe_rate_limiter` table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL. + +```sql +select name,plugin,source,status,bucket_size,fill_rate,max_concurrency from steampipe_rate_limiter +``` +```sql ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | exec | plugin | active | | | 15 | +| aws_global_concurrency | aws | config | active | | | 200 | +| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | +| sns_read_900 | aws | config | active | 810 | 810 | | +| sns_read_150 | aws | config | active | 135 | 135 | | +| sns_read_30 | aws | config | active | 27 | 27 | | ++------------------------------+--------+--------+--------+-------------+-----------+-----------------+ +``` + +You can override a limiter that is compiled into a plugin by creating an HCL limiter with the same name. In the previous example, we can see that the `exec` plugin includes a default limiter named `exec_max_concurrency_limiter` that sets the max_concurrency to 15. We can override this value at run time by creating an HCL `limiter` for this plugin with the same name. The `limiter` block must be contained in a `plugin` block. Like `connection`, Steampipe will load all `plugin` blocks that it finds in any `.spc` file in the `~/.steampipe/config` directory. For example, we can add the following snippet to the `~/.steampipe/config/exec.spc` file: + +```hcl +plugin "exec" { + limiter "exec_max_concurrency_limiter" { + max_concurrency = 20 + } +} +``` + +Querying the `steampipe_rate_limiter` table again, we can see that there are now 2 rate limiters for the `exec` plugin named `exec_max_concurrency_limiter`, but the one from the plugin is overridden by the one in the config file. + +```sql ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | exec | plugin | overridden | | | 15 | +| exec_max_concurrency_limiter | exec | config | active | | | 20 | +| aws_global_concurrency | aws | config | active | | | 200 | +| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | +| sns_read_900 | aws | config | active | 810 | 810 | | +| sns_read_150 | aws | config | active | 135 | 135 | | +| sns_read_30 | aws | config | active | 27 | 27 | | ++------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +``` + + ## Hints, Tips, & Best practices - You can use ANY scope in the `where`, even if it does not appear in the `scope` for the limiter. Remember that the `scope` defines the grouping; it acts similar to `group by` in SQL. Consider the following rate limiter: diff --git a/docs/reference/config-files/plugin.md b/docs/reference/config-files/plugin.md index f48ee4d..b848cb5 100644 --- a/docs/reference/config-files/plugin.md +++ b/docs/reference/config-files/plugin.md @@ -115,7 +115,6 @@ plugin "aws_low" { } } - connection "aws_prod_1" { plugin = plugin.aws_high profile = "prod1" @@ -169,7 +168,7 @@ connection "aws_dev" { - The `aws_dev` aggregator will include the `aws_dev_1` and `aws_dev_2` connections -This capability also allows you to run multiple plugin versions side-by-side: +You can also run multiple plugin versions side-by-side: ```hcl plugin "aws_latest" { @@ -192,7 +191,6 @@ connection "aws_prod_2" { profile = "prod2" regions = ["*"] } - ``` @@ -203,8 +201,6 @@ Limiters provide a simple, flexible interface to implement client-site rate limi - Limit the number of parallel request to reduce contention for client and network resources - Avoid hitting server limits and throttling -[link to guide here] - ### Supported options | Argument | Default | Description |-------------------|-----------|-------------------- @@ -216,9 +212,33 @@ Limiters provide a simple, flexible interface to implement client-site rate limi ## Examples -```hcl -``` +See the [Concurrency & Rate Limiting](/docs/guides/limiter) for more examples. + + +```hcl +plugin "aws" { + # run up to 250 functions concurrently across all connections + limiter "aws_global_concurrency" { + max_concurrency = 250 + } + # run up to 1000 functions per second in us-east-1 for each connection + limiter "aws_rate_limit_us_east_1" { + bucket_size = 1000 + fill_rate = 1000 + scope = ["connection", "region"] + where = "region = 'us-east-1'" + } + # run up to 200 functions per second in regions OTHER than us-east-1 + # for each connection + limiter "aws_rate_limit_non_us_east_1" { + bucket_size = 200 + fill_rate = 200 + scope = ["connection", "region"] + where = "region <> 'us-east-1'" + } +} +``` From ec1ccae00657dce85a159842c5020e0ec4e8b430 Mon Sep 17 00:00:00 2001 From: John Smyth Date: Mon, 25 Sep 2023 17:02:21 -0500 Subject: [PATCH 6/9] add sql reference for arg in plugin limiter docs. add more detail about memory soft limits --- docs/reference/config-files/plugin.md | 36 ++++++++++++++++--- .../env-vars/steampipe_memory_max_mb.md | 2 +- .../steampipe_plugin_memory_max_mb.md | 2 +- 3 files changed, 34 insertions(+), 6 deletions(-) diff --git a/docs/reference/config-files/plugin.md b/docs/reference/config-files/plugin.md index b848cb5..1ffaa50 100644 --- a/docs/reference/config-files/plugin.md +++ b/docs/reference/config-files/plugin.md @@ -22,7 +22,7 @@ plugin "aws" { | Argument | Default | Description |-|-|-|- | `source` | none | A [plugin version string](#plugin-version-strings) the specifies which plugin this configuration applies to. If not specified, the plugin block label is assumed to be the plugin source. -| `memory_max_mb` | `1024` | The soft memory limit for the plugin, in MB. +| `memory_max_mb` | `1024` | The soft memory limit for the plugin, in MB. Steampipe sets `GOMEMLIMIT` for the plugin process to the specified value. The Go runtime does not guarantee that the memory usage will not exceed the limit, but rather uses it as a target to optimize garbage collection. | `limiter` | none | Optional [limiter](#limiter) blocks used to set concurrency and/or rate limits @@ -211,6 +211,33 @@ Limiters provide a simple, flexible interface to implement client-site rate limi | `where` | none | A `where` clause to further filter the scopes to specific values. +### `where` syntax + +The `where` argument supports the following PostgreSQL comparison operators: + +| Operator | Description +|----------|-------------------------------- +| `<` | less than +| `<=` | less than or equal +| `=` | equal +| `!=` | not equal +| `<>` | not equal +| `>=` | greater than or equal +| `>` | greater than +| `like` | string like (case sensitive) +| `ilike` | string like (case insensitive) +| `is null`| null test +| `not` | logical negation +| `and` | logical conjunction +| `or` | logical disjunction +| `in` | set membership (equality) + + +You may use parentheses to force explicit lexical precedence, otherwise [standard PostgreSQL operator precedence](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-PRECEDENCE) applies. + + + + ## Examples See the [Concurrency & Rate Limiting](/docs/guides/limiter) for more examples. @@ -219,12 +246,12 @@ See the [Concurrency & Rate Limiting](/docs/guides/limiter) for more examples. ```hcl plugin "aws" { - # run up to 250 functions concurrently across all connections + # up to 250 functions concurrently across all connections limiter "aws_global_concurrency" { max_concurrency = 250 } - # run up to 1000 functions per second in us-east-1 for each connection + # up to 1000 functions per second in us-east-1 for each connection limiter "aws_rate_limit_us_east_1" { bucket_size = 1000 fill_rate = 1000 @@ -232,7 +259,7 @@ plugin "aws" { where = "region = 'us-east-1'" } - # run up to 200 functions per second in regions OTHER than us-east-1 + # up to 200 functions per second in regions OTHER than us-east-1 # for each connection limiter "aws_rate_limit_non_us_east_1" { bucket_size = 200 @@ -240,5 +267,6 @@ plugin "aws" { scope = ["connection", "region"] where = "region <> 'us-east-1'" } + } ``` diff --git a/docs/reference/env-vars/steampipe_memory_max_mb.md b/docs/reference/env-vars/steampipe_memory_max_mb.md index 0bb5f95..fd0a4d4 100644 --- a/docs/reference/env-vars/steampipe_memory_max_mb.md +++ b/docs/reference/env-vars/steampipe_memory_max_mb.md @@ -4,7 +4,7 @@ sidebar_label: STEAMPIPE_MEMORY_MAX_MB --- # STEAMPIPE_MEMORY_MAX_MB -Set a soft memory limit for the `steampipe` process. +Set a soft memory limit for the `steampipe` process. Steampipe sets `GOMEMLIMIT` for the `steampipe` process to the specified value. The Go runtime does not guarantee that the memory usage will not exceed the limit, but rather uses it as a target to optimize garbage collection. Set the `STEAMPIPE_MEMORY_MAX_MB` to `0` disable the soft memory limit. diff --git a/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md b/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md index 70aa2fd..8f08a76 100644 --- a/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md +++ b/docs/reference/env-vars/steampipe_plugin_memory_max_mb.md @@ -5,7 +5,7 @@ sidebar_label: STEAMPIPE_PLUGIN_MEMORY_MAX_MB # STEAMPIPE_PLUGIN_MEMORY_MAX_MB -Set a default memory soft limit for each plugin process. +Set a default memory soft limit for each plugin process. Steampipe sets `GOMEMLIMIT` for each plugin process to the specified value. The Go runtime does not guarantee that the memory usage will not exceed the limit, but rather uses it as a target to optimize garbage collection. Note that each plugin can have its own `memory_max_mb` set in [a `plugin` definition](/docs/reference/config-files/plugin), and that value would override this default setting. From 6402451fe6766b987e894a616a00ca5427c4577a Mon Sep 17 00:00:00 2001 From: John Smyth Date: Mon, 25 Sep 2023 17:24:39 -0500 Subject: [PATCH 7/9] update references from steampipe_rate_limiter -> steampipe_plugin_limiter, update examples for new column names --- docs/guides/limiter.md | 57 ++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 25 deletions(-) diff --git a/docs/guides/limiter.md b/docs/guides/limiter.md index 4973320..e708dd1 100644 --- a/docs/guides/limiter.md +++ b/docs/guides/limiter.md @@ -302,22 +302,24 @@ The diagnostics information includes information about each Get, List, and Hydra ## Viewing and Overriding Limiters -Steampipe includes the `steampipe_rate_limiter` table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL. +Steampipe includes the `steampipe_plugin_limiter` table to provide visibility into all the limiters that are defined in your installation, including those defined in plugin code as well as limiters defined in HCL. ```sql -select name,plugin,source,status,bucket_size,fill_rate,max_concurrency from steampipe_rate_limiter +select name,plugin,source_type,status,bucket_size,fill_rate,max_concurrency from steampipe_plugin_limiter ``` ```sql -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ -| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ -| exec_max_concurrency_limiter | exec | plugin | active | | | 15 | -| aws_global_concurrency | aws | config | active | | | 200 | -| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | -| sns_read_900 | aws | config | active | 810 | 810 | | -| sns_read_150 | aws | config | active | 135 | 135 | | -| sns_read_30 | aws | config | active | 27 | 27 | | -+------------------------------+--------+--------+--------+-------------+-----------+-----------------+ ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ +| name | plugin | source_type | status | bucket_size | fill_rate | max_concurrency | ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | hub.steampipe.io/plugins/turbot/exec@latest | plugin | active | | | 15 | +| sns_get_topic_attributes_150 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 150 | 150 | | +| sns_get_topic_attributes_30 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 30 | 30 | | +| aws_global | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 10 | 10 | | +| sns_list_topics | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 30 | 30 | | +| sns_list_tags_for_resource | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 10 | 10 | | +| sns_get_topic_attributes_us_east_1 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 3000 | 3000 | | +| sns_get_topic_attributes_900 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 900 | 900 | | ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ ``` You can override a limiter that is compiled into a plugin by creating an HCL limiter with the same name. In the previous example, we can see that the `exec` plugin includes a default limiter named `exec_max_concurrency_limiter` that sets the max_concurrency to 15. We can override this value at run time by creating an HCL `limiter` for this plugin with the same name. The `limiter` block must be contained in a `plugin` block. Like `connection`, Steampipe will load all `plugin` blocks that it finds in any `.spc` file in the `~/.steampipe/config` directory. For example, we can add the following snippet to the `~/.steampipe/config/exec.spc` file: @@ -330,20 +332,25 @@ plugin "exec" { } ``` -Querying the `steampipe_rate_limiter` table again, we can see that there are now 2 rate limiters for the `exec` plugin named `exec_max_concurrency_limiter`, but the one from the plugin is overridden by the one in the config file. +Querying the `steampipe_plugin_limiter` table again, we can see that there are now 2 rate limiters for the `exec` plugin named `exec_max_concurrency_limiter`, but the one from the plugin is overridden by the one in the config file. ```sql -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ -| name | plugin | source | status | bucket_size | fill_rate | max_concurrency | -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ -| exec_max_concurrency_limiter | exec | plugin | overridden | | | 15 | -| exec_max_concurrency_limiter | exec | config | active | | | 20 | -| aws_global_concurrency | aws | config | active | | | 200 | -| sns_read_us_east_1 | aws | config | active | 2700 | 2700 | | -| sns_read_900 | aws | config | active | 810 | 810 | | -| sns_read_150 | aws | config | active | 135 | 135 | | -| sns_read_30 | aws | config | active | 27 | 27 | | -+------------------------------+--------+--------+------------+-------------+-----------+-----------------+ +select name,plugin,source_type,status,bucket_size,fill_rate,max_concurrency from steampipe_plugin_limiter +``` +```sql ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ +| name | plugin | source_type | status | bucket_size | fill_rate | max_concurrency | ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ +| exec_max_concurrency_limiter | hub.steampipe.io/plugins/turbot/exec@latest | plugin | active | | | 15 | +| exec_max_concurrency_limiter | hub.steampipe.io/plugins/turbot/exec@latest | config | active | | | 20 | +| aws_global | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 10 | 10 | | +| sns_list_topics | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 30 | 30 | | +| sns_list_tags_for_resource | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 10 | 10 | | +| sns_get_topic_attributes_us_east_1 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 3000 | 3000 | | +| sns_get_topic_attributes_900 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 900 | 900 | | +| sns_get_topic_attributes_150 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 150 | 150 | | +| sns_get_topic_attributes_30 | hub.steampipe.io/plugins/turbot/aws@latest | config | active | 30 | 30 | | ++------------------------------------+---------------------------------------------+-------------+--------+-------------+-----------+-----------------+ ``` @@ -388,6 +395,6 @@ Querying the `steampipe_rate_limiter` table again, we can see that there are now - Use the plugin logs (`~/.steampipe/logs/plugin*.log`) to verify that the rate limiters are reducing the throttling and other errors from the API as you would expect. -- Use the `steampipe_rate_limiter` table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use `STEAMPIPE_DIAGNOSTIC_LEVEL=ALL` to enable extra diagnostic info in the `_ctx` to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the `STEAMPIPE_DIAGNOSTIC_LEVEL` variable must be set in the database service process - if you run steampipe as a service, it must be set when you run `steampipe service start` +- Use the `steampipe_plugin_limiter` table to see what rate limiters are in effect from both the plugins and the config files, as well as which are active. Use `STEAMPIPE_DIAGNOSTIC_LEVEL=ALL` to enable extra diagnostic info in the `_ctx` to discover what scopes are available and to verify that limiters are being applied as you expect. Note that the `STEAMPIPE_DIAGNOSTIC_LEVEL` variable must be set in the database service process - if you run steampipe as a service, it must be set when you run `steampipe service start` - Throttling errors from the server, such as `429 Too Many Requests` are not *inherently* bad. Most cloud SDKs actually account for retrying such errors and expect that it will sometimes occur. Steampipe plugins generally implement an exponential back-off & retry to account for such cases. You can use client side limiters to help avoid resource contention and to reduce throttling from the server, but completely avoiding server-side throttling is probably not necessary in most cases. From 51fe6ca41b681c776655516e3650a875c80a7656 Mon Sep 17 00:00:00 2001 From: John Smyth Date: Tue, 26 Sep 2023 14:39:52 -0500 Subject: [PATCH 8/9] update docs for new `steampipe plugin install` behavior to install missing plugins --- docs/managing/plugins.md | 17 ++++++++++++++++- docs/reference/cli/plugin.md | 12 +++++++++++- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/managing/plugins.md b/docs/managing/plugins.md index 3bddcb2..0c8b060 100644 --- a/docs/managing/plugins.md +++ b/docs/managing/plugins.md @@ -60,7 +60,7 @@ Steampipe plugins are packaged in OCI format and can be hosted and installed fro $ steampipe plugin install us-docker.pkg.dev/myproject/myrepo/myplugin@mytag ``` -### Installing from a file +### Installing from a File A plugin binary can be installed manually, and this is often convenient when developing the plugin. Steampipe will attempt to load any plugin that is referred to in a `connection` configuration: - The plugin binary file must have a `.plugin` extension - The plugin binary must reside in a subdirectory of the `~/.steampipe/plugins/` directory and must be the ONLY `.plugin` file in that subdirectory @@ -76,6 +76,21 @@ For example, consider a `myplugin` plugin that you have developed. To install i } ``` +### Installing Missing Plugins + +You can install all missing plugins that are referenced in your configuration files: + +```bash +$ steampipe plugin install +``` + +Running `steampipe plugin install` with no arguments will cause Steampipe to read all `connection` and `plugin` blocks in all `.spc` files in the `~/.steampipe/config` directory and install any that are referenced but are not installed. Note that when doing so, any default `.spc` file that does not exist in the configuration will also be copied. You may pass the `--skip-config` flag if you don't want to copy these files: + +```bash +$ steampipe plugin install --skip-config +``` + + ## Viewing Installed Plugins You can list the installed plugins with the `steampipe plugin list` command: diff --git a/docs/reference/cli/plugin.md b/docs/reference/cli/plugin.md index 81262c2..93ae162 100644 --- a/docs/reference/cli/plugin.md +++ b/docs/reference/cli/plugin.md @@ -18,7 +18,7 @@ steampipe plugin [command] | Command | Description |-|- -| `install` | Install or update a plugin +| `install` | Install one or more plugins | `list` | List currently installed plugins | `uninstall` | Uninstall a plugin | `update ` | Update one or more plugins @@ -39,6 +39,10 @@ steampipe plugin [command] --progress Enable or disable progress information. By default, progress information is shown - set --progress=false to hide the progress bar. Applies only to plugin install and plugin update. + + --skip-config + Applies only to plugin install, skip creating the default config file for plugin. + @@ -54,6 +58,12 @@ Install a specific version of a plugin: steampipe plugin install aws@0.107.0 ``` +Install all missing plugins that specified in configuration files. Do not download their default configuration files: + +```bash +steampipe plugin install --skip-config +``` + List installed plugins: ```bash steampipe plugin list From 470cd4d5b36b36586f1a6865cd797b36dc796d86 Mon Sep 17 00:00:00 2001 From: Binaek Sarkar Date: Thu, 28 Sep 2023 15:09:35 +0530 Subject: [PATCH 9/9] update --- docs/reference/cli/service.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/cli/service.md b/docs/reference/cli/service.md index 2db1d1c..14c1f12 100644 --- a/docs/reference/cli/service.md +++ b/docs/reference/cli/service.md @@ -31,12 +31,12 @@ steampipe service [command] | `--dashboard` | `start` | Start the `dashboard` web server with the database service | `--dashboard-listen string` | `start` | Accept dashboard connections from: `local` (localhost only) or `network` (open) | `--dashboard-port int` | `start` | Dashboard web server port (default `9194`) -| `--database-listen string` | `start` | Accept database connections from: `local` (localhost only) or `network` (open) +| `--database-listen string` | `start` | Accept connections from: local (alias for `localhost` only), `network` (alias for `*`), or a comma separated list of hosts and/or IP addresses (default `network`) | `--database-password string` | `start` | Set the steampipe database password for this session. See [STEAMPIPE_DATABASE_PASSWORD](reference/env-vars/steampipe_database_password) for additional information -| `--database-port int` | `start` | Database service port (default 9193) +| `--database-port int` | `start` | Database service port (default `9193`) | `--force` | `stop`, `restart` | Forces the service to shutdown, releasing all open connections and ports | `--foreground` | `start` | Run the service in the foreground -| `--show-password` | `start`, `status` | View database password for connecting from another machine (default false) +| `--show-password` | `start`, `status` | View database password for connecting from another machine (default `false`) | `--var stringArray` | `start` | Specify the value of a variable (only applies if '--dashboard' flag is also set) | `--var-file strings` | `start` | Specify an .spvar file containing variable values (only applies if '--dashboard' flag is also set) | `--all` | `status` | Bypass the `--install-dir` and print status of all running services