Skip to content

Latest commit

 

History

History
225 lines (176 loc) · 7.11 KB

README.md

File metadata and controls

225 lines (176 loc) · 7.11 KB

Scripts

When adding a new resource to infracost, a productFilter has to be added that uniquely identifies the product for pricing purposes (the filter is used to find a unique productHash). The filter needs a service, productFamily and attribute key/values to filter the prices. Because cloud prices have many parameter values to determine a price on a per-region or product type basis, querying for prices can look odd as many parameter values are duplicated. These scripts help to explore the pricing data and find the correct price by using graphqurl and jq to determine the productFilter that uniquely identifies the product.

Setup

  1. These scripts require jq, node and the npm module graphqurl. On MacOS, the following should work:
brew install jq
brew install node
npm install -g graphqurl
  1. Download the AWS pricing index file:
wget https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json
  1. Set the INFRACOST_API_KEY env variable:
export INFRACOST_API_KEY=my_api_key_here

Usage

  • List all available AWS service names, or find a service name by searching for a keyword or acronym in the JSON file downloaded in the setup step. AWS use many acronyms so be sure to search for those too, e.g. "ES" returns "AmazonES" for ElasticSearch. The reason we do not query the pricing service graphQL API is that it only returns a limited number of responses, and as there are lots of products in AWS, our desired service name might not be included in the returned responses.

    List all available AWS service names:

    ./1_find_services.sh

    Search for service name using a keyword/acronym:

    ./1_find_services.sh <keyword>

    For example, running ./1_find_services.sh rds returns:

    "offerCode" : "AmazonRDS",

  • Find the desired productFamily using the service name found in the previous step:

    ./2_find_product_families.sh <service-name>

    For example, running ./2_find_product_families.sh AmazonRDS returns:

    {
  "Aurora Global Database",
  "CPU Credits",
  "Database Instance"
}

  • Use scripts 3 with 4 to explore attribute keys and values separately, or script 5 to list all key-attribute pairs. This is useful if you want to grep for things such as grep -i usagetype or grep -i engine.

    • Find the desired attributes keys using the service name and productFamily found in previous steps:

      ./3_find_attribute_keys.sh <service-name> <product-family>

      For example, running ./3_find_attribute_keys.sh AmazonRDS "Database Instance" returns the following abbreviated output, which shows there are different instanceType and databaseEngine attribute:

      Found 1000 different products
------------------

@@ -18,3 +18,3 @@
    "key": "instanceType",
-    "value": "db.m3.large",
+    "value": "db.r4.4xlarge",
@@ -28,3 +28,3 @@
    "key": "databaseEngine",
-    "value": "MySQL",
+    "value": "MariaDB",
...

    • Identify desired attributes value using the service name, productFamily and attribute key found in previous steps:

      ./4_find_attribute_values.sh <service-name> <product-family> <attribute-key>

      For example, running ./4_find_attribute_values.sh AmazonRDS "Database Instance" databaseEngine returns the unique values for the databaseEngine attribute:

      "Aurora MySQL"
"Aurora PostgreSQL"
"MariaDB"
"MySQL (on-premise for Outpost)"
"MySQL"
"Oracle"
"PostgreSQL (on-premise for Outposts)"
"PostgreSQL"
"SQL Server"

    • List all key-value pairs for the desired attributes value using the service name and productFamily:

      ./5_find_all_attribute_key_values.sh <service-name> <product-family>

      For example, running ./5_find_all_attribute_key_values.sh AmazonRDS "Database Instance" returns:

      clockSpeed=2.3 GHz
...
databaseEdition=Standard
...
databaseEngine=SQL Server
...
usagetype=Multi-AZUsage:db.m6g.4xl

  • When writing integration tests, a PriceHash is used to match the price of a cost component so the tests continue to pass even if the actual price value changes. To find a PriceHash:

    1. Install the modheader browser extension so you can set custom HTTP headers in your browser.
    2. Use the modheader browser extension to set x-api-key to your Infracost API key.
    3. Browse to https://pricing.api.infracost.io/graphql and use the following query with your desired variable values:
    • query:
    query($productFilter: ProductFilter!, $priceFilter: PriceFilter) {
  products(filter: $productFilter) {
    prices(filter: $priceFilter) {
      priceHash
      USD
    }
  }
}
    • variables (edit as required):
    {
  "priceFilter": {
    "purchaseOption": "on_demand"
  },
  "productFilter": {
    "service": "AmazonRDS",
    "productFamily": "Database Instance",
    "region": "us-east-1",
    "attributeFilters": [
      {
        "key": "instanceType",
        "value": "db.t3.medium"
      },
      {
        "key": "databaseEngine",
        "value": "MariaDB"
      },
      {
        "key": "deploymentOption",
        "value": "Single-AZ"
      }
    ]
  }
}

    The above returns the following result, which you can manually verify by browsing to the AWS pricing page for the above product/filters. For example, https://aws.amazon.com/rds/mariadb/pricing/ shows the hourly price as $0.068, which matches the following JSON:

    {
  "data": {
    "products": [
      {
        "prices": [
          {
            "priceHash": "72dd84252654545b6f19c2e553efecde-d2c98780d7b6e36641b521f1f8145c6f",
            "USD": "0.0680000000"
          }
        ]
      }
    ]
  }
}

MongoDB lookups

Instead of using the scripts, you can also run distinct or regex queries on MongoDB to explore the prices.

  1. Install MongoDB version 4.
  2. Join our Community Slack channel, and send a message to Ali Khajeh-Hosseini, he'll send you a link so you can download a copy of his MongoDB dump.
  3. Import the prices to your local:
tar -xvf cloudPricing.tar
mongorestore --gzip cloudPricing
  1. You can now query you local MongoDB:
mongo
use cloudPricing;

db.products.distinct("vendorName");
> should show you: [ "aws", "gcp" ]

db.products.distinct("service", {"vendorName": "gcp", "service": {$regex: "kms", $options: "i"} });
> [ "Cloud Key Management Service (KMS)", "Thales CPL ekms-dpod-eu" ]

db.products.distinct("attributes.description", {"vendorName": "gcp", "service": "Cloud Key Management Service (KMS)" });
> [
	"Active HSM ECDSA P-256 key versions",
	"Active HSM ECDSA P-384 key versions",
	"Active HSM RSA 2048 bit key versions",
...
]