Tyk v1.8.3 and Dashboard v0.9.6.3

• Bug fixes
• Service discovery module re-built, now supports lists (a la Consul) and many more configurations
• AddedURL-based versioning (only respects first value after listen path (e.g. /listen_path/**VERSION**/resource/id)

Notes:

• For list based service discovery commands, set the endpoint_returns_list value to true or change it in the dashboard, this will treat the requested object as a list

Tyk v1.8.2 and Dashboard v0.9.6.2

Changelog

• Bug fixes
• More local assets (removed CDN from dashboard)
• Developer key management support (in-place policy updates) for better developer key lifecycle mgmt
• Catalogue entries are now no longer linked to APIs

Upgrade Warning

This update is fully backwards compatible, however:

This update migrates API Catalogue (portal) entries into a new format. Portal entries used to need to be linked to an API. As of this version, this is no longer the case. instead, API Entries are linked to a policy ID. In turn, Key requests are also linked to policies instead of to APIs.

Reasoning: Policies give access to multiple APIs, allowing users to package APIs into bundles and tiers. In the previous portal incarnation this was fine so long as you only ever gave access to a single APi per portal entry. Realistically this does not make sense.

Old API Catalogue entries and key requests will still work as normal, including links to keys and key data in the developer profile section.

However all new catalogue entries created int he dashboard will be for this new version of key requests.

This update is fully backwards compatible, so existing policies/entries and key requests will work just as before. Only new catalogue entries will be affected.

Tyk v1.8.1 and Dashboard v0.9.6.1

Changelog

• Fixed redis diconnect bug
• Fixed Mongo reload bug on host manager
• New onboarding flow in Dashboard
• Swagger documentation support in portal now respects host data and URLs as set in the dashboard conf
• when a key request is approved, the request metadata and user meta data are attached to the session object
• Various documentation improvements

Tyk 1.8 and Tyk Dashboard 0.9.6

Tyk v1.8

Large update to Tyk, major improvements and new features.
New: Dashbaord also has FreeBSD versions

Full Changelog:

• Security option added for shared nodes: Set disable_virtual_path_blobs=true to stop virtual paths from loading blob fields

• Added session meta data variables to transform middleware:

  You can reference session metadata attached to a key in the header injector using:

  $tyk_meta.KEY_NAME
  

  And in the body transform template through:

  ._tyk_meta.KEYNAME
  

  You must enable sesison parsing in the TemplateData of the body tranform entry though by adding:

  "enable_session": true
  

  To the path entry

• Added CORS support, add a CORS section to your API definition:

   CORS: {
      enable: false,
      allowed_origins: [
        "http://foo.com"
      ]
   },
  

• Full CORS Options are:

  CORS struct {
      Enable             bool     `bson:"enable" json:"enable"`
      AllowedOrigins     []string `bson:"allowed_origins" json:"allowed_origins"`
      AllowedMethods     []string `bson:"allowed_methods" json:"allowed_methods"`
      AllowedHeaders     []string `bson:"allowed_headers" json:"allowed_headers"`
      ExposedHeaders     []string `bson:"exposed_headers" json:"exposed_headers"`
      AllowCredentials   bool     `bson:"allow_credentials" json:"allow_credentials"`
      MaxAge             int      `bson:"max_age" json:"max_age"`
      OptionsPassthrough bool     `bson:"options_pasthrough" json:"options_pasthrough"`
      Debug              bool     `bson:"debug" json:"debug"`
  } `bson:"CORS" json:"CORS"`
  

• Fixed cache bug

• When using node segments, tags will be transferred into analytics data as well as any token-level tags, so for example, you could tag each node independently, and then view the trafic that went through those nodes by ID or group them in aggregate

• You can now segment gateways that use a DB-backed configurations for example if you vae APIs in different regions, or only wish to service a segment of your APIs (e.g. "Health APIs", "Finance APIs"). So you can have a centralised API registry using the dashboard, and then Tag APIs according to their segment(s), then configure your Tyk nodes to only load those API endpoints, so node 1 may only serve health APIs, while node 2 might serve a mixture and node 3 will serve only finance APIs. To enable, simply configure your node and add to tyk.conf and host_manager.conf (if using):

  "db_app_conf_options": {
          "node_is_segmented": false,
           "tags": ["test2"]
     }
  

• You will need to add a tags: [] sectino to your API definition in the DB to enable this feature, or set it in the dashboard.

• Dynamic endpoints support response middleware

• Dynamic endpoints support caching

• Dynamic endpoints also count towards analytics

• JSVM now has access to a TykBatchRequest function to make batch requests in virtual paths. Use case: Create a virtual endpoint that interacts with multiple upstream APIs, gathers the data, processes the aggregates somehow and returns them as a single body. This can then be cached to save on load.

• Added virtual path support, you can now have a JS Function respond to a request, makes mocking MUCh more flexible, TODO: expose batch methods to JSVM. To activate, add to extended paths:

  virtual: [
      {
        response_function_name: "thisTest",
        function_source_type: "file",
        function_source_uri: "middleware/testVirtual.js",
        path: "virtualtest",
        method: "GET",
        use_session: true
      }
  ]
  

• Virtual endpoint functions are pretty clean:

  function thisTest(request, session, config) {
      log("Virtual Test running")
  
      log("Request Body: ")
      log(request.Body)
  
      log("Session: ")
      log(session)
  
      log("Config:")
      log(config)
  
      log("param-1:")
      log(request.Params["param1"])
  
      var responseObject = {
          Body: "THIS IS A  VIRTUAL RESPONSE"
          Headers: {
              "test": "virtual", 
              "test-2": "virtual"
          },
          Code: 200
      }
  
      return TykJsResponse(responseObject, session.meta_data)
  
  }
  log("Virtual Test initialised")
  

• Added refresh tests for OAuth

• URL Rewrite in place, you can specify URLs to rewrite in the extended_paths seciton f the API Definition like so:

  "url_rewrites": [
      {
        "path": "virtual/{wildcard1}/{wildcard2}",
        "method": "GET",
        "match_pattern": "virtual/(.*)/(\d+)",
        "rewrite_to": "new-path/id/$2/something/$1"
      }
    ]
  

• You can now add a "tags":["tag1, "tag2", tag3"] field to token and policy definitions, these tags are transferred through to the analytics record when recorded. They will also be available to dynamic middleware. This means there is more flexibility with key ownership and reporting by segment.

• Cleaned up server output, use --debug to see more detailed debug data. Keeps log size down

• TCP Errors now actually raise an error

• Added circuit breaker as a path-based option. To enable, add a new sectino to your versions extended_paths list:

  circuit_breakers: [
      {
        path: "get",
        method: "GET",
        threshold_percent: 0.5,
        samples: 5,
        return_to_service_after: 60
      }
  

  ]

Circuit breakers are individual on a singlie host, they do not centralise or pool back-end data, this is for speed. This means that in a load balanced environment where multiple Tyk nodes are used, some traffic can spill through as other nodes reach the sampling rate limit. This is for pure speed, adding a redis counter layer or data-store on every request to a servcie would jsut add latency.

Circuit breakers use a thresh-old-breaker pattern, so of sample size x if y% requests fail, trip the breaker.

The circuit breaker works across hosts (i.e. if you have multiple targets for an API, the samnple is across all upstream requests)

When a circuit breaker trips, it will fire and event: BreakerTriggered which you can define actions for in the event_handlers section:

```
event_handlers: {
    events: {
      BreakerTriggered: [
        {
          handler_name: "eh_log_handler",
          handler_meta: {
            prefix: "LOG-HANDLER-PREFIX"
          }
        },
        {
          handler_name: "eh_web_hook_handler",
          handler_meta: {
            method: "POST",
            target_path: "http://posttestserver.com/post.php?dir=tyk-event-test",
            template_path: "templates/breaker_webhook.json",
            header_map: {
              "X-Tyk-Test-Header": "Tyk v1.BANANA"
            },
            event_timeout: 10
          }
        }
      ]
    }
  },
```

Status codes are:

```
// BreakerTripped is sent when a breaker trips
BreakerTripped = 0

// BreakerReset is sent when a breaker resets
BreakerReset = 1
```

• Added round-robin load balancing support, to enable, set up in the API Definition under the proxy section:

  ...
  "enable_load_balancing": true,
  "target_list": [
      "http://server1", 
      "http://server2", 
      "http://server3"
  ],
  ...
  

• Added REST-based Servcie discovery for both single and load balanced entries (tested with etcd, but anything that returns JSON should work), to enable add a service discovery section to your Proxy section:

  // Solo
  service_discovery : {
    use_discovery_service: true,
    query_endpoint: "http://127.0.0.1:4001/v2/keys/services/single",
    use_nested_query: true,
    parent_data_path: "node.value",
    data_path: "hostname",
    port_data_path: "port",
    use_target_list: false,
    cache_timeout: 10
  },
  
  
  // With LB
  "enable_load_balancing": true,
  service_discovery: {
    use_discovery_service: true,
    query_endpoint: "http://127.0.0.1:4001/v2/keys/services/multiobj",
    use_nested_query: true,
    parent_data_path: "node.value",
    data_path: "array.hostname",
    port_data_path: "array.port",
    use_target_list: true,
    cache_timeout: 10
  },
  

• For service discovery, multiple assumptions are made:

  • The response data is in JSON
  • The response data can have a nested value set that will be an encoded JSON string, e.g. from etcd:

  $ curl -L http://127.0.0.1:4001/v2/keys/services/solo
  
  {
      "action": "get",
      "node": {
          "key": "/services/single",
          "value": "{\"hostname\": \"http://httpbin.org\", \"port\": \"80\"}",
          "modifiedIndex": 6,
          "createdIndex": 6
      }
  }
  

  $ curl -L http://127.0.0.1:4001/v2/keys/services/multiobj
  
  {
      "action": "get",
      "node": {
          "key": "/services/multiobj",
          "value": "{\"array\":[{\"hostname\": \"http://httpbin.org\", \"port\": \"80\"},{\"hostname\": \"http://httpbin.org\", \"port\": \"80\"}]}",
          "modifiedIndex": 9,
          "createdIndex": 9
      }
  }
  

  Here the key value is actually an encoded JSON string, which needs to be decoded separately to get to the data.

  • In some cases port data will be separate from host data, if you specify a port_data_path, the values will be zipped together and concatenated into a valid proxy string.
  • If use_target_list is enabled, then enable_load_balancing msut also be enabled, as Tyk will treat the list as a target list.
  • The nested data object in a service registry key MUST be a JSON Object, not just an Array.

• Fixed bug where version parameter on POST requests would empty request body, streamlined request copies in general.

• it is now possible to use JSVM middleware on Open (Keyless) APIs

• It is now possible to configure the timeout parameters around the http server in the tyk.conf file:

  "http_server_options": {
      "override_defaults": true,
      "read_timeout": 10,
      "write_timeout": 10
  }...
  

Hotfix 1.7.3

This release fixes a bug with Oauth Refresh tokens not being valid after consecutive use

Hotfix v1.7.2

In certain configurations, Tyk can run out of open file descriptors because the golang Http server respects the keep-alive header, this can cause problems with high-trafic APIs.

With this hotfix you can add "close_connections": true to your tyk.conf file and tyk will not keep open TCP connections.

Tyk v1.7.1

Major speed and reliability optimisations.

Tyk v1.7 and Dashboard v0.9.5

Update: Dashboard 0.9.5.3

• Hotfix for listen_path issues when adding a new API

Update: Dashboard 0.9.5.2

• Fixes a compromise on editing listen path in forced API defaults mode. Listen path now editable again.

UPDATE: Dashboard 0.9.5.1

• Includes bug fix for very large data sets (stats not showing).

The full changelog for 1.7:

• Open APIs now support caching, body transforms and header transforms
• Added RPC storage backend for cloud-based suport. RPC server is built in vayala/gorpc, signature for the methods that need to be provideda are in the rpc_storage_handler.go file (see the dispatcher).
• Added oauth_refresh_token_expire setting in configuration, allows for customisation of refresh token expiry in OAuth flows
• Changed refresh token expiry to be 14 days by default
• Basic swagger file supoprt in command line, use --import-swagger=petstore.json to import a swagger definition, will create a Whitelisted API.
• Created quota monitoring for orgs and user keys, uses a webhook. To configure update tyk.conf to include the global check rate and target data:

    "monitor": {
          "enable_trigger_monitors": false,
          "configuration": {
          "method": "POST",
              "target_path": "http://posttestserver.com/post.php?dir=tyk-monitor-test",
              "template_path": "templates/monitor_template.json",
              "header_map": {"x-tyk-monitor-secret": "12345"},
              "event_timeout": 10
          },
          "global_trigger_limit": 80.0,
          "monitor_user_keys": false,
          "monitor_org_keys": true
    }

• It is also possible to add custom rate monitors on a per-key basis, SessionObject has been updated to include a "monitor" section which lets you define custom limits to trigger a quota event, add this to your key objects:

    "monitor": {
          "trigger_limits": [80.0, 60.0, 50.0]
      }

• If a custom limit is the same as a global one the event will only fire once. The output will look like this:

    {
        "event": "TriggerExceeded",
        "message": "Quota trigger reached",
        "org": "53ac07777cbb8c2d53000002",
        "key": "53ac07777cbb8c2d53000002c74f43ddd714489c73ea5c3fc83a6b1e",
        "trigger_limit": "80",
    }

• Added response body transforms (JSON only), uses the same syntax as regular transforms, must be placed into transform_response list and the trasnformer must be registered under the new response_transforms list, otherwise it will not be activated.

    {
      name: "response_body_transform",
      options: {}
    }

• Added Response middleware chain and interface to handle response middleware. Response middleware must be declared under response_processors otherwise it is not loaded. Specifying options under the extended paths section will not be enough to enable response processors

    {
      name: "header_injector",
      options: {
        "add_headers": {"name": "value"},
        "remove_headers": ["name"]
      }
    }

• Added response header injection (uses the same code as the regular injector), add your path definitions to the extended_paths.transform_response_headers filed.
• Added SupressDefaultOrgStore - uses a default redis connection to handle unfound Org lookups, this is merely patching a potential hole
• Added support for Sentry DSN - monitoring can be added and enabled by adding to your tyk.conf:

    ...
    "use_sentry": true,
    "sentry_code": "https://your-dsn-string",
    ...

• Modification: Analyitcs purger (redis) now uses redis lists, much cleaner, and purge is a transaction which means multiple gateways can purge at the same time safely without risk of duplication
• Added enforce_org_data_age config parameter that allows for setting the expireAt in seconds for analytics data on an organisation level. (Requires the addition of a data_expires filed in the Session object that is larger than 0)
• Triggers now for dashboard/portal events, can be webhook or email:

    api_event: {
      webhook: "http://posttestserver.com/post.php?dir=tyk-events",
      email: "[email protected]"
    },
    key_event: {
      webhook: "http://posttestserver.com/post.php?dir=tyk-key-events",
      email: "[email protected]"
    },
    key_request_event: {
      webhook: "http://posttestserver.com/post.php?dir=tyk-key-events",
      email: "[email protected]"
    }

• Fixed bug: Basic auth users can nw be created
• Fixed bug: Basic Auth type can now be selected in API creation

Tyk 1.6 and Tyk Dashboard 0.9.4.5

UPDATE: Dashboard version 0.9.4.5 Hotfix: Large data sets now supported in analytics. Fixes bug where analytics do not show up.

Major release - now with a portal :-)

v1.6

• Added LDAP StorageHandler, enables basic key lookups from an LDAP service

• Added Policies feature, you can now define key policies for keys you generate:

  • Create a policies/policies.json file

  • Set the appropriate arguments in tyk.conf file:

    "policies": {
        "policy_source": "file",
        "policy_record_name": "./policies/policies.json"
    }
    

  • Create a policy, they look like this:

    {
        "default": {
            "rate": 1000,
            "per": 1,
            "quota_max": 100,
            "quota_renewal_rate": 60,
            "access_rights": {
                "41433797848f41a558c1573d3e55a410": {
                    "api_name": "My API",
                    "api_id": "41433797848f41a558c1573d3e55a410",
                    "versions": [
                        "Default"
                    ]
                }
            },
            "org_id": "54de205930c55e15bd000001",
            "hmac_enabled": false
        }
    }
    

  • Add a apply_policy_id field to your Session object when you create a key with your policy ID (in this case the ID is default)

  • Reload Tyk

  • Policies will be applied to Keys when they are loaded form Redis, and the updated i nRedis so they can be ueried if necessary

• Policies can invalidate whole keysets by copying over the InActive field, set this to true in a policy and all keys that have the policy set will be refused access.

• Added granular path white-list: It is now possible to define at the key level what access permissions a key has, this is a white-list of regex keys and apply to a whole API definition. Granular permissions are applied after version-based (global) ones in the api-definition. These granular permissions take the form a new field in the access rights field in either a policy definition or a session object in the new allowed_urls field:

  {
      "default": {
          "rate": 1000,
          "per": 1,
          "quota_max": 100,
          "quota_renewal_rate": 60,
          "access_rights": {
              "41433797848f41a558c1573d3e55a410": {
                  "api_name": "My API",
                  "api_id": "41433797848f41a558c1573d3e55a410",
                  "versions": [
                      "Default"
                  ],
                  "allowed_urls": [
                      {   
                          "url": "/resource/(.*),
                          "methods": ["GET", "POST"] 
                      }
                  ]
              }
          },
          "org_id": "54de205930c55e15bd000001",
          "hmac_enabled": false
      }
  }
  

• Added hash_keys config option. Setting this to true willc ause Tyk to store all keys in Redis in a hashed representation. This will also obfuscate keys in analytics data, using the hashed representation instead. Webhooks will continue to make the full API key available. This change is not backwards compatible if enabled on an existing installation.

• Added cache_options.enable_upstream_cache_control flag to API definitions

  • Upstream cache control is exclusive, caching must be enabled on the API, and the path to listen for upstream headers must be defined in the extended_paths section, otherwise the middleware will not activate for the path
  • Modified caching middleware to listen for two response headers: x-tyk-cache-action-set and x-tyk-cache-action-set-ttl.
  • If an upstream application replies with the header x-tyk-cache-action-set set to 1 (or anything non empty), and upstream control is enabled. Tyk will cache the response.
  • If the upstream application sets x-tyk-cache-action-set-ttl to a numeric value, and upstream control is enabled, the cached object will be created for whatever number of seconds this value is set to.

• Added auth.use_param option to API Definitions, set to tru if you want Tyk to check for the API Token in the request parameters instead of the header, it will look for the value set in auth.auth_header_name and is case sensitive

• Host manager now supports Portal NginX tempalte maangement, will generate portal configuration files for NginX on load for each organisation in DB

• Host manager will now gracefully attempt reconnect if Redis goes down

• Tyk will now reload on notifications from Redis (dashboard signal) for cluster reloads (see below), new option in config SuppressRedisSignalReload will suppress this behaviour (for example, if you are still using old host manager)

• Added new group reload endpoint (for management via LB), sending a GET to /tyk/reload/group will now send a pub/sub notification via Redis which will cause all listening nodes to reload gracefully.

• Host manager can now be set to manage Tyk or not, this means host manager can be deployed alongside NGinX without managing Tyk, and Tyk nodes reloading on their own using redis pub/sub

• Rate limiter now uses a rolling window, makes gaming the limiter by staddling the TTL harder

Hotfix Release for OAuth Server

OAuth server was having problems with client creation and extraction during authentication flow, this hotfix addresses this issue.

• There is a known bug in the Tyk Dashboard, OAuth client management does not work, the REST API should be used to generate new OAuth clients. This has been remedied in the master branch and will be fully unctional in version 1.6 (0.9.4 of Dashboard)

This release only has the binaries for the main supported linux architectures. They can be applied as a drop-in replacement to the binary on your system