v0.2.1

This release contains a fix to the flaky TestRunServer test, which used to fail the CI workflow every often. The other major change is the introduction of a new package, config, that contains almost all the constants from different packages, new types for unmarshaling koanf config into structs and getters for certain fields that has complex types or multiple values from different internal and external packages. Running run command loads the default values for plugins and global configs first before loading the corresponding config files.

A breaking change is the introduction of the default config object for loggers, clients, pool and proxy config. Later I'll add support for loading multiple config objects for running multiple objects of those types for dealing with different database servers (#83).

What's Changed

• Fix flaky TestRunServer test by @mostafa in #82
• New config parser by @mostafa in #84

Full Changelog: v0.2.0...v0.2.1

v0.2.0

This release contains enhacements to the plugin subsystem.

What's Changed

• Add cmdline args to plugin cmd by @mostafa in #75
• Pass environment variabless to plugins on load by @mostafa in #77
• Use koanf getter functions by @mostafa in #80
• Implement requires field for checking if required plugins are loaded or not by @mostafa in #81

Full Changelog: v0.1.4...v0.2.0

v0.1.4

Changes in this milestone (v0.1.x)

This is the latest release of the v0.1.x milestone that significantly contributed to the overall stability of GatewayD. A lot of changes has gone into this milestone including the following:

1. Fixed concurrent connection handling and management.
2. Changed parameter type of HookConfig.Run to receive map[string]interface{} and convert it to structpb.Struct internally to remove lots of boilerplate code.
3. Improved error handling by wrapping/unwrapping package-level errors into GatewayDError with lots of custom errors for different parts of the system.
4. Added more tests.
5. Improved client code by introducing chunk reading of the buffer to prevent hanging and introduction of deadlines (timeouts).
6. Added mechanism to retry the connection after close by the server.
7. Used structured logging everywhere.
8. Added comments to code and docstrings to all functions.
9. Improved traffic hooks by removing a lot of duplicate code and cleanups.
10. Added TCP Keep-Alive to client connections.
11. Partially fixed stale connection issues until the issue I investigated is resolved, as quoted below:

    PostgreSQL has an authentication timeout of 60s that waits till the client authenticates. The current design of GatewayD makes the TCP connection and waits for the connecting clients of GatewayD to authenticate via the proxied connection to PostgreSQL. This causes connections to become stale over time due to timeout. I need to start developing PostgreSQL-specific features for authentication and other related components, either with the plugins or the core.

Changes in the current release

In this release, I've tried to improve the code as best as I can. The connection stability is greatly improved, however the connections will somehow time out and become stale, although new connections will help re-instantiate the client object, which fixes the problem, but this is not enough. My fixes in this release help detect the issue earlier and close the ingress connection abruptly, which is not what I intend to do. The other change is improvements to hook system by addressing a few edge cases. I also added a TCP keepalive to the client connections.

What's Changed

• Traffic hooks improvements by @mostafa in #64
• Add TCP keepalive by @mostafa in #68
• Fix stale connection issue by @mostafa in #69

Full Changelog: v0.1.3...v0.1.4

v0.1.3

This release contains various improvements and fixes to different parts of the code. The retry mechanism is improved, but still needs more work, which I am gonna invest in the next few releases. I used structured logging and also added as much docstring and comments as I can to clarify everything.

What's Changed

• Retry connection after close by server by @mostafa in #58
• Use structured logging everywhere by @mostafa in #63
• Add docstring to all functions by @mostafa in #60

Full Changelog: v0.1.2...v0.1.3

v0.1.2

This release contains an important fix that significantly improves stability of the server (9a090cf) 🎉. I also added more tests and improved the overall client code by introducing configurable parameters for timeouts with defaults.

What's Changed

• Add tests by @mostafa in #53
• Improve client code by @mostafa in #54

Full Changelog: v0.1.1...v0.1.2

v0.1.1

This release contains two changes:

1. Change args parameter type in HookConfig.Run to accept map[string]interface{} instead of *structpb.Struct. This effectively removed a lot of boilerplate needed to deal with structpb.NewStruct.
2. Introduction of GatewayDError error type to wrap package errors and deal with special error cases.

What's Changed

• Change parameter type of HookConfig.Run by @mostafa in #51
• Improve error handling by @mostafa in #52

Full Changelog: v0.1.0...v0.1.1

v0.1.0

This release is the first in a series from the v0.1.x milestone. I tried to fix the concurrent connection handling and sudden freezes when testing GatwayD using psql command. This is unstable as is, and it needs more time to fix the underlying problem. There original issue presented two major problems:

1. Small buffer sizes caused the truncation of the result from the server, hence psql would consider it as chunked response and wait forever, hence freezing.
2. The previous event flow was erroneous. Connection handling and closing is the second major problem. Aborting the connection abruptly causes the clients to wait forever, hence freezing or abnormal behavior.

I will revisit this throughout the current milestone, and will try to fix it and make it more stable.

What's Changed

• Fix concurrent connection handling by @mostafa in #49

Full Changelog: v0.0.9...v0.1.0

v0.0.9

This release contains an adapter that redirects logs from the internal logger of the plugin system, hclog, to the global logger of GatewayD, zerolog. This release marks the end of the milestone 0.0.x, which was focused on laying the foundation for further development. I tried to achieve correctness and integrity in this milestone, however, there are some stability issues that I'll deal with in milestone 0.1.x. Going forward, documentation becomes an integral part of the project and will be written progressively.

Looking back at the previous releases, a lot has changed. The project was bootstrapped with a few lines of code as a proof of concept to mimic a L4 gateway. Now the GatewayD has a plug-based architecture with many components to help facilitate the inner-workings of the database gateway:

• Server: for accepting connections from the database clients, like psycopg2 in Python.
• Client: for connecting to databases using a raw TCP connection.
• Generic pool: used in a few places throughout the code to manage connections and plugins. The generic pool is a wrapper around the sync.Map from Go stdlib.
• Proxy: an object that connect servers and clients and passes traffic through. There are two types of proxies:
  • Fixed: a pool of clients will be created and connected to the database. They will be waiting ready to be proxied. The pool won't grow, so extra clients will be rejected. Queuing will be added in the future.
  • Elastic: an empty pool is created and each new connection from a database client creates a new connection to the database. The connections can be reused if chosen.
• Plugin system: can tap into the hooks and try to influence the behavior of the gateway by observe and manipulating data as it passes through between client and server. Plugins are managed by a plugin registry. The plugins can be written in any language that one can produce protobuf stubs and can interface through gRPC. The protocol file is available in plugin/v1/plugin.proto and the generated stubs for Go and the plugin interface are available in the same directory. There is a test plugin that shows how to work with hooks. Upon loading of each plugin, its dynamic configuration is read by calling its GetPluginConfig function. This effectively registers the list of hooks that plugin exposes with the hook system. All plugin functions (RPC) are unary for now.
• Hooks: callback function that plugins can register to. The server will run the hooks based on events, like onConfigLoaded and it causes the configuration options to be sent to plugins registered for that specific event. Each hook can trigger many plugins and they will be called based on their priority. The returned result of the hook from the plugin will be used to update the configuration dynamically (in-memory). Other hooks might have different side-effects. Some hooks are only notification hooks, so that the plugin can tell when an event happen, plus some pieces of information. The list of hooks are available here. The returned result of each hook from each registered plugin will be passed to the next plugin in the list.
  There is a verification function to check the signature of the parameters passed to the hook against the signature of the returned results and acts based on the policy chosen in the global config. There are two modes: strict and non-strict.
  • In strict mode, the parameter and return value signatures should match 1:1. In strict mode, either of the following three actions can be taken:
    1. ignore: ignore the result of the non-matching hook and continue with the next call to another plugin.
    2. abort: abort execution of the hook and return the last returned result from the previous hook.
    3. remove: ignores the result of the current registered hook and removes it from the registry, so the next call won't hit it again.
  • In non-strict, aka. permissive, mode, the parameters and the returned result should not match and the results of the hooks will be merged. This verification policy is called passdown.
• Logging: zerolog is used throughout the code to log events as they happen with various levels of importance. Loggers of other parts of the system has adapters to send logs to zerolog, as mentioned in What's changed? below.
• CLI: a run command with two options for reading: a) the global config file and b) the plugins config file.
• Config parser: there are two config file parsers that accept configuration in YAML format:
  • Global config (gatewayd.yaml): for setting up logger, clients, pool, proxy, server and plugins objects.
  • Plugin config (gatewayd_plugins.yaml): for adding the list of plugins to be loaded. Plugins priority is determined by the appearance of the plugin config in the file.
  • Dynamic config: there is a dynamic configuration hook, onConfigLoaded, to let plugins changes the global configuration before the server boots and after the global config is read, as demonstrated in the test plugin.
• Buf for managing protobuf files and generated stubs for the plugin system.
• Unit- and integration tests.
• CI workflows for a) testing unit-tests and b) testing integration with an example plugin.

What's Changed

• Add logging adapter for the plugin system by @mostafa in #45

Full Changelog: v0.0.8...v0.0.9

v0.0.8

This release includes several changes including the plugin system, major refactoring and adding tests. A new test plugin, gatewayd-plugin-test, is developed to help developers understand how to write a plugin for GatewayD in Go. Plugins can be written in practically any language that can produce and consume protobuf stubs and expose a gRPC interface. For various reasons, the Hashicorp's go-plugin is used to develop the plugin system.

What's Changed

• Refactorings before implementation of the plugin system by @mostafa in #39
• Plugin system by @mostafa in #40
• Add more tests by @mostafa in #42

Full Changelog: v0.0.7...v0.0.8

v0.0.7

I have heavily refactored the hook system and resolved a lot of issues and bugs I found along the way and added tests wherever needed. These are the major changes:

• buf for linting, building and generate stubs from protocol buffer files.
• Hashicorp's go-plugin is used to build the plugin system, as explained in #8.
• Introduce chaining of hook results (for use in chaining results of function in the plugins registered to each hook with their priority)
• Merge config from disk with the one retrieved from the plugins registered to the OnConfigLoaded hook.

What's Changed

• Refactor hook system by @mostafa in #35
• Non-strict (permissive) verification policy by @mostafa in #38

Full Changelog: v0.0.6...v0.0.7