Skip to content

Commit

Permalink
Add ability to pass extra server args to postgres_fdw:
Browse files Browse the repository at this point in the history
```
sgr mount postgres_fdw schema -c [connstr] -o@- <<EOF
{
  "extra_server_args": {
    "use_remote_estimate": "true",
	"extensions": "postgis",
	"fetch_size": "10000"
  }
}
EOF
```

Add ability to rename object files in-engine (utility function for some ingestion).

Allow disabling `IMPORT FOREIGN SCHEMA` and passing a table schema in Postgres/MySQL FDWs. For example:

```
sgr mount mysql_fdw schema -c [connstr] -o@- <<EOF
{
  "remote_schema": "mysql_schema",
  "tables": {
    "table_1": {
      "t1_col_1": "integer",
      "t1_col_2": "text"
    },
    "table_2": {
      "t2_col_1": "bigint",
      "t2_col_2": "integer"
    }
  }
}
EOF
```

Add a fork (https://github.com/splitgraph/postgres-elasticsearch-fdw) of https://github.com/matthewfranglen/postgres-elasticsearch-fdw to `sgr mount`, letting others mount ES indexes:

```
sgr mount elasticsearch -c elasticsearch:9200 -o@- <<EOF
{
  "table_spec": {
    "table_1": {
      "schema": {
        "id": "text",
        "@timestamp": "timestamp",
        "query": "text",
        "col_1": "text",
        "col_2": "boolean",
      }
      "index": "index-pattern*",
      "rowid_column": "id",
      "query_column": "query",
    }
  }
}
EOF
```

Differences:

  * Pass qualifiers as ElasticSearch queries using the query DSL (was using the `query=...` qual as a Lucene query string, which is useless in JOINs. Now we combine both the query implied from the quals and the Lucene query string, if passed)
  * Close the search context on `end_scan` (otherwise many ES queries to the FDW in a 10 minute span would cause it to error with a "too many scroll contexts" exception)
  * Add EXPLAIN support (outputs the used ES query)
  • Loading branch information
mildbyte committed Sep 16, 2020
1 parent 57ba32f commit a1628da
Show file tree
Hide file tree
Showing 5 changed files with 16 additions and 4 deletions.
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[bumpversion]
commit = True
tag = True
current_version = 0.2.1
current_version = 0.2.2
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+))?
serialize =
{major}.{minor}.{patch}-{release}
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Changelog

## v0.2.2 (2020-09-16)

* Add ability to pass extra server args to postgres_fdw (`extra_server_args`)
* Add ability to rename object files in-engine (utility function for some ingestion).
* Allow disabling `IMPORT FOREIGN SCHEMA` and passing a table schema in Postgres/MySQL FDWs.
* Add a fork (https://github.com/splitgraph/postgres-elasticsearch-fdw) of https://github.com/matthewfranglen/postgres-elasticsearch-fdw to `sgr mount`, letting others mount ES indexes. Fork changes:
* Pass qualifiers as ElasticSearch queries using the query DSL (was using the `query=...` qual as a Lucene query string, which is useless in JOINs. Now we combine both the query implied from the quals and the Lucene query string, if passed)
* Close the search context on `end_scan` (otherwise many ES queries to the FDW in a 10 minute span would cause it to error with a "too many scroll contexts" exception)
* Add EXPLAIN support (outputs the used ES query)

Full set of changes: [`v0.2.1...v0.2.2`](https://github.com/splitgraph/splitgraph/compare/v0.2.1...v0.2.2)

## v0.2.1 (2020-09-02)

* Add ability to skip config injection at the end of config-manipulating functions (pass `-s`) and don't fail if the Docker socket isn't reachable
Expand Down
2 changes: 1 addition & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

set -eo pipefail

SGR_VERSION=${SGR_VERSION-0.2.1}
SGR_VERSION=${SGR_VERSION-0.2.2}
INSTALL_DIR=${INSTALL_DIR-$HOME/.splitgraph}
# Set IGNORE_SGR_EXISTS to keep going if sgr already exists.
# Set SKIP_BINARY=1 to skip downloading sgr
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "splitgraph"
version = "0.2.1"
version = "0.2.2"
description = "Command line library and Python client for Splitgraph, a version control system for data"
license = "Apache 2.0 modified with Commons Clause"
authors = ["Splitgraph Limited"]
Expand Down
2 changes: 1 addition & 1 deletion splitgraph/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.2.1"
__version__ = "0.2.2"

0 comments on commit a1628da

Please sign in to comment.