Initial Pass Informix DBHelper #25

feenster · 2020-02-05T20:50:02Z

first pass informix DbHelper

volcan01010 · 2020-02-06T21:32:32Z

etlhelper/db_helpers/informix.py

@@ -0,0 +1,79 @@
+"""
+Database helper for Postgres


volcan01010 · 2020-02-06T21:32:41Z

etlhelper/db_helpers/informix.py

+
+class InformixDbHelper(DbHelper):
+    """
+    Postgres db helper class


volcan01010 · 2020-02-06T21:34:08Z

etlhelper/db_helpers/informix.py

+        """
+        Returns connection string for sql alchemy
+        """
+        password = self.get_password(password_variable)


Refactor for Informix connection string. This is prime TDD territory once you have manually figured out what the SQLAlchemy string is supposed to look like.

volcan01010 · 2020-02-06T21:34:32Z

etlhelper/db_helpers/informix.py

+        return (f'postgresql://{db_params.user}:{password}@'
+                f'{db_params.host}:{db_params.port}/{db_params.dbname}')
+
+    @staticmethod


Remove this - it's only required for PostgreSQL

volcan01010 · 2020-02-06T21:34:54Z

etlhelper/db_helpers/informix.py

+    def cursor(conn):
+        """
+        Return a cursor on current connection.  This implementation allows
+        SQLite cursor to be used as context manager as with other db types.


volcan01010 · 2020-02-06T21:36:39Z

requirements.txt

-pyodbc
-psycopg2-binary
-twine
+alabaster==0.7.12


Strip version numbers as we want to be as flexible as possible. It is probably easier just to add ibm-db and pytest-sugar to the existing list.

volcan01010 · 2020-02-06T21:37:52Z

test/unit/test_db_helpers.py

@@ -1,11 +1,12 @@
 """Unit tests for db_helpers module."""
-from unittest.mock import Mock
-import pytest
+import pyodbc


I'm not sure why pyodbc moved places here, but it should be in the group with the third-party imports, rather than the standard library ones.

volcan01010 · 2020-02-06T21:39:20Z

test/unit/test_db_helpers.py

@@ -26,6 +27,8 @@

 SQLITEDB = DbParams(dbtype='SQLITE', filename='/myfile.db')

+INFORMIXDB = DbParams(dbtype='INFORMIX', database='testDb', hostname='localhost', port='1111', protocol='tcpip',uid='user' )


Line length. Flake8 is set to 120, but less is better.

volcan01010 · 2020-02-06T21:41:01Z

test/unit/test_db_helpers.py

@@ -43,7 +46,8 @@ def test_oracle_connect_exceptions():
     'DRIVER=test driver;SERVER=tcp:server;PORT=1521;DATABASE=testdb;UID=testuser;PWD=mypassword'), # NOQA
    (POSTGRESDB, psycopg2,
     'host=server port=1521 dbname=testdb user=testuser password=mypassword'),
-    (SQLITEDB, sqlite3, '/myfile.db')
+    (SQLITEDB, sqlite3, '/myfile.db'),
+    (INFORMIXDB, ibm_db_dbi, 'database=testDb;hostname=localhost;port=1111;protocol=tcpip;uid=user;pwd=mypassword')


Line length. See MSSQLDB example above for line splitting and turning off Flake8 check for line.

volcan01010 · 2020-02-06T21:56:35Z

Thanks @feenster,

There are a few tweaks to make before this is ready to merge:

Flake8 failure in CI pipeline

The CI pipeline runs flake8 against the etlhelper and test directories. It found the following:

$ flake8 etlhelper test
etlhelper/db_helpers/informix.py:68:5: E303 too many blank lines (2)
test/unit/test_db_helpers.py:30:112: E231 missing whitespace after ','
test/unit/test_db_helpers.py:30:121: E501 line too long (124 > 120 characters)
test/unit/test_db_helpers.py:30:123: E202 whitespace before ')'

SQLAlchemy conn_string test and function

DbHelpers have a method to create a SQLAlchemy connection string, but we haven't implemented that yet for InformixDbHelper. Please add another parameterized test and make it pass.

Setup.py

setup.py defines extras_require options for each database type. Please add an entry for Informix. This will allow users to install Informix driver automatically with pip install etlhelper[informix].

README

Please add a line in the database drivers instructions for [informix]. Please also add a section for "IBM Informix" underneath the "pyodbc" section that explains that the driver is hard-coded to used protocol=tcpip and that this may need to be configured on the server.

See also the review comments on the changes to the files.

It was fun hacking on this yesterday.

volcan01010 · 2020-02-06T21:58:47Z

If you were super keen, you could add integration tests against a Dockerised instance. See CONTRIBUTING.md for details and test/integration/test_db/test_sqlite.py for a template. It's quite a bit more work, though and probably constitutes a separate ticket.

volcan01010 · 2020-05-25T21:31:33Z

Informix database notes.

Docker info: https://github.com/informix/informix-dockerhub-readme/blob/master/14.10.FC1/informix-developer-database.md

Docker run command:

docker run -it --name ifx -h ifx -p 9088:9088 -e LICENSE=accept ibmcom/informix-developer-database:latest

from etlhelper import connect, DbParams, execute, get_rows
import os

os.environ['IFX_PASSWORD'] = 'in4mix'
ifxdb = DbParams(dbtype="INFORMIX",
          hostname='localhost', port=9088,
          uid='informix', database='informix')

conn = connect(ifxdb, 'IFX_PASSWORD')

Gets this error:

ETLHelperConnectionError: Error connecting to database=informix;hostname=localhost;port=9088;protocol=tcpip;uid=informix;pwd=in4mix via dbapi: ibm_db_dbi::OperationalError: Exception('[IBM][CLI Driver] SQL30081N  A communication error has been detected. Communication protocol being used: "TCP/IP".  Communication API being used: "SOCKETS".  Location where the error was detected: "127.0.0.1".  Communication function detecting the error: "recv".  Protocol specific error code(s): "*", "*", "0".  SQLSTATE=08001 SQLCODE=-30081')

Why can't I connect on TCP/IP? The port is correct. And what should the database value be?

Initial Pass Informix DBHelper

790c52a

first pass informix DbHelper

volcan01010 requested changes Feb 6, 2020

View reviewed changes

dvalters changed the base branch from master to main August 20, 2020 11:35

volcan01010 mentioned this pull request Nov 19, 2020

Added Database class #97

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial Pass Informix DBHelper #25

Initial Pass Informix DBHelper #25

feenster commented Feb 5, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 Feb 6, 2020

volcan01010 commented Feb 6, 2020

volcan01010 commented Feb 6, 2020

volcan01010 commented May 25, 2020 •

edited

Loading

		@@ -26,6 +27,8 @@

		SQLITEDB = DbParams(dbtype='SQLITE', filename='/myfile.db')

		INFORMIXDB = DbParams(dbtype='INFORMIX', database='testDb', hostname='localhost', port='1111', protocol='tcpip',uid='user' )

Initial Pass Informix DBHelper #25

Are you sure you want to change the base?

Initial Pass Informix DBHelper #25

Conversation

feenster commented Feb 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

volcan01010 commented Feb 6, 2020

Flake8 failure in CI pipeline

SQLAlchemy conn_string test and function

Setup.py

README

volcan01010 commented Feb 6, 2020

volcan01010 commented May 25, 2020 • edited Loading

Informix database notes.

volcan01010 commented May 25, 2020 •

edited

Loading