Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#3287: Update puppet5 + python3 #3288

Open
wants to merge 64 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
43f5865
#3287: base.dockerfile, install puppet-agent for 'xenial'
jeff1evesque Oct 1, 2018
512644a
#3287: base.dockerfile, update 'git' + 'wget'
jeff1evesque Oct 1, 2018
d7af0cb
#3287: base.dockerfile, install ruby + update gems
jeff1evesque Oct 1, 2018
248b537
#3287: base.dockerfile, consolidate syntax
jeff1evesque Oct 1, 2018
e3f91b2
#3287: base.dockerfile, install wget before use
jeff1evesque Oct 2, 2018
ad8eb63
#3287: base.dockerfile, 'yum update' before 'install wget'
jeff1evesque Oct 2, 2018
b463d1c
#3287: remove 'puppet-python' implementation
jeff1evesque Oct 2, 2018
952cae7
#3287: base.dockerfile, update install + r10k syntax
jeff1evesque Oct 2, 2018
4de1de1
#3287: packages.yml, update package versions
jeff1evesque Oct 2, 2018
7ec8c89
#3287: Merge branch 'feature-3287' into local feature-3287
jeff1evesque Oct 2, 2018
a1ae19f
#3287: packages.pp, use 'pip3' provider
jeff1evesque Oct 2, 2018
cf0ad87
#3287: packages.yml, ensure 'six' at '1.11.0'
jeff1evesque Oct 2, 2018
c915bd3
#3287: mariadb.dockerfile, use 'python3'
jeff1evesque Oct 2, 2018
f3f016f
#3287: entrypoint.erb, use 'python3'
jeff1evesque Oct 2, 2018
ac907b7
#3287: packages.yaml, enforce 'python3'
jeff1evesque Oct 2, 2018
871915a
#3287: update containers to '0.8'
jeff1evesque Oct 2, 2018
7b19af2
#3287: update python 'print' to python3 syntax
jeff1evesque Oct 3, 2018
6fd002b
#3287: hardcode 'system::packages' dependency
jeff1evesque Oct 3, 2018
2b66f97
#3287: install.pp, require 'system::packages'
jeff1evesque Oct 3, 2018
05c8f0e
#3287: backend, remove all docker images
jeff1evesque Oct 3, 2018
e2c402f
#3287: Merge branch 'feature-3287' into local feature-3287
jeff1evesque Oct 3, 2018
eb3311c
#3287: install mongodb using xenial references
jeff1evesque Oct 3, 2018
bfaa20b
#3287: packages.yml, fix 'flask' package typo
jeff1evesque Oct 4, 2018
aeb26dd
#3287: flatten docker 'ENV' as one line
jeff1evesque Oct 4, 2018
7710436
#3287: packages.yml, add 'apt-utils'
jeff1evesque Oct 4, 2018
2c10f79
#3287: ensure proper docker 'ENV' assignment
jeff1evesque Oct 4, 2018
0b0211d
#3287: ensure docker 'ENV' properly defined
jeff1evesque Oct 4, 2018
3e5a877
#3287: packages.yaml, add apt-transport-https'
jeff1evesque Oct 4, 2018
80e9865
#3287: install.pp, add '/bin' to 'path'
jeff1evesque Oct 4, 2018
b9a346a
#3287: conform 'exception' to python3 syntax
jeff1evesque Oct 5, 2018
971814f
#3287: install_rancher, update cli, agent, server
jeff1evesque Oct 6, 2018
825c5f5
#3287: query.py, ensure 'except' uses python3 syntax
jeff1evesque Oct 6, 2018
2c56f99
#3287: query.py, use python3 'except' syntax
jeff1evesque Oct 7, 2018
a49c0e0
#3287: query.py, use python3 'except' syntax
jeff1evesque Oct 7, 2018
7a6223f
#3287: Puppetfile, updaate packages
jeff1evesque Oct 7, 2018
68623f2
#3287: packages.yml, ensure 'MySQLdb' installs
jeff1evesque Oct 7, 2018
204e1ed
#3287: packages.yml, ensure mariadb client for python3x
jeff1evesque Oct 7, 2018
88e0034
#3287: packages.yml, remove 'mysqlclient'
jeff1evesque Oct 7, 2018
7b64a3e
#3287: server.pp, add 'service_name' per docs
jeff1evesque Oct 7, 2018
c2698cf
#3287: hset.py, ensure 'print' python3 syntax
jeff1evesque Oct 7, 2018
c05734f
#3287: model.py, ensure 'print' python3 syntax
jeff1evesque Oct 7, 2018
56886e1
#3287: README.md, fix port references
jeff1evesque Oct 8, 2018
0462e80
#3287: nginx.dockerfile, fix docstring
jeff1evesque Oct 9, 2018
98f0919
#3287: nginx-xxx.yaml, remove 'host_port'
jeff1evesque Oct 9, 2018
5c914f3
#3287: remove unused 'host_port' from puppet manifests
jeff1evesque Oct 9, 2018
64d5c9e
#3287: settings.py, ensure python 2+3 string compatibility
jeff1evesque Oct 9, 2018
4c6c0ec
#3287: dataset.py, ensure python 2+3 string compatibility
jeff1evesque Oct 9, 2018
950f10e
#3287: ensure sql arguments supplied as tuple
jeff1evesque Oct 10, 2018
05ab4b9
#3287: base_data.py, remove 'unorderable types' error
jeff1evesque Oct 10, 2018
6e8a29a
#3287: prediction.py, ensure tuple sql argument
jeff1evesque Oct 10, 2018
fad91d6
#3287: base_data.py, remove 'unorderable types' error
jeff1evesque Oct 10, 2018
7e26189
#3287: remove 'unorderable types' error
jeff1evesque Oct 10, 2018
05ed2a3
#3287: eliminate 'elif' with 'int' casting using 'or'
jeff1evesque Oct 11, 2018
8f8ad6a
#3287: properly validate 'string_types[0]'
jeff1evesque Oct 11, 2018
d75f6ee
"#3287: Merge branch 'feature-3287' into local feature-3287"
jeff1evesque Oct 11, 2018
d397806
#3287: remove 'client.pp' mariadb implementation
jeff1evesque Oct 11, 2018
0ac6bb9
#3287: mariadb.dockerfile, temporarily add client
jeff1evesque Oct 11, 2018
3cbe7c0
#3287: dataset.py, coerce 'float' before validation
jeff1evesque Oct 12, 2018
afed9d2
#3287: dataset.py, validate 'error' key
jeff1evesque Oct 12, 2018
598639e
#3287: dataset.py, add 'elif' to cast int
jeff1evesque Oct 13, 2018
17b650c
#3287: dataset.py, cast 'result' to int
jeff1evesque Oct 13, 2018
03de240
#3287: Merge branch 'feature-3287' into local feature-3287
jeff1evesque Oct 13, 2018
89fa672
#3287: eliminate collection + document count comparison
jeff1evesque Oct 13, 2018
f960675
#3287: remove redundant new-style 'object' declarations
jeff1evesque Oct 14, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Both the web-interface, and the programmatic-api, have corresponding
### Web Interface

The [web-interface](https://github.com/jeff1evesque/machine-learning/blob/master/interface/templates/index.html),
can be accessed within the browser on `https://192.168.99.101:8080`:
can be accessed within the browser on `https://192.168.99.101:8585`:

![web-interface](https://user-images.githubusercontent.com/2907085/39499223-97b96fce-4d7a-11e8-96e2-c4e31f6b8e09.JPG 'web-interface')

Expand Down Expand Up @@ -135,7 +135,7 @@ A post request, can be implemented in python, as follows:
```python
import requests

endpoint = 'https://192.168.99.101:9090/load-data'
endpoint = 'https://192.168.99.101:9595/load-data'
headers = {
'Authorization': 'Bearer ' + token,
'Content-Type': 'application/json'
Expand All @@ -151,6 +151,6 @@ requests.post(endpoint, headers=headers, data=json_string_here)

It is important to remember that the [`docker-compose.development.yml`](https://github.com/jeff1evesque/machine-learning/blob/3889788a8343a4b7cef2cf84166f9bd35d83021c/docker-compose.development.yml#L33-L43),
has defined two port forwards, each assigned to its corresponding reverse
proxy. This allows port `8080` on the host, to map into the `webserver-web`
container. A similar case for the programmatic-api, uses port `9090` on the
proxy. This allows port `8585` on the host, to map into the `webserver-web`
container. A similar case for the programmatic-api, uses port `9595` on the
host.
12 changes: 5 additions & 7 deletions brain/cache/hset.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,13 @@
from brain.cache.query import Query


class Hset(object):
class Hset:
'''

This class provides an interface to cache, and uncache the redis hash
data structure. Specifically, necessary data components is passed into the
corresponding class method.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self):
Expand All @@ -36,7 +34,7 @@ def __init__(self):
# start redis client
try:
self.myRedis.start_redis()
except Exception, error:
except Exception as error:
self.list_error.append(str(error))

def cache(self, hash_name, key, value):
Expand All @@ -48,9 +46,9 @@ def cache(self, hash_name, key, value):

try:
self.myRedis.hset(hash_name, key, value)
except Exception, error:
except Exception as error:
self.list_error.append(str(error))
print self.list_error
print(self.list_error)

def uncache(self, hash_name, key):
'''
Expand All @@ -61,6 +59,6 @@ def uncache(self, hash_name, key):

try:
return {'result': self.myRedis.hget(hash_name, key), 'error': None}
except Exception, error:
except Exception as error:
self.list_error.append(str(error))
return {'result': None, 'error': self.list_error}
12 changes: 5 additions & 7 deletions brain/cache/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,14 @@
from brain.converter.model import Model as Converter


class Model(object):
class Model:
'''

This class provides an interface to cache, and uncache the redis hash
data structure. Specifically, necessary data components is passed into the
corresponding class method, which allow computed model(s) to be stored into
a NoSQL datastore.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, model=None):
Expand All @@ -39,7 +37,7 @@ def __init__(self, model=None):
# start redis client
try:
self.myRedis.start_redis()
except Exception, error:
except Exception as error:
self.list_error.append(str(error))

def cache(self, hash_name, key):
Expand All @@ -53,9 +51,9 @@ def cache(self, hash_name, key):
try:
serialized = Converter(self.model).serialize()
self.myRedis.hset(hash_name, key, serialized)
except Exception, error:
except Exception as error:
self.list_error.append(str(error))
print self.list_error
print(self.list_error)

def uncache(self, hash_name, key):
'''
Expand Down Expand Up @@ -96,6 +94,6 @@ def get_all_titles(self, name):
'error': 'no previous model found in cache'
}

except Exception, error:
except Exception as error:
self.list_error.append(str(error))
return {'result': None, 'error': self.list_error}
4 changes: 1 addition & 3 deletions brain/cache/query.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from brain.cache.settings import Settings


class Query(object):
class Query:
'''

This class provides an interface to various Redis data structures.
Expand All @@ -32,8 +32,6 @@ class Query(object):
concept to 'lists'. Also, the above included redis data structures,
provides enough flexibility to accomplish most requirements.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, db_num=0, host=None, port=None):
Expand Down
4 changes: 1 addition & 3 deletions brain/cache/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,12 @@
from flask import current_app


class Settings(object):
class Settings:
'''

This class provides an interface to get, or set the redis host, or redis
port.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self):
Expand Down
4 changes: 1 addition & 3 deletions brain/converter/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,12 @@
from sklearn import svm, preprocessing


class Model(object):
class Model:
'''

This class provides an interface to serialize, and deserialize an SVM
object.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, model):
Expand Down
4 changes: 1 addition & 3 deletions brain/converter/operation.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,13 @@
'''


class Operation(object):
class Operation:
'''

This class provides an interface to restructure the supplied data into a
consistent structure, which allows successive parsers to implement
corresponding logic.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, settings, dataset=None):
Expand Down
4 changes: 1 addition & 3 deletions brain/converter/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,13 @@
'''


class Settings(object):
class Settings:
'''

This class provides an interface to restructure the supplied data into a
consistent structure, which allows successive parsers to implement
corresponding logic.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, settings, dataset=None):
Expand Down
12 changes: 5 additions & 7 deletions brain/database/account.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,11 @@
from brain.database.query import SQL


class Account(object):
class Account:
'''

This class provides an interface to the users account.

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self):
Expand Down Expand Up @@ -71,7 +69,7 @@ def check_username(self, username):
sql_statement = 'SELECT * '\
'FROM tbl_user '\
'WHERE username=%s'
args = (username)
args = (username,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand All @@ -95,7 +93,7 @@ def check_email(self, email):
sql_statement = 'SELECT * '\
'FROM tbl_user '\
'WHERE email=%s'
args = (email)
args = (email,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand All @@ -119,7 +117,7 @@ def get_password(self, username):
sql_statement = 'SELECT password '\
'FROM tbl_user '\
'WHERE username=%s'
args = (username)
args = (username,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand All @@ -143,7 +141,7 @@ def get_uid(self, username):
sql_statement = 'SELECT id_user '\
'FROM tbl_user '\
'WHERE username=%s'
args = (username)
args = (username,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand Down
10 changes: 7 additions & 3 deletions brain/database/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,14 @@
from brain.database.query import NoSQL


class Collection(object):
class Collection:
'''

This class provides an interface to retrieve, and store various parameters,
of the specified collection, from the mongodb.

Note: this class is invoked within 'base_data.py'

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self):
Expand Down Expand Up @@ -55,6 +53,12 @@ def query(self, collection, operation, payload=None):
if operation == 'drop_collection':
self.nosql.connect()
response = self.nosql.execute(operation, collection)

elif operation == 'count_documents':
self.nosql.connect(collection)
response = self.nosql.execute(operation, payload)
response['result'] = int(response['result'] or 0)

else:
self.nosql.connect(collection)
response = self.nosql.execute(operation, payload)
Expand Down
12 changes: 5 additions & 7 deletions brain/database/entity.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from brain.database.query import SQL


class Entity(object):
class Entity:
'''

This class provides an interface to save, retrieve an SVM entity title,
Expand All @@ -19,8 +19,6 @@ class Entity(object):
Note: this class is invoked within 'model_generate.py', 'base_data.py',
and 'data_append.py'

Note: this class explicitly inherits the 'new-style' class.

'''

def __init__(self, premodel_data=None, session_type=None):
Expand Down Expand Up @@ -106,7 +104,7 @@ def get_title(self, id_entity):
sql_statement = 'SELECT title '\
'FROM tbl_dataset_entity '\
'WHERE id_entity=%s'
args = (id_entity)
args = (id_entity,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand Down Expand Up @@ -135,7 +133,7 @@ def get_collections(self, uid):
'FROM tbl_dataset_entity '\
'WHERE uid_created=%s '\
'ORDER BY datetime_created'
args = (uid)
args = (uid,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand Down Expand Up @@ -166,7 +164,7 @@ def get_collection_count(self, uid):
'FROM tbl_dataset_entity '\
'WHERE uid_created=%s'\
') AS c'
args = (uid)
args = (uid,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand All @@ -176,7 +174,7 @@ def get_collection_count(self, uid):
if response_error:
return {'error': response_error, 'result': None}
else:
return {'error': None, 'result': int(response['result'][0][0])}
return {'error': None, 'result': int(response['result'][0][0] or 0)}

def remove_entity(self, uid, collection):
'''
Expand Down
6 changes: 3 additions & 3 deletions brain/database/model_type.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from brain.database.query import SQL


class ModelType(object):
class ModelType:
'''

Note: this class explicitly inherits the 'new-style' class.
Expand All @@ -20,7 +20,7 @@ class ModelType(object):
def __init__(self):
'''

This constructor is responsible for defining class variables.
This class provides and interface between the model_type.

'''

Expand Down Expand Up @@ -49,7 +49,7 @@ def get_model_type(self, collection):
' INNER JOIN tbl_model_type mtype'\
' ON mid.model_type = mtype.id_model'\
' WHERE mid.collection=%s'
args = (collection)
args = (collection,)
response = self.sql.execute('select', sql_statement, args)

# retrieve any error(s)
Expand Down
Loading