feat: pbench-server REST API

Added pbench-server to be executed by systemctl.
Additional changes required on RPM build spec.

feat: agent side of transfer REST

feat: moved util-scripts to own utils dir

final prototype for move_results

feat: moved read in chunks and post to copy_results_tb

reverted util-scripts changes

fixed revert

feat: refactored pbench-make-result-tb to python

feat: converted copy-results-tb

added get on pbench-server for check sum

fix: formatting on logger

feat: added first test for pbench-server

added test for post

added tests plus mod gitignore

feat: fixed and moved tests around for pbench-server REST

feat: fixes to tests plus metadata.log fixure cleanup

general refactoring

feat: reafactored and added tests

feat: gral tests refactoring

removed config invocation for agent

added host_info entrypoint test

feat: tests refactoring

feat: added pytests to travis

fix: travisyml conflicts

fix: removed new line on init

fix: travis yml python path

fix: travis yml python script

fix: added missing libraries for travis

fix: added werkzeug library to travis install

fix: moved python rpm package to be systems one

fix: added system site packages for travis virtualenv

feat: added tests and pytest.ini

fix: log formatting plus added tests

fix: fixes to TCRT tests

Addressing review comments

Moved agent utils to lib/pbench and config ammendments

fix to travis.yml and tests references

general refactoring

Added libs to pythonpath for travis

removed unused import

fix to travis configtools import

Added _PBENCH_*_CONFIG env variables to travis

Fix for fixtures paths

fix on test fixture variables

fix for travis env yaml

fix for server tests

Refactored config on server side to accomodate to pytest

moved logger to init

fixed configtools refactor

fix for config fixture install_dir

Fix for test copy results

fix to server tests

Fixes to server tests

fixed flake8 issues

black lint fixes

made config methods

blacked

Fixes to config

Added argparse for move_results plus removed check entrypoint for server

Addressing comments from PR

Fixes to agent tests

fix for test copy result

Fixes to test move results fixtures

fix to tox pythonpath

fix for pytest execution

Fixed requirements/removed pytest-flask

running pytest via python

Fixes to pythonpath for tox

fixes to tox pythonpath

moved pythonpath to envdir in tox

Added build-system to pytproject toml

added requires to build-system

Fix configtools import

fix configtools import

moved rpm package to pipy

removed sitepackages from tox

Fixes to fixtures

removed rpy-py-installer

Removed rpm for host info

modified env from VIRTUAL_ENV to PYTHONPATH

Fix for server test setup

removed fixtures scope plus redundant packages on test-requirements

Added fixture decorator

Added pytest-flask package

fixes to server conftest

blacked

fixes to server tests

Final fixes to tests

blacked

fixes to add-metalog-option

additional fixes

.gitignore

@@ -14,3 +14,5 @@ dist
 pip-wheel-metadata
 AUTHORS
 ChangeLog
+.idea/
+.pytest_cache/

.travis.yml

@@ -7,6 +7,8 @@ env:
   - TOXENV=lint
   - TOXENV=agent
   - TOXENV=server
+  - TOXENV=pytest_agent
+  - TOXENV=pytest_server
   - TOXENV=py3
 before_install:
  - sudo apt-get update

agent/config/pbench-agent-default.cfg

@@ -1,15 +1,22 @@
 [DEFAULT]
 version = 002
+pbench_web_server = pbench.example.com
 
 [results]
 user = pbench
 host_path = http://%(pbench_result_redirector)s/pbench-archive-host
 webserver = %(pbench_web_server)s
-host_info_url = http://%(webserver)s/pbench-results-host-info.versioned/pbench-results-host-info.URL002
+host_info_uri = pbench-results-host-info.versioned/pbench-results-host-info.URL002
+host_info_url = http://%(webserver)s/%(host_info_uri)s
 dir = /srv/pbench/public_html/incoming
 scp_opts = -o StrictHostKeyChecking=no
 ssh_opts = -o StrictHostKeyChecking=no
 
+# REST API entrypoint
+api_version = 1
+rest_endpoint = api/v%(api_version)s
+server_rest_url = http://%(webserver)s/%(rest_endpoint)s
+
 [pbench-agent]
 install-dir = %(pbench_install_dir)s
 pbench_user = pbench

agent/lib/pbench/__init__.py

@@ -0,0 +1,7 @@
+import logging
+import sys
+
+logger = logging.getLogger(__name__)
+logger.addHandler(logging.StreamHandler(sys.stdout))
+logger.propagate = False
+logging.basicConfig(level=logging.INFO, format="%(message)s")

agent/lib/pbench/add_metalog_option.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+
+# Usage: pbench-add-metalog-option <metadata log file> <section> <option>
+
+# Add an option to a section of the metadata.log file.
+# E.g. using an 'iterations' arg for the option
+#
+# iterations: 1-iter, 2-iter, 3-iter
+#
+# where the iterations are in the <iterations.file>, one iteration per line.
+import errno
+import sys
+from configparser import ConfigParser, NoSectionError
+
+
+def add_metalog_option(log_file, section, option, value):
+    config = ConfigParser()
+
+    try:
+        config.read(log_file)
+    except OSError as ex:
+        if ex.errno == errno.ENOENT:
+            raise Exception(f"Log file does not exist: {log_file}")
+        raise
+
+    try:
+        config.set(section, option, value)
+    except NoSectionError:
+        config.add_section(section)
+        config.set(section, option, value)
+    config.write(open(log_file, "w"))
+
+
+if __name__ == "__main__":
+    _log_file = sys.argv[1]
+    _section = sys.argv[2]
+    _option = sys.argv[3]
+    _value = sys.argv[4]
+    add_metalog_option(_log_file, _section, _option, _value)

agent/lib/pbench/config.py

@@ -0,0 +1,16 @@
+from pbench.common import configtools
+
+
+class AgentConfig:
+    def __init__(self):
+        opts, _ = configtools.parse_args()
+        pbench_config, _ = configtools.init(opts, "_PBENCH_AGENT_CONFIG")
+
+        self.agent = pbench_config["pbench-agent"]
+        self.results = pbench_config["results"]
+
+    def get_agent(self):
+        return self.agent
+
+    def get_results(self):
+        return self.results

agent/lib/pbench/copy_result_tb.py

@@ -0,0 +1,71 @@
+import os
+import sys
+from pathlib import Path
+
+import requests
+
+from agent.lib.pbench import logger
+from agent.lib.pbench.config import AgentConfig
+from werkzeug.utils import secure_filename
+
+
+class CopyResultTb:
+    def __init__(self, tarball, path, _logger=None):
+        config = AgentConfig()
+        server_rest_url = config.results.get("server_rest_url")
+
+        self.tarball = tarball
+        self.path = path
+        self.chunk_size = 4096
+        self.logger = logger if not _logger else _logger
+        self.upload_url = f"{server_rest_url}/upload"
+        self.controller_dir = os.path.dirname(self.tarball)
+        self.controller = os.path.basename(self.controller_dir)
+
+    def read_in_chunks(self, file_object):
+        while True:
+            data = file_object.read(self.chunk_size)
+            if not data:
+                break
+            yield data
+
+    def post_file(self, file):
+        filename = secure_filename(os.path.join(self.path, file))
+        with open(f"{os.path.join(self.path, file)}.md5.check", "r") as _check:
+            md5sum = _check.read()
+        headers = {"filename": filename, "md5sum": md5sum}
+        content_path = os.path.abspath(file)
+        with open(content_path, "rb") as f:
+            try:
+                response = requests.post(
+                    self.upload_url, data=self.read_in_chunks(f), headers=headers
+                )
+                self.logger.info("File uploaded successfully")
+            except Exception:
+                self.logger.exception("There was something wrong with your request")
+                sys.exit(1)
+        if not response.status_code == 200:
+            self.logger.error("There was something wrong with your request")
+            self.logger.error("Error code: %s" % response.status_code)
+            sys.exit(1)
+
+    def copy_result_tb(self):
+        if not os.path.exists(self.tarball):
+            self.logger.error("tarball does not exist, %s" % self.tarball)
+            sys.exit(1)
+        if not os.path.exists(f"{self.tarball}.md5.check"):
+            self.logger.error(
+                "tarball's .md5.check does not exist, %s.md5.check" % self.tarball
+            )
+            sys.exit(1)
+
+        files = [file for file in Path(self.controller_dir).iterdir() if file.is_file()]
+        file_count = len(files)
+        if file_count != 2:
+            self.logger.error(
+                "(internal): unexpected file count, %s, associated with tarball, %s"
+                % (file_count, self.tarball)
+            )
+            sys.exit(1)
+
+        self.post_file(self.tarball)

agent/lib/pbench/make_result_tb.py

@@ -0,0 +1,124 @@
+import hashlib
+import os
+import sys
+import tarfile
+import datetime
+
+from agent.lib.pbench import logger
+from agent.lib.pbench.add_metalog_option import add_metalog_option
+from agent.lib.pbench.config import AgentConfig
+from pathlib import Path
+from pbench.common import configtools
+
+
+class MakeResultTb:
+    def __init__(self, result_dir, target_dir, user, prefix, _logger=None):
+        self.result_dir = result_dir
+        self.target_dir = target_dir
+        self.user = user
+        self.prefix = prefix
+        self.logger = logger if not _logger else _logger
+
+    def make_result_tb(self):
+        config = AgentConfig()
+
+        if not os.path.exists(self.result_dir):
+            self.logger.error("Invalid result directory provided: %s" % self.result_dir)
+            sys.exit(1)
+
+        if not os.path.exists(self.target_dir):
+            self.logger.error("Invalid target directory provided: %s" % self.target_dir)
+            sys.exit(1)
+
+        full_result_dir = os.path.realpath(self.result_dir)
+        pbench_run_name = os.path.basename(self.result_dir)
+        if os.path.exists(f"{full_result_dir}.copied"):
+            self.logger.debug("Already copied %s" % self.result_dir)
+            sys.exit(0)
+
+        if os.path.exists(f"{full_result_dir}/.running"):
+            self.logger.debug(
+                "The benchmark is still running in %s - skipping" % pbench_run_name
+            )
+            self.logger.debug(
+                "If that is not true, rmdir %s/.running, and try again"
+                % pbench_run_name
+            )
+            sys.exit(0)
+
+        md_log = f"{full_result_dir}/metadata.log"
+        if not os.path.exists(md_log):
+            self.logger.debug(
+                "The %s/metadata.log file seems to be missing", pbench_run_name
+            )
+            sys.exit(0)
+
+        opts, _ = configtools.parse_args()
+        opts.filename = md_log
+        conf_md, _ = configtools.init(opts, None)
+        md_config = conf_md["pbench"]
+        res_name = md_config.get("name")
+        pbench_run = config.agent.get("pbench_run")
+        if res_name != pbench_run_name:
+            self.logger.warning(
+                "The run in directory %s/%s "
+                "has an unexpected metadata name, '%s' - skipping"
+                % (pbench_run, pbench_run_name, res_name)
+            )
+            sys.exit(1)
+
+        if self.user:
+            add_metalog_option(md_log, "run", "user", self.user)
+
+        if self.prefix:
+            add_metalog_option(md_log, "run", "prefix", self.prefix)
+
+        result_size = sum(
+            file.stat().st_size for file in Path(full_result_dir).rglob("*")
+        )
+        self.logger.debug(
+            "preparing to tar up %s bytes of data from %s"
+            % (result_size, full_result_dir)
+        )
+        add_metalog_option(md_log, "run", "raw_size", f"{result_size}")
+
+        timestamp = datetime.datetime.isoformat(datetime.datetime.now())
+        add_metalog_option(
+            md_log, "pbench", "tar-ball-creation-timestamp", f"{timestamp}"
+        )
+
+        tarball = os.path.join(self.target_dir, f"{pbench_run_name}.tar.xz")
+        files = [os.path.realpath(file) for file in Path(full_result_dir).rglob("*")]
+        try:
+            with tarfile.open(tarball, mode="x:xz") as tar:
+                for f in files:
+                    tar.add(f)
+        except tarfile.TarError:
+            self.logger.error(
+                "tar ball creation failed for %s, skipping" % self.result_dir
+            )
+            if os.path.exists(tarball):
+                os.remove(tarball)
+            sys.exit(1)
+
+        self.make_md5sum(tarball)
+
+        return tarball
+
+    def make_md5sum(self, tarball):
+        tarball_md5 = f"{tarball}.md5.check"
+        try:
+            hash_md5 = hashlib.md5()
+            with open(tarball, "rb") as tar:
+                for chunk in iter(lambda: tar.read(4096), b""):
+                    hash_md5.update(chunk)
+
+            with open(tarball_md5, "w") as md5:
+                md5.write(hash_md5.hexdigest())
+        except Exception:
+            self.logger.error("md5sum failed for %s, skipping" % tarball)
+            if os.path.exists(tarball):
+                os.remove(tarball)
+            if os.path.exists(tarball_md5):
+                os.remove(tarball_md5)
+            sys.exit(1)