Replication Package content (#1)

* Content of the replication package

* Added workflow build+test+static analysis

* Last changes, renaming the test folders and updating names to convenion

* Updated flow and removed old Jenkisnfile

* Last changes requested by Antonia: spelling and mismatch name-surname in some authors.

* Some spelling mistakes, added EQUAVEL project

.github/workflows/python-app.yml

@@ -0,0 +1,43 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: retorchx-rp
+
+on:
+  push:
+    branches: [ "**" ]
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  build-and-test:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python 3.10
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install flake8 pytest
+          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+      - name: Lint with flake8
+        run: |
+          # stop the build if there are Python syntax errors or undefined names
+          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+      - name: Test with pytest
+        run: |
+          pytest
+      - name: Generate Report
+        run: |
+          pip install coverage
+          coverage run -m pytest

.gitignore

@@ -0,0 +1,96 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# IPython Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# dotenv
+.env
+
+# virtualenv
+.venv/
+venv/
+ENV/
+pyvenv.cfg
+# Spyder project settings
+.spyderproject
+
+# Rope project settings
+.ropeproject
+#Folder with the datasets
+data/
+input/
+outputs/
+#Jetbrains files
+.idea

CITATION.cff

@@ -0,0 +1,50 @@
+cff-version: 1.2.0
+message: "If you cite this replication package, please cite it as below."
+authors:
+- family-names: "Augusto"
+  given-names: "Cristian"
+  orcid: "https://orcid.org/0000-0001-6140-1375"
+- family-names: "Morán"
+  given-names: "Jesús"
+  orcid: "https://orcid.org/0000-0002-7544-3901"
+- family-names: "Bertolino"
+  given-names: "Antonia"
+  orcid: "https://orcid.org/0000-0001-8749-1356"
+- family-names: "De la Riva"
+  given-names: "Claudio"
+  orcid: "https://orcid.org/0000-0001-5592-9683"
+- family-names: "Tuya"
+  given-names: "Javier"
+  orcid: "https://orcid.org/0000-0002-1091-934X"
+title: "Replication package for 'RETORCH*: A Cost and Resource aware Model for E2E Testing in the Cloud'"
+version: 1.0
+doi: "TO-DO"
+date-released: 2023-08-11
+url: "https://github.com/giis-uniovi/retorchx-rp"
+preferred-citation:
+  type: article
+  authors:
+- family-names: "Augusto"
+  given-names: "Cristian"
+  orcid: "https://orcid.org/0000-0001-6140-1375"
+- family-names: "Morán"
+  given-names: "Jesús"
+  orcid: "https://orcid.org/0000-0002-7544-3901"
+- family-names: "Bertolino"
+  given-names: "Antonia"
+  orcid: "https://orcid.org/0000-0001-8749-1356"
+- family-names: "De la Riva"
+  given-names: "Claudio"
+  orcid: "https://orcid.org/0000-0001-5592-9683"
+- family-names: "Tuya"
+  given-names: "Javier"
+  orcid: "https://orcid.org/0000-0002-1091-934X"
+  doi: "10.0000/00000" #TO-DO
+  journal: "Journal Title" #TO-DO
+  month: 9 #TO-DO
+  start: 1 # TO-DO First page number
+  end: 10 # TO-DO Last page number
+  title: "RETORCH*: A Cost and Resource aware Model for E2E Testing in the Cloud"
+  issue: 1 #TO-DO
+  volume: 1 #TO-DO
+  year: 2023 #TO-DO

Makefile

@@ -0,0 +1,5 @@
+init:
+	pip install -r requirements.txt
+
+test:
+	pytest 

README.md

@@ -1,2 +1,159 @@
-# retorchx-rp
-RETORCH* Replication package
+# Replication package for *'RETORCH\*: A Cost and Resource aware Model for E2E Testing in the Cloud'*
+
+This repository contains the replication package of the paper *RETORCH\*: A Cost and Resource aware Model for E2E
+Testing in the Cloud*
+published at [TO-DO]
+
+The replication package comprises various Execution Plan raw datasets generated by our Continuous Integration (CI)
+system, along with the necessary scripts to generate the average datasets and the spreadsheets used to generate the
+usage profiles for each Execution Plan. The following subsections introduce the package structure and the procedure.
+
+## Replication package structure and naming conventions:
+
+The naming conventions are :
+
+- **Execution Plans**  are named using the level of parallelism (e.g., 5parallel, 4parallel, 3parallel).
+- **JenkinsFiles** are named using the Execution Plan name followed by "-Jenkinsfile."
+- **Execution Plan raw datasets**  are named with their Execution Plan, followed by "-raw-dataset-" and the execution
+  number.
+
+The replication package is structured as follows:
+
+1. `/raw-datasets`: Execution Plan raw datasets generated by the Continuous Integration
+   system. The data is organized into separate sub-folders one for each different Execution Plan (
+   e.g., `./4parallel`, `./3parallel`, etc.).
+
+2. `/scripts`: contains all the necessary Python scripting code for calculating the average datasets that are used as
+   input for the cost model sheets.
+
+3. `/tests`: contains different unit test cases that check the different methods contained in the `/scripts`
+   directory processing and calculating the average datasets from the raw files.
+
+4. `/generatedatasets.sh/ps1`:  scripts used to launch the data processing and calculation tasks within the project.
+
+5. `/avg-datasets`: stores the output files generated by processing scripts. Each output file is named
+   according to the Execution Plan that belongs in `/raw-datasets` (e.g., the file for
+   `/raw-datasets/2parallel` will be saved as `./2parallel_avg.csv`).
+
+6. `/profile-template.xlsx`: spreadsheet template that enables the Usage Profile calculations and illustrations.
+
+7. `/jenkinsfiles`: holds different JenkinsFiles of the Execution Plans used in the different case studies
+   related.
+
+## Experimental Subject
+
+The experimental subject is a real-world application
+called [Fullteaching](https://github.com/codeurjc-students/2019-FullTeaching/tree/Angular-Refactor), used as a
+demonstrator of the [ElasTest EU Project](https://elastest.eu/). FullTeaching provides an education platform composed
+of several test resources, such as web servers, databases, and multimedia servers that allows to create online
+classrooms, classes or publish and create class resources.
+
+To the best of our knowledge, FullTeaching has two test suites available in different
+repositories [[1]](https://github.com/elastest/full-teaching) [[2]](https://github.com/codeurjc-students/2019-FullTeaching/tree/Angular-Refactor).
+The test suite used to generate the raw datasets provided in this replication package is a compilation of the available
+test, cases in these repositories. The test suite, along with the necessary scripts and JenkinsFile required to run the
+Execution Plan is made available in a [GitHub repository](https://github.com/giis-uniovi/retorch-st-fullteaching).
+
+## Treatment Replication Overview
+
+The process consists of two distinct parts: the generation of average Execution Plan datasets performed through several
+scripts, and the generation of the usage profile using a parameterized stylesheet with the previously calculated
+datasets. These two parts are detailed below.
+
+- **Average Execution Plan Datasets generation:** This process is accomplished through several scripts that take the raw
+  datasets of each Execution Plan as input, organized in separate folders (one for each different execution plan). The
+  raw datasets contain the start and end times of each Cloud Object Instance and TJob lifecycle. The scripts calculate
+  the average duration of each lifecycle phase and generate a new file (identified by the Execution Plan name plus '_
+  avg') with
+  the start and end times of each phase based on these calculated averages.
+
+- **Usage Profile generation:** The process takes one of the previously calculated average datasets as input. The
+  dataset
+  is manually incorporated into the stylesheet by the tester, who also parameterizes it according to the Cloud Object
+  Instance they want to analyze (one alternative per stylesheet sheet). The stylesheet produces as output the Usage
+  Profiles, as well as the different costs for each alternative.
+
+## Treatment Replication Procedure
+
+There are two options for conducting the replication process:
+
+1. **Using Paper Datasets:** provided in the `/raw-datasets` folder.
+
+2. **Execute BYS the different Execution Plans and generate your own datasets** using the made
+   available [test suite](https://github.com/giis-uniovi/retorch-st-fullteaching) and
+   the [execution plans](/jenkinsfiles) provided.
+
+By following either of these options, you can successfully replicate the treatment and conduct your evaluation.
+
+### Generate the average datasets and the Usage Profiles
+
+To replicate the generation of the average datasets and the resource profiles of a concrete parametrization please
+follow
+the instructions outlined below:
+
+1. From the scripts folder, execute the `/generatedatasets.sh/.ps1` script
+2. Create a new Usage Profile stylesheet using the [template provided](profile-template.xlsx) and rename it with an
+   appropriate name (e.g. OriginalexecutionplanProfile.xlns) and fine-tune the different parameters(outlined in red) :
+    - Replace the average lifecycle time (A1:L13) with the selected Execution Plan average file (e.g.
+      `/avg-datasets/3parallel_avg.csv`)
+    - COI Capacities and Invoiced Prices:
+        - Contracted_Capacity.Size: Units of capacity provisioned.
+        - Invoiced_Price.price: Amount in dollars billed for each unit of capacity provisioned.
+        - Billing_Option.Billing_Period: The time for which billing is calculated.
+        - Provisioned time: Calculated as COI.Prov.Time - COI.Deprov.Time.
+    - Test Suite Execution:
+        - Number of test suite executions per hour.
+
+   ![](images/COI_Exec_Parameters.png)
+
+3. Make as many copies of the Spreadsheet sheet as Cloud Object Instances, and adjust the parameters accordingly, in
+   order get its different costs and usage profiles and compare and analyze them.
+
+### Replication procedure outputs
+
+The outputs of the replication procedure are the following:
+
+- The average datasets stored into `/avg-datasets`
+- The Usage Profiles as well as the different utilization and costs (overall, set-up, test execution, tear-down and
+  overprovisioning) are shown in the bottom of the different sheets :
+  ![](images/UsageProfileEx.png)
+
+### Generate your own raw Execution Plan datasets
+
+To generate your own raw Execution Plan datasets, you need to create a multibranch pipeline in your Jenkins
+installation. The replication package provides several Execution Plans that can be used with
+the [FullTeaching Test Suite](https://github.com/giis-uniovi/retorch-st-fullteaching). To use these Execution Plans, you
+simply need to replace the JenkinsFile with the desired one and execute it.
+
+The different raw datasets are archived as artifacts and can be downloaded directly. These datasets should be placed in
+the `/raw-datasets` folder within a suitable sub-folder with the Execution Plan name (e.g.,
+`/raw-datasets/customscheduling`).
+
+Next, follow the steps provided in
+the [Generate the average files and the Usage Profiles](#generate-the-average-datasets-and-the-usage-profiles) section.
+
+## Contributing
+
+See the general contribution policies and guidelines for *giis-uniovi* at
+[CONTRIBUTING.md](https://github.com/giis-uniovi/.github/blob/main/profile/CONTRIBUTING.md).
+
+## Contact
+
+Contact any of the researchers who authored the paper; their affiliation and contact information are provided in the
+paper itself.
+
+## Citing this work
+
+[TO-DO]
+RETORCH E2E Test Orchestration framework:
+
+- Cristian Augusto, Jesús Morán, Antonia Bertolino, Claudio de la Riva, and Javier Tuya,
+  “RETORCH: an approach for resource-aware orchestration of end-to-end test cases”,
+  *Software Quality Journal*, vol. 28, no. 3, 2020.
+  https://doi.org/10.1007/s11219-020-09505-2 - [Full Paper available](https://link.springer.com/article/10.1007/s11219-020-09505-2) - [Authors version](https://digibuo.uniovi.es/dspace/bitstream/handle/10651/55405/RETORCHSQJExtension_BUO.pdf;jsessionid=0E661594C8732B8D2CA53636A31E4FD5?sequence=1) -
+  [Download citation](https://citation-needed.springer.com/v2/references/10.1007/s11219-020-09505-2?format=refman&flavour=citation)
+
+## Acknowledgments
+
+This work has been developed under the TestBUS (PID2019-105455GB-C32) and EQUAVEL (PID2022-137646OB-C32) projects, both supported
+by the [Ministry of Science and Innovation (SPAIN)](https://www.ciencia.gob.es/)

avg-datasets/.gitignore

@@ -0,0 +1,4 @@
+# Ignore everything in this directory
+*
+# Except this file
+!.gitignore

generatedatasets.ps1

@@ -0,0 +1,5 @@
+python3 -m venv .
+.\Scripts\Activate.ps1
+pip3 install -r requirements.txt
+python .\retorch\core.py
+.\Scripts\deactivate.bat

generatedatasets.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+sudo apt-get install -y python3.10 python3-pip
+pip3 install virtualenv
+virtualenv ./venv
+source ./venv/bin/activate
+pip3 install -r requirements.txt
+python3 retorch/core.py
+
+
+deactivate