diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d16825f7..32eb9fb2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -35,9 +35,21 @@ jobs: - name: Run TRLC run: | make trlc + selenium-tests: + name: Run Selenium Tests + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + - name: Install dependencies + run: | + python3 -m pip install --upgrade pip + python3 -m pip install selenium webdriver-manager + - name: Run selenium Tests + run: | + make selenium-tests test: name: TestSuite - needs: [lint, trlc] + needs: [lint, trlc, selenium-tests] strategy: matrix: os: [ubuntu-24.04, windows-2022, macos-13, macos-14] diff --git a/Makefile b/Makefile index 7810db7e..5fab934a 100644 --- a/Makefile +++ b/Makefile @@ -54,6 +54,10 @@ clang-tidy: cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra' -DCMAKE_BUILD_TYPE=Release && \ cmake --build build --target clang-tidy +selenium-tests: + @echo "Running Selenium Tests..." + python3 ./tests-system/lobster-core/html_report/selenium-test/test_html_report.py + integration-tests: packages (cd tests-integration/projects/basic; make) (cd tests-integration/projects/filter; make) diff --git a/documentation/selenium_into_CI.md b/documentation/selenium_into_CI.md new file mode 100644 index 00000000..071c1b4e --- /dev/null +++ b/documentation/selenium_into_CI.md @@ -0,0 +1,138 @@ +# **Document: Investigation on Integrating Selenium or XRAY into CI for GUI Tests** + +## **Purpose** +As a software developer, the goal is to integrate GUI testing tools (Selenium or XRAY) into the CI pipeline for the `lobster-html-report` system. This would help automate system tests, ensure reliability, and simplify maintenance. + +--- + +## **Objective** +1. Research how Selenium and XRAY can be integrated into CI tools. +2. Provide a solution for running GUI-based system tests in a Continuous Integration (CI) environment. +3. Identify the pros, cons, and suitability of each tool. +4. Recommend the most effective approach for the `lobster-html-report` system. + +--- + +## **Tools Overview** +### **1. Selenium** +Selenium is a widely used open-source framework for automating web applications. It supports multiple browsers and languages (Python, Java, C#, etc.). +- **Features:** + - Automates web GUI tests. + - Integrates well with CI tools (GitHub Actions) + - Supports headless browsers for CI environments. + +### **2. XRAY** +XRAY is a test management tool that integrates with Jira to manage test cases, execution, and reporting. +- **Features:** + - Integrates with CI/CD tools. + - Allows for test execution reporting (manual and automated tests). + - Works alongside automation frameworks (e.g., Selenium). + +--- + +## **Integration Strategies** +### **Option 1: Selenium Integration in CI** +To integrate Selenium tests into the CI pipeline: +1. **Set Up Selenium Environment:** + - Install required dependencies (Selenium libraries, drivers like ChromeDriver/GeckoDriver). + - Use a headless browser (Chrome/Firefox) for GUI tests in CI. + +2. **Add GUI Test Script:** + - Write Selenium test scripts for `lobster-html-report` (e.g., validate UI components, links, buttons). + - Save test scripts in the project repository (e.g., under a `tests/gui` folder). + +3. **Configure CI Pipeline:** + - **GitHub Actions Example**: + ```yaml + name: GUI Tests + on: [push] + jobs: + gui-tests: + runs-on: ubuntu-latest + steps: + - name: Checkout Code + uses: actions/checkout@v3 + - name: Set Up Python + uses: actions/setup-python@v3 + with: + python-version: '3.x' + - name: Install Dependencies + run: | + pip install selenium + wget https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip + unzip chromedriver_linux64.zip + - name: Run Selenium Tests + run: python tests/gui/test_lobster_html_report.py + ``` + +4. **Test Execution:** + - Execute Selenium scripts automatically during pipeline runs. + - Capture test results in CI logs. + +--- + +### **Option 2: XRAY Integration in CI** +To integrate XRAY tests into CI for test management and reporting: +1. **Set Up XRAY with CI/CD Tool:** + - Integrate XRAY with Jira and CI tools. + - Use REST APIs or XRAY CLI for uploading and executing tests. + +2. **Write Test Scripts (with Selenium):** + - Use Selenium to create GUI test scripts for `lobster-html-report`. + - Link these automated tests to XRAY test cases using XRAY's framework. + +3. **Upload and Execute Tests:** + - Example for uploading results: + ```yaml + - name: Run Selenium Tests + run: | + python tests/gui/test_lobster_html_report.py > results.xml + curl -H "Content-Type: application/json" \ + -u $JIRA_USER:$JIRA_API_TOKEN \ + -X POST "https://your-jira-instance/rest/raven/1.0/import/execution" \ + --data @results.xml + ``` + +4. **Test Reporting:** + - XRAY collects and manages test results. + - Reports can be viewed in Jira or exported for further analysis. + +--- + +## **Comparison: Selenium vs XRAY** +| Feature | Selenium | XRAY | +|---------------------------|---------------------------------------|-------------------------------------------| +| Test Automation | Full web UI automation | Works with Selenium for automation | +| CI Integration | Direct integration into pipelines | Integration via APIs or CLI | +| Test Management | Not included | Comprehensive test management | +| Reporting | Logs in CI/CD tools | Detailed reporting in Jira | +| Suitability for Project | Highly suitable for GUI automation | Suitable for tracking and reporting tests | + +--- + +## **Recommendations** +1. **Primary Approach**: Use **Selenium** for GUI test automation within the CI pipeline as it: + - Offers robust web automation capabilities. + - Easily integrates with GitHub Actions, GitLab CI/CD, or Jenkins. + - Supports headless browsers for faster CI execution. + +2. **Secondary Approach**: If test management and reporting are essential: + - Use XRAY alongside Selenium to manage, execute, and report tests in Jira. + - Integrate XRAY into the CI pipeline for test result uploads. + +--- + +## **Conclusion** +Integrating Selenium into the CI pipeline is the most effective solution for automating GUI system tests for `lobster-html-report`. If additional test management and reporting are required, XRAY can be integrated alongside Selenium. The proposed solution will simplify maintenance, improve reliability, and ensure automated testing becomes part of the development workflow. + +--- + +## **Next Steps** +1. Set up Selenium test scripts for `lobster-html-report`. +2. Configure Selenium integration into the existing CI pipeline. +3. If needed, explore XRAY integration for enhanced test management and reporting. + +--- + +**End of Document** + diff --git a/tests-system/lobster-core/html_report/selenium-test/test_html_report.html b/tests-system/lobster-core/html_report/selenium-test/test_html_report.html new file mode 100644 index 00000000..50050301 --- /dev/null +++ b/tests-system/lobster-core/html_report/selenium-test/test_html_report.html @@ -0,0 +1,386 @@ + + + +L.O.B.S.T.E.R. + + + +
+

L.O.B.S.T.E.R.

+
Lightweight Open BMW Software Traceability Evidence Report
+
+ +
+

Overview

+
+
+
+

Coverage

+ + + + + + + + + + + + + + + + + + + + + + + + +
CategoryRatioCoverageOK ItemsTotal Items
Requirements0.0% + +0.00% + +03
code0.0% + +0.00% + +03
+
+
+
+

Issues

+
+ +
+

Detailed report

+
+

Requirements and Specification

+

Requirements

+
.\main.trlc
+ +
+
TRLC Requirement vanillaEg.trlc
+
Source: + +.\main.trlc +
+
+
Treat Requirements Like Code
+
+
+
Issues: +
    +
  • missing reference to code
    • +
+
+
+
+ + +
+
TRLC Requirement vanillaEg.lobster
+
Source: + +.\main.trlc +
+
+
Lightweight Open BMW Software Traceability Evidence Report
+
+
+
Issues: +
    +
  • missing reference to code
    • +
+
+
+
+ +
Codebeamer https://codebeamer.company.net/, tracker 65426
+ +
+
codebeamer Technical requirement traffic light detection
+
Source: + +cb item 17044 'traffic light detection' +
+
+Status: Valid +
+
+
Issues: +
    +
  • missing reference to code
    • +
+
+
+
+ +

Implementation

+

code

+
.\Test_Setup\main.py
+ +
+
Python Function main.greet_user
+
Source: + +.\Test_Setup\main.py +
+
+
Issues: +
    +
  • unknown tracing target req welcome.greet
    • +
  • missing up reference
    • +
+
+
+
+ +
.\software.py
+ +
+
Python Function software.hello
+
Source: + +.\software.py +
+
+
Issues: +
    +
  • missing up reference
    • +
+
+
+
+ + +
+
Python Function software.hello1
+
Source: + +.\software.py +
+
+
Issues: +
    +
  • missing up reference
    • +
+
+
+
+ +

Verification and Validation

+
+
+ + + diff --git a/tests-system/lobster-core/html_report/selenium-test/test_html_report.py b/tests-system/lobster-core/html_report/selenium-test/test_html_report.py new file mode 100644 index 00000000..0720c013 --- /dev/null +++ b/tests-system/lobster-core/html_report/selenium-test/test_html_report.py @@ -0,0 +1,59 @@ +import os +import sys +from selenium import webdriver +from selenium.webdriver.chrome.service import Service +from webdriver_manager.chrome import ChromeDriverManager +from selenium.webdriver.chrome.options import Options + +def test_html_report_title(dynamic_dir, report_filename, expected_title): + """ + Test function to verify the title of an HTML report with dynamic file path. + + Parameters: + - dynamic_dir (str): Base directory containing the report. + - report_filename (str): Name of the HTML report file. + - expected_title (str): Expected title of the report. + + Returns: + - bool: True if the title matches, False otherwise. + """ + dir_path = os.path.dirname(os.path.abspath(__file__)) + + file_path = dir_path + "/test_html_report.html" + + report_path = os.path.join(dynamic_dir, report_filename) + file_url = f"file:///{os.path.abspath(report_path)}" + + # Headless mode options + chrome_options = Options() + chrome_options.add_argument("--headless") + chrome_options.add_argument("--no-sandbox") + chrome_options.add_argument("--disable-dev-shm-usage") + # Automatically download ChromeDriver + driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options) + try: + driver.get(file_url) + actual_title = driver.title + if actual_title == expected_title: + print(f"Test Passed: Title is '{actual_title}'") + sys.exit(0) + else: + print(f"Test Failed: Expected '{expected_title}', but got '{actual_title}'") + sys.exit(1) + + except Exception as e: + print(f"Error: {e}") + sys.exit(1) + return False + + finally: + driver.quit() + +if __name__ == "__main__": + + base_directory = "./tests-system/lobster-core/html_report/selenium-test" + report_file = "test_html_report.html" + expected_title = 'L.O.B.S.T.E.R.' + sys.exit(test_html_report_title(base_directory, report_file, expected_title)) + +