RFC-007: GitHub Workflow Reorganize

[aceforeverd]

Status: Approved

Problem

• Currently it cannot auto release for openmldb-batch java library, python SDK & macOS binary: https://github.com/4paradigm/OpenMLDB/runs/4116665257?check_suite_focus=true
  • macos binary: no job
  • python SDK deploy: version not tweaked, try to upload as version v0.2.3
    • ci(cpp): job to build/test/publish openmldb binary on macOS #258
    • version not tweaked, the script steps/release.sh is problematic
  • openmldb-batch deploy:
    • openmldb-batch/pom.xml: not tweaks
    • depends on a snapshot version of hybridse, not stable
• hybridse jobs is separated from OpenMLDB
• java SDK deploy
  • depends on hybridse java library, it is in the same code base
  • release workflow not covered in pull request test
  • has macOS and allinone variants
• build time:
  • openmldb building slow: 1hour
  • too many parallel jobs triggered by a push

Current CICD workflow

Compiler docker image compatibility

hybridsql version	0.3.0 download thirdparty compile hybridse	0.2.3 download thirdparty download hybridse	0.2.0 (same with v0.2.1, v0.2.2) assume thirdpary in /depends download hybridse
ghcr.io/4paradigm/hybridsql::latest	Y	Y	N
ghcr.io/4paradigm/hybridsql::0.4.0 centos7, gcc8, no thirdparty support aarch64	Y	Y	N
ghcr.io/4paradigm/hybridsql::0.3.0 cenots7, gcc8, bundled (pre-compiled) thirdparty	Y	Y	N
ghcr.io/4paradigm/hybridsql::0.2.0 centos7, gcc7, bundled (compile from source) thirdparty	N	Y	N

solution

1、reorganize cicd jobs

reorganized jobs, apply to .github/workflows/cicd.yml. Ideally, HybridSE, CPP, Java, Python Job can merged into single one. Problem is, free GitHub Runner have 15GB disk limit, currently not able to compile three in the same time.

• Coverage: depends on add hybridse to root CMakeLists: build(cmake): add hybridse to root CMakeLists.txt #625
• HybriSE: compile hybirdse and runs hybirdse tests, remove other steps
• CPP job: test runs OpenMLDB ut, sql_sdk_test, sql_cluster_sdk, upload test result, upload native binary (useful to download binary of every commit)
  • matrix strategy: Linux & macOS
• Java job: test runs java ut, deploy java library to maven central
  • matrix strategy: Linux & macOS
• Python job: test runs python ut, deploy library to PyPi
  • matrix strategy: Linux only
• Release Job: publish native binary uploaded by CPP jobs
  • depends on CPP job
• Test Report: publish CPP/Java/Python Test results
  • one for forked pull request, one for local branch

2. Java Library (depends on add hybridse to root CMakeLists: build(cmake): add hybridse to root CMakeLists.txt #625)

• merge hybridse-java into openmldb
• other java module depends hybridse by source
• publish in the same time

3. tweak scripts & pom.xml to support java & python deploy

• there should be single script or entry to tweak version number in code, merge file: steps/release.sh, steps/pack_openmldb_javasdk.sh, hybridse/java/prepare_release.sh

4. hybridsql docker

• hybridsql:0.3.0 or later, with OpenMLDB v0.2.3 or later, should be fine. for code commit v0.2.2 or below

Alternatives

• merge any two of HyBridSE, CPP, Python, Java job, as long as disk space is not used up

Risks

• GitHub Free Runner's disk space limit

Changes and Additions to Public Interfaces

The complication and cicd workflow change, but they depend on refactor over complication steps.

Performance Impact

• it might take more time for a job to finish, e.g the CPP job
• the concurrent number of jobs may decrease

Backwards Compatibility and Upgrade Path

NA

Related Works

• Update version in CMakeFile and setup.py when auto release package #665
• ci: openmldb-batch deploy failed on macos version #627 fix: OpenMLDB-batch java module not integrated well with it's parent module #673
• feat: move hybridse java module to OpenMLDB #674
• doc: add compatibility note and upgrade guide about compile image #678
• ci(cpp): job to build/test/publish openmldb binary on macOS #258
• ci(python): job for python sdk on macOS #675
• ci: refactor Github jobs, merge HybridSE and OpenMLDB workflow #676
• ci: include OpenMLDB cpp & java code in codecoverage report #677