Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update build system for Linux, macOS, add GitHub Actions #30

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
version: 2
updates:
# Dependabot updates for GitHub Actions
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: "weekly"
84 changes: 84 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
name: Build

on:
[push, pull_request]

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: false

defaults:
run:
shell: bash

jobs:
build-docker:
strategy:
fail-fast: false
matrix:
include:
- container: wpilib/aarch64-cross-ubuntu:bullseye-22.04
name: LinuxARM64
platform-type: linuxarm64
arch: arm64
- container: wpilib/raspbian-cross-ubuntu:bullseye-22.04
name: LinuxARM32
platform-type: linuxarm32
arch: arm32
runs-on: ubuntu-latest
name: "Build - ${{ matrix.name }}"
container: ${{ matrix.container }}
steps:
- uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install dependencies
run: npm install

- name: Pretest
run: npm run pretest

# Due to the nature of the build process, we can't run the tests in the container becauase external hardware is required
# If this were to be running on a local machine, the tests would be run here
#- name: Test
# run: npm test

build-native:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
name: "Build - ${{ matrix.os }}"
steps:
- uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install dependencies
run: npm install

- name: Pretest
run: npm run pretest

# Due to the nature of the build process, we can't run the tests in the container becauase external hardware is required
# If this were to be running on a local machine, the tests would be run here
#- name: Test
# run: npm test
52 changes: 52 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Create release

on:
push:
tags:
- 'v*'

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: false

defaults:
run:
shell: bash

jobs:
release:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
name: "Release - ${{ matrix.os }}"

steps:
- uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install dependencies
run: npm install

- name: Build
run: npm run build

- name: Create release
uses: softprops/action-gh-release@v2
with:
files: dist/*
tag_name: ${{ github.ref }}
name: ${{ github.ref }}
body: |
This is a release for version ${{ github.ref }}.
It contains the compiled files from the build process.
153 changes: 135 additions & 18 deletions scripts/download-CanBridge.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,47 +2,127 @@ import * as fs from "fs";
import * as path from "path";
import axios from 'axios';
import AdmZip from 'adm-zip';
import { platform, arch } from 'os';

const canBridgeTag = "v2.5.1";
const canBridgeTag = "v2.5.0";
const canBridgeReleaseAssetUrlPrefix = `https://github.com/REVrobotics/CANBridge/releases/download/${canBridgeTag}`;

const externalCompileTimeDepsPath = 'externalCompileTimeDeps';
const runtimeArtifactsPath = path.join('prebuilds', 'node_canbridge-win32-x64');
const runtimeArtifactsPath = {
win: path.join('prebuilds', 'node_canbridge-win32-x64'),
osx: path.join('prebuilds', 'node_canbridge-darwin-osxuniversal'),
linux: path.join('prebuilds', 'node_canbridge-linux-x64'),
linuxArm: path.join('prebuilds', 'node_canbridge-linux-arm64'),
linuxArm32: path.join('prebuilds', 'node_canbridge-linux-arm32')
};
const tempDir = 'temp';

try {
await Promise.all(Array.of(
downloadCanBridgeArtifact('CANBridge.lib', externalCompileTimeDepsPath),
downloadCanBridgeArtifact('CANBridge.dll', runtimeArtifactsPath),
downloadCanBridgeArtifact('wpiHal.lib', externalCompileTimeDepsPath),
downloadCanBridgeArtifact('wpiHal.dll', runtimeArtifactsPath),
downloadCanBridgeArtifact('wpiutil.lib', externalCompileTimeDepsPath),
downloadCanBridgeArtifact('wpiutil.dll', runtimeArtifactsPath),
downloadCanBridgeArtifact('headers.zip', tempDir),
));
// TODO: Do not hardcode the filenames, instead get them from the GitHub API -> Look at Octokit: https://github.com/octokit/octokit.js
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Octokit is great if you need it, but I don't really see the problem with hardcoding URLs here. We need to hardcode a specific tag, so using Octokit to tell us what the latest version is wouldn't be useful. Octokit could give us a full list of artifacts, but we'd still need to hardcode which ones we want to use (unless we wanted to try to use some sort of heuristics to figure out which file contains what we're looking for, which feels unnecessary). What do you see as the advantages of using Octokit in this context?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I was after was to have it always pull from latest and to iterate through every single asset that was in a release, rather than hardcoding the files and the version to be specific on what was needed to be downloaded. Memory on what that was specifically for is a bit hazy. In my mind it would cut down on commits updating the tag as well as not having to worry about missing artifacts if I just download them all from an endpoint.

Little more complex, but would be more extendable as far as what could be available in the future. This note can be totally tossed out if we don't care about those commits or hardcoding. More of a "hey check this out, might help with something in the future" kind of thing.

await Promise.all([
'CANBridge-linuxarm32.zip',
'CANBridge-linuxarm64.zip',
'CANBridge-linuxx86-64.zip',
'CANBridge-osxuniversal.zip',
'CANBridge-windowsx86-64.zip',
'headers.zip'
].map(filename => downloadCanBridgeArtifact(filename)));
console.log("CANBridge download completed");


console.log("Extracting headers");

const zipFiles = fs.readdirSync(tempDir).filter(filename => filename.endsWith('.zip') && filename !== 'headers.zip');
for (const filename of zipFiles) {
await unzipCanBridgeArtifact(filename, tempDir);
}
const headersZip = new AdmZip(path.join(tempDir, "headers.zip"));
headersZip.extractAllTo(path.join(externalCompileTimeDepsPath, 'include'));
console.log("Headers extracted");

moveRuntimeDeps();

await headersZip.extractAllTo(path.join(externalCompileTimeDepsPath, 'include'), true);
console.log(`Successfully downloaded CANBridge ${canBridgeTag}`);
moveCompileTimeDeps();
} catch (e) {
if (axios.isAxiosError(e) && e.request) {
console.error(`Failed to download CANBridge file ${e.request.protocol}//${e.request.host}/${e.request.path}`);
console.error(`Failed to download CANBridge file ${e.request.protocol}//${e.request.host}${e.request.path}`);
} else {
console.error(`Failed to download CANBridge`);
console.error(`Other error occurred: ${e.message}`);
// For non-axios errors, the stacktrace will likely be helpful
throw e;
}
process.exit(1);
} finally {
if (fs.existsSync(tempDir)) {
fs.rmSync(tempDir, { recursive: true });
fs.rmSync(tempDir, { recursive: true, force: true});
}
}

async function downloadCanBridgeArtifact(filename, destDir) {
/**
* Move external compile time dependencies to the correct directory
*
* This function is used to move the external compile time dependencies to the correct directory based on the platform and architecture from downloaded artifacts
*/
function moveCompileTimeDeps() {
console.log("Moving external compile time dependencies to correct directories");
if (!fs.existsSync(externalCompileTimeDepsPath)) {
fs.mkdirSync(externalCompileTimeDepsPath, { recursive: true });
}
if (platform() === 'win32') {
const deps = ['CANBridge.lib', 'wpiHal.lib', 'wpiutil.lib'];
deps.forEach(dep => moveExternalCompileTimeDeps(path.join('win32-x64', dep)));
} else if (platform() === 'darwin') {
const deps = ['libCANBridge.a'];
deps.forEach(dep => moveExternalCompileTimeDeps(path.join('darwin-osxuniversal', dep)));
} else if (platform() === 'linux') {
const deps = ['libCANBridge.a'];
const archDepMap = {
x64: 'linux-x64',
arm64: 'linux-arm64',
arm: 'linux-arm32'
};
deps.forEach(dep => moveExternalCompileTimeDeps(path.join(archDepMap[arch()], dep)));
}
console.log("External compile time dependencies moved to correct directories");
}

/**
* Move runtime dependencies to the correct directory
*
* This function is used to move the runtime dependencies to the correct directory based on the platform and architecture from downloaded artifacts
*/
function moveRuntimeDeps() {
console.log("Moving artifacts to correct directories");
if (!fs.existsSync('prebuilds')) {
fs.mkdirSync('prebuilds', { recursive: true });
}
if (platform() === 'win32') {
const deps = ['CANBridge.dll', 'wpiHal.dll', 'wpiutil.dll'];
deps.forEach(dep => moveRuntimeArtifactsDeps(path.join('win32-x64', dep), runtimeArtifactsPath.win));
} else if (platform() === 'darwin') {
const deps = ['libCANBridge.dylib', 'libwpiHal.dylib', 'libwpiutil.dylib'];
deps.forEach(dep => moveRuntimeArtifactsDeps(path.join('darwin-osxuniversal', dep), runtimeArtifactsPath.osx));
} else if (platform() === 'linux') {
const deps = ['libCANBridge.so', 'libwpiHal.so', 'libwpiutil.so'];
if (arch() === 'x64') {
deps.forEach(dep => moveRuntimeArtifactsDeps(path.join('linux-x64', dep), runtimeArtifactsPath.linux));
}
if (arch() === 'arm64') {
deps.forEach(dep => moveRuntimeArtifactsDeps(path.join('linux-arm64', dep), runtimeArtifactsPath.linuxArm));
}
if (arch() === 'arm') {
deps.forEach(dep => moveRuntimeArtifactsDeps(path.join('linux-arm32', dep), runtimeArtifactsPath.linuxArm32));
}
}
console.log("CANBridge artifacts moved to correct directories");
}

/**
* Download artifacts from the CANBridge GitHub release page
*
* @param {*} filename filename of the artifact to download
* @param {*} destDir destination directory to save the artifact, defaults to tempDir
*/
async function downloadCanBridgeArtifact(filename, destDir = tempDir) {
fs.mkdirSync(destDir, { recursive: true });
const response = await axios.get(`${canBridgeReleaseAssetUrlPrefix}/${filename}`, { responseType: "stream" });
const fileStream = fs.createWriteStream(`${destDir}/${filename}`);
Expand All @@ -51,3 +131,40 @@ async function downloadCanBridgeArtifact(filename, destDir) {
fileStream.on('finish', resolve);
});
}

/**
* Unzip the CANBridge artifacts
*
* @param {string} filename - filename of the artifact to unzip
* @param {string} destDir - destination directory to unzip the artifact
*/
async function unzipCanBridgeArtifact(filename, destDir) {
const zip = new AdmZip(`${destDir}/${filename}`);
let filepath;
if (filename.includes('linuxarm32')) filepath = "linux-arm32";
else if (filename.includes('linuxarm64')) filepath = "linux-arm64";
else if (filename.includes('linuxx86-64')) filepath = "linux-x64";
else if (filename.includes('osxuniversal')) filepath = "darwin-osxuniversal";
else if (filename.includes('windowsx86-64')) filepath = "win32-x64";
zip.extractAllTo(`${destDir}/${filepath}`);
}

/**
* Move runtime artifacts to the correct directory
*
* @param {*} filename filename of the artifact to move
* @param {*} destDir destination directory to save the artifact
*/
function moveRuntimeArtifactsDeps(filename, destDir) {
fs.mkdirSync(destDir, { recursive: true });
fs.renameSync(path.join(tempDir, filename), path.join(destDir, path.basename(filename)));
}

/**
* Move External Compile Time Dependencies to the correct directory
*
* @param {*} filename filename of the artifact to move
*/
function moveExternalCompileTimeDeps(filename) {
fs.renameSync(path.join(tempDir, filename), path.join(externalCompileTimeDepsPath, path.basename(filename)));
}