Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

npm init neon --lib #1014

Merged
merged 50 commits into from
May 6, 2024
Merged
Show file tree
Hide file tree
Changes from 46 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
2b6476b
Use tsconfig/node18 as base TS config
Dec 2, 2023
6633fb5
generate more modern ES, which unfortunately means we can't as easy i…
Dec 9, 2023
f3be15e
consolidate Versions type handling
Dec 9, 2023
d79a4a6
get the tests passing again with modern TS and modern node versions
Dec 9, 2023
aa6ce0f
single quotes for imports
Dec 9, 2023
bdf24a9
simplify unnecessary type test that I added during debugging
Dec 9, 2023
827016b
add top-level comment explaining the versions.ts wrapper module
Dec 9, 2023
7804e07
WIP
dherman Jan 7, 2024
23585f1
rename "release" to "remote-publish" to avoid ambiguity with GitHub R…
dherman Jan 8, 2024
3555443
Initial version of `npm init neon --lib`
dherman Feb 26, 2024
a2ce7a5
First working draft of build automation for libraries!
dherman Mar 27, 2024
7460c7c
move hbs delegate into GitHub CI plugin
dherman Mar 27, 2024
00394f8
update versions
dherman Mar 27, 2024
88e8bb7
npm run prettier
dherman Mar 27, 2024
74fe46a
bugfix: PackageSpec needs version (default is "0.1.0") and manifest t…
dherman Mar 28, 2024
2a68374
add a newline to the manifest
dherman Mar 28, 2024
e54a1de
diagnostics
dherman Mar 28, 2024
4b301e3
build fix
dherman Mar 28, 2024
d995950
export * from './index.cts'
dherman Apr 7, 2024
b2e0535
better ways to extract node and rust versions
dherman Apr 7, 2024
86cfc2f
bugfix: steps.build.outputs.filename => steps.pack.outputs.filename
dherman Apr 7, 2024
4741f2a
Update pkgs/create-neon/data/templates/ci/github/build.yml.hbs
dherman Apr 7, 2024
cda4804
bugfix: packageSpec.library not package.library
dherman Apr 7, 2024
86c7daa
node 20.x
dherman Apr 14, 2024
e965176
remove .npmignore since it's only for libraries, which already use `"…
dherman Apr 14, 2024
24e8898
complete the node v20 changes
dherman Apr 14, 2024
a36e170
generate the right README details for the more nested repo layout of …
dherman Apr 14, 2024
66148cf
fix the conditional logic for the directory layout in README
dherman Apr 14, 2024
d27655d
create-neon now generates the nested directory structure for TS libra…
dherman Apr 14, 2024
c8b804e
command-line option documentation in the main README
dherman Apr 14, 2024
4bbafb2
tee all the GITHUB_OUTPUT for simple diagnostics
dherman Apr 14, 2024
4ec45b3
no default since it's required
dherman Apr 14, 2024
eb6ac15
a few more GHA details:
dherman Apr 15, 2024
d3aa32c
replace the magic comments with a magic label
dherman Apr 15, 2024
8de9b14
remove comments.yml.hbs from templates
dherman Apr 15, 2024
1150668
comment copy-edits
dherman Apr 15, 2024
36a002f
eliminate two more magic feathers
dherman Apr 15, 2024
e6eef73
slight improvements to the generated index.cts boilerplate:
dherman Apr 15, 2024
970aff8
different hello world for libraries vs simple projects
dherman Apr 15, 2024
8830100
add some explanatory comments to the generated code
dherman Apr 16, 2024
a7168c0
oops, copy-pasta bug
dherman Apr 16, 2024
e2b3792
missed one {{#$}}
dherman Apr 16, 2024
8c59954
id: tag
dherman Apr 16, 2024
8d68793
bugfixes:
dherman Apr 19, 2024
a24ab89
more detailed instructions for setting up npm tokens
dherman Apr 19, 2024
b2c69e0
bump @neon-rs/manifest so saving manifests includes a final newline
dherman Apr 20, 2024
4091edf
use the new more ergonomic CLI syntax
dherman May 6, 2024
895361e
harden the use of third-party GH actions by using explicit SHAs for a…
dherman May 6, 2024
4ecd9b1
explain what "portable library" means in the README
dherman May 6, 2024
fd212c1
fix failing test for --yes
dherman May 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,026 changes: 3,940 additions & 86 deletions package-lock.json

Large diffs are not rendered by default.

38 changes: 37 additions & 1 deletion pkgs/create-neon/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,42 @@ The `create-neon` tool bootstraps [Neon](https://neon-bindings.com) projects, wh

You can conveniently use this tool with the [`npm init`](https://docs.npmjs.com/cli/v7/commands/npm-init) syntax:

### Creating a Simple Project

To create a simple Neon project that consists purely of Rust code:

```sh
$ npm init neon [<opts> ...] my-project
```

#### Global Options

```sh
-y|--yes Skip interactive `npm init` questionnaire.
```

### Creating a Portable Library

To create a portable npm library with pre-built binaries:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, it might not be obvious to a potential user what "portable" means.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point--I added some explanatory text.


```sh
$ npm init neon [<opts> ...] --lib [<lib-opts> ...] my-project
```

This will generate a project that can be used by pure JavaScript or TypeScript consumers without them even being aware of the use of Rust under the hood. It achieves this by publishing pre-built binaries for common Node platform architectures that are loaded just-in-time by a JS wrapper module.

This command generates the necessary npm and CI/CD configuration boilerplate to require nearly zero manual installation on typical GitHub-hosted repos. The only manual step required is to configure GitHub Actions with the necessary npm access token to enable automated publishing.

This command chooses the most common setup by default, but allows customization with fine-grained configuration options. These configuration options can also be modified later with the [Neon CLI](https://www.npmjs.com/package/@neon-rs/cli).

#### Library Options

```sh
$ npm init neon my-project
--ci none|github CI/CD provider to generate config for.
(Default: github)
--bins none|npm[:org] Cache provider to publish pre-built binaries.
(Default: npm, with org inferred from package)
--platform <platform> Binary platform to add support to this library for.
This option can be specified multiple times.
(Default: macos, linux, windows)
```
6 changes: 5 additions & 1 deletion pkgs/create-neon/data/templates/.gitignore.hbs
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,8 @@ target
index.node
**/node_modules
**/.DS_Store
npm-debug.log*
npm-debug.log*{{#eq packageSpec.library.lang compare="ts"}}
lib
dherman marked this conversation as resolved.
Show resolved Hide resolved
{{/eq}}
cargo.log
cross.log
141 changes: 81 additions & 60 deletions pkgs/create-neon/data/templates/README.md.hbs
Original file line number Diff line number Diff line change
Expand Up @@ -6,66 +6,79 @@
{{/if}}
This project was bootstrapped by [create-neon](https://www.npmjs.com/package/create-neon).

## Installing {{package.name}}
## Building {{package.name}}

Installing {{package.name}} requires a [supported version of Node and Rust](https://github.com/neon-bindings/neon#platform-support).
Building {{package.name}} requires a [supported version of Node and Rust](https://github.com/neon-bindings/neon#platform-support).

You can install the project with npm. In the project directory, run:
To run the build, run:

```sh
$ npm install
$ npm run build
```

This fully installs the project, including installing any dependencies and running the build.
This command uses the [@neon-rs/cli](https://www.npmjs.com/package/@neon-rs/cli) utility to assemble the binary Node addon from the output of `cargo`.

## Building {{package.name}}
## Exploring {{package.name}}

If you have already installed the project and only want to run the build, run:
After building {{package.name}}, you can explore its exports at the Node console:

{{#if packageSpec.library}}
```sh
$ npm i
$ npm run build
$ node
> require('.').greeting()
{ message: 'hello node' }
```

This command uses the [cargo-cp-artifact](https://github.com/neon-bindings/cargo-cp-artifact) utility to run the Rust build and copy the built library into `./index.node`.

## Exploring {{package.name}}

After building {{package.name}}, you can explore its exports at the Node REPL:

{{else}}
```sh
$ npm install
$ npm i
$ npm run build
$ node
> require('.').hello()
"hello node"
'hello node'
```
{{/if}}

## Available Scripts

In the project directory, you can run:

### `npm install`
{{#unless packageSpec.library}}
#### `npm install`

Installs the project, including running `npm run build`.

### `npm build`
{{/unless}}
#### `npm run build`

Builds the Node addon (`index.node`) from source.
Builds the Node addon (`index.node`) from source, generating a release build with `cargo --release`.

Additional [`cargo build`](https://doc.rust-lang.org/cargo/commands/cargo-build.html) arguments may be passed to `npm build` and `npm build-*` commands. For example, to enable a [cargo feature](https://doc.rust-lang.org/cargo/reference/features.html):
Additional [`cargo build`](https://doc.rust-lang.org/cargo/commands/cargo-build.html) arguments may be passed to `npm run build` and similar commands. For example, to enable a [cargo feature](https://doc.rust-lang.org/cargo/reference/features.html):

```
npm run build -- --feature=beetle
```

#### `npm build-debug`
#### `npm run debug`

Similar to `npm run build` but generates a debug build with `cargo`.

#### `npm run cross`

Similar to `npm run build` but uses [cross-rs](https://github.com/cross-rs/cross) to cross-compile for another platform. Use the [`CARGO_BUILD_TARGET`](https://doc.rust-lang.org/cargo/reference/config.html#buildtarget) environment variable to select the build target.

Alias for `npm build`.
{{#eq packageSpec.library.ci.type compare="github"}}
#### `npm run release`

#### `npm build-release`
Initiate a full build and publication of a new patch release of this library via GitHub Actions.

Same as [`npm build`](#npm-build) but, builds the module with the [`release`](https://doc.rust-lang.org/cargo/reference/profiles.html#release) profile. Release builds will compile slower, but run faster.
#### `npm run dryrun`

### `npm test`
Initiate a dry run of a patch release of this library via GitHub Actions. This performs a full build but does not publish the final result.

{{/eq}}
#### `npm test`

Runs the unit tests by calling `cargo test`. You can learn more about [adding tests to your Rust code](https://doc.rust-lang.org/book/ch11-01-writing-tests.html) from the [Rust book](https://doc.rust-lang.org/book/).

Expand All @@ -77,47 +90,55 @@ The directory structure of this project is:
{{package.name}}/
├── Cargo.toml
├── README.md
├── index.node
├── package.json
{{#if packageSpec.library}}
├── lib/
{{#eq packageSpec.library.lang compare="ts"}}
├── src/
| ├── index.mts
| └── index.cts
├── crates/
| └── {{package.name}}/
| └── src/
| └── lib.rs
{{/eq}}
├── platforms/
{{else}}
├── src/
| └── lib.rs
├── index.node
{{/if}}
├── package.json
└── target/
```

### Cargo.toml

The Cargo [manifest file](https://doc.rust-lang.org/cargo/reference/manifest.html), which informs the `cargo` command.

### README.md

This file.

### index.node

The Node addon—i.e., a binary Node module—generated by building the project. This is the main module for this package, as dictated by the `"main"` key in `package.json`.

Under the hood, a [Node addon](https://nodejs.org/api/addons.html) is a [dynamically-linked shared object](https://en.wikipedia.org/wiki/Library_(computing)#Shared_libraries). The `"build"` script produces this file by copying it from within the `target/` directory, which is where the Rust build produces the shared object.

### package.json

The npm [manifest file](https://docs.npmjs.com/cli/v7/configuring-npm/package-json), which informs the `npm` command.

### src/

The directory tree containing the Rust source code for the project.

### src/lib.rs

The Rust library's main module.

### target/

Binary artifacts generated by the Rust build.
| Entry | Purpose |
|----------------|------------------------------------------------------------------------------------------------------------------------------------------|
| `Cargo.toml` | The Cargo [manifest file](https://doc.rust-lang.org/cargo/reference/manifest.html), which informs the `cargo` command. |
| `README.md` | This file. |
{{#if packageSpec.library}}
{{#eq packageSpec.library.lang compare="ts"}}
| `lib/` | The directory containing the generated output from [tsc](https://typescriptlang.org). |
| `src/` | The directory containing the TypeScript source files. |
| `index.mts` | Entry point for when this library is loaded via [ESM `import`](https://nodejs.org/api/esm.html#modules-ecmascript-modules) syntax. |
| `index.cts` | Entry point for when this library is loaded via [CJS `require`](https://nodejs.org/api/modules.html#requireid). |
| `crates/` | The directory tree containing the Rust source code for the project. |
| `lib.rs` | Entry point for the Rust source code. |
{{else}}
| `lib/` | The directory containing The directory containing the JavaScript source files. |
{{/eq}}
| `platforms/` | The directory containing distributions of the binary addon backend for each platform supported by this library. |
{{else}}
| `src/` | The directory tree containing the Rust source code for the project. |
| `lib.rs` | Entry point for the Rust source code. |
| `index.node` | The main module, a [Node addon](https://nodejs.org/api/addons.html) generated by the build and pointed to by `"main"` in `package.json`. |
{{/if}}
| `package.json` | The npm [manifest file](https://docs.npmjs.com/cli/v7/configuring-npm/package-json), which informs the `npm` command. |
| `target/` | Binary artifacts generated by the Rust build. |

## Learn More

To learn more about Neon, see the [Neon documentation](https://neon-bindings.com).

To learn more about Rust, see the [Rust documentation](https://www.rust-lang.org).
Learn more about:

To learn more about Node, see the [Node documentation](https://nodejs.org).
- [Neon](https://neon-bindings.com).
- [Rust](https://www.rust-lang.org).
- [Node](https://nodejs.org).
3 changes: 3 additions & 0 deletions pkgs/create-neon/data/templates/Workspace.toml.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[workspace]
members = ["crates/{{package.name}}"]
resolver = "2"
5 changes: 5 additions & 0 deletions pkgs/create-neon/data/templates/ci/github/.env.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
NODE_VERSION={{versions.node}}.x
NPM_REGISTRY=https://registry.npmjs.org
RUST_VERSION=stable
ACTIONS_USER=github-actions
[email protected]
137 changes: 137 additions & 0 deletions pkgs/create-neon/data/templates/ci/github/build.yml.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
name: Build

on:
workflow_call:
inputs:
ref:
description: 'The branch, tag, or SHA to check out'
required: true
type: string
update-version:
description: 'Update version before building?'
required: false
type: boolean
default: false
version:
description: 'Version update (ignored if update-version is false)'
required: false
type: string
default: 'patch'
github-release:
description: 'Publish GitHub release?'
required: false
type: boolean
default: false
tag:
description: 'The release tag (ignored if github-release is false)'
required: false
type: string
default: ''

jobs:
matrix:
name: Matrix
runs-on: ubuntu-latest
outputs:
matrix: {{#$}} steps.matrix.outputs.result {{/$}}
steps:
- name: Checkout Code
uses: actions/checkout@{{versions.actions.checkout}}
with:
ref: {{#$}} inputs.ref {{/$}}
- name: Setup Neon Environment
uses: ./.github/actions/setup
with:
use-rust: false
- name: Look Up Matrix Data
id: matrixData
shell: bash
run: echo "json=$(npx neon ci github | jq -rc)" | tee -a $GITHUB_OUTPUT
- name: Compute Matrix
id: matrix
uses: actions/github-script@{{versions.actions.githubScript}}
with:
script: |
const platforms = {{#$}} steps.matrixData.outputs.json {{/$}};
const macOS = platforms.macOS.map(platform => {
return { os: "macos-latest", platform, script: "build" };
});
const windows = platforms.Windows.map(platform => {
return { os: "windows-latest", platform, script: "build" };
});
const linux = platforms.Linux.map(platform => {
return { os: "ubuntu-latest", platform, script: "cross" };
});
return [...macOS, ...windows, ...linux];

binaries:
name: Binaries
needs: [matrix]
strategy:
matrix:
cfg: {{#$}} fromJSON(needs.matrix.outputs.matrix) {{/$}}
runs-on: {{#$}} matrix.cfg.os {{/$}}
permissions:
contents: write
steps:
- name: Checkout Code
uses: actions/checkout@{{versions.actions.checkout}}
with:
ref: {{#$}} inputs.ref {{/$}}
- name: Setup Neon Environment
id: neon
uses: ./.github/actions/setup
with:
use-cross: {{#$}} matrix.cfg.script == 'cross' {{/$}}
platform: {{#$}} matrix.cfg.platform {{/$}}
- name: Update Version
if: {{#$}} inputs.update-version {{/$}}
shell: bash
run: |
git config --global user.name $ACTIONS_USER
git config --global user.email $ACTIONS_EMAIL
npm version {{#$}} inputs.version {{/$}} -m "v%s"
- name: Build
shell: bash
env:
CARGO_BUILD_TARGET: {{#$}} steps.neon.outputs.target {{/$}}
NEON_BUILD_PLATFORM: {{#$}} matrix.cfg.platform {{/$}}
run: npm run {{#$}} matrix.cfg.script {{/$}}
- name: Pack
id: pack
shell: bash
run: |
mkdir -p dist
echo filename=$(basename $(npm pack ./platforms/{{#$}} matrix.cfg.platform {{/$}} --silent --pack-destination=./dist --json | jq -r '.[0].filename')) | tee -a $GITHUB_OUTPUT
- name: Release
if: {{#$}} inputs.github-release {{/$}}
uses: softprops/action-gh-release@{{versions.actions.ghRelease}}
with:
files: ./dist/{{#$}} steps.pack.outputs.filename {{/$}}
tag_name: {{#$}} inputs.tag {{/$}}

main:
name: Main
needs: [matrix]
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@{{versions.actions.checkout}}
with:
ref: {{#$}} inputs.ref {{/$}}
- name: Setup Neon Environment
uses: ./.github/actions/setup
with:
use-rust: false
- name: Pack
id: pack
shell: bash
run: |
mkdir -p dist
echo "filename=$(npm pack --silent --pack-destination=./dist)" | tee -a $GITHUB_OUTPUT
- name: Release
if: {{#$}} inputs.github-release {{/$}}
uses: softprops/action-gh-release@{{versions.actions.ghRelease}}
with:
files: ./dist/{{#$}} steps.pack.outputs.filename {{/$}}
tag_name: {{#$}} inputs.tag {{/$}}
Loading
Loading