diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml new file mode 100644 index 0000000..615b7ca --- /dev/null +++ b/.github/workflows/check.yml @@ -0,0 +1,41 @@ +name: check + +on: + push: + branches: + - main + - edge + - beta + - stable + pull_request: + merge_group: + workflow_dispatch: + +jobs: + ci: + strategy: + fail-fast: false + matrix: + forklift-version: [0.5.0, 0.4.0] + runs-on: ubuntu-latest + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v4 + + - name: Setup Forklift + uses: PlanktoScope/setup-forklift@v1 + with: + version: ${{ matrix.forklift-version }} + + - name: Cache Forklift repos + run: | + mkdir -p $HOME/.forklift + forklift dev plt cache-repo + + - name: Run Forklift checks + run: forklift dev plt check + + - name: Make Forklift deployment plan:w + run: forklift dev plt plan diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..5420ac6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +# Downloaded forklift binary +forklift diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..91d2128 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,13 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), +and this project uses [Calendar Versioning](https://calver.org/) with a `YYYY.minor.patch` scheme. +All dates in this file are given in the [UTC time zone](https://en.wikipedia.org/wiki/Coordinated_Universal_Time). + +## Unreleased + +### Added + +- Added packages and deployments for a proof-of-concept of an independently-running segmenter. diff --git a/LICENSE-Apache b/LICENSE-Apache new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE-Apache @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/LICENSE-BlueOak.md b/LICENSE-BlueOak.md new file mode 100644 index 0000000..c5402b9 --- /dev/null +++ b/LICENSE-BlueOak.md @@ -0,0 +1,55 @@ +# Blue Oak Model License + +Version 1.0.0 + +## Purpose + +This license gives everyone as much permission to work with +this software as possible, while protecting contributors +from liability. + +## Acceptance + +In order to receive this license, you must agree to its +rules. The rules of this license are both obligations +under that agreement and conditions to your license. +You must not do anything with this software that triggers +a rule that you cannot or will not follow. + +## Copyright + +Each contributor licenses you to do everything with this +software that would otherwise infringe that contributor's +copyright in it. + +## Notices + +You must ensure that everyone who gets a copy of +any part of this software from you, with or without +changes, also gets the text of this license or a link to +<https://blueoakcouncil.org/license/1.0.0>. + +## Excuse + +If anyone notifies you in writing that you have not +complied with [Notices](#notices), you can keep your +license by taking all practical steps to comply within 30 +days after the notice. If you do not do so, your license +ends immediately. + +## Patent + +Each contributor licenses you to do everything with this +software that would otherwise infringe any patent claims +they can license or become able to license. + +## Reliability + +No contributor can revoke this license. + +## No Liability + +***As far as the law allows, this software comes as is, +without any warranty or condition, and no contributor +will be liable to anyone for any damages related to this +software or this license, under any kind of legal claim.*** diff --git a/README.md b/README.md index c535539..24c3a2a 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,82 @@ # pallet-segmenter An experimental Forklift pallet for running the PlanktoScope's segmenter separately on a computer + +## Introduction + +pallet-segmenter is a [Forklift](https://github.com/PlanktoScope/forklift) pallet +specifying a configuration of Forklift package deployments for running the PlanktoScope segmenter on +any Docker host. + +## Usage + +### Prerequisites + +You will need to have the Docker Engine installed on your computer. Installation instructions are +available [here](https://docs.docker.com/engine/install/). + +Then, you will set up the [`forklift`](https://github.com/PlanktoScope/forklift) tool on your +computer. Setup instructions are available +[here](https://github.com/PlanktoScope/forklift?tab=readme-ov-file#downloadinstall-forklift). Note +that currently `forklift` is only tested and built for Linux computers. + +### Deployment + +You can clone the latest commit of this Forklift pallet to your computer, by +using the `forklift` tool: +``` +forklift plt clone github.com/PlanktoScope/pallet-segmenter@main +``` + +Then you can apply the cloned pallet on your computer using the following sequence of `forklift` +CLI commands: +``` +forklift plt cache-repo +sudo -E forklift plt apply +``` + +Warning: this will replace all Docker containers on your Docker host with the package deployments +specified by this pallet and delete any Docker containers not specified by this pallet's package +deployments. + +If your user is [in the `docker` group](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user) +(so that you don't need to use `sudo` when running `docker` +commands), then you can just run a single command instead of the two the commands listed above: + +``` +forklift plt switch github.com/PlanktoScope/pallet-segmenter@main +``` + +Then you can access the Node-RED dashboard from your web browser at +<http://localhost:1880/ui>, or you can programmatically interact with the segmenter over MQTT via +<mqtt://localhost:1883> (e.g. using [github.com/PlanktoScope/cli](https://github.com/PlanktoScope/cli)). +You can also access the Node-RED dashboard editor from your web browser at <http://localhost:1880/admin>. + +The segmenter loads input datasets - and saves output files - in `~/.local/share/planktoscope/data`, +instead of the usual path on PlanktoScopes (`/home/pi/data`). Similarly, logs are saved in +`~/.local/share/planktoscope/device-backend-logs` instead of `/home/pi/device-backend-logs`. +The simplest way to add input datasets and download EcoTaxa export archives (working around issues +with file permissions between your user account and the `root` user used for running the segmenter) +will be to use the filebrowser in your web browser, at <http://localhost:9000>. + +### Forking + +To make your own copy of this repository for experimentation, you should fork this repository to a +new repository. Then, update the `path` fields of the `forklift-pallet.yml` and +`forklift-repository.yml` files to match the path of your new repository. + +## Licensing + +Forklift packages deployed by this pallet have their own software licenses, as specified in the +Forklift repositories where those packages are published. Any source code provided with this +Forklift pallet is covered by the following information, except where otherwise indicated: + +Copyright Ethan Li and PlanktoScope project contributors + +SPDX-License-Identifier: Apache-2.0 OR BlueOak-1.0.0 + +You can use the source code provided here either under the +[Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0) +or under the [Blue Oak Model License 1.0.0](https://blueoakcouncil.org/license/1.0.0); +you get to decide. We are making the software available under the Apache license because it's +[OSI-approved](https://writing.kemitchell.com/2019/05/05/Rely-on-OSI.html), +but we like the Blue Oak Model License more because it's easier to read and understand. diff --git a/deployments/filebrowser.deploy.yml b/deployments/filebrowser.deploy.yml new file mode 100644 index 0000000..946aa97 --- /dev/null +++ b/deployments/filebrowser.deploy.yml @@ -0,0 +1,2 @@ +package: /deployments/filebrowser +disabled: false diff --git a/deployments/filebrowser/compose.yml b/deployments/filebrowser/compose.yml new file mode 100644 index 0000000..e6164b6 --- /dev/null +++ b/deployments/filebrowser/compose.yml @@ -0,0 +1,17 @@ +services: + server: + image: filebrowser/filebrowser:v2.27.0 + ports: + - 9000:80 + volumes: + - ~/.local/share/planktoscope/data/:/home/pi/data/ + - ~/.local/share/planktoscope/data/:/srv # this prevents filebrowser from making an unnamed volume + - server-data:/database + environment: + - FB_DATABASE=/database/filebrowser.db + - FB_ROOT=/home/pi/data + - FB_PORT=80 # for some reason filebrowser doesn't work if this is port 8080 + - FB_NOAUTH=true + +volumes: + server-data: {} diff --git a/deployments/filebrowser/forklift-package.yml b/deployments/filebrowser/forklift-package.yml new file mode 100644 index 0000000..ca92328 --- /dev/null +++ b/deployments/filebrowser/forklift-package.yml @@ -0,0 +1,19 @@ +package: + description: In-browser file manager for PlanktoScope datasets + maintainers: + - name: Ethan Li + email: lietk12@gmail.com + license: Apache-2.0 + sources: + - https://github.com/filebrowser/filebrowser + +deployment: + compose-files: [compose.yml] + description: Provides a filebrowser server + provides: + services: + - description: File browser & manager for PlanktoScope's datasets + port: 9000 + protocol: http + paths: + - /* diff --git a/deployments/mosquitto.deploy.yml b/deployments/mosquitto.deploy.yml new file mode 100644 index 0000000..3b77ebc --- /dev/null +++ b/deployments/mosquitto.deploy.yml @@ -0,0 +1,4 @@ +package: github.com/PlanktoScope/device-pkgs/core/infra/mosquitto +features: + - mqtt-broker +disabled: false diff --git a/deployments/node-red-dashboard.deploy.yml b/deployments/node-red-dashboard.deploy.yml new file mode 100644 index 0000000..1bb883f --- /dev/null +++ b/deployments/node-red-dashboard.deploy.yml @@ -0,0 +1,3 @@ +package: /deployments/node-red-dashboard +features: +disabled: false diff --git a/deployments/node-red-dashboard/compose.yml b/deployments/node-red-dashboard/compose.yml new file mode 100644 index 0000000..c0f6150 --- /dev/null +++ b/deployments/node-red-dashboard/compose.yml @@ -0,0 +1,10 @@ +services: + server: + image: ghcr.io/planktoscope/segmenter-node-red-dashboard:sha-df8fc69 + ports: + - 1880:1880 + volumes: + - ~/.local/share/planktoscope/data/img/:/home/pi/data/img/ + - ~/.local/share/planktoscope/data/export/ecotaxa/:/home/pi/data/export/ecotaxa/ + extra_hosts: + - host.docker.internal:host-gateway diff --git a/deployments/node-red-dashboard/forklift-package.yml b/deployments/node-red-dashboard/forklift-package.yml new file mode 100644 index 0000000..ca92536 --- /dev/null +++ b/deployments/node-red-dashboard/forklift-package.yml @@ -0,0 +1,45 @@ +package: + description: Web GUI for operating the PlanktoScope's segmenter + maintainers: + - name: Ethan Li + email: lietk12@gmail.com + license: GPL-3.0-or-later + sources: + - https://github.com/PlanktoScope/PlanktoScope + +deployment: + compose-files: [compose.yml] + requires: + services: + - tags: [mqtt-broker] + port: 1883 + protocol: mqtt + nonblocking: true + - tags: [planktoscope-api-v2] + port: 1883 + protocol: mqtt + paths: + - /segmenter/segment + - /status/segmenter + - /status/segmenter/name + - /status/segmenter/object_id + - /status/segmenter/metric + # TODO: add fileset dependencies for /home/pi/PlanktoScope/config.json, /home/pi/data/img, + # /home/pi/.local/etc/pkscope-distro/installer-versioning.yml, + provides: + listeners: + - description: Node-RED server providing an interactive dashboard to operate the segmenter + port: 1880 + protocol: tcp + services: + - description: The Node-RED editor for the Node-RED dashboard + port: 1880 + protocol: http + paths: + - / + - description: Node-RED graphical user interface for the segmenter + port: 1880 + protocol: http + paths: + - /ui + - /ui/* diff --git a/deployments/segmenter.deploy.yml b/deployments/segmenter.deploy.yml new file mode 100644 index 0000000..5d0ca9a --- /dev/null +++ b/deployments/segmenter.deploy.yml @@ -0,0 +1,4 @@ +package: /deployments/segmenter +features: + # - dev-src +disabled: false diff --git a/deployments/segmenter/compose-dev-src.yml b/deployments/segmenter/compose-dev-src.yml new file mode 100644 index 0000000..8e8abcd --- /dev/null +++ b/deployments/segmenter/compose-dev-src.yml @@ -0,0 +1,4 @@ +services: + worker: + volumes: + - ~/.local/share/planktoscope/src/device-backend/processing/segmenter:/home/pi/device-backend/processing/segmenter diff --git a/deployments/segmenter/compose.yml b/deployments/segmenter/compose.yml new file mode 100644 index 0000000..3e14f56 --- /dev/null +++ b/deployments/segmenter/compose.yml @@ -0,0 +1,9 @@ +services: + worker: + image: ghcr.io/planktoscope/device-backend-processing-segmenter:sha-ebd357f + volumes: + - ./machine-name:/var/lib/planktoscope/machine-name + - ~/.local/share/planktoscope/data:/home/pi/data + - ~/.local/share/planktoscope/device-backend-logs/processing/segmenter:/home/pi/device-backend-logs/processing/segmenter + extra_hosts: + - host.docker.internal:host-gateway diff --git a/deployments/segmenter/forklift-package.yml b/deployments/segmenter/forklift-package.yml new file mode 100644 index 0000000..734a404 --- /dev/null +++ b/deployments/segmenter/forklift-package.yml @@ -0,0 +1,34 @@ +package: + description: PlanktoScope object segmenter for export to EcoTaxa + maintainers: + - name: Ethan Li + email: lietk12@gmail.com + license: GPL-3.0-or-later + sources: + - https://github.com/PlanktoScope/device-backend + +deployment: + compose-files: [compose.yml] + requires: + services: + - tags: [mqtt-broker] + port: 1883 + protocol: mqtt + nonblocking: false # absence of the MQTT server is not yet handled + provides: + services: + - description: MQTT handling of segmenter commands and broadcasting of segmenter statuses + tags: [planktoscope-api-v2] + port: 1883 + protocol: mqtt + paths: + - /segmenter/segment + - /status/segmenter + - /status/segmenter/name + - /status/segmenter/object_id + - /status/segmenter/metric + +features: + dev-src: + description: Allows the source code to be overwritten from an existing codebase for development & troubleshooting + compose-files: [compose-dev-src.yml] diff --git a/deployments/segmenter/machine-name b/deployments/segmenter/machine-name new file mode 100644 index 0000000..0acc657 --- /dev/null +++ b/deployments/segmenter/machine-name @@ -0,0 +1 @@ +standalone-unknown diff --git a/forklift-pallet.yml b/forklift-pallet.yml new file mode 100644 index 0000000..f730967 --- /dev/null +++ b/forklift-pallet.yml @@ -0,0 +1,6 @@ +forklift-version: v0.5.2 + +pallet: + path: github.com/PlanktoScope/pallet-segmenter + description: An experimental Forklift pallet for running the PlanktoScope's segmenter separately on a computer + readme-file: README.md diff --git a/forklift-repository.yml b/forklift-repository.yml new file mode 100644 index 0000000..6345b38 --- /dev/null +++ b/forklift-repository.yml @@ -0,0 +1,6 @@ +forklift-version: v0.4.0 + +repository: + path: github.com/PlanktoScope/pallet-segmenter + description: Packages to help with running the PlanktoScope's segmenter separately on a computer + readme-file: README.md diff --git a/requirements/repositories/github.com/PlanktoScope/device-pkgs/forklift-version-lock.yml b/requirements/repositories/github.com/PlanktoScope/device-pkgs/forklift-version-lock.yml new file mode 100644 index 0000000..8a411a6 --- /dev/null +++ b/requirements/repositories/github.com/PlanktoScope/device-pkgs/forklift-version-lock.yml @@ -0,0 +1,4 @@ +type: pseudoversion +tag: v2024.0.0-alpha.0 +timestamp: "20240227034045" +commit: 3809b4ed589ee267ee37e7f57d6dc1f2b4465eb4