fix: addressing feedback from testing (#4)

* feat: detailed docs
NeonGeckoCom · Oct 21, 2024 · d8251bf · d8251bf
1 parent fff9ecc
commit d8251bf
Show file tree

Hide file tree

Showing 24 changed files with 473 additions and 327 deletions.
diff --git a/.gitignore b/.gitignore
@@ -2,3 +2,4 @@
 neon_hub_secrets.yaml
 xdg/
 .DS_Store
+.venv/
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # Neon Hub
 
-Neon Hub is a central server for artificial intelligence, powered by Neon.AI. It is designed to be a private, offline, and secure alternative to cloud-based AI assistants like Alexa, Google Assistant, and Siri. A Neon Hub can run on any consumer computer built within the last 5 years, running the Linux operating system, and can be accessed from any device on the same network. Neon Hub is designed to be easy to set up and use, with a web interface for managing services and a RESTful API for developers. A GPU is not required and is currently not supported, but future versions will support GPU acceleration.
+Neon Hub is a central server for artificial intelligence, powered by Neon AI®. It is designed to be a private, offline, and secure alternative to cloud-based AI assistants like Alexa, Google Assistant, and Siri. A Neon Hub can run on any consumer computer built within the last 5 years, running the Linux operating system, and can be accessed from any device on the same network. Neon Hub is designed to be easy to set up and use, with a web interface for managing services and a RESTful API for developers. A GPU is not required and is currently not supported, but future versions will support GPU acceleration.
 
 A Neon Hub can be used with any number of Neon Nodes, which can be as small as a Raspberry Pi Zero W. Nodes can be placed throughout a home or office, and can be used to interact with the Hub using voice commands, text, or a web interface. The Hub can be used to control smart home devices, answer questions, play music, and more.
 
@@ -13,206 +13,8 @@ Neon Hub is perfect for:
 - Hospitals
 - Hotels
 
-## System Requirements
+## Documentation
 
-| Component | Minimum | Recommended    |
-| --------- | ------- | -------------- |
-| CPU       | 2 cores | 4 cores        |
-| Memory    | 4GB     | 8GB            |
-| Disk      | 77GB    | 100GB SSD/NVME |
+Detailed documentation is available at [https://neongeckocom.github.io/neon-hub-installer](https://neongeckocom.github.io/neon-hub-installer).
 
-~25GB for Docker images, 7GB for OS, 45GB for data
-
-Neon Hub runs on both x86 and ARM CPUs.
-
-### Approximate Docker image sizes
-
-| Service          | Size     |
-| ---------------- | -------- |
-| neon-gui         | 826MB    |
-| neon-enclosure   | 1.2GB    |
-| neon-audio       | 3GB      |
-| neon-skills      | 2.6GB    |
-| neon-speech      | 3.6GB    |
-| neon-messagebus  | 778MB    |
-| neon-api-proxy   | 746MB    |
-| neon-hana        | 443MB    |
-| neon-iris        | 1.78GB   |
-| neon-iris-websat | 1.61GB   |
-| libretranslate   | 5.1GB    |
-| coqui            | 1.3GB    |
-| fasterwhisper    | 1.94GB   |
-| yacht            | 415MB    |
-| **Total**        | 25.338GB |
-
-## Exposed services and their ports
-
-### Speech-To-Text (STT)
-
-Also known as Automatic Speech Recognition (ASR), this is what enables the assistant to take your recorded voice and turn it into text that it can parse.
-
-Different STT engines have different tradeoffs. For example, Neon's custom NeMo citrinet model is extremely fast (even on Raspberry Pi), but its quality is not as good as FasterWhisper, and it cannot handle heavily accented English as well. FasterWhisper has several model sizes available and, depending on your tolerance for waiting on the assistant, you can trade speed for quality and vice versa.
-
-At this time, Neon Hub only ships with fasterwhisper.
-
-- fasterwhisper: `http://neon-hub.local:8080`
-- nemo: `http://neon-hub.local:8081`
-
-### Text-To-Speech (TTS)
-
-TTS allows the assistant to talk to you. Currently, Neon Hubs only ship with a custom Coqui model, which is optimized for Raspberry Pi CPU inference and performs extremely well on x86 processors.
-
-- coqui: `http://neon-hub.local:9666`
-
-### Translation
-
-Neon Hub ships with Libretranslate for fully private and offline language translation.
-
-- libretranslate: `http://neon-hub.local:5000`
-
-### HTTP Services
-
-Neon Hub uses HANA for RESTful API communications among different services. Non-developers will never need to use it, but it is available. Endpoint documentation and testing is available at `http://neon-hub.local:8082/docs`.
-
-Neon Hub also has two web variations of Iris. At `http://neon-hub.local:7860` there is a Gradio interface where you can type questions to Neon, speak directly to Neon, drop WAV files to speak to Neon, or change your personal information.
-
-At `http://neon-hub.local:8001` there is a chat interface that includes a wakeword ("Hey Neon") for full voice interaction, similar to a smart speaker like Alexa. This interface is compatible with any modern smartphone, tablet, or computer. _At this time, Apple iOS devices do not have wake word support with audio playback due to security constraints imposed by Apple._
-
-In order to fully leverage the Iris websat, you must either enable HTTPS and accept the self-signed certificate warning in your browser, or provide your own HTTPS certificate and configure DNS.
-
-- hana: `http://neon-hub.local:8082`
-- iris: `http://neon-hub.local:7860`
-- iris-websat: `http://neon-hub.local:8001`
-
-#### Custom SSL certificate
-
-The `nginx` service expects a public and private key pair to be located at `/home/neon/$HOSTNAME.local.crt` and `/home/neon/$HOSTNAME.local.key`, with the default value of `$HOSTNAME` being neon-hub.local. If you would like to use your own certificate, you can replace the existing files with your own and restart the nginx service in Yacht.
-
-### Service management
-
-Neon Hub leverages Docker Compose for container management. For ease of viewing logs and managing services, Yacht is installed on the server at `http://neon-hub.local:8000/#/apps`. # TODO: [How to set your password](https://yacht.sh/docs/Pages/User_Settings), how to view and use, etc.
-
-If you'd like to disable default services, this is the place to do it. This is your Hub - use only what you want!
-
-## Post-Installation
-
-### Configuration
-
-All configuration is done in the `neon.yaml` file. This file is located in the `/home/neon/xdg/neon/neon.yaml` directory by default. After making changes to the file, you must restart the Neon core services in Yacht for the changes to take effect.
-
-Neon core services:
-
-- neon-gui
-- neon-enclosure
-- neon-audio
-- neon-skills
-- neon-speech
-- neon-messagebus
-
-### Installing additional skills and plugins
-
-Each Neon Core service is a Docker container. To install a new service, you need to add it to the `neon.yaml` file and then restart the services.
-
-For each package you want to install, add a [pip-compatible package name](https://pip.pypa.io/en/stable/reference/requirements-file-format/) to the `default_skills`, `extra_dependencies.voice`, `extra_dependencies.audio`, or `extra_dependencies.phal` lists in the `neon.yaml` file.
-
-Examples:
-
-```yaml
-default_skills: # Add skills to install here
-  - neon-skill-alerts
-  # - other pip-compatible package names
-extra_dependencies:
-  global: # Installs in all containers/services
-    - requests
-  skills: # Add neon-skill dependencies to install here
-    - neon-skill-alerts
-  voice: # Add neon-speech dependencies to install here
-    - ovos-stt-server
-  audio: # Add neon-audio dependencies to install here
-    - ovos-tts-plugin-beepspeak
-  enclosure: # Add neon-enclosure dependencies to install here
-    - ovos-PHAL-plugin-homeassistant
-```
-
-### Skill settings
-
-Each skill has its own settings file located in the `/home/neon/xdg/neon/skills/$SKILL_NAME/settings.json` directory. These settings can be modified to change the behavior of the skill. Some skills require you to restart the neon-skills service in Yacht for the changes to take effect, although many skills automatically read the settings file each time and do not require a restart.
-
-### External services
-
-TODO:
-alpha_vantage:
-api_key: CUSTOM_KEY_HERE
-open_weather_map:
-api_key: CUSTOM_KEY_HERE
-wolfram_alpha:
-api_key: CUSTOM_KEY_HERE
-
-### Troubleshooting
-
-- rmq-admin: `http://neon-hub.local:15672`
-- mana
-  - `docker exec -it $(docker ps -q -f name=neon-messagebus) mana -h`
-
-#### Failure to install Docker
-
-`sudo rm /etc/apt/sources.list.d/docker*`
-
-### Services
-
-Neon Hub comes with an nginx reverse proxy that routes traffic to the appropriate service. The following paths are available:
-
-| Service        | Friendly URL                            | URL with port                 |
-| -------------- | --------------------------------------- | ----------------------------- |
-| Libretranslate | `https://libretranslate.neon-hub.local` | `http://neon-hub.local:5000`  |
-| Fasterwhisper  | `https://fasterwhisper.neon-hub.local`  | `http://neon-hub.local:8080`  |
-| Coqui          | `https://coqui.neon-hub.local`          | `http://neon-hub.local:9666`  |
-| HANA           | `https://hana.neon-hub.local`           | `http://neon-hub.local:8082`  |
-| Iris           | `https://iris.neon-hub.local`           | `http://neon-hub.local:7860`  |
-| Iris-Websat    | `https://iris-websat.neon-hub.local`    | `http://neon-hub.local:8001`  |
-| Yacht          | `https://yacht.neon-hub.local`          | `http://neon-hub.local:8000`  |
-| RMQ-Admin      | `https://rmq-admin.neon-hub.local`      | `http://neon-hub.local:15672` |
-
-Please note that the Iris-Websat service will only work with HTTPS, although you can see your chat history and the Iris interface at `http://neon-hub.local:8001`.
-
-### Local access to addresses
-
-#### Hosts file
-
-Add the following to your `/etc/hosts` file:
-
-```bash
-10.10.10.10 neon-hub.local fasterwhisper.neon-hub.local coqui.neon-hub.local libretranslate.neon-hub.local hana.neon-hub.local iris.neon-hub.local iris-websat.neon-hub.local yacht.neon-hub.local rmq-admin.neon-hub.local
-```
-
-Replace `10.10.10.10` with the IP address of your Neon Hub. (TODO: Maybe. We can probably just exclude neon-hub.local from the hosts file for appliances.)
-
-#### DNS
-
-If you have a DNS server, you can add the following records:
-
-```bash
-neon-hub.local. IN A 10.10.10.10
-fasterwhisper.neon-hub.local. IN A 10.10.10.10
-coqui.neon-hub.local. IN A 10.10.10.10
-libretranslate.neon-hub.local. IN A 10.10.10.10
-hana.neon-hub.local. IN A 10.10.10.10
-iris.neon-hub.local. IN A 10.10.10.10
-iris-websat.neon-hub.local. IN A 10.10.10.10
-yacht.neon-hub.local. IN A 10.10.10.10
-rmq-admin.neon-hub.local. IN A 10.10.10.10
-```
-
-Replace `10.10.10.10` with the IP address of your Neon Hub.
-
-## NEXT STEPS
-
-- [ ] Installer script with whiptail
-- [x] Self-signed TLS certificate
-- [x] Traefik/nginx
-- [x] Dependencies - we probably want neon_messagebus to be dependent on neon_rabbitmq so Iris works
-
-## Non-root containers
-
-- Iris and Iris-Websat
-- FasterWhisper
+It can be accessed locally by [installing mkdocs](https://www.mkdocs.org/getting-started/) and running `mkdocs serve` in the root of the repository.
diff --git a/ansible/hub.yaml b/ansible/hub.yaml
@@ -105,59 +105,28 @@
         append: yes
         groups: docker
     # Set up prerequisite folders
-    - name: Create {{ xdg_dir }} folder with 2775 permissions
+    - name: Create Neon folders with 2775 permissions
       ansible.builtin.file:
-        path: "{{ xdg_dir }}"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create /home/neon/compose folder with 2775 permissions
-      ansible.builtin.file:
-        path: "/home/neon/compose"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create /home/neon/.config/pulse folder with 2775 permissions
-      ansible.builtin.file:
-        path: "/home/neon/.config/pulse"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create {{ xdg_dir }}/config folder with 2775 permissions
-      ansible.builtin.file:
-        path: "{{ xdg_dir }}/config"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create {{ xdg_dir }}/local folder with 2775 permissions
-      ansible.builtin.file:
-        path: "{{ xdg_dir }}/local"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create {{ xdg_dir }}/share folder with 2775 permissions
-      ansible.builtin.file:
-        path: "{{ xdg_dir }}/share"
-        state: directory
-        mode: "2775" # Keeps the group sticky
-        owner: neon
-        group: neon
-    - name: Create {{ xdg_dir }}/config/neon folder with 2775 permissions
-      ansible.builtin.file:
-        path: "{{ xdg_dir }}/config/neon"
+        path: "{{ item }}"
         state: directory
         mode: "2775" # Keeps the group sticky
         owner: neon
         group: neon
+      loop:
+        - "{{ xdg_dir }}"
+        - "{{ xdg_dir }}/local"
+        - "{{ xdg_dir }}/share"
+        - "{{ xdg_dir }}/config"
+        - "{{ xdg_dir }}/config/neon"
+        - "/home/neon/compose"
+        - "/home/neon/.config/pulse"
     - name: Create {{ xdg_dir }}/config/rabbitmq folder
       ansible.builtin.file:
         path: "{{ xdg_dir }}/config/rabbitmq"
         state: directory
+        mode: "2775"
+        owner: 999
+        group: 999
     # Certificate generation
     - name: Generate self-signed certificate
       import_tasks: generate-certificate.yaml
@@ -166,13 +135,17 @@
       include_vars:
         file: "{{ secrets_file }}"
       # no_log: true
-    - name: Create rabbitmq.json
+    - name: Create rabbitmq files
+      loop:
+        - rabbitmq.json
+        - rabbitmq.conf
+        - enabled_plugins
       ansible.builtin.template:
-        src: templates/rabbitmq.json.j2
-        dest: "{{ xdg_dir }}/config/rabbitmq/rabbitmq.json"
+        src: "templates/{{ item }}.j2"
+        dest: "{{ xdg_dir }}/config/rabbitmq/{{ item }}"
         mode: "0644"
-        owner: neon
-        group: neon
+        owner: 999
+        group: 999
     - name: Create neon.yaml
       ansible.builtin.template:
         src: templates/neon.yaml.j2
@@ -187,13 +160,6 @@
         mode: "0644"
         owner: neon
         group: neon
-    - name: Create rabbitmq.conf
-      ansible.builtin.template:
-        src: templates/rabbitmq.conf
-        dest: "{{ xdg_dir }}/config/rabbitmq/rabbitmq.conf"
-        mode: "0644"
-        owner: neon
-        group: neon
     # Docker services
     - name: Copy Yacht compose file
       ansible.builtin.template:
@@ -231,7 +197,7 @@
         owner: neon
         group: neon
     # Deploy
-    - name: Start Core and Hub
+    - name: Download and Start Core and Hub
       community.docker.docker_compose_v2:
         project_src: /home/neon/compose
         project_name: neon
@@ -255,3 +221,8 @@
       delay: 3
       environment:
         DOCKER_TIMEOUT: "900"
+    - name: Create file indicating successful deployment
+      ansible.builtin.file:
+        path: "{{ xdg_dir }}/neon-installed"
+        state: touch
+        mode: "2775"
diff --git a/ansible/templates/diana.yaml.j2 b/ansible/templates/diana.yaml.j2
@@ -65,11 +65,10 @@ websocket:
   host: neon-messagebus
 listener:
   enable_voice_loop: False
-# keys:
-#   api_services:
-#     alpha_vantage:
-#       api_key: CUSTOM_KEY_HERE
-#     open_weather_map:
-#       api_key: CUSTOM_KEY_HERE
-#     wolfram_alpha:
-#       api_key: CUSTOM_KEY_HERE
+# api_services:
+#   alpha_vantage:
+#     api_key: CUSTOM_KEY_HERE
+#   open_weather_map:
+#     api_key: CUSTOM_KEY_HERE
+#   wolfram_alpha:
+#     api_key: CUSTOM_KEY_HERE
diff --git a/ansible/templates/enabled_plugins.j2 b/ansible/templates/enabled_plugins.j2
@@ -0,0 +1 @@
+[rabbitmq_management].