DAQ uses a simple Firebase-hosted web page to provide a dynamic dashboard of test results. There's three parts, which are needed at different stages of deployment/updates.
- Initial setup (for deploying a new GCP project)
- Code deploy (to deploy a new or updated version of the code)
- Authentication (to authenticate any new users to the web page)
After this setup, or if using an already existing setup with a new DAQ install, follow the service account setup instructions.
Note that these instructions keep changing everytime the web-flow for Firebase changes... it's neigh impossible to keep everything synchronized. The basic steps are always the same, but just the details and ordering shifts now and then. Don't panic.
- Identify a GCP project that you can to host the system. Acquire a new one if necessary.
- Goto the Firebase Console and add a new project.
- Add the hosting GCP project to link it to this Firebase setup.
- Chose an appropriate billing option. The Blaze plan should be fine.
- Disable Google Analytics, unless you also want to setup an account for that.
- Continue on to add Firebase to your GCP project.
- Enable a native mode database
- https://console.firebase.google.com/
- Select your project.
- Select
Databasefrom the options on the left. - Create Database -- you do not want the realtime data base or datastore mode.
- Create it in production mode.
- Add a web-app to the Firebase project.
- https://console.firebase.google.com/
- Select your project.
- Select "+ Add app" (or this may be auto-selected for you).
- Select "</>" (Web) to add a new web-app.
- Use a clever nickname and register app. Not sure if/how/when this matters.
- Enable Firebase Hosting
- Skip the part about adding the Firebase SDK.
- Follow the instructions for installing the Firebase CLI.
- Ignore the bit about "Deploy to Firebase Hosting" for now.
- Get your firebase config object for a web app.
- Copy the
const firebaseConfig = { ... }snippet tolocal/firebase_config.js
- Copy the
- Add an API key restriction:
- Go to API Key Restriction
- There should be an API Keys as a Browser key (auto creatd by Firebase)
- Edit this, and add an HTTP Referrer, which will be the https:// address of the daq hosted web app:
https://your-project-name.firebaseapp.com/*
- Enable Google sign-in from
- https://console.firebase.google.com/
- Select your project.
- Select "Authentication"
- Select "Sign-in method"
- Enable "Google" sign-in.
- Install Firebase function dependencies.
- Make sure
npmis installed. - Go to
firebase/functionsand runnpm install.
- Make sure
- Enable appengine on GCP
- Go to GCP AppEngine console
- Enable!
- In
firebase/, run./deploy.shto deploy to your project.- Follow the link to the indicated Hosting URL to see the newly installed pages.
- You will likely see an Authentication failed message. See next section for details.
Firestore rules are enforced requiring enabled user login to access data and reports. There's two phases to this process:
- Web-app needs to be configured and deployed with appropriate web-app credentials (see above).
- Users need to access the assigned web-app, and sign in. Initially, they will be denied access.
- The system administrator will need to run
bin/user_enableany time there is a new user.
The data for DAQ reporting through to Firebase goes through a number of distinct steps. This section outlines the basic steps to help diagnose and debug the system, past "it doesn't work!"
- DAQ GCP Connection:
When the system starts up and runs, there should be some descriptive log messages that highlight
operation, with obvious error messages when something goes wrong:
... INFO:gcp:Loading gcp credentials from local/daq-testlab-de56aa4b1e47.json INFO:gcp:Initialized gcp pub/sub daq-project:daq-testlab INFO:gcp:Initialized gcp firestore daq-project:daq-testlab ... INFO:gcp:Uploaded test report to inst/report_9a02571e8f00.txt ...
-
PubSub Topic & Subscription: On the GCP PubSub Topics page, there should be an entry for a
projects/daq-project/topics/daq_runnertopic, with at least one subscription to something likeprojects/daq-project/subscriptions/gcf-daq_firestore-daq_runner, which is the linked Firestore cloud function. -
Cloud Function: The GCP Cloud Functions page should show a
daq_firestorefunction, and if you look at the logs there should beinfoevents for each reported stage of DAQ test, e.g.:I daq_firestore 210626605171528 updating 1537561337615 daq-laptop port-undefined undefined status daq_firestore 210626605171528 I daq_firestore 210625487917242 updating 1537561337804 daq-laptop port-1 5ba552f2 sanity daq_firestore 210625487917242 I daq_firestore 210621644751867 updating 1537561338017 daq-laptop port-1 5ba552f2 info daq_firestore 210621644751867 I daq_firestore 210625202516197 updating 1537561338223 daq-laptop port-1 5ba552f2 dhcp daq_firestore 210625202516197 I daq_firestore 210621107285036 updating 1537561362562 daq-laptop port-1 5ba552f2 info daq_firestore 210621107285036 I daq_firestore 210617247716870 updating 1537561362716 daq-laptop port-1 5ba552f2 dhcp daq_firestore 210617247716870 I daq_firestore 210615717506431 updating 1537561362986 daq-laptop port-1 5ba552f2 base daq_firestore 210615717506431
-
Firebase Use Logs: The same Cloud Functions are represented in Firebase and can be viewed as part of the Firebase Functions Usage Logs (note that you'll have to manually replace
daq-projectwith the appropriate{projectId}in order for that link to work). The logs there will also show any signficant errors highlighting most problems. -
Firestore Database: The data ends up in the Firestore Database (again replacing
daq-projectwith the appropraite{projectId}) and filed underorigin/{accountId}(the name of the service account) of the DAQ install supplying the data. -
Web Application: The test Web Application (again, will have to substitute the appropriate
projectId), should show a list of all accounts with ingested data. If nothing is showing here, or theaccountIdis missing, check the web dev console to see if there's any obvious errors.
The high-level flow for configuration data is shown below. This is not a complete outlay, since it doesn't cover all the cases, but it gives a rough indicator of how the flow works. This same overall process applies to both the base system configuration as well as device-specific configs. Don't take this diagram as gospel, the only true truth comes from the source.
+-------+ +-----+ +-------+ +-----------+ +---------+
| Disk | | DAQ | | Test | | Firebase | | WebApp |
+-------+ +-----+ +-------+ +-----------+ +---------+
| --------------\ | | | |
| | DAQ startup |-| | | |
| |-------------| | | | |
| | | | |
| Raw Config | | | |
|------------------------>| | | |
| | | | |
| Merged Config (aux/) | | | |
|<------------------------| | | |
| | | | |
| | Raw & Merged | | |
| |----------------------------->| |
| | -------------\ | | |
| |-| Test Start | | | |
| | |------------| | | |
| | | | |
| Merged Config (/config/device/) | | |
|------------------------------------------->| | |
| | | | |
| | Merged (active) | | |
| |----------------------------->| |
| | | | ----------------\ |
| | | | | Page (re)load |-|
| | | | |---------------| |
| | | | |
| | | | Raw & Merged |
| | | |--------------------->|
| | | | | ------------\
| | | | |-| JSON Edit |
| | | | | |-----------|
| | | | |
| | | | Raw |
| | | |<---------------------|
| | | | | -----------------\
| | | | |-| "Saving" state |
| | | | | |----------------|
| | | | |
| | | Raw | |
| |<-----------------------------| |
| | | | |
| Raw & Merged | | | |
|<------------------------| | | |
| | | | |
| | Merged (next) | | |
| |----------------------------->| |
| | | | |
| | | | Merged (next) |
| | | |--------------------->|
| | | | | ----------------------\
| | | | |-| "Provisional" state |
| | | | | |---------------------|
| | -------------\ | | |
| |-| Test Start | | | |
| | |------------| | | |
| | | | |
| Merged Config (/config/device/) | | |
|------------------------------------------->| | |
| | | | |
| | Merged (active) | | |
| |----------------------------->| |
| | | | |
| | | | Merged (active) |
| | | |--------------------->|
| | | | | -----------------\
| | | | |-| "Normal" state |
| | | | | |----------------|
| | | | |
Generated with Text Art Sequence Generator with the contents:
object Disk DAQ Test Firebase WebApp
note left of DAQ: DAQ startup
Disk -> DAQ: Raw Config
DAQ -> Disk: Merged Config (aux/)
DAQ -> Firebase: Raw & Merged
note right of DAQ: Test Start
Disk -> Test: Merged Config (/config/device/)
DAQ -> Firebase: Merged (active)
note left of WebApp: Page (re)load
Firebase -> WebApp: Raw & Merged
note right of WebApp: JSON Edit
WebApp -> Firebase: Raw
note right of WebApp: "Saving" state
Firebase -> DAQ: Raw
DAQ -> Disk: Raw & Merged
DAQ -> Firebase: Merged (next)
Firebase -> WebApp: Merged (next)
note right of WebApp: "Provisional" state
note right of DAQ: Test Start
Disk -> Test: Merged Config (/config/device/)
DAQ -> Firebase: Merged (active)
Firebase -> WebApp: Merged (active)
note right of WebApp: "Normal" state
Generated with dbdiagram.io with this file