How to test front-end changes?

[jnm]

I would like to test front-end changes to KPI code, e.g. while reviewing PRs. I do not want to use npm run watch, because no production build uses this. I would prefer not to have to rebuild and recreate all containers each time I change something in KPI, because this is slow (in excess of 3.5 minutes on my machine).

Here's what I tried, to no avail:

1. Cloned fresh copies of the KPI and KoBoCAT repositories;
2. Cloned a fresh kobo-install;
3. Set up my environment with run.py, instructing it to use developer mode and giving it the paths to the KPI and KoBoCAT source directories I'd just cloned;
4. Waited for everything to start up and verified that it worked;
5. Made a single change to an ES6 file in KPI;
6. Ran run.py --build-kpi;
7. Checked to see if my changes were visible in the browser or had made it into the compiled static files (they hadn't);
8. After that completed, ran run.py again, which recreated all containers;
9. Checked again for my changes in the browser and in the static files: they had not taken effect.

❓ Am I doing something wrong, or am I trying to do something we haven't made possible yet?

Here is the (very long) transcript of my shell session:

Starting fresh and cloning the source repositories:

john@scrappy:/tmp/kit$ sudo find -delete
john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kpi
Cloning into 'kpi'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 35185 (delta 0), reused 0 (delta 0), pack-reused 35184
Receiving objects: 100% (35185/35185), 11.94 MiB | 7.96 MiB/s, done.
Resolving deltas: 100% (26851/26851), done.
john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobocat
Cloning into 'kobocat'...
remote: Enumerating objects: 216, done.
remote: Counting objects: 100% (216/216), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 55268 (delta 132), reused 164 (delta 114), pack-reused 55052
Receiving objects: 100% (55268/55268), 23.14 MiB | 7.35 MiB/s, done.
Resolving deltas: 100% (39126/39126), done.

Cloning and running kobo-install:

john@scrappy:/tmp/kit$ git clone [email protected]:kobotoolbox/kobo-install
Cloning into 'kobo-install'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 549 (delta 19), reused 25 (delta 11), pack-reused 513
Receiving objects: 100% (549/549), 129.61 KiB | 3.50 MiB/s, done.
Resolving deltas: 100% (364/364), done.
john@scrappy:/tmp/kit$ cd kobo-install/
john@scrappy:/tmp/kit/kobo-install$ python run.py
╔═══════════════════════════════════════════════════════════════╗
║ Welcome to `kobo-install`!                                    ║
║                                                               ║
║ You are going to be asked some questions that will            ║
║ determine how to build the configuration of `KoBoToolBox`.    ║
║                                                               ║
║ Some questions already have default values (within brackets). ║
║ Just press `enter` to accept the default value or enter `-`   ║
║ to remove previously entered value.                           ║
║ Otherwise choose between choices or type your answer.         ║
╚═══════════════════════════════════════════════════════════════╝
Where do you want to install?
[/tmp/kit/kobo-docker]: 
Do you want to see advanced options?
	1) Yes
	2) No
[2]: 1
What kind of installation do you need?
	1) On your workstation
	2) On a server
[2]: 1
Please choose which network interface you want to use?
	other) Other
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
[wlan0]: other
	lo) 127.0.0.1
	br-206827b65b4d) 172.24.0.1
	lxcbr0) 10.0.3.1
	wlan0) 192.168.1.39
	docker0) 172.17.0.1
[wlan0]: docker0
SMTP server: 
SMTP port [25]: 
SMTP user: 
From email address [[email protected]]: 
Super user's username [super_admin]: super
Super user's password [hdshIONMPviP]: duper
Docker Compose prefix? (leave empty for default): 
Web server port?
[80]: 
Developer mode?
	1) Yes
	2) No
[2]: 1
╔═══════════════════════════════════════════════════════════╗
║ Where are the files located locally? It can be absolute   ║
║ or relative to the directory of the installation.         ║
║ Leave empty if you don't need to overload the repository. ║
╚═══════════════════════════════════════════════════════════╝
KoBoCat files location: /tmp/kit/kobocat
KPI files location: /tmp/kit/kpi
Enable DEBUG?
	1) True
	2) False
[2]: 1
Postgres database [kobotoolbox]: 
Postgres user [kobo]: 
Postgres password [YJnEdBFnm5]: 
Do you want to tweak PostgreSQL settings?
	1) Yes
	2) No
[2]: 
Do you want to customize service ports?
	1) Yes
	2) No
[2]: 
Do you want to use AWS S3 storage?
	1) Yes
	2) No
[2]: 
Google Analytics Identifier: 
Google API Key: 
Intercom App ID: 
Do you want to use Sentry?
	1) Yes
	2) No
[2]: 
Do you want to tweak uWSGI settings?
	1) Yes
	2) No
[2]: 
Do you want to activate backups?
	1) Yes
	2) No
[2]: 
Cloning into '/tmp/kit/kobo-docker'...
remote: Enumerating objects: 2546, done.
remote: Total 2546 (delta 0), reused 0 (delta 0), pack-reused 2546
Receiving objects: 100% (2546/2546), 1.84 MiB | 6.38 MiB/s, done.
Resolving deltas: 100% (1580/1580), done.
Switched to a new branch 'kobo-install'
From https://github.com/kobotoolbox/kobo-docker
 * branch            kobo-install -> FETCH_HEAD
╔═══════════════════════════════════════════════════════════════════╗
║ Administrative privileges are required to update your /etc/hosts. ║
╚═══════════════════════════════════════════════════════════════════╝
Do you want to review your /etc/hosts file before overwriting it?
	1) Yes
	2) No
[2]: 
Stopping kobo-docker_redis_cache_1 ... done
Stopping kobo-docker_redis_main_1  ... done
Stopping kobo-docker_rabbit_1      ... done
Stopping kobo-docker_mongo_1       ... done
Removing kobo-docker_redis_cache_1 ... done
Removing kobo-docker_redis_main_1  ... done
Removing kobo-docker_postgres_1    ... done
Removing kobo-docker_rabbit_1      ... done
Removing kobo-docker_mongo_1       ... done
Removing network kobo-docker_default
Removing network kobo-docker_kobo-fe-network
WARNING: Network kobo-docker_kobo-fe-network not found.
Launching environment
Creating network "kobo-docker_default" with the default driver
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_redis_cache_1 ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_redis_main_1  ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_postgres_1, kobo-docker_rabbit_1, kobo-docker_redis_main_1, kobo-docker_redis_cache_1, kobo-docker_mongo_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Pulling nginx (kobotoolbox/nginx:latest)...
Building kobocat
WARNING: Image for service kobocat was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Pulling enketo_express (kobotoolbox/enketo-express-extra-widgets:1.72.2)...
Building kpi
WARNING: Image for service kpi was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_kobocat_1        ... done
Creating kobo-docker_enketo_express_1 ... done
Waiting for environment to be ready. It can take a few minutes.
..
`KoBoToolbox` has not started yet, wait for another 600 minutes!
......................
╔════════════════════════════╗
║ Ready                      ║
║ URL: http://kf.kobo.local/ ║
║ User: super                ║
║ Password: duper            ║
╚════════════════════════════╝

Looking for the string creating your first project, which I will later change:

john@scrappy:/tmp/kit/kobo-install$ cd ../kobo-docker/
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first project' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
./.vols/static/kpi/js/components/searchcollectionlist.es6
./.vols/static/kpi/compiled/app-5a551488c6647ae95445.js
./.vols/static/kpi/compiled/app-05390ab1917b6af887af.js
grep: ./.vols/db: Permission denied

I make the following change to KPI, outside this shell:

diff --git a/jsapp/js/components/searchcollectionlist.es6 b/jsapp/js/components/searchcollectionlist.es6
index 965a251e..449bd3fd 100644
--- a/jsapp/js/components/searchcollectionlist.es6
+++ b/jsapp/js/components/searchcollectionlist.es6
@@ -249,7 +249,7 @@ class SearchCollectionList extends Reflux.Component {
                         return (
                           <bem.Loading>
                             <bem.Loading__inner>
-                              {t('Let\'s get started by creating your first project. Click the New button to create a new form.')}
+                              {t('Let\'s get started by creating your first Martian. Click the New button to create a new form.')}
                               <div className='pro-tip'>
                               {t('Advanced users: You also drag and drop XLSForms here and they will be uploaded and converted to projects.')}
                               </div>

Rebuilding KPI and seeing if the change made it through:

john@scrappy:/tmp/kit/kobo-docker$ cd -
/tmp/kit/kobo-install
john@scrappy:/tmp/kit/kobo-install$ python run.py --build-kpi
Building kpi
john@scrappy:/tmp/kit/kobo-install$ cd -
/tmp/kit/kobo-docker
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first Martian' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
grep: ./.vols/db: Permission denied

Since no results were found after rebuilding KPI, I'm now using run.py with no arguments to recreate all containers:

john@scrappy:/tmp/kit/kobo-docker$ cd -
/tmp/kit/kobo-install
john@scrappy:/tmp/kit/kobo-install$ time python run.py
Stopping kobo-docker_postgres_1    ... done
Stopping kobo-docker_rabbit_1      ... done
Stopping kobo-docker_redis_main_1  ... done
Stopping kobo-docker_redis_cache_1 ... done
Stopping kobo-docker_mongo_1       ... done
WARNING: Found orphan containers (kobo-docker_kobocat_1, kobo-docker_kpi_1, kobo-docker_nginx_1, kobo-docker_enketo_express_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Removing kobo-docker_postgres_1    ... done
Removing kobo-docker_rabbit_1      ... done
Removing kobo-docker_redis_main_1  ... done
Removing kobo-docker_redis_cache_1 ... done
Removing kobo-docker_mongo_1       ... done
Removing network kobo-docker_default
Stopping kobo-docker_kobocat_1        ... done
Stopping kobo-docker_kpi_1            ... done
Stopping kobo-docker_nginx_1          ... done
Stopping kobo-docker_enketo_express_1 ... done
Removing kobo-docker_kobocat_1        ... done
Removing kobo-docker_kpi_1            ... done
Removing kobo-docker_nginx_1          ... done
Removing kobo-docker_enketo_express_1 ... done
Removing network kobo-docker_kobo-fe-network
Launching environment
Creating network "kobo-docker_default" with the default driver
Creating kobo-docker_redis_cache_1 ... done
Creating kobo-docker_postgres_1    ... done
Creating kobo-docker_mongo_1       ... done
Creating kobo-docker_rabbit_1      ... done
Creating kobo-docker_redis_main_1  ... done
Creating network "kobo-docker_kobo-fe-network" with driver "bridge"
WARNING: Found orphan containers (kobo-docker_postgres_1, kobo-docker_redis_main_1, kobo-docker_rabbit_1, kobo-docker_mongo_1, kobo-docker_redis_cache_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Creating kobo-docker_nginx_1          ... done
Creating kobo-docker_enketo_express_1 ... done
Creating kobo-docker_kpi_1            ... done
Creating kobo-docker_kobocat_1        ... done
Waiting for environment to be ready. It can take a few minutes.
..................
╔════════════════════════════╗
║ Ready                      ║
║ URL: http://kf.kobo.local/ ║
║ User: super                ║
║ Password: duper            ║
╚════════════════════════════╝

real	3m33.192s
user	0m2.287s
sys	0m0.376s

Now looking again for my modification in the compiled static files:

john@scrappy:/tmp/kit/kobo-install$ cd -
/tmp/kit/kobo-docker
john@scrappy:/tmp/kit/kobo-docker$ grep -lFR 'creating your first Martian' .
grep: ./log/postgres/postgresql-2019-01-20_000000.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-20_024825.log: Permission denied
grep: ./log/postgres/postgresql-2019-01-19_232518.log: Permission denied
grep: ./.vols/db: Permission denied

Since there are still no results, here's a sanity check in the source directory:

john@scrappy:/tmp/kit/kobo-docker$ cd ../kpi
john@scrappy:/tmp/kit/kpi$ grep -lFR 'creating your first Martian' .
grep: ./jsapp/fonts: No such file or directory
./jsapp/js/components/searchcollectionlist.es6
grep: ./jsapp/compiled: No such file or directory
grep: ./node_modules: No such file or directory

In case it's helpful: run.conf (formatted).zip

[jnm]

Here's what I've been doing:

kpi$ cat static_shock.sh 
rm -fr staticfiles
npm run copy-fonts &&
npm run build &&
./manage.py collectstatic --noinput &&
rsync -aq --delete --chown=www-data "${KPI_SRC_DIR}/staticfiles/" "${NGINX_STATIC_DIR}/"

I run that script inside the kpi container and regenerates the static files.

[jnm]

update: make sure you restart uWSGI after this (or just restart the whole kpi container). also, purge jsapp/compiled before rebuilding as stale app-*.js bundles can hide out there as well.

rm -fr staticfiles
rm -fr jsapp/compiled
npm run copy-fonts &&
npm run build &&
./manage.py collectstatic --noinput &&
rsync -aq --delete --chown=www-data "${KPI_SRC_DIR}/staticfiles/" "${NGINX_STATIC_DIR}/"

[jnm]

here's a script that takes advantage of the up-to-date files in the most recently built Docker image instead of calling npm run build again. see the echo text inside the script for a more detailed explanation.

(moved to gist)

[noliveleger]

@jnm what about calling your script in the --build commands?

[jnm]

Sounds good!