165 Set-up GS JDBC role service

.env.demo

@@ -16,7 +16,7 @@ LAYMAN_AUTHN_MODULES=layman.authn.http_header
 LAYMAN_AUTHN_HTTP_HEADER_NAME=a0468616f9968eaecdc3377988aba650
 GRANT_CREATE_PUBLIC_WORKSPACE=EVERYONE
 GRANT_PUBLISH_IN_PUBLIC_WORKSPACE=EVERYONE
-LAYMAN_ROLE_SERVICE_URI=postgresql://docker:docker@postgresql:5432/layman_test?schema=_role_service
+LAYMAN_ROLE_SERVICE_URI=postgresql://docker:docker@postgresql:5432/gis?schema=_role_service
 
 # connection parameters to PostgreSQL database
 LAYMAN_PG_HOST=postgresql

.env.dev

@@ -16,7 +16,7 @@ LAYMAN_AUTHN_MODULES=layman.authn.oauth2,layman.authn.http_header
 LAYMAN_AUTHN_HTTP_HEADER_NAME=a0468616f9968eaecdc3377988aba650
 GRANT_CREATE_PUBLIC_WORKSPACE=EVERYONE
 GRANT_PUBLISH_IN_PUBLIC_WORKSPACE=EVERYONE
-LAYMAN_ROLE_SERVICE_URI=postgresql://docker:docker@postgresql:5432/layman_test?schema=_role_service
+LAYMAN_ROLE_SERVICE_URI=postgresql://docker:docker@postgresql:5432/gis?schema=_role_service
 
 # connection parameters to PostgreSQL database
 LAYMAN_PG_HOST=postgresql

CHANGELOG.md

@@ -9,6 +9,7 @@
   ```
   It was already required in v1.22.3.
 - Set new environment variable [LAYMAN_ROLE_SERVICE_URI](doc/env-settings.md#LAYMAN_ROLE_SERVICE_URI)
+- Stop using environment variable `LAYMAN_GS_ROLE_SERVICE`, it has no effect to Layman anymore. Role service called `layman_role_service` is used now.
 ### Migrations and checks
 #### Schema migrations
 - [#165](https://github.com/LayerManager/layman/issues/165) Add column `role_name` to table `rights` in prime DB schema. Add constraint that exactly one of columns `role_name` and `id_user` is not null.
@@ -856,7 +857,7 @@ There is a critical bug in this release, posting new layer breaks Layman: https:
 - [#62](https://github.com/LayerManager/layman/issues/62) GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set on Layman's startup according to [LAYMAN_GS_PROXY_BASE_URL](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PROXY_BASE_URL). If you do not set the variable, value is calculated as [LAYMAN_CLIENT_PUBLIC_URL](doc/env-settings.md#LAYMAN_CLIENT_PUBLIC_URL)+[LAYMAN_GS_PATH](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
 - [#83](https://github.com/LayerManager/layman/issues/89) All layers are created as `GEOMETRY` type, so any other type can be added (for example polygons can be added to points).
 - [#73](https://github.com/LayerManager/layman/issues/73) Layman users are automatically created on GeoServer (either at start up of Layman or when reserved) with separate role and workspace. Username is the same as in Layman, name of role is `"USER_"+username`, name of workspace is the same as username. Read and write permissions for workspace are set according to Layman's authorization (as of now read-everyone-write-everyone or read-everyone-write-owner).
-- New environment variables [LAYMAN_GS_USER_GROUP_SERVICE](doc/env-settings.md#LAYMAN_GS_USER_GROUP_SERVICE) and [LAYMAN_GS_ROLE_SERVICE](doc/env-settings.md#LAYMAN_GS_ROLE_SERVICE) enable to control which user/group and role services are used at GeoServer. Not setting these variables means to use default services. 
+- New environment variables [LAYMAN_GS_USER_GROUP_SERVICE](doc/env-settings.md#LAYMAN_GS_USER_GROUP_SERVICE) and [LAYMAN_GS_ROLE_SERVICE](https://github.com/LayerManager/layman/blob/v1.22.0/doc/env-settings.md#LAYMAN_GS_ROLE_SERVICE) enable to control which user/group and role services are used at GeoServer. Not setting these variables means to use default services. 
 - [#69](https://github.com/LayerManager/layman/issues/69) Three separate identical settings files (`layman_settings_demo.py`, `layman_settings_dev.py`, `layman_settings_test.py`) were merged into one file `layman_settings.py`.
 - If username used in REST API request path is not yet reserved, HTTP requests other than POST returns (e.g. GET) HTTP error 404 (Layman code 40). Previously in case of GET request, empty list was returned.
 - List of GeoServer reserved workspace names was moved from `layman_settings.py` into source code (`src\layman\common\geoserver\__init__.py`)

Makefile

@@ -3,24 +3,28 @@
 start-demo:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
-	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate postgresql qgis geoserver redis layman celery_worker flower timgen layman_client nginx
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate qgis geoserver redis layman celery_worker flower timgen layman_client nginx
 
 start-demo-full:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
-	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate postgresql qgis geoserver redis layman celery_worker flower timgen layman_client micka nginx
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate qgis geoserver redis layman celery_worker flower timgen layman_client micka nginx
 
 start-demo-only:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate --no-deps layman celery_worker flower timgen layman_client nginx
 
 start-demo-full-with-optional-deps:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate
 
@@ -33,15 +37,17 @@ build-demo:
 upgrade-demo:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
-	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate --no-deps postgresql qgis geoserver redis timgen layman_client nginx
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate --no-deps qgis geoserver redis timgen layman_client nginx
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps layman bash -c "cd src && python3 layman_flush_redis.py && python3 wait_for_deps.py && python3 standalone_upgrade.py"
 
 upgrade-demo-full:
 	mkdir -p layman_data deps/qgis/data
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml build layman layman_client timgen
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d postgresql
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps -u root layman bash -c "cd src && python3 -B setup_geoserver.py"
-	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate --no-deps postgresql qgis geoserver redis timgen layman_client micka nginx
+	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml up -d --force-recreate --no-deps qgis geoserver redis timgen layman_client micka nginx
 	docker compose -f docker-compose.deps.demo.yml -f docker-compose.demo.yml run --rm --no-deps layman bash -c "cd src && python3 layman_flush_redis.py && python3 wait_for_deps.py && python3 standalone_upgrade.py"
 
 upgrade-after-timeout:
@@ -55,6 +61,7 @@ deps-stop:
 
 start-dev:
 	mkdir -p layman_data layman_data_test tmp deps/qgis/data
+	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d postgresql
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml run --rm --no-deps -u root layman_dev bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up --force-recreate -d
 
@@ -64,6 +71,7 @@ stop-dev:
 start-dev-only:
 	mkdir -p layman_data layman_data_test tmp deps/qgis/data
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml rm -fsv layman_dev celery_worker_dev flower timgen layman_client
+	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d postgresql
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml run --rm --no-deps -u root layman_dev bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d layman_dev celery_worker_dev flower timgen layman_client
 
@@ -93,8 +101,9 @@ restart-celery-dev:
 
 upgrade-dev:
 	mkdir -p layman_data layman_data_test tmp deps/qgis/data
+	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d postgresql
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml run --rm --no-deps -u root layman_dev bash -c "cd src && python3 -B setup_geoserver.py"
-	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d timgen layman_client postgresql qgis nginx-qgis geoserver redis micka
+	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d timgen layman_client qgis nginx-qgis geoserver redis micka
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml run --rm --no-deps layman_dev bash -c "cd src && python3 layman_flush_redis.py && python3 wait_for_deps.py && python3 standalone_upgrade.py"
 
 prepare-dirs:
@@ -194,7 +203,8 @@ test:
 		docker compose -f docker-compose.deps.yml -f docker-compose.test.yml build layman_client_test ; \
 	fi;
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml rm -f layman_test
-	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh"
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up -d postgresql
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh && python3 src/assert_db.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps -u root layman_test bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up --force-recreate --no-deps -d celery_worker_test
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --name layman_test_run_1 layman_test
@@ -214,7 +224,8 @@ test-separated:
 		docker compose -f docker-compose.deps.yml -f docker-compose.test.yml build layman_client_test ; \
 	fi;
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml rm -f layman_test
-	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh"
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up -d postgresql
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh && python3 src/assert_db.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps -u root layman_test bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up --force-recreate --no-deps -d celery_worker_test
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --name layman_test_run_1 -e "TEST_TYPE=$(test_type)" layman_test bash -c "bash test_separated.sh $(max_fail)"
@@ -234,7 +245,8 @@ test-static:
 		docker compose -f docker-compose.deps.yml -f docker-compose.test.yml build layman_client_test ; \
 	fi;
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml rm -f layman_test
-	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh"
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up -d postgresql
+	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps layman_test bash -c "bash ensure-test-data.sh && python3 src/assert_db.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --no-deps -u root layman_test bash -c "cd src && python3 -B setup_geoserver.py"
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml up --force-recreate --no-deps -d celery_worker_test
 	docker compose -f docker-compose.deps.yml -f docker-compose.test.yml run --rm --name layman_test_run_1 layman_test bash -c "bash test_static.sh"
@@ -283,6 +295,7 @@ geoserver-exec:
 	docker compose -f docker-compose.deps.yml exec geoserver bash
 
 geoserver-ensure-authn:
+	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml up -d postgresql
 	docker compose -f docker-compose.deps.yml -f docker-compose.dev.yml run --rm --no-deps -u root layman_dev bash -c "cd src && python3 -B setup_geoserver.py"
 
 get-current-user:

doc/env-settings.md

@@ -176,9 +176,6 @@ Name of [GeoServer role](https://docs.geoserver.org/2.21.x/en/user/security/weba
 ### LAYMAN_GS_USER_GROUP_SERVICE
 Name of [user/group service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/usergroupservices.html) used for managing users at GeoServer. If not set (default), the service named `default` is chosen. Usually it's [XML user/group service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/usergroupservices.html#xml-user-group-service).
 
-### LAYMAN_GS_ROLE_SERVICE
-Name of [role service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/roleservices.html) used for managing roles and user-role associations at GeoServer. If not set (default), the service named `default` is chosen. Usually it's [XML user/group service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/roleservices.html#xml-role-service).
-
 ### LAYMAN_GS_AUTHN_HTTP_HEADER_ATTRIBUTE
 Secret value of [GeoServer HTTP authentication request header attribute](https://docs.geoserver.org/2.21.x/en/user/security/tutorials/httpheaderproxy/index.html) used for WFS proxy. Only combination of lowercase characters and numbers must be used for the value. If you change an existing value, you have to change it also in GeoServer GUI manually.
 

doc/publish-map.md

@@ -13,9 +13,9 @@ In QGIS, you need to implement following steps.
 First, compose JSON valid against [map-composition schema](https://github.com/hslayers/map-compositions). For Layman, especially `describedBy`, `name`, `title`, `abstract`, `layers`, `projection`, and `extent attributes are important. Each layer must have `className` attribute equal to `HSLayers.Layer.WMS` or `WMS`.
 
 Then save the file to Layman using [POST Workspace Maps](rest.md#post-workspace-maps) endpoint. Well-known [requests](https://requests.readthedocs.io/en/latest/) module can be used for sending HTTP requests. See especially
-- [More complicated POST requests](https://requests.readthedocs.io/en/latest/user/quickstart/#more-complicated-post-requests)
-- [POST a Multipart-Encoded File](https://requests.readthedocs.io/en/latest/user/quickstart/#post-a-multipart-encoded-file)
-- [POST Multiple Multipart-Encoded Files](https://requests.readthedocs.io/en/latest/user/advanced/#post-multiple-multipart-encoded-files)
+- [More complicated POST requests](https://requests.readthedocs.io/en/latest/user/quickstart.html#more-complicated-post-requests)
+- [POST a Multipart-Encoded File](https://requests.readthedocs.io/en/latest/user/quickstart.html#post-a-multipart-encoded-file)
+- [POST Multiple Multipart-Encoded Files](https://requests.readthedocs.io/en/latest/user/advanced.html#post-multiple-multipart-encoded-files)
 
 In response of [POST Workspace Maps](rest.md#post-workspace-maps) you will obtain
  - `name` of the map unique within all maps in used [workspace](models.md#workspace)

src/clear_layman_data.py

@@ -120,24 +120,6 @@ def main():
             )
             response.raise_for_status()
 
-            for role in roles:
-                response = requests.delete(
-                    urljoin(geoserver.GS_REST_ROLES, f'role/{role}/user/{user}/'),
-                    headers=headers_json,
-                    auth=auth,
-                    timeout=settings.DEFAULT_CONNECTION_TIMEOUT,
-                )
-                response.raise_for_status()
-
-            response = requests.delete(
-                urljoin(geoserver.GS_REST_ROLES, 'role/' + f"USER_{user.upper()}"),
-                headers=headers_json,
-                auth=auth,
-                timeout=settings.DEFAULT_CONNECTION_TIMEOUT,
-            )
-            if response.status_code != 404:
-                response.raise_for_status()
-
             response = requests.delete(
                 urljoin(geoserver.GS_REST_USER, user),
                 headers=headers_json,

src/geoserver/geoserver_test.py

@@ -24,29 +24,6 @@ def gs_user():
     assert gs_util.delete_user(TEST_USER, GS_AUTH)
 
 
-@pytest.fixture()
-def gs_role():
-    roles = gs_util.get_roles(GS_AUTH)
-    assert TEST_ROLE not in roles
-    assert gs_util.ensure_role(TEST_ROLE, GS_AUTH)
-    yield TEST_ROLE
-    assert gs_util.delete_role(TEST_ROLE, GS_AUTH)
-
-
-def test_role_management():
-    init_roles = gs_util.get_roles(GS_AUTH)
-    new_role = TEST_ROLE
-    assert new_role not in init_roles
-    assert gs_util.ensure_role(new_role, GS_AUTH)
-    roles = gs_util.get_roles(GS_AUTH)
-    assert new_role in roles
-    assert len(init_roles) + 1 == len(roles)
-    assert gs_util.delete_role(new_role, GS_AUTH)
-    roles = gs_util.get_roles(GS_AUTH)
-    assert new_role not in roles
-    assert len(init_roles) == len(roles)
-
-
 def test_user_management():
     init_usernames = gs_util.get_usernames(GS_AUTH)
     new_user = TEST_USER
@@ -62,21 +39,6 @@ def test_user_management():
     assert len(init_usernames) == len(usernames)
 
 
-def test_user_role_management(gs_user, gs_role):
-    user = gs_user[0]
-    init_user_roles = gs_util.get_user_roles(user, GS_AUTH)
-    role = gs_role
-    assert role not in init_user_roles
-    assert gs_util.ensure_user_role(user, role, GS_AUTH)
-    user_roles = gs_util.get_user_roles(user, GS_AUTH)
-    assert role in user_roles
-    assert len(init_user_roles) + 1 == len(user_roles)
-    assert gs_util.delete_user_role(user, role, GS_AUTH)
-    user_roles = gs_util.get_user_roles(user, GS_AUTH)
-    assert role not in user_roles
-    assert len(init_user_roles) == len(user_roles)
-
-
 @pytest.mark.parametrize('service', gs_util.SERVICE_TYPES)
 def test_service_srs_list_management(service):
     init_service_epsg_codes = gs_util.get_service_srs_list(service, GS_AUTH)