868 Add X-Forwarded-Proto / -Host

CHANGELOG.md

@@ -15,15 +15,15 @@
 - [#765](https://github.com/LayerManager/layman/issues/765) Remove `authn.txt` files from workspace directories. The same information as in `authn.txt` files is saved in prime DB schema.
 - [#868](https://github.com/LayerManager/layman/issues/868) Fill table `map_layer` with relations between maps and [internal layers](doc/models.md#internal-map-layer) (layers published on this Layman instance). Relations to [external layers](doc/models.md#internal-map-layer) (layers of other servers) are not imported into the table.
 ### Changes
-- [#868](https://github.com/LayerManager/layman/issues/868) Responses to many requests respects [HTTP header `X-Forwarded-Prefix`](doc/client-proxy.md#x-forwarded-prefix-http-header) of the request. Those requests are:
+- [#868](https://github.com/LayerManager/layman/issues/868) Responses to many requests respect [HTTP X-Forwarded headers](doc/client-proxy.md#x-forwarded-http-headers) of the request. Those requests are:
   - GET [Publications](doc/rest.md#get-publications), [Layers](doc/rest.md#get-layers), [Workspace Layers](doc/rest.md#get-workspace-layers), [Maps](doc/rest.md#get-maps), and [Workspace Maps](doc/rest.md#get-workspace-maps)
   - [GET](doc/rest.md#get-workspace-layer), [PATCH](doc/rest.md#patch-workspace-layer), and [DELETE](doc/rest.md#delete-workspace-layer) Workspace Layer
   - [GET](doc/rest.md#get-workspace-map), [PATCH](doc/rest.md#patch-workspace-map), and [DELETE](doc/rest.md#delete-workspace-map) Workspace Map
   - [GET Workspace Map File](doc/rest.md#get-workspace-map-file)
   - [POST](doc/rest.md#post-workspace-layers) and [DELETE](doc/rest.md#delete-workspace-layers) Workspace Layers
   - [POST](doc/rest.md#post-workspace-maps) and [DELETE](doc/rest.md#delete-workspace-maps) Workspace Maps
   - requests to [WMS](doc/endpoints.md#web-map-service) and [WFS](doc/endpoints.md#web-feature-service) endpoints
-- [#868](https://github.com/LayerManager/layman/issues/868) Responses to [GET Workspace Layer Metadata Comparison](doc/rest.md#get-workspace-layer-metadata-comparison) and [GET Workspace Map Metadata Comparison](doc/rest.md#get-workspace-map-metadata-comparison) do not respect [HTTP header `X-Forwarded-Prefix`](doc/client-proxy.md#x-forwarded-prefix-http-header) of the request intentionally, in order to keep URLs in canonical form.
+- [#868](https://github.com/LayerManager/layman/issues/868) Responses to [GET Workspace Layer Metadata Comparison](doc/rest.md#get-workspace-layer-metadata-comparison) and [GET Workspace Map Metadata Comparison](doc/rest.md#get-workspace-map-metadata-comparison) do not respect [HTTP X-Forwarded headers](doc/client-proxy.md#x-forwarded-http-headers) of the request intentionally, in order to keep URLs in canonical form.
 - [#868](https://github.com/LayerManager/layman/issues/868) Relations between map and [internal layers](doc/models.md#internal-map-layer) are updated in `map_layer` table when calling [POST Workspace Maps](doc/rest.md#post-workspace-maps), [PATCH Workspace Map](doc/rest.md#patch-workspace-map), [DELETE Workspace Map](doc/rest.md#delete-workspace-map), and [DELETE Workspace Maps](doc/rest.md#delete-workspace-maps).
 - [#927](https://github.com/LayerManager/layman/issues/927) Send styles to GeoServer with [`raw`](https://docs.geoserver.org/2.21.x/en/user/rest/api/styles.html#raw) param set to `True`.
 - [#880](https://github.com/LayerManager/layman/issues/880) Use Docker Compose v2 (`docker compose`) in Makefile without `compatibility` flag and remove `Makefile_docker-compose_v1` file. Docker containers are named according to Docker Compose v2 and may have different name after upgrade.

doc/client-proxy.md

@@ -19,28 +19,36 @@ Imagine request e.g. to [GET Publications](rest.md#get-publications) sent throug
 ]
 ```
 
-By default, Layman will not adjust URLs in its response to contain also URL path prefix of the client proxy (`/layman-proxy` in above example). If you prefer to adjust URLs in Layman responses to contain also URL path prefix of the client proxy, you need to send also `X-Forwarded-Prefix` HTTP header with the request.
+By default, Layman will not adjust URLs in its response to contain also URL path prefix of the client proxy (`/layman-proxy` in above example). If you prefer to adjust URLs in Layman responses to contain also URL path prefix of the client proxy (or even host and protocol), you need to send also [X-Forwarded HTTP headers](#x-forwarded-http-headers) with the request.
 
-## X-Forwarded-Prefix HTTP header
+## X-Forwarded HTTP headers
 
-The value of the `X-Forwarded-Prefix` HTTP header will be used as prefix in some URL paths of Layman response and is required to match regular expression `^(?:/[a-z0-9_-]+)*$`.
+Layman supports three optional X-Forwarded HTTP headers, whose values will be used in some URLs in Layman responses:
+- `X-Forwarded-Proto`: The value will be used as protocol in some URLs, and it is required to be `http` or `https`.
+- `X-Forwarded-Host`: The value will be used as host in some URLs, and it is required to match regular expression `^(?=.{1,253}\.?(?:\:[0-9]{1,5})?$)(?:(?!-|[^.]+_)[a-z0-9-_]{1,63}(?<!-)(?:\.|(?:\:[0-9]{1,5})?$))+$`.
+- `X-Forwarded-Prefix`: The value will be used as prefix in some URL paths, and it is required to match regular expression `^(?:/[a-z0-9_-]+)*$`.
 
-For example, if you send request to `/layman-client-proxy/rest/publications` with HTTP header `X-Forwarded-Prefix=/layman-client-proxy` then response will change to
+For example, consider there is layman running at `https://enjoychallenge.tech/rest` and client proxy running at `https://laymanproxy.com/layman-client-proxy`. If you send request to your Layman proxy `https://laymanproxy.com/layman-client-proxy/rest/publications` with HTTP headers
+```
+X-Forwarded-Host=laymanproxy.com
+X-Forwarded-Prefix=/layman-client-proxy
+```
+then response will change to
 
 ```json
 [
   {
     "workspace": "my_workspace",
     "publication_type": "layer",
     "name": "my_layer",
-    "url": "https://mylaymandomain.com/layman-client-proxy/rest/publications",
+    "url": "https://laymanproxy.com/layman-client-proxy/rest/publications",
     ...
   },
   ...
 ]
 ```
 
-Currently, value of `X-Forwarded-Prefix` affects following URLs:
+Currently, value of X-Forwarded headers affects following URLs:
 * [GET Publications](rest.md#get-publications)
   * `url` key
 * [GET Layers](rest.md#get-layers)
@@ -69,7 +77,7 @@ Currently, value of `X-Forwarded-Prefix` affects following URLs:
     * `protocol`.`url` key
     * each `legends` key if its HTTP protocol and netloc corresponds with `url` or `protocol`.`url`
     * `style` key if its HTTP protocol and netloc corresponds with `url` or `protocol`.`url`
-  * NOTE: If client proxy prefix was used in URLs in uploaded file, then such prefix is also replaced with prefix according to `X-Forwarded-Prefix` header value. Such prefix is removed for requests without `X-Forwarded-Prefix` header.
+  * NOTE: If client proxy protocol, host, or URL path prefix was used in URLs in uploaded file, then such values are also replaced with values according to X-Forwarded header values. Default values are used for requests without X-Forwarded headers (protocol is the one from [LAYMAN_CLIENT_PUBLIC_URL](env-settings.md#layman_client_public_url), host is [LAYMAN_PROXY_SERVER_NAME](env-settings.md#layman_proxy_server_name), and path prefix is empty string).
 * [POST Workspace Layers](rest.md#post-workspace-layers)
   * `url` key
 * [DELETE Workspace Layer](rest.md#delete-workspace-layer)
@@ -95,11 +103,11 @@ Currently, value of `X-Forwarded-Prefix` affects following URLs:
   * `thumbnail`.`url` key
   * `metadata`.`comparison_url` key
 * [OGC endpoints](endpoints.md)
-  * Headers `X-Forwarded-Proto`, `X-Forwarded-Host`, `X-Forwarded-For`, `X-Forwarded-Path`, `Forwarded` and `Host` are ignored
+  * Headers `X-Forwarded-For`, `X-Forwarded-Path`, `Forwarded` and `Host` are ignored
   * [WMS endpoints](endpoints.md#web-map-service)
     * all requests URLs
     * all legend URLs
   * [WFS endpoints](endpoints.md#web-feature-service)
     * all operations URLs
 
-Value of `X-Forwarded-Prefix` does not affects response values of [GET Workspace Layer Metadata Comparison](rest.md#get-workspace-layer-metadata-comparison) and [GET Workspace Map Metadata Comparison](rest.md#get-workspace-map-metadata-comparison).
+Values of X-Forwarded headers does not affect response values of [GET Workspace Layer Metadata Comparison](rest.md#get-workspace-layer-metadata-comparison) and [GET Workspace Map Metadata Comparison](rest.md#get-workspace-map-metadata-comparison).

doc/rest.md

@@ -578,7 +578,9 @@ Content-Type: `multipart/form-data`
 Body parameters:
 - **file**, JSON file
    - must be valid against [map-composition schema](https://github.com/hslayers/hslayers-ng/wiki/Composition-schema)
-   - URLs of [internal layers](models.md#internal-map-layer) may contain [client-proxy prefix](client-proxy.md)
+   - layer is considered [internal](models.md#internal-map-layer) if
+     - its URL points to the Layman instance (with or without client-proxy URL path prefix),
+     - or its URL protocol and its URL host corresponds with [X-Forwarded headers](client-proxy.md#x-forwarded-http-headers) (with or without client-proxy URL path prefix)
 - *name*, string
    - computer-friendly identifier of the map
    - must be unique among all maps of one workspace
@@ -739,7 +741,7 @@ Notice that some JSON properties are automatically updated by layman, so file ob
    - **email** set to email of the owner, or empty string if not known
    - other properties will be deleted
 - **groups** are removed
-- [some layer URLs](client-proxy.md#x-forwarded-prefix-http-header) according to `X-Forwarded-Prefix` header
+- [some layer URLs](client-proxy.md#x-forwarded-http-headers) according to X-Forwarded HTTP headers
 
 #### Request
 No action parameters.

sample/layman.map/internal_url_wms_layers_with_client_proxy.json → sample/layman.map/internal_url_two_wms_layers_with_client_proxy.json

@@ -37,11 +37,24 @@
       "title": "Mista",
       "className": "HSLayers.Layer.WMS",
       "singleTile": true,
-      "url": "http://localhost:8000/client-proxy/client-proxy-subdir/geoserver/ows",
+      "url": "https://laymanproxy.com/client-proxy/client-proxy-subdir/geoserver/ows",
       "params": {
         "LAYERS": "testuser1_wms:mista",
         "FORMAT": "image\/png"
       }
+    },
+    {
+      "metadata": {},
+      "visibility": true,
+      "opacity": 1,
+      "title": "Hranice",
+      "className": "HSLayers.Layer.WMS",
+      "singleTile": true,
+      "url": "https://anotherlayman.org/client-proxy/geoserver/testuser1_wms/ows",
+      "params": {
+        "LAYERS": "hranice",
+        "FORMAT": "image\/png"
+      }
     }
   ]
 }

src/layman/common/rest.py

@@ -157,7 +157,7 @@ def get_integer_from_param(request_args, param_name, negative=True, zero=True, p
     return result
 
 
-def get_publications(publication_type, actor, request_args=None, workspace=None, *, x_forwarded_prefix=None):
+def get_publications(publication_type, actor, request_args=None, workspace=None, *, x_forwarded_items=None):
     request_args = request_args or {}
     known_order_by_values = [consts.ORDER_BY_TITLE, consts.ORDER_BY_FULL_TEXT, consts.ORDER_BY_LAST_CHANGE,
                              consts.ORDER_BY_BBOX, ]
@@ -251,7 +251,7 @@ def get_publications(publication_type, actor, request_args=None, workspace=None,
             'workspace': res_workspace,
             'publication_type': info['type'].split('.')[1],
             'title': info.get("title"),
-            'url': layman_util.get_workspace_publication_url(info['type'], res_workspace, name, x_forwarded_prefix=x_forwarded_prefix),
+            'url': layman_util.get_workspace_publication_url(info['type'], res_workspace, name, x_forwarded_items=x_forwarded_items),
             'uuid': info["uuid"],
             'access_rights': info['access_rights'],
             'updated_at': info['updated_at'].isoformat(),

src/layman/geoserver_proxy.py

@@ -227,12 +227,15 @@ def proxy(subpath):
         'X-Forwarded-Host',
         'X-Forwarded-For',
         'X-Forwarded-Path',
+        'X-Forwarded-Prefix',
         'Forwarded',
         'Host',
     ]:
         headers_req.pop(header, None)
-    x_forwarded_prefix = layman_util.get_x_forwarded_prefix(request.headers)
-    headers_req['X-Forwarded-Path'] = x_forwarded_prefix or ''
+    x_forwarded_items = layman_util.get_x_forwarded_items(request.headers)
+    headers_req['X-Forwarded-Proto'] = x_forwarded_items.proto or settings.LAYMAN_PUBLIC_URL_SCHEME
+    headers_req['X-Forwarded-Host'] = x_forwarded_items.host or settings.LAYMAN_PROXY_SERVER_NAME
+    headers_req['X-Forwarded-Path'] = x_forwarded_items.prefix or ''
 
     # ensure layer attributes in case of WFS-T
     app.logger.info(f"{request.method} GeoServer proxy, headers_req={headers_req}, url={url}")

src/layman/layer/__init__.py

@@ -154,5 +154,5 @@ def get_layer_info_keys(*, geodata_type, original_data_source):
 NO_STYLE_DEF = STYLE_TYPES_DEF[0]
 
 
-def get_workspace_publication_url(workspace, publication_name, *, x_forwarded_prefix=None):
-    return url_for('rest_workspace_layer.get', layername=publication_name, workspace=workspace, x_forwarded_prefix=x_forwarded_prefix)
+def get_workspace_publication_url(workspace, publication_name, *, x_forwarded_items=None):
+    return url_for('rest_workspace_layer.get', layername=publication_name, workspace=workspace, x_forwarded_items=x_forwarded_items)

src/layman/layer/filesystem/thumbnail.py

@@ -32,14 +32,14 @@ def ensure_layer_thumbnail_dir(workspace, layername):
     return thumbnail_dir
 
 
-def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
+def get_layer_info(workspace, layername, *, x_forwarded_items=None):
     thumbnail_path = get_layer_thumbnail_path(workspace, layername)
     if os.path.exists(thumbnail_path):
         return {
             'thumbnail': {
                 'url': url_for('rest_workspace_layer_thumbnail.get', workspace=workspace,
                                layername=layername,
-                               x_forwarded_prefix=x_forwarded_prefix),
+                               x_forwarded_items=x_forwarded_items),
                 'path': os.path.relpath(thumbnail_path, common_util.get_workspace_dir(workspace))
             },
             '_thumbnail': {

src/layman/layer/geoserver/sld.py

@@ -41,10 +41,10 @@ def delete_layer(workspace, layername):
     return result
 
 
-def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
+def get_layer_info(workspace, layername, *, x_forwarded_items=None):
     response = get_style_response(workspace, layername, gs_util.headers_sld['1.0.0'], settings.LAYMAN_GS_AUTH)
     if response and response.status_code == 200:
-        url = url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername, x_forwarded_prefix=x_forwarded_prefix)
+        url = url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername, x_forwarded_items=x_forwarded_items)
         info = {
             'style': {
                 'url': url,

src/layman/layer/geoserver/util.py

@@ -1,17 +1,20 @@
 import logging
 from urllib.parse import urlparse
-
+from layman import settings
+from layman.util import XForwardedClass
 from geoserver.util import wms_direct, wfs_direct
 
 logger = logging.getLogger(__name__)
 CACHE_GS_PROXY_BASE_URL_KEY = f'{__name__}:GS_PROXY_BASE_URL'
 DEFAULT_EXTERNAL_DB_STORE_PREFIX = 'external_db'
 
-from layman import settings
-
 
-def get_gs_proxy_server_url():
-    proxy_base_url = f'{settings.LAYMAN_PUBLIC_URL_SCHEME}://{settings.LAYMAN_PROXY_SERVER_NAME}'
+def get_gs_proxy_server_url(*, x_forwarded_items=None):
+    x_forwarded_items = x_forwarded_items or XForwardedClass()
+    protocol = x_forwarded_items.proto or settings.LAYMAN_PUBLIC_URL_SCHEME
+    host = x_forwarded_items.host or settings.LAYMAN_PROXY_SERVER_NAME
+    path_prefix = x_forwarded_items.prefix or ''
+    proxy_base_url = f'{protocol}://{host}{path_prefix}'
     return proxy_base_url
 
 

src/layman/layer/geoserver/wfs.py

@@ -58,10 +58,10 @@ def delete_layer(workspace, layername):
     return {}
 
 
-def get_wfs_url(workspace, external_url=False, *, x_forwarded_prefix=None):
-    assert external_url or not x_forwarded_prefix
-    x_forwarded_prefix = x_forwarded_prefix or ''
-    base_url = urljoin(get_gs_proxy_server_url(), x_forwarded_prefix) + settings.LAYMAN_GS_PATH if external_url else settings.LAYMAN_GS_URL
+def get_wfs_url(workspace, external_url=False, *, x_forwarded_items=None):
+    assert external_url or not x_forwarded_items
+    base_url = get_gs_proxy_server_url(x_forwarded_items=x_forwarded_items) + settings.LAYMAN_GS_PATH \
+        if external_url else settings.LAYMAN_GS_URL
     return urljoin(base_url, workspace + '/wfs')
 
 
@@ -80,6 +80,8 @@ def get_wfs_direct(workspace):
 def get_wfs_proxy(workspace):
     headers = {
         settings.LAYMAN_GS_AUTHN_HTTP_HEADER_ATTRIBUTE: settings.LAYMAN_GS_USER,
+        'X-Forwarded-Proto': settings.LAYMAN_PUBLIC_URL_SCHEME,
+        'X-Forwarded-Host': settings.LAYMAN_PROXY_SERVER_NAME,
         'X-Forwarded-Path': '',
     }
     key = get_flask_proxy_key(workspace)
@@ -125,11 +127,11 @@ def clear_cache(workspace):
     mem_redis.delete(key)
 
 
-def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
+def get_layer_info(workspace, layername, *, x_forwarded_items=None):
     wfs = get_wfs_proxy(workspace)
     if wfs is None:
         return {}
-    wfs_proxy_url = get_wfs_url(workspace, external_url=True, x_forwarded_prefix=x_forwarded_prefix)
+    wfs_proxy_url = get_wfs_url(workspace, external_url=True, x_forwarded_items=x_forwarded_items)
 
     wfs_layername = f"{workspace}:{layername}"
     if wfs_layername not in wfs.contents:

src/layman/layer/geoserver/wms.py

@@ -82,11 +82,11 @@ def delete_layer(workspace, layername):
     return {}
 
 
-def get_wms_url(workspace, external_url=False, *, x_forwarded_prefix=None):
-    assert external_url or not x_forwarded_prefix
-    x_forwarded_prefix = x_forwarded_prefix or ''
+def get_wms_url(workspace, external_url=False, *, x_forwarded_items=None):
+    assert external_url or not x_forwarded_items
     geoserver_workspace = get_geoserver_workspace(workspace)
-    base_url = urljoin(get_gs_proxy_server_url(), x_forwarded_prefix) + settings.LAYMAN_GS_PATH if external_url else settings.LAYMAN_GS_URL
+    base_url = get_gs_proxy_server_url(x_forwarded_items=x_forwarded_items) + settings.LAYMAN_GS_PATH \
+        if external_url else settings.LAYMAN_GS_URL
     return urljoin(base_url, geoserver_workspace + '/ows')
 
 
@@ -106,6 +106,8 @@ def get_wms_proxy(workspace):
     headers = {
         settings.LAYMAN_GS_AUTHN_HTTP_HEADER_ATTRIBUTE: settings.LAYMAN_GS_USER,
         'X-Forwarded-Path': '',
+        'X-Forwarded-Proto': settings.LAYMAN_PUBLIC_URL_SCHEME,
+        'X-Forwarded-Host': settings.LAYMAN_PROXY_SERVER_NAME,
     }
     key = get_flask_proxy_key(workspace)
 
@@ -170,11 +172,11 @@ def get_timeregex_props(workspace, layername):
     return result
 
 
-def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
+def get_layer_info(workspace, layername, *, x_forwarded_items=None):
     wms = get_wms_proxy(workspace)
     if wms is None:
         return {}
-    wms_proxy_url = get_wms_url(workspace, external_url=True, x_forwarded_prefix=x_forwarded_prefix)
+    wms_proxy_url = get_wms_url(workspace, external_url=True, x_forwarded_items=x_forwarded_items)
 
     if layername not in wms.contents:
         return {}

src/layman/layer/micka/csw.py

@@ -27,7 +27,7 @@
 post_layer = empty_method
 
 
-def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
+def get_layer_info(workspace, layername, *, x_forwarded_items=None):
     uuid = get_layer_uuid(workspace, layername)
     try:
         csw = common_util.create_csw()
@@ -50,7 +50,7 @@ def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
                 'csw_url': settings.CSW_PROXY_URL,
                 'record_url': common_util.get_metadata_url(uuid, url_type=common_util.RecordUrlType.BASIC),
                 'comparison_url': url_for('rest_workspace_layer_metadata_comparison.get', workspace=workspace, layername=layername,
-                                          x_forwarded_prefix=x_forwarded_prefix),
+                                          x_forwarded_items=x_forwarded_items),
             }
         }
     return {}