Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding CROSSORIGIN & django-csp handling #405

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ workflows:
matrix:
parameters:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
django-version: ["3.2", "4.2", "5.0", "5.1"]
django-version: ["4.2", "5.0"]
exclude:
- python-version: "3.8"
django-version: "5.0"
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,19 @@ For more general information, view the [readme](README.md).
Releases are added to the
[github release page](https://github.com/ezhome/django-webpack-loader/releases).

## --- INSERT VERSION HERE ---

- Automatically add `crossorigin` attributes to tags with `integrity` attributes when necessary
- Use `request.csp_nonce` from [django-csp](https://github.com/mozilla/django-csp) if available and configured

## [3.1.1] -- 2024-08-30

- Add support for Django 5.1

## [3.2.0] -- 2024-07-28

- Remove support for Django 3.x (LTS is EOL)

## [3.1.0] -- 2024-04-04

Support `webpack_asset` template tag to render transformed assets URL: `{% webpack_asset 'path/to/original/file' %} == "/static/assets/resource-3c9e4020d3e3c7a09c68.txt"`
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,11 @@ WEBPACK_LOADER = {

- `TIMEOUT` is the number of seconds webpack_loader should wait for Webpack to finish compiling before raising an exception. `0`, `None` or leaving the value out of settings disables timeouts

- `INTEGRITY` is flag enabling [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) on rendered `<script>` and `<link>` tags. Integrity hash is get from stats file and configuration on side of `BundleTracker`, where [configuration option](https://github.com/django-webpack/webpack-bundle-tracker#options) `integrity: true` is required.
- `INTEGRITY` is a flag enabling [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) on rendered `<script>` and `<link>` tags. Integrity hash is fetched from the stats of `BundleTrackerPlugin`. The [configuration option](https://github.com/django-webpack/webpack-bundle-tracker#options) `integrity: true` is required.

- `CROSSORIGIN`: If you use the `integrity` attribute in your tags and you load your webpack generated assets from another origin (that is not the same `host:port` as the one you load the webpage from), you can configure the `CROSSORIGIN` configuration option. The default value is `''` (empty string), where an empty `crossorigin` attribute will be emitted when necessary. Valid values are: `''` (empty string), `'anonymous'` (functionally same as the empty string) and `use-credentials`. For an explanation, see https://shubhamjain.co/2018/09/08/subresource-integrity-crossorigin/. A typical case for this scenario is when you develop locally and your webpack-dev-server runs with hot-reload on a local host/port other than that of django's `runserver`.

- `CSP_NONCE`: Automatically generate nonces for rendered bundles from [django-csp](https://github.com/mozilla/django-csp). Default `False`. Set this to `True` if you use `django-csp` and and `'strict-dynamic'` [CSP mode](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src#strict-dynamic).

- `LOADER_CLASS` is the fully qualified name of a python class as a string that holds the custom Webpack loader. This is where behavior can be customized as to how the stats file is loaded. Examples include loading the stats file from a database, cache, external URL, etc. For convenience, `webpack_loader.loaders.WebpackLoader` can be extended. The `load_assets` method is likely where custom behavior will be added. This should return the stats file as an object.

Expand Down
13 changes: 10 additions & 3 deletions tests/app/tests/test_webpack.py
Original file line number Diff line number Diff line change
Expand Up @@ -224,11 +224,18 @@ def test_integrity(self):

self.assertIn((
'<script src="/static/django_webpack_loader_bundles/main.js" '
'integrity="sha256-1wgFMxcDlOWYV727qRvWNoPHdnOGFNVMLuKd25cjR+o=" >'
'integrity="sha256-1wgFMxcDlOWYV727qRvWNoPHdnOGFNVMLuKd25cjR+'
'o= sha384-3RnsU3Z2OODW6qaMAPVpNC5lBb4M5I1+joXv37ACuLvCO6gQ7o'
'OD7IC1zN1uAakD sha512-9nLlV4v2pWvgeavHop1wXxdP34CfYv/xUZHwVB'
'N+1p+pAvHDmBw4XkvvciSGW4zQlWhaUiIi7P6nXmsLE+8Fsw==" >'
'</script>'), result.rendered_content)
self.assertIn((
'<link href="/static/django_webpack_loader_bundles/main.css" rel="stylesheet" '
'integrity="sha256-cYWwRvS04/VsttQYx4BalKYrBDuw5t8vKFhWB/LKX30=" />'),
'<link href="/static/django_webpack_loader_bundles/main.css" '
'rel="stylesheet" integrity="sha256-cYWwRvS04/VsttQYx4BalKYrB'
'Duw5t8vKFhWB/LKX30= sha384-V/UxbrsEy8BK5nd+sBlN31Emmq/WdDDdI'
'01UR8wKIFkIr6vEaT5YRaeLMfLcAQvS sha512-aigPxglXDA33t9s5i0vRa'
'p5b7dFwyp7cSN6x8rOXrPpCTMubOR7qTFpmTIa8z9B0wtXxbSheBPNCEURBH'
'KLQPw==" />'),
result.rendered_content
)

Expand Down
2 changes: 1 addition & 1 deletion webpack_loader/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
__author__ = "Vinta Software"
__version__ = "3.1.1"
__version__ = "3.2.0"

import django

Expand Down
6 changes: 6 additions & 0 deletions webpack_loader/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,15 @@
'IGNORE': [r'.+\.hot-update.js', r'.+\.map'],
'LOADER_CLASS': 'webpack_loader.loaders.WebpackLoader',
'INTEGRITY': False,
# See https://shubhamjain.co/2018/09/08/subresource-integrity-crossorigin/
# See https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/crossorigin
# type is Literal['anonymous', 'use-credentials', '']
'CROSSORIGIN': '',
# Whenever the global setting for SKIP_COMMON_CHUNKS is changed, please
# update the fallback value in get_skip_common_chunks (utils.py).
'SKIP_COMMON_CHUNKS': False,
# Use nonces from django-csp when available
'CSP_NONCE': False
}
}

Expand Down
101 changes: 89 additions & 12 deletions webpack_loader/loaders.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
import json
import time
import os
import time
from functools import lru_cache
from io import open
from typing import Dict, Optional
from urllib.parse import urlparse
from warnings import warn

from django.conf import settings
from django.contrib.staticfiles.storage import staticfiles_storage
from django.http.request import HttpRequest

from .exceptions import (
WebpackError,
Expand All @@ -13,6 +18,30 @@
WebpackBundleLookupError,
)

_CROSSORIGIN_NO_REQUEST = (
'The crossorigin attribute might be necessary but you did not pass a '
'request object. django_webpack_loader needs a request object to be able '
'to know when to emit the crossorigin attribute on link and script tags. '
'Chunk name: {chunk_name}')
_CROSSORIGIN_NO_HOST = (
'You have passed the request object but it does not have a "HTTP_HOST", '
'thus django_webpack_loader can\'t know if the crossorigin header will '
'be necessary or not. Chunk name: {chunk_name}')
_NONCE_NO_REQUEST = (
'You have enabled the adding of nonce attributes to generated tags via '
'django_webpack_loader, but haven\'t passed a request. '
'Chunk name: {chunk_name}')
_NONCE_NO_CSPNONCE = (
'django_webpack_loader can\'t generate a nonce tag for a bundle, '
'because the passed request doesn\'t contain a "csp_nonce". '
'Chunk name: {chunk_name}')


@lru_cache(maxsize=100)
def _get_netloc(url: str) -> str:
'Return a cached netloc (host:port) for the passed `url`.'
return urlparse(url=url).netloc


class WebpackLoader:
_assets = {}
Expand Down Expand Up @@ -42,19 +71,67 @@ def get_asset_by_source_filename(self, name):
files = self.get_assets()["assets"].values()
return next((x for x in files if x.get("sourceFilename") == name), None)

def get_integrity_attr(self, chunk):
if not self.config.get("INTEGRITY"):
return " "

integrity = chunk.get("integrity")
def _add_crossorigin(
self, request: Optional[HttpRequest], chunk: Dict[str, str],
integrity: str, attrs_l: str) -> str:
'Return an added `crossorigin` attribute if necessary.'
def_value = f' integrity="{integrity}" '
if not request:
message = _CROSSORIGIN_NO_REQUEST.format(chunk_name=chunk['name'])
warn(message=message, category=RuntimeWarning)
return def_value
if 'crossorigin' in attrs_l:
return def_value
host: Optional[str] = request.META.get('HTTP_HOST')
if not host:
message = _CROSSORIGIN_NO_HOST.format(chunk_name=chunk['name'])
warn(message=message, category=RuntimeWarning)
return def_value
netloc = _get_netloc(url=chunk['url'])
if netloc == '' or netloc == host:
# Crossorigin not necessary
return def_value
cfgval: str = self.config.get('CROSSORIGIN')
if cfgval == '':
return f'{def_value}crossorigin '
return f'{def_value}crossorigin="{cfgval}" '

def get_integrity_attr(
self, chunk: Dict[str, str], request: Optional[HttpRequest],
attrs_l: str) -> str:
if not self.config.get('INTEGRITY'):
# Crossorigin only necessary when integrity is used
return ' '

integrity = chunk.get('integrity')
if not integrity:
raise WebpackLoaderBadStatsError(
"The stats file does not contain valid data: INTEGRITY is set to True, "
'but chunk does not contain "integrity" key. Maybe you forgot to add '
"integrity: true in your BundleTracker configuration?"
)

return ' integrity="{}" '.format(integrity.partition(" ")[0])
'The stats file does not contain valid data: INTEGRITY is set '
'to True, but chunk does not contain "integrity" key. Maybe '
'you forgot to add integrity: true in your '
'BundleTrackerPlugin configuration?')
return self._add_crossorigin(
request=request, chunk=chunk, integrity=integrity,
attrs_l=attrs_l)

def get_nonce_attr(
self, chunk: Dict[str, str], request: Optional[HttpRequest],
attrs: str) -> str:
'Return an added nonce for CSP when available.'
if not self.config.get('CSP_NONCE'):
return ''
if request is None:
message = _NONCE_NO_REQUEST.format(chunk_name=chunk['name'])
warn(message=message, category=RuntimeWarning)
return ''
nonce = getattr(request, 'csp_nonce', None)
if nonce is None:
message = _NONCE_NO_CSPNONCE.format(chunk_name=chunk['name'])
warn(message=message, category=RuntimeWarning)
return ''
if 'nonce=' in attrs.lower():
return ''
return f'nonce="{nonce}" '

def filter_chunks(self, chunks):
filtered_chunks = []
Expand Down
25 changes: 15 additions & 10 deletions webpack_loader/templatetags/webpack_loader.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
from typing import Optional
from warnings import warn

from django.http.request import HttpRequest
from django.template import Library
from django.utils.safestring import mark_safe

Expand All @@ -20,33 +22,36 @@ def render_bundle(
if skip_common_chunks is None:
skip_common_chunks = utils.get_skip_common_chunks(config)

url_to_tag_dict = utils.get_as_url_to_tag_dict(
bundle_name, extension=extension, config=config, suffix=suffix,
attrs=attrs, is_preload=is_preload)
request: Optional[HttpRequest] = context.get('request')
tags = utils.get_as_url_to_tag_dict(
bundle_name, request=request, extension=extension, config=config,
suffix=suffix, attrs=attrs, is_preload=is_preload)

request = context.get('request')
if request is None:
if skip_common_chunks:
warn(message=_WARNING_MESSAGE, category=RuntimeWarning)
return mark_safe('\n'.join(url_to_tag_dict.values()))
return mark_safe('\n'.join(tags.values()))

used_urls = getattr(request, '_webpack_loader_used_urls', None)
if not used_urls:
used_urls = request._webpack_loader_used_urls = set()
if used_urls is None:
used_urls = set()
setattr(request, '_webpack_loader_used_urls', used_urls)
if skip_common_chunks:
url_to_tag_dict = {url: tag for url, tag in url_to_tag_dict.items() if url not in used_urls}
used_urls.update(url_to_tag_dict.keys())
return mark_safe('\n'.join(url_to_tag_dict.values()))
tags = {url: tag for url, tag in tags.items() if url not in used_urls}
used_urls.update(tags)
return mark_safe('\n'.join(tags.values()))


@register.simple_tag
def webpack_static(asset_name, config='DEFAULT'):
return utils.get_static(asset_name, config=config)


@register.simple_tag
def webpack_asset(asset_name, config='DEFAULT'):
return utils.get_asset(asset_name, config=config)


@register.simple_tag(takes_context=True)
def get_files(
context, bundle_name, extension=None, config='DEFAULT',
Expand Down
48 changes: 30 additions & 18 deletions webpack_loader/utils.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
from collections import OrderedDict
from functools import lru_cache
from importlib import import_module
from typing import Optional, OrderedDict

from django.conf import settings
from .config import load_config
from django.http.request import HttpRequest

_loaders = {}
from .config import load_config
from .loaders import WebpackLoader


def import_string(dotted_path):
Expand All @@ -21,12 +24,11 @@ def import_string(dotted_path):
raise ImportError('%s doesn\'t look like a valid module path' % dotted_path)


def get_loader(config_name):
if config_name not in _loaders:
config = load_config(config_name)
loader_class = import_string(config['LOADER_CLASS'])
_loaders[config_name] = loader_class(config_name, config)
return _loaders[config_name]
@lru_cache(maxsize=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is changing the API w/o a clear benefit. While _loaders is private, someone might be using it.
I would refrain from changing this unless there's a very clear benefit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 'very clear benefit' is to speed up config loading immensely, because template generation runs this function countless times when generating a single template, by looking at the various code paths.

_loader is a name that implies you shouldn't use that variable outside of the module, and it's everybody's responsibility to do just that, or face the consequences if they don't.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function already does the same thing lru_cache will do. _loaders is exactly the cache.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me demonstrate you why reaching for a global variable is slower than using lru_cache that uses a cache table locally:

import timeit
from functools import lru_cache
from random import randint

_some_global_variable = {}


def expensive_function(var1):
    return randint(0, 5)


def get1(var1):
    if var1 not in _some_global_variable:
        result = expensive_function(var1)
        _some_global_variable[var1] = result
        return result
    return _some_global_variable[var1]


@lru_cache(maxsize=None)
def get2(var1):
    return expensive_function(var1)


print(timeit.timeit(
    stmt='get1("x")', setup='from __main__ import get1', number=10000000))
print(timeit.timeit(
    stmt='get2("x")', setup='from __main__ import get2', number=10000000))

A couple run outputs with various python versions I have currently installed:

sh-5.2$ python3.8 test.py
0.7265332540009695
0.3478194800009078

sh-5.2$ python3.9 test.py
0.6371905300002254
0.3206007739972847

sh-5.2$ python3.10 test.py
0.48459655799888424
0.2498635900010413

sh-5.2$ python3.11 test.py
0.48944291500083636
0.38999983699977747

sh-5.2$ python3.12 test.py
0.425895505999506
0.3634855400014203

def get_loader(config_name) -> WebpackLoader:
config = load_config(config_name)
loader_class = import_string(config['LOADER_CLASS'])
return loader_class(config_name, config)


def get_skip_common_chunks(config_name):
Expand Down Expand Up @@ -57,7 +59,10 @@ def get_files(bundle_name, extension=None, config='DEFAULT'):
return list(_get_bundle(loader, bundle_name, extension))


def get_as_url_to_tag_dict(bundle_name, extension=None, config='DEFAULT', suffix='', attrs='', is_preload=False):
def get_as_url_to_tag_dict(
bundle_name, request: Optional[HttpRequest] = None, extension=None,
config='DEFAULT', suffix='', attrs='', is_preload=False
) -> OrderedDict[str, str]:
'''
Get a dict of URLs to formatted <script> & <link> tags for the assets in the
named bundle.
Expand All @@ -70,7 +75,8 @@ def get_as_url_to_tag_dict(bundle_name, extension=None, config='DEFAULT', suffix

loader = get_loader(config)
bundle = _get_bundle(loader, bundle_name, extension)
result = OrderedDict()
result = OrderedDict[str, str]()
attrs_l = attrs.lower()

for chunk in bundle:
if chunk['name'].endswith(('.js', '.js.gz')):
Expand All @@ -80,25 +86,29 @@ def get_as_url_to_tag_dict(bundle_name, extension=None, config='DEFAULT', suffix
).format(''.join([chunk['url'], suffix]), attrs)
else:
result[chunk['url']] = (
'<script src="{0}"{2}{1}></script>'
'<script src="{0}"{2}{3}{1}></script>'
).format(
''.join([chunk['url'], suffix]),
attrs,
loader.get_integrity_attr(chunk),
loader.get_integrity_attr(chunk, request, attrs_l),
loader.get_nonce_attr(chunk, request, attrs_l),
)
elif chunk['name'].endswith(('.css', '.css.gz')):
result[chunk['url']] = (
'<link href="{0}" rel={2}{3}{1}/>'
'<link href="{0}" rel={2}{3}{4}{1}/>'
).format(
''.join([chunk['url'], suffix]),
attrs,
'"stylesheet"' if not is_preload else '"preload" as="style"',
loader.get_integrity_attr(chunk),
loader.get_integrity_attr(chunk, request, attrs_l),
loader.get_nonce_attr(chunk, request, attrs_l),
)
return result


def get_as_tags(bundle_name, extension=None, config='DEFAULT', suffix='', attrs='', is_preload=False):
def get_as_tags(
bundle_name, request=None, extension=None, config='DEFAULT', suffix='',
attrs='', is_preload=False):
'''
Get a list of formatted <script> & <link> tags for the assets in the
named bundle.
Expand All @@ -108,7 +118,7 @@ def get_as_tags(bundle_name, extension=None, config='DEFAULT', suffix='', attrs=
:param config: (optional) the name of the configuration
:return: a list of formatted tags as strings
'''
return list(get_as_url_to_tag_dict(bundle_name, extension, config, suffix, attrs, is_preload).values())
return list(get_as_url_to_tag_dict(bundle_name, request, extension, config, suffix, attrs, is_preload).values())


def get_static(asset_name, config='DEFAULT'):
Expand All @@ -125,6 +135,7 @@ def get_static(asset_name, config='DEFAULT'):

return '{0}{1}'.format(public_path, asset_name)


def get_asset(source_filename, config='DEFAULT'):
'''
Equivalent to Django's 'static' look up but for webpack assets, given its original filename.
Expand All @@ -136,6 +147,7 @@ def get_asset(source_filename, config='DEFAULT'):
'''
loader = get_loader(config)
asset = loader.get_asset_by_source_filename(source_filename)
if not asset: return None
if not asset:
return None

return get_static(asset['name'], config)