Development | Release v0.23 - 8/8/22

[johanjanssens]

I'm very happy to announce Pages v0.23. This release been in the cooker for the past 6 months. It brings a lot of stabilty fixes, especially for the http cache, some great new features and also adds 3 new Pages extensions.

• Overview
• Releases
• Changelog
• Breaking

Overview

This release concludes the decoupling from Joomla core, the Joomla integration has been moved into the Joomla extension. It also brings some great new features:

• data and content collection
• entity routing
• jsonapi routing and discoverability
• domparser extension

There are a number of additional breaking changes. The changes are all simple find/replace so nothing too hard.

Data Collection

A major new feature in this release is the additional a new data collection type which allows to easily wrap pages data in a collection. This makes it possible to easily create custom collections from custom structured data.

Config

namespace

The namespace config option allows to configure the data namespace to use, the default namespace use the /joomlatools-pages/data directory. For more info on data namespaces see: #256

path

The path config option defines the data path (relative to the namespace root directory) to use.

---
@collection:
    model: data?path=foo/bar
---

The path can also be an array, providing multiple paths to get data from, and combine them in a single collection. This is useful if you have multiple directories that contain the data files. It also allows to filter data per directory.

---
@collection:
    model: data
    path:
        - /foo
        - /bar
---

Content Collection

The content collection extends from the data collection and adds additional functionality making it behave similar to the pages collection.

The content collection is using the new content entity, which provides the same entity properties then the page entity, the page entity is now extending from the content entity. See: ../model/entity/content.php

States

The content collection offers the same states as the pages collection making it very easy to switch between both without the need to make any changes to routing, or templates.

• folder: Allows to filter the collection for one of the provided paths
• slug: Allows to filter the collection for a specific file, returning only a single item.

Properties

The following properties are automatically added to each entity:

• hash: the crc32b hash if the file content (in case of markdown files), or the filesize
• date: the file last modified date
• slug: the name of the file
• folder: the relative folder to the data namespace root
• path: the full data path (including the namespace)

Example

A more complete example to setup a blog collection from multiple data directories, with support for json and rss.

Blog post would go into the /joomlatools-pages/data/blog/foo and /joomlatools-pages/data/blog/bar directories, files should end on .md, not .html.md (only applies to pages).

file: /joomlatools-pages/pages/blog/index.html.php

---
@route: '/blog/[digit:page]?.[json:format]?'

@collection:
    type: blog
    model: content
    config:
        path:
            - /blog/foo
            - /blog/bar
    state:
        limit: 10
        sort: date
        order: desc
    format: json
    route: /blog/post

title: Blog
summary: There's lots happening, our blog will keep you up-to-date.
---

<?= partial('/blog/posts', [
    'posts' => collection(),
    'route' => page()->collection->route
]); ?>

file: /joomlatools-pages/pages/blog/post.html.php

---
@route: '[/blog/.+?:folder]/[*:slug].[json:format]?'

@collection:
    extend: /blog
---

<?= partial('/blog/post', [
    'post' => collection(),
]); ?>

file: /joomlatools-pages/pages/blog/index.rss.php

---
@layout: template://pages/newsfeed.rss

@collection:
    extend: /blog
    state:
        limit: 20
        sort: date
        order: desc

title: Blog
summary: There's lots happening, our blog will keep you up-to-date.
---

Entity routing

Routing is very hard to get right and it’s even harder to make just work. Pages already removed the concept of Itemid's but I still wanted route resolving to be simpler.

This release adds entity based routing, meaning that internally pages is now aware of the relationship between a page and a collection. You can just push an entity into the route() function and it will resolve the route for you.

• old: route([/path/to/page], ['foo' => x, 'bar' => y]) (still supported)
• new: route($entity)

The route() template function can now accept collection entities and will handle routing for it. The function can still accept a path to a page and query and remains compatible with the existing routing approach.

Sounds simple but it’s not, this problem has me stumped for years, in Joomla 1.5 we tried to get around it by implementing so called route helpers that made it easier to route a content item but that only worked for a component and was an API that had to be developed for each component separate, for a large part because Joomla’s routing system isn’t smart enough to handle routing transparently.

The system in Pages works everywhere and it works all the time. There are a few outstanding PR’s related to redirect and file and rewrite routing that are already completed, but are slated for 0.24: https://github.com/joomlatools/joomlatools-pages/projects/6 With all that, the routing in Pages is now completed

Automatic routing

If no dynamic route is provided for the entity, Pages will resolve the entity to a url using the entities primary key.

For example in case of the pages or data collection the primary key is a composite key based on the slug and folder states, when both are provided the state is considered unique and pages knows the collection will only return a single entity for that state.

If no dynamic route is provided, Pages will use the primary key to generate an entity url, as follows: http://mysite.com/blog?folder=folder&slug=slug

By providing a dynamic route

---
@route: /blog/[:folder]/[:slug]
---

This url will be resolved to: http://mysite.com/blog/folder/slug

Collection routing

To ensure a collection knows what page to route entities to a new route collection property has been added which defines the path of the page that is used internally to resolve the route for an entity. If no route is provided the collection will route to the page it's defined on.

For example to route a blog post collection to a /blog/post page that will ender the post.

---
@collection:
    type: blog
    model: data
    config:
        path:
            - /blog/foo
            - /blog/bar
        entity: content
    state:
        limit: 10
        sort: date
        order: desc
    format: json
    route: /blog/post

---

JSONAPI routing and discoverability

The jsonapi contains format specific links to allow discovering and crawling site content. The entity links are now auto-routed using the collection primary keys which are defined through the model unique and composite states, see also: #569

Specifying a collection route

A collection can define also additional route property, which allows to set the pages who's routes to be used to route the collection entities.

@collection:
    model: ext:foo.model.articles
    format: json
    route: /news/article

If the collection route is false, no entity links will be included in the jsonapi

@collection:
    model: ext:foo.model.bar
    format: json
    route: false

This allows you to create relations between pages for example in a list / item setup, where you have a page /news that renders a list of all news articles and a page /news/article that renders the individual article. The routes for both would be:

• /news: /news
• /news/article: /news/[*:slug]

To make the news listing page and individual article pages discoverable through the JSONAPI the collection needs to specify json as an additional collection format.

To make both routes route the format as .json instead of ?format=json you can specify an optional .[json:format]? constraint in the route as follows:

• /news: /news.[json:format]?
• /news/article: /news/[*:slug].[json:format]?

JSONAPI endpoint

To make the site content discoverable you can add a specific api endpoint, this endpoint similar to the /api/cache.json endpoint would allow to discover and crawl a site by following the links in the json.

For example, add a file called /api/content.json with following content:

---
@collection:
    state:
        level: 0
        limit: 100
    filter:
        type: 'neq:redirect'
        folder: 'neq:/api'

@process:
    cache: true
---

You can now use http://www.mysite.com/api/content.json to discover all the content of the site.

DOMParser extension

The DomParser extension for Pages allows to easily manipulate html, for example html you get from another site, a webservice, or just html rendered by a partial you still like to make changes too, think for example about a use-case where you have pre-rendered and cached html (*), for a menu system for example (which is the same for every page) but you need to add css classes or id’s to mark the active element. The dom parser is a perfect fit for that.

Example

The Following example will extract the content from a published Google Doc and inject it in the page. The code uses the Symfony CssSelector to query and to clean up the html retrieved from Google, removing all class, id and style attributes.

---
title: Google Doc
url: https://docs.google.com/document/d/e/2PACX-1vTK9nkPIsoq6ZJeVm86ns4BX7Q0bOcMHV5jYoaGlsREXUxt22kDW2zt0Zh3wSF9mquuVQKlPFyRw9HK/pub
---

<? $html = html($url); ?>

<title><?= $html->query('html > head > title')->getText(); ?></title>

<article>
    <?= $html->filter('html > body > div[id="contents"]')
        ->query('*')->each(fn($node) =>
            $node->removeAttributes(['style', 'class', 'id'])
                  ->setAttribute('class', ['x', 'y'])
        )->query('style')->remove();
    ?>
</article>

This will produce the following results

Releases

Released v0.23.0 More info:

• v0.23.0: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.23.0
• v0.23.1: https://github.com/joomlatools/joomlatools-pages/releases/tag/v0.23.1

Changelog

Extensions

1. [ext:joomla] - Joomla Extension #856

Pages has been further decoupled from Joomla. All Joomla specific integration logic has been moved into the Joomla extension in preparation of adding support for Joomla 4.

If you are installing Pages in Joomla using the pre-build packages nothing will change, everything will continue to work as the Joomla extension will be pre-bundled with the package.

This approach is quite similar to the concept of distributions in Drupal: https://www.drupal.org/docs/drupal-distributions, by bundling Pages with the Joomla extension it becomes a Joomla distribution, the decoupling paves the way for other application specific distributions like for example Wordpress.

2. [ext:sentry] - Sentry.io extension #846

Implemented support for Sentry exception tracking. Sentry tracking works async and is configured to make use of Guzzle 7 which is installed by default in Joomlatools Server.

Sentry gives you proper error and exception tracking, tracing etc. Sentry can be run on your own hardware or you can use the SAAS version, they offer a nice free plan which is good enough to get started.

Pages provides both a backend (PHP) and frontend (javascript) client integration for Sentry.

• PHP: https://docs.sentry.io/platforms/php/
• Javascript: https://docs.sentry.io/platforms/javascript/

3. [ext:domparser] - html and xml parser #860

Implemented a HTML and XML DOM parser. The parser extends DOMDocument and uses, DOMXPath and Symfony CssSelector (if available).

The implementation improves upon #264 and moves the html() and xml() template functions out of the Pages core data object and into the ext:domparser extension

Collections

1. Data and Content collection #919

Added a new data and content collection type which allows to easily wrap pages data in a collection. For example:

---
@collection:
    model: data?path=foo/bar
    state:
        limit: 20
---

2. Improved cache collection #864

Improves the work done in #311 to improve the performance of the cache collection. Key improvements:

• Use a fixed limit and paginate the cache collection max 100 items
• Improve memory load by using readdir()
• Do not validate the cache

The cache collection has also been simplified, all you need now is:

Example: /cache.json

---
collection:
    extend: cache
---

3. Hardened cache collection #871

A new 'debug' config option has been added. If debug is disabled the cache will catch any exceptions when loading, any files generating errors will be deleted from disk. Enable it to expose cache parsing issues.

You can enable debug using the global Pages debug setting or you can enable it specifically for the cache as follows.

<?php
return [
    'extensions'   => [
        'com:pages.model.cache' => [ 
             'debug' => true
        ]
    ]
]        

Routing

1. Entity routing #933

Support for entity based routing has been added. Instead of requiring to provide a page path and query.

• old: route([/path/to/page], ['foo' => x, 'bar' => y])
• new: route($entity)

The route() template function can now accept collection entities and will handle routing for it. The function can still accept a path to a page and query and remains compatible with the existing routing approach.

2. Improved ComPagesPageEntity::getParent() #939

The page <=> parent relationship discovery has been improved. By default the parent is found by traversing the page path on level up, if however the page is a collection extending another collection the extended collection page will be returned as parent.

JSON

1. Improved page entity json output #915

Improved the page entity json output to prevent leaking php code through the jsonapi . The page content will only be included in single entity jsonapi endpoint if the content type is one of following: md, html, txt.

2. Improved jsonapi link routing and discoverability #911

The jsonapi contains format specific links to allow discovering and crawling site content. The entity links are auto-routed using the collection primary keys which are defined through the model unique and composite states, see also: #569

A collection can define also additional route property, which allows to set the pages who's routes to be used to route the collection entities.

@collection:
    model: ext:foo.model.articles
    format: json
    route: /news/article

Other

1. ComPagesConfig refactoring #853

A new ComPagesConfigAbstract class and ComPagesConfigInterface have been added to make it easier for extensions to implement their own config objects.

The ComPagesConfigAbstract offers support for magic getters:

• $config->get[option])
• $config->get[Option]()

It also offers a getOptions() method which returns a ComPagesConfigOptions extending from ComPagesConfigOptions to make it easy to output the options as JSON.

2. Improved code highlighter #947

The code highlighter has been updated to latest version: 11.6.0 and the default theme changed to base16/flat. The copy button script has also been improved and no longer relies on third party code.

3. Added onAfterPagesBootstrap event #851

Added a new onAfterPagesBootstrap which is fired after extension bootstrapping has been completed. The event allows extensions to perform post bootstrapping tasks.

13. Error handling refactoring #848

Synced changes with joomlatools/joomlatools-framework#569

Breaking

1. Improved image filter attribute handling #945

Following changes have been made:

• <ktml:images></ktml:images> has been renamed to <ktml:image></ktml:image>

• The image filter now allows to add custom data- attributes to image elements and they are re-included when the <img /> element is replaced with a responsive version.

• It's no longer possible to use data- attributes to provide filter config options, instead a <ktml:image></ktml:image> wrapper should be used. For example:

 <ktml:image lazyload="progressive,inline" width="400">
     <img src="/images/foo.jpg" / >
     <img src="/images/bar.jpg"  />
 </ktml:image>

2. Improved collection filter #935

Defining a filter

Improved the collection filter making it possible to define multiple fluent filters and extending filters of base collections. A collection filter is now no longer defined as part of the state but as a separate collection attribute. Example:

---
@collection:
    state:
        filter:
           foo: bar
---

to

---
@collection:
     filter:
          foo: bar
---

When using the collection() template function a filter can now defined as the third parameter.

<? collection('pages', ['folder' => foo, 'filter' => ['visible' => 'gte:0']])  ?>

to

<? collection('pages', ['folder' => foo],  ['visible' => 'gte:0']) ?>

3. Improved data object #921

Removed the following method

• find(): use filter() method instead
• toJson(): to transform a data object to json just cast it to a string, eg <?= data([path]) ?>

4. Renamed pages collection state path to folder #913

Renamed the Pages collection path state to folder to prevent conflicts with the path page property.

5. Renamed filesystem collection to file #909

Renamed the filesystem collection to file to better reflect that is operates on a single structured data file.

6. Require cache for json to be explicitly enabled #903

Changed the caching logic for json output. Caching for json now needs to be explicitly enabled in the page frontmatter by setting:

@process:
    cache: true

7. Removed xml/html data support #866

Removed the xml/html data support. Replaced by the new ext:domparser extension #860