Searching for Content in Pages Sites

[JoeJoomla]

Joomlers have taken search for granted since the CMS a database driven and search is baked in. When creating a Pages site what consideration should be given for search functionality if the site is not using a database?

[johanjanssens]

Search is still a work in progress.

The implementation is not hard really, extract data (content, title, description, ...) from page when it’s rendered into a data store of choice that is optimised for search.

Different database solutions are outlined here: https://github.com/joomlatools/joomlatools-pages/projects/2#card-37435563 I am leaning towards TNTSearch, https://github.com/teamtnt/tntsearch which can run using sqlite which makes it very lean.

Approach

In case you want to give this as go. The approach is similar to how the static caching works only you write the data to a db instead of the filesystem and you extra only that data you need. With search you would extract the data you need and insert it in the datastore of choice.

The code for the staticcache can be found here: https://github.com/joomlatools/joomlatools-pages/blob/master/code/event/subscriber/staticcache.php

The onAfterDispatcherCache and onAfterDispatcherPurge events are perfect for this, the cache event is only called when a page is cached, or when the cache is updated, (if the page remains the same it’s not called), purge is called when the page is removed from cache.

You can either grab the page content by re-rendering the page, just getting the html, or you run a little regex over the html to filter out a specific html element. Description, title and other metadata you can fetch from the page.

Thats it, and your good to go 🙂

[johanjanssens]

@JoeJoomla Little update on this. I have made quite a bit of improvements in Pages v0.23 to make it possible to implement search. Most specifically, the improvements to the entity routing and jsonapi discoverability.

I have chosen to use Meilisearch instead of TNTSearch as I previously mentioned. Meilisearch is installed in Joomlatools Server which we use to run all our sites.

Here is how this works:

1. Create a /api/search.json endpoint

Searchable content that needs to be indexed is exposed through a /api/search.json endpoint.

---
@collection:
    state:
        limit: 100
    filter:
        type: collection
        state:
            searchable: true

@process:
    cache: false
---

This endpoint will provide a jsonapi of all the data to be indexed with additional metadata for the crawler to crawl, extract and index the data. All you do is adding:

state:
    searchable: true

to page, or a page containing a collection to have it included in /api/search.json.

2. Scrape /api/search.json

I build a scraper microservice using Swoole (Swoole is by default installed in Joomlatools Server) which crawls and scrapes the jsonapi exposed by /api/search.json The scraper will parse the content, transform it to be compliant with the Docsearch.js widget offered by Meilisearch and then insert it in Meilisearch.

The crawler is pretty smart, it will not re-index pages that have already been indexed, unless it's told to do so, or unless the content of the page has changed. It uses the http caching layer in Pages to make decisions if a page was updated or not. This makes the re-indexing process very fast, in most cases only a few pages need updating, which takes less then a second.

Results are very accurate and very fast search solution that can index, any content exposed by Pages. Check to see this in action here: https://www.joomlatools.com/extensions/docman/documentation