TAO taoAdvancedSearch extension

Extension required to advanced search integration with TAO platform oat-sa/extension-tao-advanced-search

Requirements

• ElasticSearch 8.13+ installed.
• Have this extension installed in TAO.

Installation instructions

Activate advanced search

Activate for using credentials from environment variables

php index.php 'oat\taoAdvancedSearch\scripts\tools\Activate'

Activate without credentials needed

php index.php 'oat\taoAdvancedSearch\scripts\tools\Activate' --host <host url> --port <host port> [--indexPrefix <optional>]

Activate with credentials needed

php index.php 'oat\taoAdvancedSearch\scripts\tools\Activate' --host <host url> --port <host port> --user <host user> --pass <host pass> [--indexPrefix <optional>]

Activate for ElasticCloud

php index.php 'oat\taoAdvancedSearch\scripts\tools\Activate' --elasticCloudId <cloud id> --elasticCloudApiKeyId <key id> --elasticCloudApiKey <key> [--indexPrefix <optional>]

Create indexes

php index.php 'oat\taoAdvancedSearch\scripts\tools\IndexCreator'

ATTENTION: In case the indexes already exist and the command above is returning error, you can delete the indexes by running the command bellow:

php index.php 'oat\taoAdvancedSearch\scripts\tools\IndexDeleter'

Create/Update aliases for indexes

php index.php 'oat\taoAdvancedSearch\scripts\tools\AliasesUpdater'

Indexation

Warmup cache

This is necessary to optimize indexation:

./taoAdvancedSearch/scripts/tools/CacheWarmup.sh --help

To populate ALL indexes, execute:

./taoAdvancedSearch/scripts/tools/IndexPopulator.sh --help

To populate only resources indexes (Items, tests, etc), execute:

./taoAdvancedSearch/scripts/tools/IndexResources.sh --help

To populate only resources from one class, execute:

./taoAdvancedSearch/scripts/tools/IndexClassResources.sh --help

To populate only class metadata indexes, execute:

./taoAdvancedSearch/scripts/tools/IndexClassMetadata.sh --help

To populate only delivery results, execute:

./taoAdvancedSearch/scripts/tools/IndexDeliveryResults.sh --help

Index Migration

To avoid having to delete and recreate all the indices, it is possible to migrate index mapping by executing the following:

php index.php 'oat\taoAdvancedSearch\scripts\tools\IndexMigration' -i tests -q '{"properties": {"test_qti_structure": {"type": "object","enabled": false}}}'

And also by creating migrations that can perform the update.

declare(strict_types=1);

namespace oat\taoAdvancedSearch\migrations;

use Doctrine\DBAL\Schema\Schema;
use oat\tao\scripts\tools\migrations\AbstractMigration;
use oat\taoAdvancedSearch\model\SearchEngine\Contract\IndexerInterface;
use oat\taoAdvancedSearch\scripts\tools\IndexMigration;

final class Version202307251344000000_taoAdvancedSearch extends AbstractMigration
{
    private const INDEX_UPDATE_BODY = '{"properties": {"test_qti_structure": {"type": "object","enabled": false}}}';

    public function getDescription(): string
    {
        return sprintf(
            'Migrate index "%s" with "%s"',
            IndexerInterface::TESTS_INDEX,
            self::INDEX_UPDATE_BODY
        );
    }

    public function up(Schema $schema): void
    {
        $this->runAction(
            new IndexMigration(),
            [
                '-i',
                IndexerInterface::TESTS_INDEX,
                '-q',
                self::INDEX_UPDATE_BODY,
            ]
        );
    }

    public function down(Schema $schema): void
    {
        $this->throwIrreversibleMigrationException();
    }
}

Garbage collection

To clean old documents in the indexes:

./taoAdvancedSearch/scripts/tools/GarbageCollector.sh --help

And to index missing records

php index.php 'oat\taoAdvancedSearch\scripts\tools\IndexMissingRecords' -h

Statistics

To retrieve information about indexed vs expected data and others, execute:

php index.php '\oat\taoAdvancedSearch\scripts\tools\IndexSummary'

Environment variables

Variable	Description	Example
FEATURE_FLAG_ADVANCED_SEARCH_DISABLED	In case you do not want to have AdvancedSearch enabled even if this extension is installed	true
ADVANCED_SEARCH_METADATA_BLACK_LIST	To avoid indexing metadata that is used in the criteria filter	URI1,URI2,URI3
ELASTICSEARCH_HOSTS	ElasticSearch hosts (use space as delimiter)	http://localhost:9200
ELASTICSEARCH_USERNAME	ElasticSearch user	user
ELASTICSEARCH_PASSWORD	ElasticSearch password	pass
ELASTICSEARCH_PREFIX	Prefix to be used in the index name	tao
ELASTICSEARCH_CLOUD_ID	ElasticSearch cloud id	cloud_id
ELASTICSEARCH_API_KEY_ID	ElasticSearch api key id	api_key_id
ELASTICSEARCH_API_KEY	ElasticSearch api key	api_key

How to create custom indexers?

Here we need to specify 3 required classes and create them:

• Normalizer: Convert search result support format of AdvancedSearch.
• Result Searcher: Execute the search for a paginated index execution.
• Result Filter Factory: The filter used to segregate the index within many workers.

<?php
namespace oat\taoAdvancedSearch\model\DeliveryResult\Service;

use oat\taoAdvancedSearch\model\DeliveryResult\Factory\DeliveryResultFilterFactory;
use oat\taoAdvancedSearch\model\DeliveryResult\Normalizer\DeliveryResultNormalizer;
use oat\taoAdvancedSearch\model\Index\Service\AbstractIndexMigrationTask;

class DeliveryResultMigrationTask extends AbstractIndexMigrationTask
{
    protected function getConfig(): array
    {
        return [
            self::OPTION_NORMALIZER => DeliveryResultNormalizer::class,
            self::OPTION_RESULT_SEARCHER => DeliveryResultSearcher::class,
            self::OPTION_RESULT_FILTER_FACTORY => DeliveryResultFilterFactory::class,
        ];
    }
}