feat: CMD-187 autocreate search page #885

wesleyboar · 2024-10-22T22:08:47Z

Overview

Support a Google Search page with less effort.

What Effort?

Every Time:

Create search engine.
Configure search engine to match others.
Set owners of search engine.
Copy search ~~code~~ → ID.
Set relevant SEARCH settings.

Previously:

Restart the server.
Create /search page.
Create snippet on page.
Paste search code in snippet.
Duplicate-or-Neglect search styles from TACC

Now, Instead:

Restart the server.

Caution

Do not merge.

Use new solution.

Warning

Successful run requires two deploys. Help wanted.

Changes

added search_page app
added SEARCH_AUTO_SETUP setting
added GOOGLE_SEARCH_ENGINE_ID setting

Testing

Register a Google Programmable Search Engine.
Add real GOOGLE_SEARCH_ENGINE_ID setting e.g.
^{GOOGLE_SEARCH_ENGINE_ID = 'b099996c09ebd4ece'}
Add SEARCH_AUTO_SETUP = True setting.
Run/Deploy website that has no search page.
Verify "Search (Auto-Generated)" page exists.
Search using search bar.
✅ Verify search page shows with results.
Remove GOOGLE_SEARCH_ENGINE_ID setting.
Run/Deploy site.
Revisit search page.
✅ Verify warning appears on page.

UI

with `GOOGLE_SEARCH_ENGINE_ID`	sans `GOOGLE_SEARCH_ENGINE_ID`

So that it can be loaded from CDN. Use cases are: - if search template is broken - for custom search page - for non-Core website

Compared to "cms-search", "google-search" is: - more accurate - more obvious to those not familiar with Core-CMS

So dev sees 'google-search' from CSS before seeing it in HTML `id` attr.

Caused server error.

This had create dbreadcrumsb before.

Final markup undecided, so keep it simple, and easy to understand.

wesleyboar

Any thoughts on why create_page() is run twice?

I have checks for existing page and fallbacks for missing page.

wesleyboar · 2024-10-25T15:15:49Z

apps/search_page/apps.py

+    def ready(self):
+        if settings.SEARCH_PAGE_AUTO_SETUP:
+            from .utils import create_page
+            create_page()


Caution
This create_page function is run twice on website startup.

1. Initial Deploy on Existing Site (When App Has Not Yet Been Installed).

Internal Server Error. Re-deploy required.

Log

Note: treebeard is an application that manages pages.

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 732, in execute raise ex.with_traceback(None) psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "cms_treenode_path_key" DETAIL: Key (path)=(0004) already exists. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/code/taccsite_cms/wsgi.py", line 16, in <module> application = get_wsgi_application() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application django.setup(set_prefix=False) File "/usr/local/lib/python3.11/site-packages/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python3.11/site-packages/django/apps/registry.py", line 124, in populate app_config.ready() File "/code/apps/search_page/apps.py", line 10, in ready create_page() File "/code/apps/search_page/utils.py", line 46, in create_page page = create_cms_page( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/contextlib.py", line 81, in inner return func(*args, **kwds) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/cms/api.py", line 183, in create_page page.set_tree_node(site=site, target=target_node, position=position) File "/usr/local/lib/python3.11/site-packages/cms/models/pagemodel.py", line 474, in set_tree_node self.node = TreeNode.add_root(instance=new_node) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/treebeard/mp_tree.py", line 626, in add_root return MP_AddRootHandler(cls, **kwargs).process() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/treebeard/mp_tree.py", line 344, in process newobj.save() File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save self.save_base( File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table results = self._do_insert( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert return manager._insert( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute return self._execute_with_wrappers( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute with self.db.wrap_database_errors: File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 732, in execute raise ex.with_traceback(None) django.db.utils.IntegrityError: duplicate key value violates unique constraint "cms_treenode_path_key" DETAIL: Key (path)=(0004) already exists. unable to load app 0 (mountpoint='') (callable not found or import error) *** no app loaded. going in full dynamic mode *** uWSGI running as root, you can use --uid/--gid/--chroot options *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 1) spawned uWSGI worker 1 (pid: 16, cores: 20) spawned uWSGI worker 2 (pid: 24, cores: 20) spawned uWSGI worker 3 (pid: 37, cores: 20) spawned uWSGI worker 4 (pid: 66, cores: 20) --- no python application found, check your startup logs for errors --- [pid: 66|app: -1|req: -1/1] 129.114.111.125 () {68 vars in 2061 bytes} [Fri Oct 25 10:01:21 2024] GET /admin/cms/page/?language=en => generated 21 bytes in 0 msecs (HTTP/2.0 500) 2 headers in 83 bytes (0 switches on core 2) --- no python application found, check your startup logs for errors --- [pid: 16|app: -1|req: -1/2] 129.114.111.125 () {66 vars in 1951 bytes} [Fri Oct 25 10:01:26 2024] GET / => generated 21 bytes in 0 msecs (HTTP/2.0 500) 2 headers in 83 bytes (0 switches on core 1)

2. Re-Deploy.

Page is created and works. Logs note existing page.

Log

[DJANGO] INFO 2024-10-25 10:03:53,120 UTC utils portal.apps.search_page.utils.create_page:62: Found existing search page at "search"

3. Delete Search Page. Deploy Again.

Page is created and works. Logs note existing page.

Log

[DJANGO] INFO 2024-10-25 10:03:53,120 UTC utils portal.apps.search_page.utils.create_page:62: Found existing search page at "search"

Tip
Try creating a custom migration instead.

wesleyboar

@jarosenb gave me a good internal review:

It sounds like what you're trying to create is a custom migration, if this thing really needs to be a CMS page.

@taoteg suggested a guide:

A good migrations primer

A.I. has given me starter code.

I have the page creation function I can adapt. I believe I can replace the complexity of the app with one file.

I'll help you create a Django CMS Page using a database migration. This is useful for automatically creating pages during deployment.

Here's how to create a new migration file that will create a CMS page:
XXXX_create_custom_page.py
from django.db import migrations
from cms.api import create_page
from cms.models import Page

def create_cms_page(apps, schema_editor):
    # Skip if page already exists
    if not Page.objects.filter(title_set__title='Your Page Title').exists():
        page = create_page(
            title='Your Page Title',
            template='page.html',  # Use your template
            language='en',
            slug='your-page-slug',
            published=True,
            in_navigation=True,
        )
        page.publish('en')

def remove_cms_page(apps, schema_editor):
    # Delete the page if it exists
    Page.objects.filter(title_set__title='Your Page Title').delete()

class Migration(migrations.Migration):
    dependencies = [
        ('cms', '0022_auto_20180620_1551'),  # Adjust this to your last migration
    ]

    operations = [
        migrations.RunPython(create_cms_page, remove_cms_page)
    ]
To use this:

Create a new migration file in your CMS app's migrations folder

Replace XXXX in the filename with the next available migration number

Adjust the dependencies to point to your last migration

Customize the page properties in create_cms_page:

title: The page title

template: Your template name

slug: URL-friendly version of the title

in_navigation: Whether to show in menus

Then run:
python manage.py migrate
The migration will create the page if it doesn't exist already. The remove_cms_page function handles rollback if you need to reverse the migration.

Note: Make sure you have the necessary imports and that your template exists before running the migration.

wesleyboar added 7 commits October 22, 2024 17:04

feat: template & setting for Google search

4515b85

Merge branch 'main' into feat/template-and-setting-for-google-search

2883e49

fix: styles and fallback warning for Google Search

8607c90

enhance: move sarch CSS to its own file

f3dc4dc

So that it can be loaded from CDN. Use cases are: - if search template is broken - for custom search page - for non-Core website

enhance: rename search stylesheet and ID

3947bc2

Compared to "cms-search", "google-search" is: - more accurate - more obvious to those not familiar with Core-CMS

refactor: move css to top of template

d0d5e00

So dev sees 'google-search' from CSS before seeing it in HTML `id` attr.

fix: missing default GOOGLE_SEARCH_ENGINE_ID

b74e2a0

Caused server error.

wesleyboar marked this pull request as ready for review October 22, 2024 23:37

wesleyboar added 12 commits October 22, 2024 19:11

Merge branch 'main' into feat/template-and-setting-for-google-search

7c93b7e

feat: search template is standard

f80ebc8

Merge branch 'main' into feat/template-and-setting-for-google-search

5208bde

feat: convert search template into a search app

9457519

feat(search_page): slimmer views.py

16d34c3

feat(search): create page if it does not exist

da3a73a

fix(search): restore breadcrumbs

5fb5a32

chore(search_page): remove now-unnecessary file

1fef4f8

This had create dbreadcrumsb before.

enhance(search_page): flag page as auto-generated

e8072e6

enhance(search): attach apphook to manage urls

3291015

style(search): polish

ca11f07

fix(search): searchbar not using curr. search path

445010c

wesleyboar changed the title ~~feat: template & setting for Google search~~ feat: search_page app Oct 24, 2024

wesleyboar added 9 commits October 23, 2024 19:42

chore(search): remove cruft

8d812c8

chore(search): reduce silly complexity of tag

3793bb9

chore(search): remove cruft 2

92e9ecf

feat(search): auto isntall and setup

adf6ad2

feat(search): simplify an isolate markup

d9db279

Final markup undecided, so keep it simple, and easy to understand.

core(search): remove cruft markup

fda65e5

chore(search): simlify utils.py

9d88be6

style(search): new line

148f661

refactor(search): polish

7cbb96e

wesleyboar added 3 commits October 24, 2024 15:19

refactor(search): rename a function

f57d4e1

style(search): ending slash for example path

55b84da

style(search): ending slash for example path

3ee4fe1

wesleyboar requested review from taoteg and jarosenb October 24, 2024 20:27

Merge branch 'main' into feat/template-and-setting-for-google-search

b4d6ac3

wesleyboar commented Oct 25, 2024

View reviewed changes

Merge branch 'main' into feat/template-and-setting-for-google-search

0f16d38

wesleyboar commented Oct 31, 2024

View reviewed changes

wesleyboar marked this pull request as draft October 31, 2024 19:24

wesleyboar changed the title ~~feat: search_page app~~ feat: CMD-187 autocreate search page Oct 31, 2024

wesleyboar added 4 commits October 31, 2024 19:25

Merge branch 'main' into feat/template-and-setting-for-google-search

ba2b091

Merge branch 'main' into feat/template-and-setting-for-google-search

959ba8e

Merge branch 'main' into feat/template-and-setting-for-google-search

7dd0402

Merge branch 'main' into feat/template-and-setting-for-google-search

a12a70f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: CMD-187 autocreate search page #885

feat: CMD-187 autocreate search page #885

wesleyboar commented Oct 22, 2024 •

edited

Loading

wesleyboar left a comment

wesleyboar Oct 25, 2024

wesleyboar Oct 31, 2024

wesleyboar left a comment •

edited

Loading

feat: CMD-187 autocreate search page #885

Are you sure you want to change the base?

feat: CMD-187 autocreate search page #885

Conversation

wesleyboar commented Oct 22, 2024 • edited Loading

Overview

Related

Changes

Testing

UI

wesleyboar left a comment

Choose a reason for hiding this comment

wesleyboar Oct 25, 2024

Choose a reason for hiding this comment

wesleyboar Oct 31, 2024

Choose a reason for hiding this comment

wesleyboar left a comment • edited Loading

Choose a reason for hiding this comment

wesleyboar commented Oct 22, 2024 •

edited

Loading

wesleyboar left a comment •

edited

Loading