.. contents:: :local:
+Kitsune supports an advanced search syntax in all its search boxes.
+A number of search operators and tokens,
+described below,
+take a {field_name} as an argument.
+The valid field names for each of our document types are explained in the Document Fields section.
The most basic search token is simply an individual word.
+For example,
+the query firefox crashes contains two tokens firefox and crashes.
By default all simple tokens specified must exist in the same field in a document for it to be matched.
+So our firefox crashes query will only match a document with both firefox and crashes in the same field.
+This behaviour can be modified using the operators below.
"x y z"
Matches the phrase, +disabling synonym matching.
+e.g. "firefox accounts"
+will match "firefox accounts" but not "firefox can't find my account".
e.g. add-ons NOT "themes"
+will match add-ons and extensions, but not themes.
field:{field_name}:{query}
Specifies that {query} should be run against {field_name}.
+If enclosed in brackets,
+{query} can have nested operators.
e.g. field:keywords.en-US:(firefox NOT android)
+will find KB documents with firefox and without android in their keywords field.
Each document type has a number of aliases, +shown below, +which allows easy searching across document types.
+e.g. using field:title:firefox in instant search would find both articles and questions with firefox in the title.
.. list-table::
+ :header-rows: 1
+
+ * - Alias
+ - Maps To
+ * - ``title``
+ - ``title.{locale}``
+ * - ``content``
+ - ``content.{locale}``
+.. list-table::
+ :header-rows: 1
+
+ * - Alias
+ - Maps To
+ * - ``title``
+ - ``question_title.{current_locale}``
+ * - ``content``
+ - ``question_content.{current_locale}`` or ``answer_content.{current_locale}``
+ * - ``question``
+ - ``question_content.{current_locale}``
+ * - ``answer``
+ - ``answer_content.{current_locale}``
+.. list-table::
+ :header-rows: 1
+
+ * - Field
+ - Maps To
+ * - ``title``
+ - ``thread_title``
+NOT x
Specifies that the token must not exist in the document.
+x AND y
Specifies that both tokens must exist in the document.
+.. note::
+ A quirk of our current implementation is that ``x AND y`` can return more results than ``x y``.
+ This is because ``x y`` requires both tokens to exist in the same field,
+ whereas ``x AND y`` will match across fields.
+
+ e.g. with a document like so:
+ ::
+ {
+ "title": "x",
+ "content": "y"
+ }
+
+ ``x y`` wouldn't match, but ``x AND y`` would.
+x OR y
Specifies that either token must exist in the document.
+exact:{field_name}:{value}
Specifies that {value} should exactly match the value of (or token in) {field_name}.
e.g. exact:question_has_solution:false will only match questions without solutions.
These fields take a value which will be mapped to the appropriate ID internally.
+.. list-table::
+ :header-rows: 1
+
+ * - Field
+ - Valid values
+ * - ``category``
+ - ``troubleshooting``,
+ ``how-to``,
+ ``how-to-contribute``,
+ ``navigation``
+range:{field_name}:{operator}:{value}
Specifies that {value} should fall within the range of the value of {field_name}.
+{field_name} must be a date or numeric field.
{operator} can take the following values:
+* gt - greater than
+* gte - greater than or equal
+* lt - less than
+* lte - less than or equal
{value} can take the following formats:
+* a basic number e.g. 1
+* a date in the form yyyy-MM-dd
+* a date and time in the form yyyy-MM-dd'T'HH:mm:ss
+* a date math value e.g. now, or now-1d
e.g. range:question_created:gte:2021-05-27 AND range:question_created:lt:2021-05-28
+will match questions created on 27th May 2021.
e.g. range:question_created:gte:now-2d
+will match questions created in the last two days.
The operators above are documented in the order they're evaluated. +To adjust the order of evaluation, +you can make use of brackets.
+e.g. NOT firefox OR crashes will match documents which either don't have firefox in them, or have crashes in them. (This can also be expressed like (NOT firefox) OR crashes).
e.g. NOT (firefox OR crashes) will match documents which have neither firefox nor crashes in them.
Fields below with a Yes in the Locale? column require a locale code after their field name.
+e.g. to match against question content in English, you'll need to use question_content.en-US.
e.g. to match against answer content in German, you'll use to use answer_content.de.
.. note::
+ Currently you can't search in a locale other than the one you're using SUMO in.
+ For instance, attempting to search for ``field:question_content.en-US:firefox`` on http://support.mozilla.org/de will return zero results.
+.. list-table::
+ :header-rows: 1
+
+ * - Field
+ - Locale?
+ - Type
+ * - ``updated``
+ -
+ - Date
+ * - ``product_ids``
+ -
+ - Array of Keywords
+ * - ``topic_ids``
+ -
+ - Array of Keywords
+ * - ``category``
+ -
+ - Keyword
+ * - ``title``
+ - Yes
+ - Text
+ * - ``content``
+ - Yes
+ - Text
+ * - ``summary``
+ - Yes
+ - Text
+ * - ``keywords``
+ - Yes
+ - Text
+ * - ``slug``
+ - Yes
+ - Keyword
+ * - ``doc_id``
+ - Yes
+ - Keyword
+.. list-table::
+ :header-rows: 1
+
+ * - Field
+ - Locale?
+ - Type
+ * - ``question_id``
+ -
+ - Keyword
+ * - ``question_title``
+ - Yes
+ - Text
+ * - ``question_creator_id``
+ -
+ - Keyword
+ * - ``question_content``
+ - Yes
+ - Text
+ * - ``question_created``
+ -
+ - Date
+ * - ``question_updated``
+ -
+ - Date
+ * - ``question_updated_by_id``
+ -
+ - Keyword
+ * - ``question_has_solution``
+ -
+ - Boolean
+ * - ``question_is_locked``
+ -
+ - Boolean
+ * - ``question_is_archived``
+ -
+ - Boolean
+ * - ``question_product_id``
+ -
+ - Keyword
+ * - ``question_topic_id``
+ -
+ - Keyword
+ * - ``question_taken_by_id``
+ -
+ - Keyword
+ * - ``question_taken_until``
+ -
+ - Date
+ * - ``question_tag_ids``
+ -
+ - Array of Keywords
+ * - ``question_num_votes``
+ -
+ - Integer
+ * - ``answer_content``
+ - Yes
+ - Array of Text
+ * - ``locale``
+ -
+ - Keyword
+.. list-table::
+ :header-rows: 1
+
+ * - Field
+ - Type
+ * - ``thread_title``
+ - Text
+ * - ``thread_forum_id``
+ - Keyword
+ * - ``forum_slug``
+ - Keyword
+ * - ``thread_id``
+ - Keyword
+ * - ``thread_created``
+ - Date
+ * - ``thread_creator_id``
+ - Keyword
+ * - ``thread_is_locked``
+ - Boolean
+ * - ``thread_is_sticky``
+ - Boolean
+ * - ``content``
+ - Text
+ * - ``author_id``
+ - Keyword
+ * - ``created``
+ - Date
+ * - ``updated``
+ - Date
+ * - ``updated_by_id``
+ - Keyword
+SUMO has a series of API endpoints to access data.
+::: contents +:::
+/api/2/search/suggest/
GET
application/json
application/json
The search suggest API allows you to get back kb documents and aaq +questions that match specified arguments.
+Arguments can be specified in the url querystring or in the HTTP request +body.
+argument type notes
+q string This is the text you\'re querying for.
++---------------+------+----------------------------------------------+
+| argument | type | notes |
++===============+======+==============================================+
+| locale | st | default: settings.WIKI_DEFAULT_LANGUAGE |
+| | ring | |
+| | | The locale code to restrict results to. |
+| | | |
+| | | Examples: |
+| | | |
+| | | - en-US |
+| | | - fr |
++---------------+------+----------------------------------------------+
+| product | st | default: None |
+| | ring | |
+| | | The product to restrict results to. |
+| | | |
+| | | Example: |
+| | | |
+| | | - firefox |
++---------------+------+----------------------------------------------+
+| max_documents | int | default: 10 |
+| | eger | |
+| | | The maximum number of documents you want |
+| | | back. |
++---------------+------+----------------------------------------------+
+| max_questions | int | default: 10 |
+| | eger | |
+| | | The maximum number of questions you want |
+| | | back. |
++---------------+------+----------------------------------------------+
All response bodies are in JSON.
+With an HTTP 200, you\'ll get back a set of results in JSON.
+{
+ "documents": [
+ {
+ "id": ... # id of kb article
+ "title": ... # title of kb article
+ "url": ... # url of kb article
+ "slug": ... # slug of kb article
+ "locale": ... # locale of the article
+ "products": ... # list of products for the article
+ "topics": ... # list of topics for the article
+ "summary": ... # paragraph summary of kb article (plaintext)
+ "html": ... # html of the article
+ }
+ ...
+ ],
+ "questions": [
+ {
+ "id": ... # integer id of the question
+ "answers": ... # list of answer ids
+ "content": ... # content of question (in html)
+ "created": ... # datetime stamp in iso-8601 format
+ "creator": ... # JSON object describing the creator
+ "involved": ... # list of JSON objects describing everyone who
+ participated in the question
+ "is_archived": ... # boolean for whether this question is archived
+ "is_locked": ... # boolean for whether this question is locked
+ "is_solved": ... # boolean for whether this question is solved
+ "is_spam": ... # boolean for whether this question is spam
+ "is_taken": ... # FIXME:
+ "last_answer": ... # id for the last answer
+ "num_answers": ... # total number of answers
+ "locale": ... # the locale for the question
+ "metadata": ... # metadata collected for the question
+ "tags": ... # tags for the question
+ "num_votes_past_week": ... # the number of votes in the last week
+ "num_votes": ... # the total number of votes
+ "product": ... # the product
+ "solution": ... # id of answer marked as a solution if any
+ "taken_until": ... # FIXME:
+ "taken_by": ... # FIXME:
+ "title": ... # title of the question
+ "topic": ... # FIXME:
+ "updated_by": ... # FIXME:
+ "updated": ... # FIXME:
+ },
+ ...
+ ]
+}
+Using curl:
+curl -X GET "http://localhost:8000/api/2/search/suggest/?q=videos"
+
+curl -X GET "http://localhost:8000/api/2/search/suggest/?q=videos&max_documents=3&max_questions=3"
+
+curl -X GET "http://localhost:8000/api/2/search/suggest/" \
+ -H "Content-Type: application/json" \
+ -d '
+{
+ "q": "videos",
+ "max_documents": 3,
+ "max_questions": 0
+}'
+++ + + + + + + + + + + + + + + + +All locales supported by SUMO.
+Example request:
++GET /api/2/locales/ HTTP/1.1 +Accept: application/json +Example response:
++HTTP/1.0 200 OK +Vary: Accept, X-Mobile, User-Agent +Allow: OPTIONS, GET +X-Frame-Options: DENY +Content-Type: application/json + +{ + "vi": { + "name": "Vietnamese", + "localized_name": "Ti\u1ebfng Vi\u1ec7t", + "aaq_enabled": false + }, + "el": { + "name": "Greek", + "localized_name": "\u0395\u03bb\u03bb\u03b7\u03bd\u03b9\u03ba\u03ac", + "aaq_enabled": false + }, + "en-US": { + "name": "English", + "localized_name": "English", + "aaq_enabled": true + } +} ++
+- reqheader Accept
+- +
+application/json
+- resheader Content-Type
+- +
+application/json
+- statuscode 200
+- +
+no error
+
We record major architectural decisions for Kitsune/SUMO in Architecture +Decision Records (ADR), as described by Michael +Nygard. +Below is the list of our current ADRs.
+::: {.toctree maxdepth="1" glob=""} +architecture/decisions/* +:::
+ + + + + + + + + + + + + + + + +Date: 2019-01-10
+Accepted
+We need to record the architectural decisions made on this project.
+We will use Architecture Decision Records, as described by Michael Nygard.
+See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's adr-tools.
+ + + + + + + + + + + + + +Date: 2020-10-27
+Pending
+Kitsune supports many locales, +and has content which we want to be searchable in those locales.
+Elasticsearch has support for many language-specific analyzers: +https://www.elastic.co/guide/en/elasticsearch/reference/7.9/analysis-lang-analyzer.html
+Search v1 used per-document analyzers, +that is to say, within the same index:
+doc_1: { "content": "Hello world" }
+doc_2: { "content": "Hallo Welt" }
+doc_1.content could be analyzed using an english analyzer,
+and doc_2.content could be analyzed using a german analyzer.
Well before version 7 ES removed this feature, +and now all fields of the same name across an index must be analyzed the same, +so we must take a different approach with the current Search implementation.
+We can either place separate locales in their own index, +and set up locale-specific analyzers for the same field name across indices. +Or we can keep separate locales within the same index, +and define unique field names for each field which needs to be analyzed under a specific locale.
+Heavily influenced by: https://www.elastic.co/blog/multilingual-search-using-language-identification-in-elasticsearch
+We will store all documents within the same index and use an Object field for fields which need to use locale-specific analyzers.
+We will call this field SumoLocaleAwareTextField and will have a key for each locale,
+with the appropriate analyzer defined on that key,
+such that:
doc_1: { "content": { "en-US": "Hello world" }}
+doc_2: { "content": { "de": "Hallo Welt" }}
+doc_1.content.en-US is analyzed using an english analyzer,
+and doc_2.content.de is analyzed using a german analyzer.
We won't need to manage many indeces of wildly different sizes, +as we would if we used one index per locale.
+Documents within a specific locale can be searched for by searching on the field.locale-name field,
+for instance content.en-US.
Searching across all locales can be performed with a wildcard (like content.*).
Date: 2020-10-27
+Pending
+As we are re-implementing our search in ElasticSearch v7, +we must re-implement Ask a Question (AAQ) search.
+There is one primary use-case for storing AAQ documents in ES which Search v1 supports, +which we must continue to be able to do in the redesigned Search: +searching for an AAQ thread as a unit.
+There are other secondary use-cases which we may want to support when storing AAQ documents in ES. +A non-exhaustive list of these are:
+We also want search to be fast, +so should model our data to avoid nested fields and parent-child relationships, +and use de-normalization wherever possible: +https://www.elastic.co/guide/en/elasticsearch/reference/7.9/tune-for-search-speed.html#_document_modeling
+We will model our data in ES based on what makes most sense for our expected use-cases, +and what will make those fast and efficient, +rather than feeling like we must have a 1:1 copy of our data structure in our database.
+In this vein, we will use a structure of two document "types" within one index,
+QuestionDocument and AnswerDocument,
+where a QuestionDocument will exist for each Question which exists in the database,
+and an AnswerDocument will exist for each Answer which exists in the database.
AnswerDocument will be a subclass of QuestionDocument so will inherit all of its fields,
+and we will set the value of those fields to the value of the Question associated with its Answer.
For instance, if in database:
+answer.created => 2020-10-27
+answer.question.created => 2020-11-01
+in elastic:
+answer_document.created => 2020-10-27
+answer_document.question_created => 2020-11-01
+QuestionDocument will also have an answer_content field,
+which contains the content of all a Question's Answers.
+We will set this to null in the AnswerDocument.
QuestionDocuments can be distinguished from AnswerDocuments by whether the created value is null or not.
This structure gives us great flexibility over the type of searches and aggregations we can do:
+AAQ threads can be searched over as a unit by searching over the question_content and answer_content fields on a QuestionDocument.
+ Collapsing or aggregations aren't required to ensure that threads only appear once,
+ which wouldn't be the case if we didn't have an answer_content field.
Searches within AAQ threads are possible by searching over the question_content fields on a QuestionDocument,
+ and the content fields on an AnswerDocument.
Searches can be limited to Question content and the content of Answers which are solutions by filtering by is_solved: True,
+ and searching over question_content and content fields.
Aggregations on Answers,
+ such as gathering contribution metrics,
+ can be performed by aggregating AnswerDocuments.
There may be some Question data indexed in AnswerDocuments which we never use.
Date: 2023-02-15
+Pending
+With support for type hints +and static type-checking tools solidly entrenched in +the Python eco-system, we'd like to take advantage of both within Kitsune. The +goal is to support +gradual typing, +to support the addition of type-hints only where they're helpful, where their +addition adds more value than their burden.
+Type hints can be helpful to quickly understand the types of values expected by
+functions and methods, as well as the types of their results, easing the burden
+of using those functions/methods for the first time. It encourages modularity,
+since one doesn't have to dig into the details of the underlying code to discover
+what's expected. Once in place, they also enable the use of static type-checking
+tools, which can flag basic interface errors, for example, when you're passing a
+str into a function that expects an int.
Type hints can sometimes be an undue burden as well. In some cases determining +the correct type can be difficult, time-consuming, and the outcome unintuitive. +In those cases, it's probably better to skip them.
+In the end, it's important to remember that type hints and static type-checking +are never a substitute for testing. Writing good tests for your code is essential +whether you add type hints or not.
+We recommend that all pull requests that modify/create Python functions and/or +methods, add a type hint for each of the function/method arguments as well as the +result. There's a judgement call here though. If it's an "undue burden", skip it. +If not, do it.
+Also, we will run a static type-checking tool, either prior to commits, or during +CI, or both.
+There are two primary consequences.
+An easier, less error-prone coding experience when working with unfamiliar Python + functions/methods.
+We reap the benefits of using a static type-checking tool. For example, the + detection, prior to run time, of interface errors, where the types passed into + functions/methods or the results returned are not what's expected.
+There is the potential for this to have a negative consequence as well. It could slow +us down. It could cost more than its worth. So, it's important to keep in mind the tradeoff. Type hints should only be added where they're helpful.
+ + + + + + + + + + + + + +