How to Make Jira Ticket Titles Searchable in Azure AI Search Using LlamaIndex?

[koberghe]

I am working on indexing Jira content into Azure AI Search using LlamaIndex. A critical requirement for my use case is to make the Jira ticket title searchable, as titles often include essential information necessary for effective retrieval.

What I’ve Tried:

Using JiraReader to load Jira content:

• The ticket title is included at the beginning of the text field.
• However, when documents are long and get chunked into multiple nodes, only the first node retains the title content, while other nodes lose it.

Exploring metadata fields:

• I attempted to use the title as a metadata field, hoping to make it searchable.
• Unfortunately, all metadata fields in LlamaIndex are defined as filterable only, with no apparent way to make them searchable outside the library.

Comparison to LangChain:

• In LangChain, I can explicitly define fields (like the title) as searchable using the fields parameter.
• However, LangChain lacks a Jira loader, which led me to explore LlamaIndex.

My Question:

Am I missing an obvious solution to this issue (also addressed here)? Is there a way in LlamaIndex to make metadata fields like title searchable, or is there a recommended workaround for ensuring Jira titles remain searchable across chunked documents, without modifying the library?

This feature feels crucial for effectively indexing Jira content in Azure AI Search. I’d appreciate any guidance or suggestions on how to handle this. Thanks!

[dosubot[bot]]

Hey @koberghe! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together!

To make Jira ticket titles searchable in Azure AI Search using LlamaIndex, you can utilize the AzureAISearchVectorStore to configure metadata fields as searchable. Although LlamaIndex metadata fields are typically filterable, you can specify them to be searchable by setting up the filterable_metadata_field_keys parameter.

Here's how you can achieve this:

1. Define Metadata Fields: Specify the Jira ticket title as a metadata field and configure it to be searchable.

2. Use AzureAISearchVectorStore: Set up the vector store with the appropriate configuration to ensure the title is indexed and searchable.

Here's an example setup:

from azure.core.credentials import AzureKeyCredential
from azure.search.documents.indexes import SearchIndexClient
from llama_index.vector_stores.azureaisearch import AzureAISearchVectorStore
from llama_index.vector_stores.azureaisearch import IndexManagement, MetadataIndexFieldType

# Azure AI Search setup
search_service_api_key = "YOUR-AZURE-SEARCH-SERVICE-ADMIN-KEY"
search_service_endpoint = "YOUR-AZURE-SEARCH-SERVICE-ENDPOINT"
credential = AzureKeyCredential(search_service_api_key)

# Index name to use
index_name = "jira-index"

# Use index client to demonstrate creating an index
index_client = SearchIndexClient(
    endpoint=search_service_endpoint,
    credential=credential,
)

# Define metadata fields to be searchable
metadata_fields = {
    "title": ("title", MetadataIndexFieldType.STRING),
    # Add other fields as needed
}

# Creating an Azure AI Search Vector Store with searchable metadata fields
vector_store = AzureAISearchVectorStore(
    search_or_index_client=index_client,
    filterable_metadata_field_keys=metadata_fields,
    hidden_field_keys=["embedding"],
    index_name=index_name,
    index_management=IndexManagement.CREATE_IF_NOT_EXISTS,
    id_field_key="id",
    chunk_field_key="chunk",
    embedding_field_key="embedding",
    embedding_dimensionality=1536,
    metadata_string_field_key="metadata",
    doc_id_field_key="doc_id",
    language_analyzer="en.lucene",
    vector_algorithm_type="exhaustiveKnn",
)

This setup allows you to make the Jira ticket title searchable by configuring it as a metadata field with the appropriate type. This approach should help you effectively index Jira content in Azure AI Search and ensure that titles remain searchable across chunked documents ^[1].

To continue talking to Dosu, mention @dosu.

[koberghe]

@dosu I don't think your code makes the title field searchable. You have just defined title as a metadata field of type string. But the method _create_metadata_index_fields turns this is into a field which is filterable only.

[dosubot[bot]]

To make the title field searchable in LlamaIndex, you need to modify the _create_metadata_index_fields method to use SearchableField instead of SimpleField. This change will allow the metadata fields to be searchable. Here's how you can adjust the code:

from azure.search.documents.indexes.models import SearchableField

def _create_metadata_index_fields(self) -> List[Any]:
    """Create a list of index fields for storing metadata values."""
    index_fields = []

    # create search fields
    for v in self._metadata_to_index_field_map.values():
        field_name, field_type = v

        # Skip if the field is already mapped
        if field_name in self._field_mapping.values():
            continue

        if field_type == MetadataIndexFieldType.STRING:
            index_field_type = "Edm.String"
        elif field_type == MetadataIndexFieldType.INT32:
            index_field_type = "Edm.Int32"
        elif field_type == MetadataIndexFieldType.INT64:
            index_field_type = "Edm.Int64"
        elif field_type == MetadataIndexFieldType.DOUBLE:
            index_field_type = "Edm.Double"
        elif field_type == MetadataIndexFieldType.BOOLEAN:
            index_field_type = "Edm.Boolean"
        elif field_type == MetadataIndexFieldType.COLLECTION:
            index_field_type = "Collection(Edm.String)"

        field = SearchableField(
            name=field_name,
            type=index_field_type,
            searchable=True,
            hidden=field_name in self._hidden_field_keys,
        )
        index_fields.append(field)

    return index_fields

This modification ensures that the title field, and any other metadata fields you define, are searchable in Azure AI Search ^[1].

To continue talking to Dosu, mention @dosu.

[koberghe]

@dosu As stated in the original question, I am looking for a solution which works without modifying the library.

[dosubot[bot]]

To make metadata fields like the Jira ticket title searchable in Azure AI Search using LlamaIndex without modifying the library, you can configure the AzureAISearchVectorStore to include the title as a searchable field. This can be done by specifying the metadata fields in the filterable_metadata_field_keys parameter, which allows them to be used in search queries.

Here's an example setup:

from azure.core.credentials import AzureKeyCredential
from azure.search.documents.indexes import SearchIndexClient
from llama_index.vector_stores.azureaisearch import AzureAISearchVectorStore
from llama_index.vector_stores.azureaisearch import IndexManagement, MetadataIndexFieldType

# Azure AI Search setup
search_service_api_key = "YOUR-AZURE-SEARCH-SERVICE-ADMIN-KEY"
search_service_endpoint = "YOUR-AZURE-SEARCH-SERVICE-ENDPOINT"
credential = AzureKeyCredential(search_service_api_key)

# Index name to use
index_name = "jira-index"

# Use index client to demonstrate creating an index
index_client = SearchIndexClient(
    endpoint=search_service_endpoint,
    credential=credential,
)

# Define metadata fields to be searchable
metadata_fields = {
    "title": ("title", MetadataIndexFieldType.STRING),
    # Add other fields as needed
}

# Creating an Azure AI Search Vector Store with searchable metadata fields
vector_store = AzureAISearchVectorStore(
    search_or_index_client=index_client,
    filterable_metadata_field_keys=metadata_fields,
    hidden_field_keys=["embedding"],
    index_name=index_name,
    index_management=IndexManagement.CREATE_IF_NOT_EXISTS,
    id_field_key="id",
    chunk_field_key="chunk",
    embedding_field_key="embedding",
    embedding_dimensionality=1536,
    metadata_string_field_key="metadata",
    doc_id_field_key="doc_id",
    language_analyzer="en.lucene",
    vector_algorithm_type="exhaustiveKnn",
)

This setup allows you to make the Jira ticket title searchable by configuring it as a metadata field with the appropriate type, without needing to modify the core library code ^[1].

To continue talking to Dosu, mention @dosu.