Skip to content

Release v6.0.0
Compare
Choose a tag to compare
@jhamon jhamon released this 07 Feb 16:00
· 26 commits to main since this release
v6.0.0
d91e31b

What's new in this release?

Indexes with Integrated Inference

This release adds a new create_index_for_model method as well as upsert_records, and search methods. Together these methods provide a way for you to easily store your data and let us manage the process of creating embeddings. To learn about available models, see the Model Gallery.

Note: If you were previously using the preview versions of this functionality via the pinecone-plugin-records package, you will need to uninstall that package in order to use the v6 pinecone release.

from pinecone import (
    Pinecone,
    CloudProvider,
    AwsRegion,
    EmbedModel,
)

# 1. Instantiate the Pinecone client
pc = Pinecone(api_key="<<PINECONE_API_KEY>>")

# 2. Create an index configured for use with a particular model
index_config = pc.create_index_for_model(
    name="my-model-index",
    cloud=CloudProvider.AWS,
    region=AwsRegion.US_EAST_1,
    embed=IndexEmbed(
        model=EmbedModel.Multilingual_E5_Large,
        field_map={"text": "my_text_field"}
    )
)

# 3. Instantiate an Index client
idx = pc.Index(host=index_config.host)

# 4. Upsert records
idx.upsert_records(
    namespace="my-namespace",
    records=[
        {
            "_id": "test1",
            "my_text_field": "Apple is a popular fruit known for its sweetness and crisp texture.",
        },
        {
            "_id": "test2",
            "my_text_field": "The tech company Apple is known for its innovative products like the iPhone.",
        },
        {
            "_id": "test3",
            "my_text_field": "Many people enjoy eating apples as a healthy snack.",
        },
        {
            "_id": "test4",
            "my_text_field": "Apple Inc. has revolutionized the tech industry with its sleek designs and user-friendly interfaces.",
        },
        {
            "_id": "test5",
            "my_text_field": "An apple a day keeps the doctor away, as the saying goes.",
        },
        {
            "_id": "test6",
            "my_text_field": "Apple Computer Company was founded on April 1, 1976, by Steve Jobs, Steve Wozniak, and Ronald Wayne as a partnership.",
        },
    ],
)

# 5. Search for similar records
from pinecone import SearchQuery, SearchRerank, RerankModel

response = index.search_records(
    namespace="my-namespace",
    query=SearchQuery(
        inputs={
            "text": "Apple corporation",
        },
        top_k=3
    ),
    rerank=SearchRerank(
        model=RerankModel.Bge_Reranker_V2_M3,
        rank_fields=["my_text_field"],
        top_n=3,
    ),
)

Call the Inference API

You can now interact with Pinecone's Inference API without the need to install any extra plugins.

Note: If you were previously using the preview versions of this functionality via the pinecone-plugin-inference package, you will need to uninstall that package.

from pinecone import Pinecone

pc = Pinecone(api_key="<<PINECONE_API_KEY>>")

inputs = ["Who created the first computer?"]
outputs = pc.inference.embed(
    model="multilingual-e5-large", 
    inputs=inputs, parameters={"input_type": "passage", "truncate": "END"}
)
print(outputs)
#  EmbeddingsList(
#      model='multilingual-e5-large',
#      data=[
#          {'values': [0.1, ...., 0.2]},
#        ],
#      usage={'total_tokens': 6}
#  )

New client variants with support for asyncio

The v6 Python SDK introduces a new client variants, PineconeAsyncio and IndexAsyncio, which provide async methods for use with asyncio. This should unblock those who wish to use Pinecone with modern async web frameworks such as FastAPI, Quart, Sanic, etc. Those trying to onboard to Pinecone and upsert large amounts of data should significantly benefit from the efficiency of running many upserts in parallel.

To use these, you will need to install pinecone[asyncio] which pulls in an extra depdency on aiohttp. See notes on installation.

You can expect more documentation and information on how to use these asyncio clients to follow soon.

import asyncio

from pinecone import (
    PineconeAsyncio,
    IndexEmbed,
    CloudProvider,
    AwsRegion,
    EmbedModel
)

async def main():
    async with PineconeAsyncio() as pc:
        if not await pc.has_index(index_name):
            desc = await pc.create_index_for_model(
                name="book-search",
                cloud=CloudProvider.AWS,
                region=AwsRegion.US_EAST_1,
                embed=IndexEmbed(
                    model=EmbedModel.Multilingual_E5_Large,
                    metric="cosine",
                    field_map={
                        "text": "description",
                    },
                )
            )

asyncio.run(main())

Interactions with a deployed index are done via IndexAsyncio class, which can be instantiated using helper methods on either Pinecone or PineconeAsyncio:

import asyncio
from pinecone import Pinecone

async def main():
    pc = Pinecone(api_key='<<PINECONE_API_KEY>>')    
    async with pc.IndexAsyncio(host="book-search-dojoi3u.svc.aped-4627-b74a.pinecone.io") as idx:
        await idx.upsert_records(
            namespace="books-records",
            records=[
                {
                    "id": "1",
                    "title": "The Great Gatsby",
                    "author": "F. Scott Fitzgerald",
                    "description": "The story of the mysteriously wealthy Jay Gatsby and his love for the beautiful Daisy Buchanan.",
                    "year": 1925,
                },
                {
                    "id": "2",
                    "title": "To Kill a Mockingbird",
                    "author": "Harper Lee",
                    "description": "A young girl comes of age in the segregated American South and witnesses her father's courageous defense of an innocent black man.",
                    "year": 1960,
                },
                {
                    "id": "3",
                    "title": "1984",
                    "author": "George Orwell",
                    "description": "In a dystopian future, a totalitarian regime exercises absolute control through pervasive surveillance and propaganda.",
                    "year": 1949,
                },
            ]
        )


asyncio.run(main())

Organize your indexes with tags

Tags are key-value pairs you can attach to indexes to better understand, organize, and identify your resources. Tags are flexible and can be tailored to your needs, but some common use cases for them might be to label an index with the relevant deployment environment, application, team, or owner.

Tags can be set during index creation by passing an optional dictionary with the tags keyword argument to the create_index and create_index_for_model methods. Here's an example demonstrating how tags can be passed to create_index.

from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    GcpRegion,
    Metric
)

pc = Pinecone(api_key='<<PINECONE_API_KEY>>')

pc.create_index(
    name='my-index',
    dimension=1536,
    metric=Metric.COSINE,
    spec=ServerlessSpec(
        cloud=CloudProvider.GCP,
        region=GcpRegion.US_CENTRAL1
    ),
    tags={
        "environment": "testing",
        "owner": "jsmith",
    }
)

See this page for more documentation about how to add, modify, or remove tags.

Sparse indexes early access support

Sparse indexes are currently in early access. This release will allow those with early access to create sparse indexes and view those configurations with the describe_index and list_indexes methods.

These are created using the same create_index method as other index types but with different configuration options. For sparse indexes, you must omit dimension while passingmetric="dotproduct" and vector_type="sparse".

from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    AwsRegion,
    Metric,
    VectorType
)

pc = Pinecone()
pc.create_index(
    name='sparse-index',
    metric=Metric.DOTPRODUCT,
    spec=ServerlessSpec(
        cloud=CloudProvider.AWS,
        region=AwsRegion.US_WEST_2
    ),
    vector_type=VectorType.SPARSE
)

# Check the description to get the host url
desc = pc.describe_index(name='sparse-index')

# Instantiate the index client
sparse_index = pc.Index(host=desc.host)

Upserting and querying a sparse index is very similar to before, except now the values field of a Vector (used when working with dense values) may be unset.

import random
from pinecone import Vector, SparseValues

def unique_random_integers(n, range_start, range_end):
    if n > (range_end - range_start + 1):
        raise ValueError("Range too small for the requested number of unique integers")
    return random.sample(range(range_start, range_end + 1), n)

# Generate some random sparse vectors
sparse_index.upsert(
    vectors=[
        Vector(
            id=str(i),
            sparse_values=SparseValues(
                indices=unique_random_integers(10, 0, 10000),
                values=[random.random() for j in range(10)]
            )
        ) for i in range(10000)
    ],
    batch_size=500,
)

# Querying sparse
sparse_index.query(
    top_k=10,
    sparse_vector={"indices":[1,2,3,4,5], "values": [random.random()]*5}
)

Configuration UX with enums

Many enum objects have been added to help with the discoverability of some configuration options. Type hints in your editor will now suggest enums such as Metric, AwsRegion, GcpRegion, PodType, EmbedModel, RerankModel and more to help you quickly get going without having to go looking for documentation examples. This is a backwards compatible change and you should still be able to pass string values for fields exactly as before if you have preexisting code.

For example, code like this

from pinecone import Pinecone, ServerlessIndex

pc = Pinecone()
pc.create_index(
    name='my-index',
    dimension=1536,
    metric='cosine',
    spec=ServerlessSpec(cloud='aws', region='us-west-2'),
    vector_type='dense'
)

Can now be written as

from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    AwsRegion,
    Metric,
    VectorType
)

pc = Pinecone()
pc.create_index(
    name='my-index',
    dimension=1536,
    metric=Metric.COSINE,
    spec=ServerlessSpec(
        cloud=CloudProvider.AWS,
        region=AwsRegion.US_WEST_2
    ),
    vector_type=VectorType.DENSE
)

Both ways of working are equally valid. Some may prefer the more concise nature of passing simple string values, but others may prefer the support your editor gives you to tab complete when working with enums.

Breaking changes in 6.x

  • pinecone-plugin-records and pinecone-plugin-inference are no longer needed because the functionality has been incorporated into the pinecone package itself. If you attempt to load the v6 client with these plugins present in your development environment, you will see an exception directing you to uninstall those plugins.
  • Dropped support for Python 3.8, which has now reached official end of life. We added support for Python 3.13.
  • Removed the explicit dependency on tqdm which is used to provide a nice progress bar when upserting lots of data into Pinecone. If tqdm is available in the environment the Pinecone SDK will detect and use it but we will no longer require tqdm to be installed in order to run the SDK. Popular notebook platforms such as Jupyter and Google Colab already include tqdm in the environment by default so for many users this will not require any changes, but if you are running small scripts in other environments and want to continue seeing the progress bars you will need to separately install the tqdm package.
  • Removed some previously deprecated and rarely used keyword arguments (config, openapi_config, and index_api) to instead prefer dedicated keyword arguments for individual settings such as api_key, proxy_url, etc. These keyword arguments were primarily aimed at facilitating testing but were never documented for the end-user so we expect few people to be impacted by the change. Having multiple ways of passing in the same configuration values was adding significant amounts of complexity to argument validation, testing, and documentation that wasn't really being repaid by significant ease of use, so we've removed those options.
Assets 2
Loading