Pinecone Integration

[neon-sunset]

I have made a library for working with pinecone.io in C# (with both gRPC and REST transports): https://github.com/neon-sunset/Pinecone.NET

The next step is to provide fully-fledged integration with SemanticKernel. However, I wanted to hear out the suggestions on the best course of action to achieve this.

Considering the existing integrations in SK, would it be appropriate to contribute the integration referencing Pinecone.NET directly to SK or would it be better to publish it as a separate package that implements adapters for embeddings interfaces?

[neon-sunset]

It seems someone is already in progress of a Pinecone contribution in #770
:(

[glahaye]

That's not necessarily a bad thing! Both can work in conjunction and get the best of both worlds!

[neon-sunset]

My plan was to wait for the #770 to get merged and then refactor implementation to switch it to reference the official gRPC .proto manifest and use the generated client. This should reduce the maintenance effort and LOC required for SK to have Pinecone integration, reduce the overhead versus JSON serialization and also benefit from not having "lossy" serialization of double/float types since Protobuf spec uses bit format and doesn't suffer from precision inconsistencies that are a risk when using JSON (see ScoredVector with double Score perfect match in protobuf produces Score == 1D, with REST Score > 1D).

[neon-sunset]

@dluc This is a follow-up to our conversation during office hours. The above comment already has some context but I would like to add some more just to make sure.

As of now, Pinecone.NET is feature-complete as far as parity with the official libraries goes. However, it targets .NET 7+, which is incompatible with NS2.0 target of SK. This is warranted by the intention to adhere to the modern C# experience (null safety, required properties, no reflection emit, concise/lean code style). After all, if someone works with cutting edge technologies, they should use newest .NET 😄

However, this means SK cannot use it directly. Current connector implementations in general could be significantly streamlined by naturally leaning on C# and its corelib, with Pinecone being the first since I could adapt Pinecone.NET to SKs codebase directly.
e.g.: Azure Cognitive Search vs Pinecone, Qdrant Weaviate...

Therefore, I was wondering if rewrites of this kind are accepted. Alternatively, I could look into adjusting Pinecone.NET to support NS2.0, and then referencing it from SemanticKernel's connector, if it's possible without sacrificing performance/features.