GPTs and Assistants for Semantic Kernel

[matthewbolanos]

Share your thoughts about how you'd like to see GPTs integrated into Semantic Kernel.

[iangithub]

Will the AssistantKernel be a feature released with SK v1.0 ?

[joslat]

IMHO it must be released with SK v 1.0 as it is a core feature now...

[matthewbolanos]

We're trying to determine if adding Assistants will require any breaking changes. So far, we don't believe so, so we'll likely ship v1.0 without Assistants and instead provide Assistants support via an experimental branch/package.

[joslat]

Noooooo... (exagerating a bit lol)

IMHO the message that the SK is aligned with OpenAI and the ecosystem is consistent is really big with all your communication so far... This is huge and, to my opinion changes completely the way of working with LLMs.. or providing a big advantage of using GPTs/Assistants versus not using them.

If it is not part of the kernel it's kind of "We provide the functionality but not from the core, but released or in a side assembly" whereas if officially released as 1.0.0 the message is "This is our ecosystem and we officially and straightforward support it from the core"

I like the idea of the AssistantKernel and can't wait to see how this goes forward, I think that the implementation is key and its support of the IPlugin interface makes it super neat and extensible - and that other AssistantKernel can have a set of AssistantKernels to its service to delegate work... that looks pretty mature and to me should go into 1.0.0.

[matthewbolanos]

We definitely want it in the our main nugget packages, so we have a stretch goal of including it in V1.0. If we don’t hit that goal though, we want to at least expose it as an experimental capability ASAP and get it baked into main nuggets by V1.1 as a non breaking change.

[joslat]

Love hearing that, thanks! I appreciate your time in sharing the big picture 👍

[dsanmart]

It would be cool, and fairly simple, to show how you can create your own flexible backend with semantic kernel and make it deployable as a GPT (only having endpoints for the input and output of the kernel).

Having seamless integrations of all the new API functionalities (especially JSON mode) would be nice too. Are these coming soon to Azure?

[joslat]

I'd love to see how the "responsibility division" is set between GPTs and Plugins, like for example I train a GPTs to "properly format and validate JSON" but I have a Plugin with formatting abilities... which ones will be used in the planner? How do I tell that I want this to be done by the GPTs (no need for tokens and execution/validations - it will be done in a single step).
Or if I have two or more GPTs, how are those orchestrated? Like, I have a custom Blog Writer GPTs - aka a Blog writer in my personal style and a GPTs specialized in markdown generation... and then some Plugins and skills... I'd like to see this combination working together for example...

[matthewbolanos]

In our current prototypes, we're leveraging function calling for all plugins and their functions (whether they're assistants or native functions). What this means is that the LLM will be responsible for choosing between the following:

1. Responding back to the user with a message
2. Call a native function with the correct arguments
3. Or ask for help from an agent with the correct ask

We've found this is the most optimal approach because you can have the LLM decide on the next best action in a single LLM call (instead of multiple).

[joslat]

So, the LLM is the "Planner" or acting like it... which makes sense as it is indeed responsible to respond to the "ask" it has.
I miss the "4. Or make a plan by using his skills (GPTs, Assistant, LLM) or a concrete planner skills and then orchestrate the execution of the task between them (execute, recheck, replan if needed...) and responding to the user with the final result."

[matthewbolanos]

A lot of that is now automatically handled by the Assistance API now. Once a thread is invoked/run, the assistant enters a loop where it will make a number of function calls until the request has been satisfied. This could be a single function, or multiple functions.

if you want the Assitant to create more complex plans faster, you could either provide it with the code interpreter tool (which runs Python in a sandbox in the cloud) or a planner (for local function invocation).

[TBag]

Would love to see a GPTs Store ready deployable GPTs chat sample (without RAG) that demonstrates SK:

• consuming a handful of parameters from the chat input
• using a planner and calling two functions (one to generate text context, another to generate an image)
• to generate a combined output of text and an image.

I'm seeing a lot of use cases that fit this pattern very well.

This sample has some of the same concepts found in the 04-DynamicRag v1 sample, but adds image generation and simplifies things a bit by eliminating the coded Plugin (Math.cs).

[matthewbolanos]

We recently added a sample 05 that shows how multi-modal features could work. In the sample, we demonstrate how you could use one model to create an image and another to describe the image that was generated. Let us know what you think of it!

[TBag]

Woah! I cannot wait to check this out. Thanks, Matt!

[TBag]

Already digging in! :) Do you have any instructions that describe what the HuggingFace setup should look like and how to configure things in HuggingFace?

[matthewbolanos]

That's a great request! I'll pull something together tomorrow and get it published :)

[Kee2u]

Id love to see an updated version of Chat Copilot with these features.

[wma4432]

How to allow users to use natural language to ask for business metrics. I want to be able to provide schema about tables and convert requests into sql that can be used to query data tables.

Can it do joins and group bys?

[matthewbolanos]

We've found that the OpenAI LLMs perform best with functions calling, so I think this scenario would be best supported by creating an API or function on top of your business data that the LLM can call with input parameters.

[Hansdlastreet]

Share your GPTs on https://gptspedia.io and apply to be GPT of the week! ❤️

[ewagner70]

Gents, may I propose a 180° u-turn approach. Currently everybody puts Agents into the centre. Honestly, the User (and their proxies) do not and actually shouldn't care about which Agent/subfunction to choose.

So why not not putting the task/query into the centre by turning the "thread" into a "queue" where the original task and its sub-completions are taken by listening Agents and its sub-functions to take over what they think can best achieve/contribute to the results of the main or at least a sub-problem until one Agent comes to the final conclusion or assembles the final answer(s) from the sub-results?

PS: I am by nature python centric (and I presume the majority within the community) ... please put that first as python has the most leverage/scaling factor in the community.

[matthewbolanos]

This is a great proposal! (And very similar to what Autogen does with joint chats). We want to support this type of solution because we believe it’ll be faster for certain classes of problems.

One of the challenges we’re facing is that OpenAI threads get “locked” whenever they’re run by a single agent (it’s also not possible to duplicate threads yet), so we’re trying to figure out the most efficient way of passing a thread to different agents so the “best” agent can contribute to the thread without going through a proxy agent.

[matthewbolanos]

We also want to acknowledge the importance of Python to the broader community. Our current goal is to get our current Python library at parity with the C# version. Once it is, we’d like to use it as the primary library for adopting new AI capabilities since SDKs typically ship with Python support first.

[ewagner70]

thank you for your positive reply ... yes, I know and already use Autogen (actually a great approach!), but these Agents/Functions are also "locked" in the "group chat" and cannot be re-used (from cost/operational perspective a "conditio sine qua non"). I would also appreciate multiple "similar" Agents providing answers to a specific (sub-)query and then the best reply can be chosen (or a even a mix) - similar to the langchain MultiQuery.

as for the solution approach, why not use a streaming approach (f.ex. apache pulsar/kafka) and then create the Agents (incl. functions) as microservice on top of it? the "threads" can be replaced with the respective pulsar/kafka topics and can have more than one "listenening" Agent-Microservice ... what do you think?

[matthewbolanos]

I think what this conversation leads me to believe is that the OpenAI Assistants API likely isn't the end-all-be-all for Assistants/Threads. There's likely even more scalable approaches using things like apache pulsar/kafka. To capture the current momentum of OpenAI Assistants, our first implementation will use their APIs (which have "locked" threads), but I want to make sure our interfaces for things like Threads is generic enough for us to support these more distributed scenarios.

[bashimr]

Great discussion and ideas :)

[Eddie-Hartman]

Personally, when I was getting started I found this course particularly helpful: https://www.deeplearning.ai/short-courses/microsoft-semantic-kernel/
I'd love to see another updated one focusing on assistants and all of the other new functionality.

[radrad]

I would like to find out if I can use semantic kernel with all of its powers (now including assistants) to come up with a DDD (Domain Driven Design) modeling tool using NLP processing capabilities of GPT chat to extract entities, properties and its relationship (ERD cardinalities: NavigationProperties represents 1-n and NavigationConnections represents n-n). You can see how one such entity can be serialized to .json structure: https://github.com/IsmailEsadKilic/Vehman2/blob/main/aspnet-core/.suite/entities/Vehicle.json
This and other .json files in this repository are the output of abp.io suite tool (https://docs.abp.io/en/commercial/latest/abp-suite/generating-crud-page) that can design these entities using the web GUI, but I would like to create it using GPT chat.

And also I would like to visualize these ERD relationships using a visualization javascript library like GoJS while I am discovering these entities and relationships among them. See this page for visualization: https://gojs.net/latest/samples/entityRelationship.html

I would like to say something like this:
Create entities: Category and Product with name and description properties, where description has max length of 250 and name has standard length.
Category has note, isActive, Product has photo string, price, quantity

Category and Product have NavigationConnection relationships.

And somehow the engine should conclude (based on some predefined configuration like standard string length: 100)
what are data types for some fields based on industry standards/naming conversion (quantity is int, price is decimal, isActive is book)

I would like to see how to connect an external dynamic visualization to the current in-memory structure I am developing.

I found that in python there is a way to use Pydantic library, Instructor and Marvin to force GPT engine to provide structured json response.
IT WOULD BE GREAT TO INTRODUCE THESE METHODS IN C# VERSION AND TO ADOPT SOME OF THESE TECHNIQUES.

https://medium.com/@jxnlco/bridging-language-model-with-python-with-instructor-pydantic-and-openais-function-calling-f32fb1cdb401
Pydantic is all you need: Jason Liu
https://www.youtube.com/watch?v=yj-wSRJwrrc&t=2s&ab_channel=AIEngineer

https://www.askmarvin.ai
Marvin lets you define dynamic prompts using code, eliminating the need for cumbersome template management. With this approach, you can easily create reusable and modular prompts, streamlining the development process.

[alessandro-gi]

Hey Matthew, very nice project! Will the assistant package be available for python?

[matthewbolanos]

Yes! @moonbox3 is currently working on the Python implementation and should have something to share soon :)

[igochkov]

Kudos to the Semantic Kernel team! Keep up the great work. I like that the kernel was made service agnostic in v1. Does this apply to the AssistantKernel, too? Is it designed with the same service-agnostic mindset, or does it serve as a specialized layer tailor-made for the OpenAI assistant?

[matthewbolanos]

We're trying to get the patterns right using OpenAI. Afterwards, we'll focus on building a model agnostic version using the same interface.

[RyanWebsterWSS]

I was wondering if the Assistants implementation currently has or will in the future have support for uploading files to assistants and threads (as opposed to templated assistants as discussed in a previous blog post). I could be wrong, but one reason why this would be useful is that when using file's instead of stuffing in the instructions, OpenAI will return annotations which mention which file it pulled its answer from VIA RAG. One of the reasons I wanted to try assistants is to lean on OpenAI's RAG implementation, but citations/annotations are important for grounding and user fact checking. Also, file upload could load non-text filetypes such as PDF's without need for the programmer to pre-parse into text.

[damian-666]

Regarding accommodation for generalized Start menus, completion, desktop OS, file manager, apps, LLM preprocessing for synonym building with offline domain-specific NLP command query, with responsive hot words and fast, context-aware, mission-critical effectiveness with some voice cues and visual discovery and 2-way COMs, and multiple hotwords:

I'm not sure if this is the right place for this topic, and I haven't had time to thoroughly evaluate its semantic kernel, but I want to put it here because you guys are doing something you should know about, one of these use cases that I think is probably very important, can revolutionize the way we command, compete, query our pc apps and OS's. at least for destop.

I asked GPT to clean it up because my hand is half-paralyzed now from computer overuse, but it takes out too much of the points I'm trying to make after working on bloated UI for Autodesk and custom game level design IDEs and visualization tools for audio scientists and physicists for 20 years. I've already gone through the voice access community feedback, but really I would rather just explain it to a senior product designer and hope that there's consensus on this whole thing. Telling you because I would think that semantic kernel might provide a lot of the necessary infrastructure to implement it, or something similar. I think they understand at least some of what I've been trying to push for. I really still be consensus on this and it's not that hard.

So sorry if this is TLDR or off-topic, but I'm starting with just what I think might have to go into the semantic kernel. The rest is just details of the use case that I'm envisioning. You don't have to read all that, it's something that's or if it just pushes, puts too much strain on what you've already done.

-Domain reduction for voice recognition that's low latency offline as in voice assist. This is like taking the really wonderful real-time offline voice access feature that's in Windows 11 Insider and making it context-aware, so that you don't have to say everything exactly and so that you don't have to switch between dictate and command modes because it should know. Also, the ability for hot words to be quickly applied to these completions when they're only partially spoken. So I've spoken to the product design and they put teardrops but for the most part, they put the voice access over the UI. Or keys if they click when you're not clicking anything, you just want to be activating a function in the active application domain.

-Preprocessing by LLMs to make dictionaries of synonyms after it has scraped and go over the object models of the file explorer and all the application UI, settings, value enums, and types, test APIs that are exposed, that can be accessed from out of process or by plugin. You usually use by automating test systems or automation.

-Hooks to allow for the system to have a natural voice response or an audio response and or visual response and a user dossier are sort of something that it's learning about the user's current preferences. This is something that an assistant should be doing if you say "never speak to me out loud or call me this name" or "Don't speak until spoken to until further notice," don't warn me unless I'm doing something that might cause data loss or something that can't be undone, or it should just go ahead and do it.

So one example of what I tried is I talked to the LLM chatGPT4 with Dalle, and OCR, and it said the best it could do is take what the voice access wrong context unaware guesses that I wanted and then take a screenshot of the menu and then tell me what I really wanted. And that worked, but that's slow. So I would love it if there is a way to hook in something that can limit the guesses of voice to text while expanding synonyms. That was pretty much more geared towards whisper, which isn't really a real-time completion or offline DSPs. So I'm not sure which where this should go, but I'm going to run it by you.

-So ideally, precise mission-critical app command and also -QUERY- if I want to say "is HDR on?" I don't even need to go ask Bing where the HDR switches are that I have to click. I don't have hands! It will just tell me yes or no. And if I say "never speak to me again out loud until further notice," it will show me on my projector the setting and bring me there. And then I can say, turn HDR off for example. Another great example is what's my WiFi password we know in windows that you have to take through really old settings that are hard to find and are duplicated and it could just say "let me verify who you are." Get closer to the camera and then tell you.

And when you implement a plug-in, you're actually less safe if you damage the process.. So what I'm proposing is a really generalized assistant that somebody in Microsoft could make, even if it's closed sorce using all the things that you've got. because it's going to as soon as it opens that file with a nice start to say something. It's going to begin to do completion and how it does it now. that if you say instead of a you say alphaGeta Charlie Delta foxtrot and they did start to integrate this NATO bit into the product. Maybe there's several reasons. One disabled people often have served in the Armed Services., and its standard on radio , maritime and internation radio 2 way coms. less likely to be misinterpreted so can be used for actions wiht consequences.

If you want to change the name of an app. It should let me, but I shouldn't have to set all these things up. I should just be telling it. From now on you whenever I say dev studio whenever I say dev studio, I want you to use the 2022 preview version. So after that initial LLM dictionary building, you'll be able to my voice make adjustments to it. I'm probably visualize it. But I can't see end users type into JSON files or coding any of this stuff. And again, these assistants. They don't need to be that many of them. It can't be perfect. There's going to be several, but I could picture a team of like 5 people at Microsoft foot nailing together the best UI. That's right out of Star Trek.

With everything that Microsoft has already spent billions on this would only be a few months of work for maybe five or six very people, 1-2 data scientists and some DSP experts, someone with pretty deep knowledge of windows and automation and testing and interfacing and multiple languages and such, silk, Clang and semantic Kernel.

Well, if I say Windows, Start launch dev studio. It shows me a letter and I say alpha and it launches that then I'm at a menu context. So if I'd say it's only I'll open close exit or whatever that will be fed into the small list to guess from. I guess semantic kernel so that something which isn't an LLM but this is another system can use that again. It's DSP or its voice to text and use that to greatly limit the amount of the space of guessing that it must do. So is that actually could be a lot using a lot fewer cycles and more sensitive to being interrupted by multiple hot word listeners so you can say "no scratch that." If you're saying something that it's being misunderstood as soon as you say something that's relevant, it'll just put up something and then you hit that hot word and then it does something so right then.

And this could be greatly generalized and it doesn't have to be a plugin or in the process. It could be a plugin to semantic kernel, but it could be something that runs separately and only done once. It's general. It works for the OS, and it works for the apps. And it does kind of set a model for every app as it sort of shows that as soon as there's a. That the system has scraped. Did it somehow inserts a microphone. And a one-line command. a short command text. field to existing apps or encourages the app developer to place that in there. But I think it could do it to legacy at a little mic or. it could have that little mic in one place and have it float near the app that you're on right near where the old ribbons file used to be. doesn't have to be done, especially for every app. And in fact, it could be done. for OSX. and Linux. It could pretty much get universal UI for everything. One of the reasons why they had so many plane crashes in World War 2 is that every airplane lever had may and opposite effect, flaps up, down level down, or such, resulting in user error and unrecoverable crash. And with this, you could sit at a friend's Mac and not worry about accidentally with a misplaced drag and drop that I thought would merge, would instead it did a replace.

So the way they've implemented this feature is sad because it's commanding the mouse or click this or that button, we want to talk directly to those applications and the UI is bound to that. The other thing I thought that could be combined into this feature is the great indexing of code feature search that's in developer studio 2000 and two. Combines the discoverability of user interfaces that are visual with the direct access of the command line without having to remember all those switches and such.

Voice-driven hands-free 100% mission critical 99.9% effective context-aware large language model plus domain-specific language scraping. UI Text spying, UI indexing, OCR on some UI to command, this isn't something for disabled people, it's for everybody. And I'm seeing, I've been putting in some feedback, but I thought I would run it by people more technical, so that they can give more thought to some of the possible complexities in it, and provide any hooks or necessary APIs in this kernel if it's relevant.

These aren't separate plugins at all. These would be operating as out-of-process test drivers. I would think it might be more safe than actually having a lot of plugins that go in the process.

The file open, the view, all the windows, the commands, anything that you can do via automation, automated testing, if it has to go through the UI, that's unfortunate, but it isn't a big problem. If there not COM API, C, C Lang, parameter types and with other scraping methods that we see in silk. It's also combining the command and code search that's in Visual Studio 2022 which unfortunately has been split into two separate search buttons And that's when I stopped using it. It only takes 1 extra hunt an peck orclick to make a feature not worth using. Again, it's overly complex and it's something that should be able to figure out. Or if if it's both in the code and in the features, it just should bring them both up and put markers on them and I get to choose. Or if I don't have any text open, obviously it's going to be a code, It's going to be a feature. If it matches some of the features, it's a lot more limited than the domain of all the possible codes they might be seeing.

LLM preprocessing can happen periodically when you first set up the system. This would go over the existing ui. And most people would probably stop using that if it's successful. If it's not, they'll, they'll go back to the way they were. There's no harm in it. The UI. mess, people don't have to care because they're going to access it by saying wahr they see oee want. the initial exposure to the UI might be by. by browsing through it or. saying what you thought you saw before it presented to you and then say a hot word which is placed near it as it's done in voice access with those little teardrops that say 123 or A,B,C,D. So you could say alpha bravo. I think they did add this, but they added it mostly for dictation And you have to say, select this and then start. But I'm not sure sure that they understood what I meant.

Sitting with users , They see a feature and say it. . and if you say send it because you're because you're going to hear a little beep. Beep means it's going to upload data and then you have a sec to say, No, don't. Don't. And if there's something not safe for work, it's going to warn you. Or if it's something that you haven't don't usually do, it might say it.. I'm not so sure that they've added it to the mission critical command execution that I was hoping for. Once the synonym and completion dictionaries are set up. beforehand, synonym dictionary, construction. as a preprocessing.step by an LLM. and then at runtime the existing.,amazing low latency offline voice access feature that appeared in windows Eleven insider builds., will be have a lot less work to do because instead of a lot of different possibilities, it's probably going to have five states on the file menu. It's going to have open, close, exit, and it's going to start showing those indicators as soon as you say anything that sounds like what it already has. So this makes it that low latency even more low latency and the ability for it to make a mistake or to not do anything is almost nil. And you definitely can step all over yourself. You can just keep talking until you see what you want showing the screen right now. You have to stop talking and wait until it like resets or say scratch that or escape or whatever. It's terrible. It's a big mess. It's something that you rage quit.

But really, if it's just partially done, it would be a huge improvement. . Right now I am using the microphone. I'm using an SM58 voice microphone, which is a cardioid dynamic mic that tends to soften on only pick up the voice and it works really well at blocking out everything else. And if you I. either ship a really good mic and headset or tie mic, but something that isn't a condenser because those tend to pick up high frequencies and you apply digital filter. I don't think when you're already applying a convolution filter to do the text. basic DSP level. Stuff intuitively, I think that that wouldn't be a good idea. To have two filters on it.

The use case, the UI is just a Windows logo with a microphone icon next to it, one line. Icon that has. that has some yellow circles that can glow to indicate that it's getting a signal. A cardiod dynamic mic It cuts out a lot of the high frequency filter on you don't have to put any digital voiced bandpass filter processing on it because you're going to be doing that For the voice to text. It would have to multiple hot words,
so that you can interrupt yourself, with offline system. So what the problem is is that If you try voice access, the current design is built over the mouse and keyboard.
And when I'm talking to application or my OS, I don't have a mouse or keyboard. I don't care about that., Although the existing features ares probably useful to I've raised this with the community and such, and they've taken notice, but I don't think they realize how much of a revamp I really intend that I hope to see.

I mean, this is to work 100% or not at all. I do not want to touch or even always look at my computer and I don't see any reason to If it's context aware and I say launch devastudio and then I go to file, there's five things on there. There's no way that it can guess wrong. What I'm trying to say. So I told chat GPT about this problem and it said, well, give me the guess from Microsoft voice access, and then give me a screen capture of the file menu and I'll tell you what you meant. And it did as a Post process, it was correct. I think it should be doing is everything with an llm as a preprocess.

it is as though the The product designer. designer did the entire almost completely implemented it.,as if they didn't know that for every user interface element that there's a property in the object model or the DOM of every app that's accessible via com and Is accessible in English and usually as the user interface directly bound to it and it's typed, used by testing rigs or plugin, and that the is onften bound to it.
right now wiht this assistant I have to change modes from command default to dictate. And that's lame because if I'm not in a text box, I'm not dictating. If I just loaded an app, the menus are in context.

[Ryan-Britton]

I am experimenting with the Experimental Agents package - and can't seem to figure out how to upload a file in a particular user message. In my use case my assistant should read and answer questions on an uploaded file in a user message (not seeded into OpenAI storage for Retrieval, preferably). How would I go about doing that? Even if I do upload the file into OpenAI storage for retrieval - how do I reference the file id in my user message?

basically mimicking this:

[jamesmkfoo23]

The threads must be severely injured (Ref: 3:47 on the video )
:)