ML based skills

[kha84]

Hey guys,

I'm truly thrilled to see what kind of crazy things you can now do with publicly available ML models. Especially in the light of some great advancements on the field of transformers like Llama and Alpaca, and some good reports on how lightweight they are, so you can self-host all that on the same device, where your voice assistant will be living.

I was doing some amateurish experiments at home and probably will have something usable at the end, but these are some ideas of what kind of skills I want a voice assistant to have:

hey "assistant" go scrap top N pages from "search_engine_name" on the topic of "keywords_to_search". Read them all, and summarize it for me.

Some combination of beautifulsoup for web scrapping + NLP models to summarize texts (BERT / BART / T5 / PEGASUS / ASGARD / Alpaca whatever)

hey "assistant_name", based on what you have just read, tell me, "some_question_about_the_text"?

Fine-tune (Alpaca?) with the given texts and let it answer that question.
Or find some question-answering models that can be fed with texts and getting your question answered. I've seen quite many of such models on HuggingFace, just need to spend some time to benchmark them properly

hey "assistant_name", what AI thinks of "any_random_question"?

Feed that to Alpaca or any other Instruct GPT-like model that will appear on public, available for self-hosting.

Some thoughts about the hardware

I'm not sure if less powerful ARM SBCs like Raspberry Pi 4B or similar will do any good job here. Need to be honest to ourselves: they're already outdated in terms of performance but their pricetag is high above the roof. I'm now more aimed to use Intel-based Mini PC devices, our dearest Chinese friends (Minisforum, Beelink, etc) are making like crazy. They are super affordable (within $150 for a decent CPU + 16 Gb RAM + 512 Gb SSD), they are silent, very upgradable and also energy efficient. I think they are much better candidates to be a heart of your voice assistant, if you need it to do something more advanced, than controlling your climate or electrical appliance via HomeAssistant or playing music. Probably some more powerful ARM chips like RK3588 can be somewhat on par with Intel N5095/5105/N95/N105 but let's see.

[ChanceNCounter]

I have several hundred words to say on the topic, but I don’t want to leave you hanging until I’m done with an indeterminately long morning, so I’ll drop the main points up front, food for thought.

First, hardware. You’re spot on. These things won’t run locally on a lightweight device anytime soon. They also require AVX, which doesn’t sound like a big ask for “real” computers, but it means repurposing old hardware for these purposes is simply impossible.

That means, for any deployment other than local on a beefy machine, we’ll need to serve this stuff. And that’s okay, because there’s going to be a huge “distributed assistant” initiative, reviving old projects and adding some new ones.

But a lot of other things need to happen first. We need to finish incorporating, rebrand the Assistant, and put a bow on it before we make a large addition to the framework. We need to establish the long-term design and dev processes for the Assistant itself, which probably doesn’t sound like much, but we’re still trying to work out how much of the former Mycroft we’re about to inherit.

In short,

1. Please don’t hurt yourself designing API, as the API we go with will be the result of a very public design phase in which you’ll be welcome to take part
2. Someday™️

[kha84]

Thanks for coming!

All your points are fair, I ain't gonna doubt that. It was more like me thinking out loud and sharing my own thoughts and ideas. So far I'm quite comfortable to play with all that stuff myself. It will take me some time, before I have some ready-to-use prototype at hands, so I could only then to think how to wrap it to be some kind of a plugin to OVOS. And chances are great, someone much more smarter and quicker than me, will do it first - so I'll be only glad.

I played with few projects (https://github.com/antimatter15/alpaca.cpp and https://github.com/ggerganov/llama.cpp). Well, they do run quite nicely on that my cheap mini pc, powered by N5095. But they're lots of "buts" there. I guess these days and probably next following weeks will be the hottest for dev teams of those two projects: I can see a lot of parts moving there and a lot of breaking changes. I'll just sit and wait to see when it stabilizes.

[ChanceNCounter]

At minimum, when my insane day calms, we’ll get you some tl;drs on the dormant projects in question. If I linked, you’d just see a confusing pile of stale code.

[ChanceNCounter]

Well, my insane day never did end. Sorry about that. Tl;dr on the dormant projects:

HiveMind is distributed Mycroft. Our earliest initiatives will concern IoT and a unified Assistant across all your devices, which are actually one and the same initiative. In the process, we'll introduce a compatibility and interchange layer for the ages (if that goes forward, in a few months, it'll be my magnum opus.)