[Platform] Introduce Speech support

[Guikingone]

Q	A
Bug fix?	no
New feature?	yes
Docs?	yes
Issues	--
License	MIT

Summary:

• Support for stt / tts and sts
• Interfaces for SpeechProviders and SpeechListeners
• Introduction of a SpeechConfiguration
• New configuration block for configuring providers (requires a platform each time)
• Introduce SpeechBag and SpeechAwareTrait

[OskarStark]

To me we maybe should introduce capabilities also to platforms rather than having a voice component. As far as I understand I cannot use the Voice component standalone, right?

I don't think a dedicated component is the way to go here

[Guikingone]

We can introduce it via the Platform, could be easier, the voice can be used without agents but it will requires the Platform at least.

Will update the PR to match this approach 👍🏻

[OskarStark]

I agree, Agent scope is not needed 👍🏻

[chr-hertel]

Hi @Guikingone, i agree that week lack some kind of guidance on how voices work - but same goes for other binary stuff like creating images or videos.

so two things i would like to understand

• what's the high-level goal here - like what do you want to build?
• why is it an extra component and not part of Platform?

btw, "speech" is more common than "vioce" isn't it?
btw2, have you seen the demo around audio and video?

[Guikingone]

what's the high-level goal here - like what do you want to build?

The main goal is to add the capacity to have an agent/platform that can "listen" and answer to inputs thanks to voice / speech (voice is used as a sugar here, could be renamed to speech), creating a workflow where you can submit voice, call the platform that transforms it to speech / text (depending on the situation you're in) and returning it to the user without frictions.

why is it an extra component and not part of Platform?

It is now part of Platform, I just pushed an update on it following the comment from @OskarStark.

btw, "speech" is more common than "voice" isn't it?

Agreed, could be renamed to Speech.

btw2, have you seen the demo around audio and video?

Yes, the goal is to ease it with a "built-in" approach / API that stays transparent for the user.

[chr-hertel]

just realized we should the "audio" demo to "speech" as well - and i'm def not really happy with that solution there.

can we make it as easy as the structured output - like with an listener?

i like that starting point:

$result = $platform->invoke('eleven_multilingual_v2', new Text('Hello world'), [
    'voice' => 'Dslrhjl3ZpzrctukrQSN', // Brad (https://elevenlabs.io/app/voice-library?voiceId=Dslrhjl3ZpzrctukrQSN)
]);

echo $result->asVoice();

what would be the return type here? would it be same as asBinary() or asDataUri()

[Guikingone]

can we make it as easy as the structured output - like with an listener?

Could be something to explore, the API is not locked for now.

what would be the return type here? would it be same as asBinary() or asDataUri()

My first approach was to do the same thing as asBinary to ease the usage.

[Guikingone]

Well, might seems weird but here we go, stt, tts and sts are working like a charm ... 👀

[chr-hertel]

Haven't made my mind up here yet - it's not clicking with me - currently that change spreads across the entire platform.

is it an option to have it as subscriber for the platform, like STT on invoke and TTS on result? instead of extending the platform interface?

[Guikingone]

Haven't made my mind up here yet - it's not clicking with me - currently that change spreads across the entire platform.

I have a solution for this point, thought about this and I have a solution, it would remove adding SpeechConfiguration in every platform.

is it an option to have it as subscriber for the platform, like STT on invoke and TTS on result? instead of extending the platform interface?

If it's related to previous point, let me push a refactoring and we can push further on this one 🙂

[Guikingone]

Ok, here's the refactoring:

• A new SpeechAwarePlatform is introduced, the goal is to decorate the "platforms" that supports STT / TTS. In the current state, ElevenLabs is the only one implemented and tested, we expose a getSpeechConfiguration method that returns the current configuration for STT / TTS, no more methods in existing platforms, extra arguments, nothing, just a decorator.

• A new SpeechAwarePlatformInterface is introduced to enforce the usage of the method and provide a cleaner API for "speech-supporting" platforms and non-supporting ones, if your platform is not decorated, well, you're not supporting it.

• This refactoring allows to keep the current API "as it" and introduce the speech support without breaking the public API, plus, it's only enabled if one platform that supports STT / TTS is configured, no extra services if not needed.

• This also improve the current API for providers / listeners plus, it reduce the current usage of providers/listeners to Invocation / Result as you mentioned @chr-hertel.

PS: Not to mention that it reduce the usages of null in methods 😅