feat(plugin): add Twilio plugin for voice and text interactions

[boolkeys]

Relates to:

Closes #1631

Risks

Low - This plugin is self-contained and doesn't modify any existing functionality.

• API keys are required and handled securely through environment variables
• Rate limiting and error handling are implemented for all external services

Background

What does this PR do?

This PR adds a new Twilio plugin that provides voice and text interaction capabilities to Eliza through three main features:

• Speech-to-text transcription using Deepgram
• Text-to-speech generation using ElevenLabs
• SMS messaging using Twilio

What kind of change is this?

Features (non-breaking change which adds functionality)

• Adds new plugin for voice and text interactions
• Implements three new services
• Provides comprehensive testing suite

Documentation changes needed?

My changes do not require a change to the project documentation as this is a new plugin with self-contained documentation in its README.

Testing

Where should a reviewer start?

1. Review the plugin structure in plugins/plugin-twilio/src/
2. Check the service implementations:
   • services/transcription.ts for Deepgram integration
   • services/voice.ts for ElevenLabs integration
   • services/twilio.ts for SMS functionality
3. Review the test suite in test/test.ts

Detailed testing steps

1. Clone the repository
2. Copy .env.example to .env and add your API keys
3. Install dependencies: pnpm install
4. Build the project: pnpm build
5. Run tests: pnpm test

Expected results:

• SMS test should successfully send a message
• Voice test should generate an audio file
• All error handling cases should be covered

Screenshots

Test Output

Deploy Notes

This plugin requires the following environment variables to be set:

TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_PHONE_NUMBER=
ELEVENLABS_API_KEY=
DEEPGRAM_API_KEY=

Discord username

.boolkeys

[github-actions]

Hi @boolkeys! Welcome to the ai16z community. Thanks for submitting your first pull request; your efforts are helping us accelerate towards AGI. We'll review it shortly. You are now a ai16z contributor!

[xR0am]

Was halfway through this! Great job @boolkeys

[xR0am]

@boolkeys you need to ask to merge into develop, not main

[boolkeys]

@boolkeys you need to ask to merge into develop, not main

Ok thanks, currently recording video demo, I will ask for another merge into develop

[xR0am]

@monilpat not sure this actually relates to Coinbase commerce you referenced ?

[Freytes]

1. Can you show a demo of this functionality working outside of your test?
   a. It seems that the plugin cannot be used in a character file, or engaging with the agent at all.

   @boolkeys

[boolkeys]

@Freytes working on it. shipping soon. some code modified. thanks for your patience.

[Freytes]

@Freytes working on it. shipping soon. some code modified. thanks for your patience.

Thanks I appreciate it. Just remember the core workflow should be.

User calls or texts Twilio number -> agent responses via text or voice.

[boolkeys]

Ok @Freytes for the core workflow.

quick update:

1. Out-of-the-box:

• Two-way SMS Conversations: User texts Twilio number -> agent responses via text.
• Voice Call Capabilities: User calls Twilio number -> agent responses via voice (Deepgram + ElevenLabs)

2. In Discord:

• Voice Channel Integration: Bot joins voice channels on request

3. In Twitter Spaces:

• SMS Notifications: Users subscribe to specific speakers > Receive text when speaker starts a Space
• Text-to-join: Reply to notification with "JOIN" command > Participate through voice or SMS

WIP

• Twilio phone numbers must be approved and A2P Campaign registration is in progress...

[Freytes]

Ok @Freytes for the core workflow.

quick update:

1. Out-of-the-box:

• Two-way SMS Conversations: User texts Twilio number -> agent responses via text.

• Voice Call Capabilities: User calls Twilio number -> agent responses via voice (Deepgram + ElevenLabs)

2. In Discord:

• Voice Channel Integration: Bot joins voice channels on request

3. In Twitter Spaces:

• SMS Notifications: Users subscribe to specific speakers > Receive text when speaker starts a Space

• Text-to-join: Reply to notification with "JOIN" command > Participate through voice or SMS

WIP

• Twilio phone numbers must be approved and A2P Campaign registration is in progress...

Was there a recent release that makes this change? Also do you have a demo

[Freytes]

I would also say release what you have as the workflow meets the bounty specifications if tested

[boolkeys]

Closing this PR as it was targeted to the wrong branch. Will submit a new PR targeting develop branch