Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(plugin): add Twilio plugin for voice and text interactions #1665

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

boolkeys
Copy link

@boolkeys boolkeys commented Jan 1, 2025

Relates to:

Closes #1631

Risks

Low - This plugin is self-contained and doesn't modify any existing functionality.

  • API keys are required and handled securely through environment variables
  • Rate limiting and error handling are implemented for all external services

Background

What does this PR do?

This PR adds a new Twilio plugin that provides voice and text interaction capabilities to Eliza through three main features:

  • Speech-to-text transcription using Deepgram
  • Text-to-speech generation using ElevenLabs
  • SMS messaging using Twilio

What kind of change is this?

Features (non-breaking change which adds functionality)

  • Adds new plugin for voice and text interactions
  • Implements three new services
  • Provides comprehensive testing suite

Documentation changes needed?

My changes do not require a change to the project documentation as this is a new plugin with self-contained documentation in its README.

Testing

Where should a reviewer start?

  1. Review the plugin structure in plugins/plugin-twilio/src/
  2. Check the service implementations:
    • services/transcription.ts for Deepgram integration
    • services/voice.ts for ElevenLabs integration
    • services/twilio.ts for SMS functionality
  3. Review the test suite in test/test.ts

Detailed testing steps

  1. Clone the repository
  2. Copy .env.example to .env and add your API keys
  3. Install dependencies: pnpm install
  4. Build the project: pnpm build
  5. Run tests: pnpm test

Expected results:

  • SMS test should successfully send a message
  • Voice test should generate an audio file
  • All error handling cases should be covered

Screenshots

Test Output

250101_plugin-twilio_test-output

Deploy Notes

This plugin requires the following environment variables to be set:

TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_PHONE_NUMBER=
ELEVENLABS_API_KEY=
DEEPGRAM_API_KEY=

Discord username

.boolkeys

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @boolkeys! Welcome to the ai16z community. Thanks for submitting your first pull request; your efforts are helping us accelerate towards AGI. We'll review it shortly. You are now a ai16z contributor!

@xR0am
Copy link

xR0am commented Jan 2, 2025

Was halfway through this! Great job @boolkeys

@xR0am
Copy link

xR0am commented Jan 2, 2025

@boolkeys you need to ask to merge into develop, not main

@boolkeys
Copy link
Author

boolkeys commented Jan 2, 2025

@boolkeys you need to ask to merge into develop, not main

Ok thanks, currently recording video demo, I will ask for another merge into develop

@xR0am
Copy link

xR0am commented Jan 3, 2025

@monilpat not sure this actually relates to Coinbase commerce you referenced ?

@lalalune lalalune changed the base branch from main to develop January 4, 2025 13:34
@Freytes
Copy link
Contributor

Freytes commented Jan 4, 2025

  1. Can you show a demo of this functionality working outside of your test?
    a. It seems that the plugin cannot be used in a character file, or engaging with the agent at all.

    @boolkeys

@boolkeys
Copy link
Author

boolkeys commented Jan 4, 2025

@Freytes working on it. shipping soon. some code modified. thanks for your patience.

@Freytes
Copy link
Contributor

Freytes commented Jan 5, 2025

@Freytes working on it. shipping soon. some code modified. thanks for your patience.

Thanks I appreciate it. Just remember the core workflow should be.

User calls or texts Twilio number -> agent responses via text or voice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Twilio voice / text integration [BOUNTY - $5k ai16z]
3 participants