Proposal Idea for GSOC 2025

[hxrshxz]

Project Title: Voice-Controlled JupyterLab Interface

What is the issue your project idea addresses?

JupyterLab is a versatile platform but relies on traditional input methods like keyboard and mouse. For users with physical disabilities, multitaskers, or anyone dealing with complex workflows, this can be inefficient. This project aims to integrate voice control into JupyterLab, allowing users to interact with their workspace entirely through voice commands 🎙️.
Modern apps like VS code's ( voice code),apple's X code, IntelliJ IDEA - JetBrains's voice bot etc all use voice recognition to make their platforms more accessible and efficient. These apps have transformed user experiences by enabling hands-free control, making multitasking easier and providing a more inclusive interaction model. Similarly, integrating voice control into JupyterLab will open up new possibilities for users,It will improve accessibility and productivity by enabling tasks like running code cells, navigating between notebooks, and managing files 📑. The feature will be optional, so users can toggle it on and off as needed .

2, What I will be building

Key Features:

Voice Recognition Integration :

• Integrate Web Speech API (or alternatives like Mozilla DeepSpeech or Google Cloud Speech-to-Text) to convert speech into text in real-time .
• Voice commands will trigger actions like "run cell," "insert new cell," "next cell," etc., allowing users to navigate and interact through voice .
• Customizable Voice Commands 🛠️:
• Users will define, modify, and delete their voice commands based on preferences (e.g., “run all cells,” “save notebook,” “insert markdown cell”)
• Toggle feature to turn voice control on/off without interfering with traditional input methods .
• Real-Time Feedback for Voice Commands :
• Implement visual feedback (e.g., floating notification “Command: Run cell”) and voice feedback (e.g., “Command executed: Cell run successfully”) to keep users informed .
• Error Handling and Command Suggestions ❌💡:
• When commands aren't recognized, provide feedback like “I didn’t catch that” with suggested commands .
• Accessibility Mode :
• For users with visual impairments, provide larger font sizes and highlighted voice commands for better visibility .
• Offline Mode Support :
• Ensure voice recognition works offline, maintaining privacy and functionality.
• Voice Command History and Recall :
• Add command history and recall features for easy access to previously used commands.
• Additional Features (Optional) 🎁:
• Support multiple user profiles for voice commands, enabling personalization .
• Voice-controlled file management (e.g., open, rename, create new folder) 📂.
• Multilingual support for broader usability .

How will this benefit JupyterLab and its community?

Improved Accessibility : Makes JupyterLab more accessible, particularly for users with physical disabilities, allowing voice-based interaction .
Productivity 🚀: Speeds up workflows, especially for multitaskers, by executing tasks without manual input ⏩.
Attracting New Users 🌟: Expands JupyterLab’s appeal to those who prefer hands-free operation 🤖, differentiating it from other environments.

How am I going to work?

• Speech-to-Text Integration : I will integrate and customize speech recognition using APIs like Web Speech API or alternatives.
• Voice Command Mapping : I’ll map voice commands to JupyterLab actions (e.g., run cells, navigate notebooks).
• Fine-tuning 🎯: Ensure accuracy and responsiveness for better voice recognition.
• Familiar with JupyterLab Frontend : My knowledge of typescript /React and APIs will help implement this integration.

Estimate of hours for project completion

Estimated Total Hours ⏳: 250-300 hours
Breakdown:

• Research & Planning : (20-30 hours): Research speech recognition APIs and define the project’s features.
• Speech Recognition Integration (40-60 hours): Implement core voice recognition features and set up real-time transcription.
• Voice Command Mapping & Customization (50-70 hours): Map basic commands and allow customization for user preferences.
• UI/UX Design & Real-Time Feedback (40-50 hours): Implement visual and voice feedback for accessibility.
• Advanced Features & Error Handling (40-50 hours): Support natural language and intelligent error handling for commands.
• Documentation & Testing (30-40 hours): Document the installation, usage, and troubleshooting, and test for bugs.
• Optimization & Feedback (20-30 hours): Refine voice recognition and optimize performance based on feedback.
• I will be enhacing and maintaining the project even after it's done

[krassowski]

I guess it would be good if the proposal addressed how it fits into existing voice control extensions for JupyterLab:

• https://github.com/krassowski/jupyterlab-voice-control
• https://github.com/JovanVeljanoski/jupyter-voicepilot
• https://github.com/MooersLab/jupyter-voice-in (https://proceedings.scipy.org/articles/MCYV2126)

and if you would like to contribute to any of those or create a custom extension.

Of note, I am not sure if it was confirmed whether JupyterLab will participate in GSoC this year.

[hxrshxz]

okay if the list is announced and JupyterLab is listed i will start working on this project