[Feature] Phase 3 image/audio enhancements to MultimodalQnA in v1.3

[dmsuehir]

Priority

P2-High

OS type

Ubuntu

Hardware type

Gaudi2 and Xeon

Running nodes

Single Node

Description

This is a follow up to issue #790 - Image and Audio Support for MultimodalityQnA.

It will track the phase 3 development defined in the 24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA RFC, as well as a few other related enhancements.

• Support for spoken responses (GenAIComps core api_protocol for optional audio fields, MMQnA backend, MMQnA UI)
  Added audio dict field to ChatMessage and modalities to ChatCompletionRequest mhbuehler/GenAIComps#24
  Backend Implementation for Text to Speech mhbuehler/GenAIExamples#53
• Support for image ingestion with a speech audio caption (Multimodal redis dataprep and the MMQnA UI)
  Update the redis multimodal dataprep microservice to support ingestion for an image with a spoken audio caption mhbuehler/GenAIComps#25
• Add support for .mp3 files in data ingestion (Multimodal redis dataprep)
  Added .mp3 support to redis multimodal data prep mhbuehler/GenAIComps#23
  Added tests + updated docs for data prep mp3 support mhbuehler/GenAIExamples#51
• UI enhancement to list and clear the contents of the vector store (MMQnA UI)
  MMQnA UI option to show/delete files from vector store mhbuehler/GenAIExamples#52
• UI enhancement to provide the relevant audio clip in the query tab (MMQnA UI)

@mhbuehler @okhleif-IL @HarshaRamayanam @ashahba