Project Deep Dive

Caribbean Spanish Speech-to-Speech Pipeline

An end-to-end, real-time speech-to-speech system leveraging WebRTC and fine-tuned LLM/TTS architectures to train healthcare students in colloquial Puerto Rican Spanish.

Speech-to-Speech Coqui XTTS-v2 Gemma Optimization LiveKit / WebRTC OpenStack Cloud
View Live Beta

Note: The live deployment utilizes our active, fine-tuned Coqui XTTS-v2 pipeline via a quantized inference setup. The companion Gemma LLM backbone is currently running on a vanilla baseline while our customized, dialect-tuned reasoning layers undergo strict sociolinguistic evaluation to prevent cultural misrepresentation.

The deployment is restricted to authorized users. Please email me for access credentials.

The Problem: The Clinical Dialect Gap

In New York, Caribbean Spanish, specifically the colloquial Puerto Rican dialect, is heavily spoken. This vernacular differs significantly in phonology, lexicon, and prosody from the Standard Spanish traditionally taught in academic settings.

For nursing and pharmacy students, this linguistic mismatch creates severe communication barriers during clinical intakes, often leading to medical misinterpretations. Because mainstream Voice AI technologies focus almost exclusively on high-resource, standardized variants, a specialized pipeline was required to help students practice with these rich regional variations.

The Solution: Real-Time S2S Architecture

We built a low-resource dialect adaptation framework that nudges established base models toward colloquial variations. This avoids the unsustainable compute constraints of training foundation models from scratch.

Acoustic Target Alignment (TTS)

Utilized audio corpora from the University of Puerto Rico. Built a data staging pipeline performing normalization, diarization, and forced token-to-audio alignment to clean the transcripts. Fine-tuned Coqui XTTS-v2 on this data, blending it with high-quality general corpora to preserve voice stability and eliminate voice synthesis hallucinations.

Dialectal Inversion (LLM)

To generate colloquial responses from scarce text data, we reversed the traditional translation vector. We took the limited colloquial data available and converted it back to Standard Spanish using commercial LLMs. Training the Gemma backbone on this inverse mapping yielded vastly superior contextual generation.

Model Output Comparison

Listen to the difference between the baseline model and our fine-tuned version. While standard models produce clearly spoken, textbook Spanish, they often don't reflect real-world patient interactions. We specifically trained our model to capture the natural speed and everyday pronunciation of colloquial Puerto Rican Spanish, helping students get comfortable with the authentic way their future patients will actually speak.

"Chacho, si tú supieras lo que pasó ayer en la fiesta, te caes pa' atrás."

Base Model
Fine-Tuned Model

"Yo te lo dije, pero tú nunca haces caso, siempre estás en la tuya."

Base Model
Fine-Tuned Model

"Nos fuimos a chinchorrear el domingo por la montaña y la pasamos súper nítido."

Base Model
Fine-Tuned Model

"Estaba lloviendo a cántaros y tuvimos que meternos pa' adentro corriendo."

Base Model
Fine-Tuned Model

"¿Tú estás seguro de eso? Porque a mí me dijeron otra cosa totalmente distinta."

Base Model
Fine-Tuned Model

Technical Deep Dive & My Role

As the lead engineer on this project, I architected and deployed the entire system. My concrete core contributions include:

  • End-to-End Data Staging: Developed modular Python pipelines for raw data ingestion, executing speaker diarization, chunking, and forced transcript alignment to maximize the value of highly constrained regional datasets.
  • Iterative Post-Training: Conducted rounds of post-training across the Gemma and XTTS blocks. To guarantee clinical relevance and phonetic validity, I co-designed human-in-the-loop evaluation protocols alongside linguistics PhD researchers.
  • LiveKit & WebRTC Infrastructure: Shifted the pipeline from a batch request pattern to a streaming architecture. Deployed LiveKit on private OpenStack cloud instances to stream the unified ASR → LLM → TTS pipeline directly through authenticated WebRTC sessions.
  • Cross-Functional Leadership: Acted as the primary technical bridge, partnering with clinical faculty to translate ambiguous educational simulator requirements into scalable backend API constraints.

Future Roadmap & Next Steps

Latency Optimization

Transitioning the generation layers to fully streaming chunk-by-chunk decoding to reduce Time-to-First-Phoneme (TTFP) under tight WebRTC window limitations.

Conversational Quality

Developing conversational data augmentation strategies to model natural turn-taking behavior, regional speech pacing, and authentic dialectal fillers to maximize dialog flow.

Clinical Impact Study

Co-designing a formal study to measure student fluency gains, tracking conversational confidence and error rates among nursing and pharmacy cohorts.