Building a voice AI that answers the phone in Bangla

Mahadi Zulfiker — Sun, 20 Sep 2026 07:55:18 GMT

Most "voice AI" demos are a microphone in a browser tab. A phone call is a different problem. The moment you put an AI on a real phone line, three things a demo hides become the entire engineering problem: latency, telephony, and a voice that sounds human in the caller's own language.

I spent a good while on this building a Bangla-speaking call agent at SeloraX. Here is what actually mattered.

The latency budget is smaller than you think

On a real call, people start to feel something is wrong at around 300 milliseconds of dead air. That is your entire round-trip budget: speech in → transcription → model reasoning → speech out → back down the phone line. Every hop spends part of it.

The naive pipeline record a turn, send it up, wait for a full response, synthesize it, play it blows the budget on the first sentence. The fix is to stream every stage and overlap them: begin synthesizing the start of the answer while the model is still generating the end of it.

You cannot bolt this on later. It decides your architecture on day one.

Telephony is the unglamorous hard part

The model is the easy 20%. Bridging a SIP telephony gateway into a realtime audio pipeline is where the real work lives call setup, codecs, jitter, and the fact that phone audio is narrow-band and lossy.

A model that sounds great on studio audio can fall apart on 8 kHz phone-band sound. This surprises people every time.

Why a custom voice

No hosted TTS produced conversational Bangla we were willing to ship. So the voice is a custom-trained model, tuned for phone-band audio specifically.

That is a real tradeoff, and I want to be clear it is not free: a custom voice is far more work and has to be maintained, versus a hosted one you never think about again. But there was no hosted option that made the call feel like a person and on a phone call, the voice is the product. Everything else the caller never sees.

What I'd tell someone starting

Design the whole thing around the 300 ms budget from day one. You cannot bolt streaming on afterwards.
Spend your hardest week on the SIP/telephony bridge, not the prompt.
Test on real phone audio early. Studio audio lies to you.

The full case study, including how this became a multi-tenant platform merchants run themselves, is on my site: mahadizulfiker.dev/work/selorax-voice-ai

I'm Mahadi Zulfiker, a full-stack engineer in Dhaka building voice-AI and commerce systems. More writing at mahadizulfiker.dev/notes.
https://mahadizulfiker.dev/notes/building-a-bangla-voice-agent