LastWorker: an AI agent that answers your phone

The problem: the phone rings and nobody picks up

Every small business has the same leak. The phone rings at 7:40pm, after the last person has gone home. Or it rings at 12:15pm when both people on shift are already with customers. Or a message lands in the website chat widget on a Saturday, or a text comes in from a number nobody recognizes, or an email asking "are you open tomorrow?" sits unread until Monday.

None of these look like emergencies. Collectively they are the single largest source of lost revenue for a small operator. A missed call from a new customer is not deferred revenue — it is revenue that goes to whoever picks up next. The caller does not leave a voicemail. They call the next result.

The conventional fixes all fail in the same direction. An answering service is expensive per call and generic in tone. A voicemail box is a polite way of saying "we will lose this one." An IVR phone tree makes the caller do the work of routing themselves, and most of them hang up before the second menu. A chatbot on the website answers the three questions in its FAQ list and stonewalls everything else. And hiring a person to cover evenings and weekends costs more than the calls are worth for most businesses at this size.

LastWorker exists for that gap: the volume of inbound contact that is too valuable to drop and too small to justify a headcount.

What LastWorker actually does

LastWorker is an AI agent that answers the phone, replies to chats, texts customers back, and handles email — 24/7, across 97 languages.

The important word in that sentence is "and." Most tools in this space pick a single channel. A voice AI product answers calls and knows nothing about the email thread the same customer started yesterday. A chat widget lives on the website and has no idea a text message arrived. Businesses do not experience inbound contact as four separate systems. The customer who called at 7pm is the customer who texted at 8pm because nobody called back, and an agent that treats those as two unrelated conversations produces exactly the experience the business was trying to avoid.

Practically, that means LastWorker is a single agent with a single view of the business — what it does, hours, services, what it can promise and what it must not — reached through four different transports. Phone is the hardest of the four by a wide margin, so most of the engineering effort went there.

The voice pipeline

A voice agent is a loop, and every stage of the loop is a place where the illusion breaks.

Telephony. The call arrives over the phone network and has to be terminated somewhere that can stream audio into software in real time rather than handing over a recording afterward. This is the CPaaS layer — Twilio in the stack behind LastWorker — and it is responsible for the call leg, the media stream, and the primitives that matter later: transfer, hold, hangup, DTMF.

Speech to text. The caller's audio is transcribed continuously, not in one shot at the end of a turn. Streaming transcription with partial results is what makes it possible to know a sentence is ending before it has ended, which is the only way to start thinking before the caller stops talking.

Reasoning. The transcript goes to an LLM — Claude and GPT are the models in the stack — with the business's context, the conversation so far, and a set of tools it is allowed to call. LangChain handles the orchestration around that: prompt assembly, tool definitions, the state of a multi-turn conversation. This is where the agent decides whether it is answering a question, taking a booking, capturing a lead, or getting out of the way.

Text to speech. The response is synthesized back to audio and streamed onto the call, ideally beginning before the full sentence has been generated.

Written out, this looks like four boxes and three arrows. The difficulty is that all four have to run concurrently and speculatively, because run sequentially they add up to a pause long enough that the caller says "hello?"

Latency is the whole product

Human conversation has a turn-taking rhythm measured in a few hundred milliseconds. Go much past that and the caller does not think "this system is slow." They think "the line dropped," or "this is a robot," and both thoughts end the call.

So the engineering target is not "fast." It is a budget, allocated per stage, where every stage's share is defended:

• Getting audio in and out over the telephony leg is fixed cost you mostly cannot optimize, only choose well.
• Transcription must be streaming and must emit usable partials, so the agent is not waiting on a final transcript.
• The model must start streaming its first tokens quickly — time-to-first-token matters far more here than total generation speed, because speech synthesis can begin on the first clause.
• Speech synthesis must stream its first audio chunk rather than rendering the whole utterance.

Chained naively, those stages serialize into a gap the caller can hear. Overlapped — transcribing while the caller is still speaking, generating while transcription finalizes, synthesizing the first clause while the rest is still being generated — the same work fits inside a gap that reads as a normal conversational pause.

Two related problems fall out of this. The first is endpointing: deciding when the caller has actually finished, as opposed to pausing mid-thought. Cut in too early and you interrupt them; wait too long and you feel slow. The second is barge-in: when the caller starts talking over the agent, playback has to stop immediately and the partially-spoken response has to be discarded from the conversation state, because the caller heard only the part that played.

The specific latency figures LastWorker hits in production are not published here on purpose — see the fact-check note at the top of this page.

Knowing when to hand off

The most important thing a voice agent does is recognize the calls it should not be handling.

An agent that tries to resolve everything will eventually make a promise the business cannot keep — a price it invented, a date it cannot honor, an answer to a question that legally requires a person. The failure mode is not an unanswered call; it is an answered call with a wrong answer, which is strictly worse, because the business does not find out until the customer arrives expecting something.

So handoff is designed as a first-class outcome, not an error path. That means: explicit escalation triggers rather than a general instruction to be helpful; a bounded set of things the agent is allowed to commit to; escalation when the caller asks for a person, when the caller is upset, or when the agent's own confidence in the topic is low; and, when there is nobody to transfer to at 2am, a clean capture of the request with everything the business needs to call back — rather than a fabricated resolution.

The handoff also has to carry context. Transferring a caller to a human who then asks them to repeat everything undoes the value of having answered at all.

97 languages, and what that actually costs

Multi-language support sounds like a checkbox and is not one. Detecting the language from the first utterance rather than making the caller pick from a menu, holding it stable when a caller code-switches mid-sentence, choosing a voice that sounds native rather than an accented default, and keeping latency inside budget when the model is reasoning in one language and speaking in another — each of those is its own problem.

The payoff is disproportionate for exactly the businesses LastWorker serves. A local business whose customers are not all fluent in one language has, until now, had no affordable option at all. There was no "hire a bilingual receptionist for the evening shift" tier.

What building it taught me

Voice is a real-time systems problem wearing an AI costume. The interesting work is scheduling, streaming, and buffering. The model is one component in a pipeline where every component is on the critical path, and treating it as "an LLM app" produces something that demos beautifully and fails on a real call.

Scope discipline beats capability. A narrow agent that reliably does six things and escalates everything else is more valuable than a broad one that attempts thirty and is wrong on four. Businesses do not want an agent that is impressive. They want one they do not have to check up on.

The transport layer is a real architectural decision. Voice quality, media streaming, transfer behavior, and per-minute cost are determined by choices made at the telephony layer, not the model layer — which is why choosing between voice CPaaS providers is worth doing carefully up front.

Perceived latency is not measured latency. A well-placed acknowledgement, a natural filler, or simply starting to speak on the first clause changes how fast the same pipeline feels. Half of the latency work is engineering; the other half is conversational design.

If you are building something in this shape, the AI agents and automations work is the same plumbing applied to a different problem, and the voice agent latency guide goes deeper on the pipeline specifically.