AI agents that actually do the work

The distinction that matters: answering vs. doing

Most things sold as "AI agents" are answer machines. You give them a document, they retrieve from it, and they reply. That is a useful product — it is just not an agent. It has no ability to change anything about the world it is describing.

An agent takes actions. It books the appointment into the actual calendar and holds the slot. It transfers the caller to the right human with the context already attached. It updates the record in the CRM so the next person sees what happened. It sends the confirmation, cancels the old booking, checks whether the part is in stock, opens the ticket.

The difference is not a matter of degree. An answer machine that gets something wrong produces a bad sentence. An agent that gets something wrong produces a double-booked Saturday, a refund that shouldn't have been issued, or a customer record overwritten with the wrong phone number. That asymmetry drives every design decision below: what the agent is allowed to touch, what it must confirm before touching, what it refuses and escalates, and how you find out afterward what it did.

So the work is only half model work. The other half is the plumbing — the tool definitions, the permission boundaries, the idempotency on every write, the audit trail, the fallback path when a downstream API is slow or down. That plumbing is the part that determines whether an agent is something you deploy or something you demo.

What I build

Voice agents. Inbound and outbound phone. The agent answers, understands, decides, acts, and where necessary hands the call to a person without making the caller start over.

Chat agents. Website widgets and in-app assistants that are wired into your systems, not just your help centre.

Email and SMS agents. The asynchronous channels, where the constraint is not speed but thread state — knowing what has already been said, what is still outstanding, and when to stop replying.

Internal automations. The unglamorous version: an agent that reads the inbound form, checks three systems, and either resolves it or routes it, so nobody's morning is spent copying fields between tabs.

The stack

Claude and GPT for reasoning, LangChain for orchestration, Twilio for telephony.

Two models rather than one because they are not interchangeable in practice. Their failure modes differ, their instruction-following differs under pressure, and their latency and cost profiles differ per task. Picking per workload — and being able to move a workload when a new version changes the arithmetic — is worth more than standardising on a favourite.

LangChain handles the orchestration layer: tool definitions, the loop between reasoning and action, and the memory of what has happened in this conversation. Twilio handles the telephony: the call leg, the media stream, transfers.

None of that is exotic. The stack is deliberately boring, because the interesting risk in an agent is never "did we pick the right framework" — it is "what happens on the call where the customer's phone number has a typo and the calendar API times out."

Voice, chat, and email are three different engineering problems

They look like one problem with three skins. They are not.

Voice is a real-time systems problem. Every stage is on the critical path: telephony leg, speech-to-text, model reasoning, text-to-speech, audio back down the line. Silence is a failure state — a pause the user reads as "it hung up" or "it didn't understand." That forces streaming everywhere: partial transcripts before the utterance ends, tokens streamed into speech synthesis before the sentence is finished. It also forces conversational design work that has nothing to do with models — handling interruptions, acknowledging before thinking, keeping turns short. The voice agent latency guide goes through where the milliseconds actually go.

Chat is a context problem. Latency budget is generous by comparison, so the difficulty moves elsewhere: the user can paste a wall of text, switch topics mid-thread, come back four days later, and expect continuity. Chat is also where users test boundaries hardest, because typing is cheap. The hard parts are retrieval quality and knowing when to stop guessing and get a person.

Email is a state problem. There is no turn-taking discipline. Threads fork, people reply to the wrong message, someone else gets CC'd halfway through, and a reply arrives ten days late. The agent needs a durable notion of what this thread is for and what is still outstanding, or it produces the worst failure mode in the category: a polite, confident loop that emails a customer forever.

Building all three well means treating them as three systems that happen to share one brain — one definition of what the business does, what it may promise, and what it must never say — rather than one system with three front doors.

LastWorker, as proof

LastWorker is the flagship version of this work: an AI agent that answers the phone, replies to chats, texts customers back, and handles email — 24/7, across 97 languages. It is the same plumbing described above, built to production standards and running against real inbound traffic rather than a demo script.

It is the reference I point at because it forced every hard part at once — real-time voice, multi-channel state, escalation to humans, and language coverage — instead of letting me pick the comfortable subset. The full write-up covers the architecture, the latency work, and the handoff design.

The honest limits

Anyone selling agents without this section is selling you a demo.

Latency is not solved. A response that lands inside about a second feels conversational; past roughly two seconds it reads as a system, not a person. Getting there means real tradeoffs — a faster model is sometimes a slightly less capable one, and shorter agent turns are sometimes less thorough. You can buy speed with capability or capability with speed. You cannot have both for free, and any vendor implying otherwise is quoting best-case numbers from an idle system.

Handoff-to-human is a design problem, not a fallback. The question is not whether the agent will hit something it can't handle — it will, on day one. The question is what happens then. A transfer that drops the caller into a queue where they repeat everything is worse than no agent at all. Handoff has to carry context, and it has to trigger early: on repeated confusion, on emotional signals, on anything touching money or health, and on an explicit request for a person. Designing the escalation path is not an afterthought; it is roughly a third of the build.

Language coverage is not uniform. Broad language support is real and genuinely valuable — but coverage is not the same as parity. Recognition accuracy, voice naturalness, and reasoning quality all vary by language, and the long tail is meaningfully weaker than the top handful. For most businesses, "good in your top three languages and workable in the rest" is exactly the right trade. It is just not the same claim as "fluent in all of them," and I would rather say so before the build than after.

Agents are narrow on purpose. An agent that reliably does six things and escalates everything else beats one that attempts thirty and is wrong on four. Scoping is where most of the value is created, and it is the first conversation, not the last.

How this is priced

Same as everything else here: $1,000 a day, flat. Not by the hour, not by the seat, not by the conversation.

Agent work fits that model well, because the honest answer to "how long will this take" depends almost entirely on your integrations. An agent with one calendar and one CRM is a short build. An agent that has to write into a legacy system with no real API is a longer one. Buying days rather than a fixed-scope estimate means we can find that out in week one and adjust, instead of discovering it in week six of a fixed bid.

Small tweaks after launch — a prompt tightened, an opening-hours change, a new escalation trigger — anything under twenty minutes stays free, for as long as the thing is running.

If you want to size a build before committing, the cost breakdown for voice agents walks through what actually drives the number.