workflows: spell the value back once the caller refuses a confirmation - #6990
Conversation
…efused ReadBack picks the read-back instruction a Get*Task hands its model after recording a value: the natural form the first time, the spelled form on every later attempt. A task records a value again only when the caller did not confirm it, and a value that sounds like another cannot be told apart by hearing it once more. Not wired into any task yet.
GetEmailTask, GetNameTask, GetPhoneNumberTask, GetAddressTask and GetDOBTask read a recorded value back the same way however many times the caller refuses it. A caller whose name sounds like another spelling can never accept the read-back: the agent had shayne.cole@gmail.com right three times in one call and each time said it as a word, which is the same sound as shane, so the caller re-spelt it and the call ended without a booking. Each task now hands its read-back through ReadBack: natural on the first attempt, spelled on every later one. The card tasks stay out, their values are never read back. verify_spelling on GetNameTask keeps spelling from the first attempt.
| natural="Repeat the address back to the user.", | ||
| spelled=( | ||
| f"Repeat the address field by field, spelling the street name letter by " | ||
| f"letter: {address_fields}" |
There was a problem hiding this comment.
should we provide spaced input so they don't hallucinate letters, like the strawberry quiz?
|
|
||
| def instruction(self, *, natural: str, spelled: str) -> str: | ||
| self._attempts += 1 | ||
| return natural if self._attempts == 1 else spelled |
There was a problem hiding this comment.
I wonder if we should go one step further like "A as in alpha" for the 3rd time.
A single bool per task replaces the ReadBack helper: once a value has been recorded, every later read-back of it is spelled. GetNameTask seeds the bool from verify_spelling, so its separate branch goes away.
|
I benched both review questions (should the prompt carry pre-spaced letters so the model doesn't hallucinate them, and should a third attempt escalate to "S as in Sierra"). Bench is How it works. Part A drives the real 1. Pre-spaced letters: yes, but not because of hallucinationAcross ~500 spelled read-backs over five models there was exactly one wrong letter (gpt-4.1-mini dropped the Y in Krzysztof, Exact-spelling rate, names:
Address (street name, 2. NATO on a later attempt: small audio gain, only viable with pre-built textTTS→STT round trip, 20 names, one trial each:
The three letter misses were single-letter confusions (G→A, M→N, U→E). The one NATO miss was STT turning "T as in Tango" into "niti as in tango". When I regenerated the samples below for this comment every round trip came back exact, so treat the 85/95 split as indicative, not settled. On the LLM side, the model produced zero wrong code words whenever it did spell, and hit 100% on the first turn given spaced input. After a refusal though, gpt-4.1 ignored the NATO instruction in 17 of 20 Audio samplesFiles live on an orphan branch of my fork (u9g/agents@e08e241). The mp4 links open GitHub's inline player; the wav links are the bare audio. Cartesia sonic-3 speaking the read-back the agent would say, followed by what Deepgram nova-3 heard. Shayne Cole letters:
NATO:
Dhruv Venkataraghavan (letter miss in the bench run: G heard as A) letters:
NATO: Mikaela Thibodeaux (NATO miss in the bench run: "T as in Tango" heard as "niti as in tango") letters: NATO:
Tomasz Wojciechowski (letter miss in the bench run: M heard as N) letters: NATO: Side finding on this PROn a plain refusal without re-spelling ("No, that's not right. It's Shayne Cole."), gpt-4.1 called Caveats: one trial per cell, n of 12 to 20, and STT is a proxy for a human listener. |
Word boundaries in the spelled form produced three consecutive spaces
("S h a y n e C o l e"). Strip spaces before spelling so every
character is separated by exactly one.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| ) | ||
|
|
||
| read_back = ( | ||
| f"Spell out the name letter by letter for verification: {' '.join(full_name.replace(' ', ''))}" |
There was a problem hiding this comment.
🟡 Name boundaries vanish during spelling
Names containing spaces inside or between parts lose every boundary through replace. Ann A Bell and Anna Bell produce the same character sequence, so callers cannot identify the stored name. Street-address boundaries disappear through replace too.
Prompt for agents
Preserve audible and textual boundaries between words and name or address components while still pre-spacing characters for reliable letter-by-letter speech. Update GetNameTask._update_name_impl and GetAddressTask._update_address_impl so distinct values such as “Ann A Bell” versus “Anna Bell”, and “North Cliff” versus “Northcliffe”, cannot produce the same spelled read-back. Extend tests/test_workflow_readback.py with ambiguous multi-word pairs and assert that their spelling instructions remain distinguishable without relying on doubled spaces.
Was this helpful? React with 👍 or 👎 to provide feedback.
3 participants
Callers can get stuck correcting a value that sounds identical to another spelling. In hotel-receptionist simulation
SR_yxUq2cWd3C3D(jobSRJ_8GbAvzvRUXAF), the agent recordedshayne.cole@gmail.comcorrectly three times but kept reading it as a word; the caller could not distinguish it from “shane” and the call ended without a booking.Each capture task now switches its confirmation instruction after the first successful update that requests confirmation:
verify_spelling=TrueThe state is an inline boolean in each task. Name and address spelling preserves punctuation and accented characters; address unit, locality, and country remain separate fields. Stored values retain their original formatting. Credit-card tasks do not read their values back and are outside this change.
Escalation depends on the model calling the update tool again. A spoken refusal alone does not guarantee that call: the benchmark found that callers re-spelling a value triggered updates more reliably than simply restating it. This change addresses repeated updates; it does not establish refusal detection.
Validation:
tests/test_workflow_readback.py: 10 hermetic cases cover the transition, repeated spelled attempts, spelling from the first attempt, punctuation/accents, and preservation of stored names and address fields.make checkpasses (formatting, lint, and type checking);make fixleaves all files unchanged.ConnectError/ invalid HTTP version. Those tests pass in CI; the local full-suite run is not green.GetEmailTaskonopenai/gpt-4.1changed the second and third read-backs from words to separated characters.Simulation check investigation:
8232cb414registered the worker successfully and ran all 10 scenarios: 9 passed, 1 failed. The name, address, and DOB correction scenarios passed. The remaining failure is “Card number too short on the first try”: the simulated caller declined to provide the test card number for privacy reasons, contrary to its scenario instructions. This matches the existing caller-refusal failure on main noted above.