Six questions to ask for whether a conversational layer will speed experts up or quietly put a keyboard between them and the work they already know.
Nearly every B2B application I’ve seen has a chat box on it. Sometimes a sidebar, sometimes a rebuild where the prompt becomes the front door.
The feature is usually well designed and has an element of sparkly craft in it, because that’s how we are defining craft in AI now, right?
The question underneath it — who is this for, and what were they doing thirty seconds ago, how long have they been doing it in their career — usually goes unanswered.
That question matters more in professional applications than anywhere else, because these users aren’t visitors, they expert users in very specific domains; they have spent decades learning their domain and their software.
I’ve spend decades, literally, watching them and have come to the conclusion that some use cases should never be replaced by AI because it will never beat them.
They’re a claims adjuster on file 82 of the week, an attorney who has read 10,000 contracts and knows the difference between assignment and change of control is just a few words they wrote in a contract 7 years ago and reuse the same pattern today, a supply planner who knows every exception code by heart. They open the same six screens every morning and know exactly where everything lives.
For them, conversation is a detour through a mine laden text field for a group of users that have off the charts cognitive limits because they’ve been in their domain, forever.
I know, I’ve seen it — in real time.
These UX decisions are downstream of one missing step: nobody drew the line first and that line is written in pencil. The reality is that where the line is for a professional application is a bit smudged, but we still have to draw it, especially in the time of AI.
Here are six considerations that tell you where the line belongs.
Expertise is a skill someone already paid for. A redesign asks them to pay for it twice.Does It Break Existing Skills?
An expert has built a skill inside your product — keyboard paths, saved views, the workaround they invented in 2019 they have written on post it notes shaves 90 seconds off a daily task.
I’ve seen that last one much more than I would like.
None of this shows up in your analytics, and all of it gets written off the day you replace the surface it lives on.
The write-off is bigger than it looks, because they aren’t only learning your new screen, they’re learning a new way of working. Jakob Nielsen argues AI is the “first new UI paradigm in 60 years”, after batch processing in 1945 and command-based interaction in 1964. An industry teaches a paradigm slowly, and your users pay tuition to every product that tries.
You’ll hear the resistance called sunk cost fallacy, usually as grounds to override it.
Wrong label.
A sunk cost shouldn’t touch the next decision. The months your user spent learning your product aren’t spent — they’re invested, and they pay out every morning, and they won’t even upgrade software to change their patterns.
The users are protecting an asset. The roadmap is protecting a decision.
The fallacy sits on the other side of the table. A team that funded a chat surface for two quarters, watched adoption stall and shipped it anyway is running the pattern Barry Staw documented in “Knee-Deep in the Big Muddy”: people commit the most resources to a failing course precisely when they caused the earlier losses.
The expertise reversal effect names the mechanism: support that helps beginners turns costly for anyone who already knows the domain. A conversational layer is scaffolding, a gift on day one and an obstruction in year four.
For example, natural language might seem cool, but the recruiter or lawyer that has 400 saved searches? They’ll want the knobs and dials because they know better than AI.
The bar isn’t “better.”
Say the multiple out loud: three times faster, or something the old path couldn’t do at all. If you can’t name a number, ship it beside the old path so you can learn.
Graphical interfaces kept the command line for 20 years, and the people who needed it never stopped using it.
Conversation lowers a barrier for people who don’t know the system. Experts have the opposite problem.Is The Chat Only for Novices?
Conversation solves recall.
If you don’t know a system, every menu is a vocabulary test you’re failing in public, which is why most enterprise applications are information architecture mine fields. Type a sentence instead and the burden moves to the machine. That’s a real improvement, and it’s why chat lands so well in a demo — demos are always someone’s first day.
Tesler’s law names what’s actually happening: complexity is conserved. Every system has an irreducible amount of it, and the only question is who absorbs it.
Chat doesn’t remove the complexity of your product, it relocates it and leverages the system to help with AI.
For someone who doesn’t know the system that’s a good trade — the machine takes the part they can’t carry. For the expert it runs backwards. They absorbed it years ago, in the form of the field name and the status code, and the prompt hands it back to them as a phrasing problem.
Direct manipulation solved the same problem from the other side. Ben Shneiderman’s 1983 formulation, still the cleanest definition anyone has written, put objects on screen where you could see them, act on them and undo what you did. Recognition instead of recall.
- For the novice, the prompt is a rescue.
- For the expert, it’s a translation layer over a system they already speak fluently, no matter how complex.
NN/g calls the novice’s version of this the articulation barrier — people know what they want, can’t phrase it, and get generic results back. Experts have the reverse problem. They know the field name, the status code, the exact filter, and are much faster with it.
Asking them to describe it in a sentence doesn’t lower a barrier, it builds one in many cases. Test accordingly.
Do Users Need to See the Whole Set?
Here’s what filters do that a prompt can’t: When an adjuster narrows a queue to 12 open files, they don’t just get 12 files, they learn there are 12 and not 400, that three are past SLA, and that the pattern looks like a Tuesday last November.
None of that was the question, all of it is the answer and scanning is diagnosis.
Filtering is a hypothesis you can run in half a second and abandon just as fast, which is why experts sweep through six before committing to one. A chat response hands back a result without the shape of what surrounds it.
A chat answer tells you what you asked for; a screen tells you what you should have asked.
Linus Lee made the harder version of this point in 2022: conversations are a bad way to keep track of information, because the state you need ends up in scroll back instead of in front of you.
Screens also carry something transcripts don’t — a shared model. Two people looking at the same view see the same thing, and they can point. A conversation is private by construction, so every user builds their own path and nobody can audit it.
Fine for a consumer app but a real problem in an operations team of 40. That’s why AI search may not be the answer.
Experienced developers in METR’s trial were 19% slower with AI, and believed the opposite.Does It Slow Experts Down?
METR ran a randomized trial with experienced open-source developers on their own repositories and found them 19% slower when allowed to use AI tools. The part that should worry you more: they expected a 24% speedup going in and still believed they’d gotten one coming out. They could not feel it happening.
Small sample, one kind of work, so don’t over-read it. It rhymes with the largest field experiment we have. Dell’Acqua, Mollick, Lakhani and their co-authors put 758 consultants through the jagged technological frontier study: inside AI’s capability, subjects were 25.1% faster at higher quality; on tasks just outside it, 19% less likely to be correct than the control.
Expertise is already compression. A prompt asks the expert to decompress it into English so the machine can compress it back.
Watch a partner who has drafted the same indemnity clause 200 times. They don’t compose it, they recall it, and the recall is close to instant, nuance is more real than AI understands, and they have to write it in the spirit of the original language.
The prompt-read-correct loop costs them more than that and leaves them less certain than when they started, because now they’re checking someone else’s work instead of trusting their own.
The model isn’t bad. They’re very good, and the interface asked them to slow to the speed of a sentence.
How Hard Is It to Verify?
In professional work an answer isn’t finished when it appears. It’s finished when someone has checked it and will put their name on it. That cost never shows up in the demo.
Stanford researchers tested the purpose-built legal research tools — the ones sold specifically as grounded and reliable — and found Lexis+ AI and Thomson Reuters’ tools hallucinating between 17% and 33% of the time.
Better than a general chatbot.
Nowhere near good enough for an associate to stop reading every citation, which means the associate is still reading every citation.
The more an output has to be defended to a third party, the worse a chat box performs as the place to make it.
Chat is structurally poor at supporting the check. It returns prose, and prose flattens the difference between what was retrieved and what was generated. A filtered result is auditable by construction — the query sits in the chips above the table, and anyone can rerun it.
Ask who has to defend the output: the further an answer travels, the more the surface has to show its work.
Three destinations, and only one of them is a chat box.Where Should the Work Go?
None of this means don’t build it. It means work has three destinations and only one of them is a chat box.
Routine Tasks Stay in the Interface
The first destination is the queue somebody works 200 times a day, the saved views, the keyboard path they could walk in the dark. Repetition is what buys speed here, and the speed is the product.
Cost runs the same direction and catches teams out: a click costs nothing to serve, while a prompt against the core loop bills you every time someone does what they used to do for free — which is why cost heads Gartner’s list of reasons these projects get cancelled.
Put a text field in front of the daily queue and you’ve charged your best users a tax to use their own product.
Rare Tasks Go to Chat
The second is the long tail nobody would design a screen for. Which of these 400 contracts have a change-of-control clause and a non-standard liability cap. One-offs that cross the whole schema and become a ticket to an analyst who answers in three days. Answers that are anomalies deep in the financial analysis.
That’s where conversation is worth real money, because the alternative isn’t a filter because it’s a wait, and a question asked once a quarter is worth paying per call for.
Call it the ticket test.
If the user would have filed a ticket, chat is competing against three days and the bar is low. If they would have clicked, it’s competing with muscle memory, and muscle memory is undefeated.
Ongoing Tasks Go to an Agent
The third destination is the one teams keep missing, and it isn’t a conversation at all.
- A contract arrives with a non-standard cap.
- A shipment slips its window.
- An account’s usage drops three weeks running.
Nobody needs to type a question about any of these, because the system already knows. That work belongs to an ambient agent running against the same data your screens display, interrupting only when a person has to decide something.
The idea isn’t new. Christopher Noessel called these systems agentive a decade ago in “Designing Agentive Technology”, and Golden Krishna argued for background processes over screens in “The Best Interface Is No Interface” in 2015. The models are finally good enough to make it work.
Chat waits to be asked. An ambient agent already knows.
Chat is pull and needs the user to know a question exists. Ambient is push and needs the system to be right about what deserves attention, which is the harder problem and the better product. Get it wrong and you’ve built a notification nobody trusts, so tune for precision and let the agent stay quiet. Get it right and it does the scanning nobody can do at 3 a.m., then hands back a filtered queue — which is to say it ends where the interface begins.
Conclusion
The line has nothing to do with how you feel about conversation. It’s a question about who’s standing in front of the screen and how many times they’ve done this before.
The truth:
- Novices need language —they don’t have the map nor the dictionary.
- Experts have the map — they built it, they maintain it, and every screen in your product is a landmark on it.
Handing them a text box asks them to give up the map and describe the terrain instead, to someone who’s never been there and occasionally makes places up.
So run the checklist before you run the sprint. Answer the five questions honestly, on real usage data rather than the demo in your head, and the roadmap sorts itself.
What’s left is a smaller chat feature than the one you were planning, aimed at work your users currently can’t do at all, sitting beside an interface you didn’t have to apologize for and an agent quietly working the queue while nobody’s watching. That’s a better product than a prompt on top of everything. It’s also harder to demo, which is most of the reason it doesn’t get built.
Checklist
The six tests reduce to five questions. They’re abstract on purpose — they should still work after the next model release, and the one after that. Saleema Amershi, Mihaela Vorvoreanu and their colleagues at Microsoft Research validated 18 guidelines for human-AI interaction against 20 shipping products in 2019. These five are narrower, and only settle whether a conversation belongs in front of an expert at all.
- How well does this person already know the system? Everything else follows from this one. Language helps whoever lacks the map and gets in the way of whoever drew it.
- Is this task routine, rare, or continuous? Routine work belongs to the interface, where repetition buys speed. Rare work belongs to conversation, because nobody was ever going to build a screen for it. Continuous work belongs to an agent that watches and interrupts, because nobody should have to remember to ask.
- Does the user need an answer, or the shape of the whole set? Retrieval is a question with a reply. Diagnosis is a question about what surrounds the reply, and it needs a surface that shows the surroundings.
- Who is faster here, and who is more certain? Speed without certainty isn’t speed. It’s a cost that shows up later, in the time it takes someone to trust the output enough to sign their name to it.
- Is it worth what switching costs? Every new path bills the user for the old one they’d already mastered. If you can’t say plainly how much better this is, leave the old path where it is and let them choose.
Action Items
- Put chat where the tickets are. Pull 90 days of requests your customers sent to analysts and operations; that queue is a validated feature list with volumes already attached.
- Give every answer a screen to land on. Render results as a filtered view the user can adjust, save and share, so the conversation hands off to the interface instead of replacing it.
- Measure verification time, not generation time. Track how long it takes a user to trust an answer enough to act on it, and whether that number is falling.
- Find the alerts nobody set up. Ask your users what they check by hand every morning, and what they wish someone had told them last week. Each answer is an ambient agent with a job and an audience already attached that wants to be helped.
- Watch the fallback ratio. Keep the clicking path and track which one users reach for. That ratio is what they actually think of the feature, and it won’t match what they say in a research session.
Where to draw the line before AI reaches professional users was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.