Practitioners know what good output looks like and should own the final call, with operations handling the plumbing.
Picture a researcher who drops 12 interview transcripts into a synthesis tool and gets a tidy theme and quote. She ran six of those interviews, so she knows within a minute nobody said it.
The manager who negotiated the contract wouldn’t have caught it, because some people lack the hands-on experience that judgment takes.
Design and research operations grew up handling licenses, consent forms, and vendor contracts. When artificial intelligence tools arrived, buying them became deciding what they’re for.
That’s backwards, because this isn’t like choosing Figma or other tools, where the same input gave the same output and a checklist could judge them.
Until the field shakes out, the people using these tools should be the directly responsible individuals for the strategy, owners answering for each decision. They know what to look for. Operations should own the plumbing that makes their choices safe.
This is the canary in the coal mine — it means someone internal lost the story.Job listings already show the skills gap where some teams clearly don’t have the skills for this world and lack the initiative to follow the industry, which is sad:
- Okta is hiring a Senior Design Operations Program Manager, Agentic Design who works in Claude Code and Cursor.
- Yahoo posted a Design Operations Manager, Systems and AI Enablement role to own its tooling roadmap.
- Etsy wanted a Director, Design Operations fluent in Claude and Cursor.
- Mistral wants a Product Operations Manager (DesignOps) who automates the team’s coordination and reporting.
I’ve spent more than 20 years in UX as a player-coach and wrote uxGPT: Mastering AI Assistants for User Experience Designers and Product Management Professionals. This is like the early days of the web, when the people coming in didn’t understand the tools and made many poor decisions.
We should avoid repeating that mistake.
A designer works at a laptop with several personal tools open in separate windows beside the single tool the company provides.Count the Shadow Tools
The first sign strategy is being set outside operations is the shadow stack. In Microsoft’s 2024 Work Trend Index, 78% of people using artificial intelligence at work brought their own tools. In the State of Prototyping: Spring 2026 survey from UX Tools, 59.1% of designers had built their own tool, app, or utility with artificial intelligence in the previous six months.
Every shadow tool is a vote cast by someone who can tell good output from bad. It’s also a vote that operations doesn’t understand the tools.Slack was a shadow tool (and still is) until it reached critical mass.
Many operations teams are struggling, and they’re falling behind because they aren’t putting in the time to follow the trends. In The State of Research Operations, a December 2025 User Interviews study of 21 specialists, six questioned how many of their roles will survive.
I’m seeing some organizations rename these teams agent operations, because existing groups haven’t learned the tools or lack the skill sets.
Tool bloat is across all business functions, not just design.This isn’t just about UX.
Across industries, teams are figuring out how to reorganize around artificial intelligence. In McKinsey’s The State of AI in 2026: On the Road to ROI, nearly three-quarters of high performers had redesigned workflows, against a quarter of others.
People who don’t know the tools get replaced by people who do because they understand how to use it for their workflows.
A 2023 Harvard Business Review interview with Karim Lakhani made it the title: “AI Won’t Replace Humans, But Humans With AI Will Replace Humans Without AI.”
Learn From Past Waves
In 1979, VisiCalc shipped for the Apple II, and buyers understood it in minutes. Peter Jennings, a co-founder of its publisher, recalls in VisiCalc 1979 finding Apple computers on managers’ desks at Tandy’s headquarters, bought to run the spreadsheet.
Tandy sold its own computers through Radio Shack, and its managers still went around them.
The managers knew what a budget model had to do, so they could judge VisiCalc in a few minutes.
Software as a service repeated the first pattern. In its 2019 Form S-1, Slack told investors that usage inside organizations typically started bottom-up. Slack spread team by team because whoever ran internal tools didn’t understand the use cases, or didn’t try to.
The early web showed the other side.
By decade’s end Jakob Nielsen had measured what big companies got wrong. In his 1999 article Who Commits the “Top Ten Mistakes” of Web Design?, the 10 largest U.S. companies averaged a 20% violation rate on his list, against 13% for the 10 most-visited sites.
The biggest budgets bought more mistakes than the people living on the web. Artificial intelligence tools are running both patterns at once, on a shorter fuse.
Name the Evaluation Gap
I call this the evaluation gap: the distance between the person who approves a tool and the person who can tell when its output is wrong. Generative tools carry more nuance: which model a tool runs, how it handles a long transcript, and whether it shows its sources.
Anyone lacking the expertise misses that nuance and the failures inside it.
In legal technology, a contract summary that drops an indemnity exception or doesn’t understand the difference between assignment and change of control reads fine to anyone who has never negotiated one.
Generative tools produce something plausible every time, and the errors hide in details only a practitioner checks.
Subject matter experts (SMEs), the people who know a domain because they do the work, carry the advantage. Anthropic’s preliminary June 2026 study of about 400,000 Claude Code sessions, Agentic coding and persistent returns to expertise, found expert sessions reached verified success more than twice as often as novice ones. The authors credit domain expertise rather than coding skill. A product manager who knows the pricing model catches generated requirements that drop the enterprise tier’s usage cap. A designer who knows the design system spots the prototype that invents a fourth button style.
My history runs almost four years with generative tools and seven before that with other artificial intelligence. It tells me which outputs to check first on a new tool.
Stop Counting Seats
Operations teams measure what they can see: seats, spend, renewals, and logins. Those numbers fit a license and fail a strategy. Better ones come from SMEs, such as the share of synthesized claims that survive a check against the transcripts and the token cost of each finished study.
In the User Interviews study, eight specialists described tool management as a growing share of the job. Money is tight: 17 said their scope had grown, while 12 said budget, staff, or resources stayed flat or shrank.
The case — if the business can’t see business impact, that set of roles are eliminated.
Seats are giving way to tokens. In Gartner Predicts AI Coding Costs Will Surpass Average Developer’s Salary by 2028, from June 2026, Gartner says coding vendors are moving from seats to consumption pricing. Deciding which work deserves the priciest model takes someone who knows the work.
Tool bloat is the canary in the coal mine from past operations mistakes. Those mistakes are costing the field jobs, and operations teams have to own this before they have credibiity.
In the UX Tools survey, 53% of designers said too many tools were slowing them down. A stack that keeps growing means nobody with judgment decided what each tool is for, and that’s a competence problem for whoever owns it.
Operations can cut the count once SMEs say which tools earn their place because that’s their right.
Hand Operations the Plumbing
Operations still matters.
Peter Merholz made the case in Peter Merholz: How Agile Can Turn Designers Into Pixel Pushers, a 2020 Superside interview. “If you don’t pay attention to the operations, the wheels come off,” he said.
Artificial intelligence adds plumbing, including contracts and maintenance for tools and workflows practitioners build. Even after years in legal technology, I don’t want to be in charge of the contracts. I was on a team whose internal workflow application became a nightmare to maintain. So split decision rights along the evaluation gap:
- Own the judgment. SMEs who use the tools are the DRIs for which ones the team adopts and what counts as acceptable output. They set strategy.
- Run the plumbing. Operations handles security and privacy review, contracts and data boundaries, access, budget, and upkeep. They run the trains.
Practitioners should decide what good looks like, and operations should make it safe to find out.
Bad calls are coming, and someone has to own them. Many operations people perceive more influence over these choices than their experience supports, and setting strategy from that seat is running with scissors.
A researcher who picked a tool that invents quotes can explain the failure and fix the review step.
An operations manager who never ran a study can only cancel the contract and usually acts too slow in the process. Treat the arrangement as temporary, because we’re moving at the speed of light, and revisit ownership once the field settles as in all innovation cycles.
Conclusion
Tandy had its own computers to sell, and its managers still bought Apple IIs to run a spreadsheet. The official plan stopped mattering once the people doing the budgets had found a better tool. Design and research teams are there now, and the shadow stack is the evidence.
This is a good time to reevaluate operations teams the way we’re reevaluating design systems and other roles: what value they provide and what the job requires. UX, like every field, has to evolve toward fewer roles and more skills, because role titles are selfish: they protect turf rather than the work, and that’s hurting the field. The job growing underneath operations is the plumbing that lets good choices scale.
Practitioners have work to do, too. Owning strategy means writing down the quality bar, sharing tools that work instead of hiding them, and retiring ones that stop paying off. A designer who wants the decision must defend it in a budget review, with numbers about the work instead of the logins.
Picture that researcher again, the one who caught the invented quote in under a minute. Give her the decision, and give operations the job of making it safe to act on.
Action items
- Write the decision charter. Name three to five SMEs who use artificial intelligence tools every week as the DRIs for which tools design and research adopt, renew, or drop, and make them answerable for the results. Put the split on one page, with operations owning security, privacy, contracts, access, token limits, and maintenance, and revisit it every six months while the field settles.
- Build a task benchmark. Collect 10 real tasks with answers your team trusts, such as a coded interview or a finished flow, and run every candidate tool against them before anyone signs a contract, scoring errors caught, edits needed, hours saved, and token cost. Settle disputes over a tool with the results. It’s the kind of internal benchmark Ethan Mollick recommends in Making AI Work: Leadership, Lab, and Crowd.
- Surface the shadow stack. Ask the team which personal tools they use and for what, promise there’s no penalty for answering, and treat the list as a shortlist for evaluation and a count of the bloat to cut.
- Rewrite the operations job descriptions. Reevaluate what design and research operations is for, the way you would a design system, and hire or train for hands-on tool skills, token and budget governance, and vendor management alongside the contract work.
The people using AI should set its strategy was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.