Field Notes | Technical Humanist

Six Things No AI Tool Comes With

Allison Marx — Fri, 18 Sep 2026 12:00:00 GMT

Someone has asked what the company is doing with AI, and now there’s a number attached. A subscription to approve. A vendor quote. A line in next quarter’s budget.

You don’t have a data team to ask. You have a few people who’ve tried ChatGPT or Copilot, a demo that made the current process look painfully manual, and a suspicion that the money could land somewhere that changes nothing.

The expensive failure happens before anyone compares tools. Something gets bought before the company has decided what work should change, what information the system can use, who’s accountable for the result, and how anyone will know the work got better.

Six things have to be true before AI spending has a fair chance of improving the work: usable data and context, a clear workflow, a way to measure the change, a named owner, a review step with real authority, and a plan for adoption. Each has a test you can run this week without technical help, and each failure points to a fix that’s usually less technical than the tool on the table.

Start With One Workflow, Not the Company

“Are we ready for AI?” has no useful answer at company level. Readiness is a property of one workflow. A team may be ready to summarize approved internal material or search a well-maintained knowledge base while being nowhere near ready for a workflow that handles sensitive customer information or writes to operational systems.

So pick one. Choose a recurring piece of work that’s slow, inconsistent, or repetitive today: the weekly report someone assembles from several exports, say, or the inbound inquiries someone sorts by hand. Every test below is asked about that one workflow.

The six answers will come out uneven. They have different owners and different fixes, so keep them separate long enough to see which one needs attention first.

Know What Goes In and Where It Came From

An AI tool works from the information it’s given: records, documents, exports, and the definitions behind them. The data and context constraint asks whether that information is current, permitted for this use, and understood by someone responsible for it. It doesn’t ask for perfectly clean data.

The problem nobody notices is usually a definition. “Customer” may mean a closed-won deal in the CRM, an active subscriber in billing, or anyone who ever submitted a form. An AI assistant won’t resolve that disagreement. It’ll reproduce whichever definition was easiest to connect.

The test. If you had to hand someone the data this workflow depends on, could you? Find the source of the last input the work used. Ask who owns that source, when it was last checked for duplicates or gaps, and whether the information was collected under terms that permit this use.

If it fails. Repair before you buy. Resolve one definition dispute. Trace one important field to its source. Name the person who can approve or halt its use. If the information would go to a vendor, find out what leaves your systems and where it goes next.

Go deeper. The Browser Didn’t Force Your Hand. Your AI Roadmap Will. puts the data work in order.

Draw the Work Before You Change It

The workflow constraint is whether anyone can see the work as it happens today: the inputs, the handoffs, the exceptions, what happens downstream, and which steps a person owns. When exceptions are handled from memory and the answer depends on who’s working that day, automation adds a disconnected process rather than improving the existing one.

The test. Sit down with the person who runs the workflow every week. They know where it breaks. Map what happens now, who does it, what information it depends on, and where the exceptions go. Then ask the team to describe what a bad answer would look like. If nobody can, the tool’s output has nothing to be checked against.

If it fails. The next move is the process, and it may involve no AI at all. Sometimes the answer is a better intake form, a documented procedure, or cleaner reporting definitions. Once the map exists, each step gets its own decision: automate, assist, simplify first, or keep human.

Go deeper. You’re Not AI-Ready. One Workflow Might Be. sets out the possible moves.

Decide How You’ll Know It Got Better

Measurement means the team can name what should improve and how it’ll be observed. Without that, the pilot becomes a demo with no viable success condition.

A draft can arrive faster while the workflow gets harder to manage once review, exceptions, and rework are counted. “We saved time” is the beginning of the analysis.

The test. Ask three people close to the workflow to write down, separately, the problem the tool is meant to solve. Compare the three answers. Then try to finish two sentences: “This will have worked when…” and “The failure we couldn’t accept is…” Note how the work goes today (how long it takes, how often it comes back for rework) so there’s something to compare against later.

If it fails. Write the decision statement before anything else: the problem, the intended users, the desired outcome, the constraints, and how success gets measured. If three people named three different problems, settle that first. Every later decision about the tool gets checked against it.

Go deeper. AI Makes the First Draft Cheaper. Trust Still Costs Money. separates task, workflow, and outcome evidence.

Put a Name on It

Ownership means a named person owns the decision, the maintenance, the monitoring, and the escalation. A mailing list doesn’t count. Without a name, the workflow becomes an orphaned pilot, and every question about it routes to whoever’s available rather than whoever’s accountable.

The test. Write one name beside each of these: who decides whether the output is good enough, who maintains it when a system or a definition changes, who pays for it, and who can pause it when the output is unreliable. Then check that person has the time. An owner with forty other things to do is a name in a document.

If it fails. Assign the owner, the support path, and a review cadence before launch. License cost is the visible part, so count the hours before declaring a tool cheap. You don’t need a large data team for this. You do need someone responsible.

Go deeper. The Integration Tax Just Got Cheaper counts what a tool costs after the subscription.

Give the Reviewer the Power to Say No

Governance, in its useful form, is the visible ownership, definitions, permissions, quality checks, and escalation paths that let a team use information in ways it can explain and defend. A small company doesn’t need an enterprise program. It needs controls in proportion to the risk, and more of them when the data is sensitive, the decision carries consequence for a person, or an error would be hard to catch.

The control most plans lean on is human review, and it works only under five conditions. The reviewer needs expertise and context, time, authority, an escalation path, and incentives that permit refusal.

A review step missing any of the five can still produce a signature. It can’t be counted on to produce oversight.

The test. Ask the person you’d name as reviewer three questions. What are you looking for? What evidence do you need to decide? What can you do when you find a problem? Then ask the plan one more: what happens if the reviewer does nothing? If the output proceeds anyway, you have a timeout with a human decoration on it.

If it fails. Choose a review model on purpose: every output, a sample, exceptions only, or no routine review with logging and fast rollback. Write down who can pause the workflow and who gets told. And keep AI out of the final decision where the result materially affects someone’s livelihood, safety, rights, health, or access to essential services.

Go deeper. You Put a Human in the Loop. Can They Say No? has both lists in full.

Count Adoption as Evidence

A system people route around hasn’t created much value. A system people use enthusiastically but can’t challenge or safely correct isn’t a success either. Adoption has to be planned: who needs to learn what, where people request a change, and whether the team can run the thing without calling whoever built it.

The test. For a tool already in use, check the last two weeks of usage against the list of people who were expected to use it. Then ask one regular user what they did with the time it saved last week. If the answer is “checked its output,” write that down. For a tool you haven’t bought, read the plan to its last page. If it ends at the demo, adoption hasn’t been planned.

If it fails. Find out whether the cause is the tool, the data, or the habit before spending more. And bring the person who does the work into the decision, alongside the person who owns the budget.

Go deeper. A Great Solution Design Makes the Next One Easier covers writing for the people who have to use it.

Six Tests on One Page

Run these against one workflow. Each line is a test, then what to do first if it fails.

Data and context. Could you hand someone the data today, and say who owns the source? If not: settle one definition, trace one field, and name who can approve its use.
Workflow. Can the person who runs it map the steps and exceptions, and can the team describe a bad answer? If not: fix or document the process before adding technology.
Measurement. Do three people name the same problem, and can you finish “this will have worked when…”? If not: write the decision statement, with a success measure.
Ownership. Is there one name, with time, for deciding, maintaining, paying, and pausing? If not: assign the owner and count the hours before launch.
Governance and review. Can the reviewer say what they’re looking for and what they can do about it? If not: pick a review model and name who can stop it.
Adoption. Does usage match the list of expected users, and does the plan go past the demo? If not: find out whether it’s the tool, the data, or the habit.

Fix the First Thing That Failed

A workflow can fail more than one test. Resist the long roadmap. Name the one failure the others depend on and start there. A review step can’t be designed for a workflow nobody has drawn.

If you’d like a quick reflection first, the free Snapshot asks five questions about one tool or workflow your team already uses. It takes about five minutes, needs no email, and names one thing to check first, based on your answers rather than an evidence review.

If one workflow and one decision are in front of you, a Working Session works through it in ninety minutes with the person who runs the work. If several efforts are competing for the same budget, or people disagree about what’s blocking progress, that’s what the AI Foundation Diagnostic is for. It ends with a named primary blocker and a sequenced 90-day plan.

A new model, platform, or subscription can’t decide what the workflow is for, settle competing definitions, assign ownership, or create the evidence that the work improved.

None of the six comes with the tool. All six can be started this week.

Before You Sign the AI Proposal, Read It for What’s Missing

Allison Marx — Fri, 18 Sep 2026 12:00:00 GMT

There’s a proposal in your inbox. An agency, a freelancer, or a software vendor has offered to build or set up something with AI. It has a price, a timeline, a list of features, and a few tool names you half recognize.

It reads well. Proposals are written to. And you’re the one deciding this week whether it’s sound, without a technical team to hand it to.

Much of what decides whether an AI build works after launch is something a non-technical owner can check on the page: the problem, the workflow, the data, the review step, the definition of done, and who owns the result when the builder leaves.

A sound proposal describes your work. A weak one describes the tool. The standard I’d hold it to: if a vendor can’t explain how the system will be evaluated, maintained, corrected, and retired when conditions change, the proposal is incomplete however impressive the demo was.

The Demo Got You This Far

A demo shows what a tool can do under prepared conditions. It doesn’t show what the workflow will do with your definitions, exceptions, data quality, permissions, and users.

The proposal is where that gap should close. Look for an evaluation plan: realistic test cases drawn from your own work, known failure conditions, and the conditions that require human review or a pause. If the only evidence on offer is the demo, you’re being asked to rely on a confident output rather than proof.

Perfect prediction was never on offer. You’re looking for a process that shows where the system fails and whether your team can catch it.

Scale the Scrutiny to What the Tool Touches

Price is a poor guide to how hard to read. What matters is what the tool touches. Raise the level of scrutiny when the data is sensitive, the output carries consequence for a person, a vendor receives your information, an action is hard to reverse, or an error would be hard to catch.

Still, a subscription and a custom build ask for different amounts of paper.

For a $50-a-Seat Tool

Problem and success measure. One written sentence on what it’s for and how you’ll know.
Data. Whether your data trains a shared model, where it’s stored, and whether you can keep specific data out.
Review. A bounded, low-risk drafting aid can run on sampling and feedback.
Ownership. Who administers the seats, and who’d notice if people stopped using it.
Running cost. Seats, training, and the hours people spend checking output.
Exit. Whether you can export what people put into it.

For a $20,000 Build

Problem and success measure. A decision statement, acceptance criteria, and a named person who signs off.
Data. Everything a seat-based tool needs, plus written access rules for every system it reads from or writes to.
Review. A named reviewer, an exception route, and a stated answer for what happens if review doesn’t occur.
Ownership. Whose accounts it runs in, who holds the credentials, who has the documentation.
Running cost. Model and platform usage, monitoring, maintenance, and changes after launch.
Exit. What leaves with the builder or the vendor if you stop in eighteen months.

The data row doesn’t shrink with the price. A cheap tool that receives customer or employee information deserves the same questions as an expensive one. The rest of this note is written for the right-hand column.

It Should Describe Your Work, Not the Tool

Does the proposal name a specific task or decision that’s slow, repetitive, or inconsistent today, the people who do it, and what should improve? Does it say how that improvement will be observed? A proposal that lists capabilities without a problem is selling a capability, and the pilot it produces will be a demo with no viable success condition.

Then look for your workflow as it runs today: the inputs, the handoffs, the downstream actions, and above all the exceptions. Exceptions need a route, not a dropped record. A sound proposal says where unmatched, uncertain, or failed items go, and to whom, with the original attached.

If it’s missing, the builder is quoting against a workflow nobody has drawn. That’s a design step, and it belongs before the build quote. On my own builds an approved workflow design is the first precondition, whether it came from me, your team, or another builder.

It Should Say What the System Reads, Writes, and Sends Elsewhere

A sound proposal names each system the build connects to, confirms each one has a supported, documented way in, and states what the automation may read and what it may write. Access rules belong in the specification before anything is connected, with only the access the work requires. A custom connector for a system with no supported interface is separate work. Ask whether it’s priced.

Look for data quality too. A proposal that’s silent on the state of your data is assuming it’s fine. Ask whether cleanup is in scope, and whether anything checks the data before go-live.

And when information leaves your systems, the data decision stays yours. A contract allocates obligations. It can’t replace your understanding of where the data goes. Ask:

Will our data train a shared model or improve the service for other customers?
Where is it processed and stored, and which subprocessors receive it?
What controls exist for access, retention, deletion, and export?
Can we prevent specific data classes from entering the workflow at all?

If nobody on your side can explain what leaves your systems and where it goes next, the proposal isn’t ready to sign.

It Should Give the Reviewer a Job

Almost every proposal says there’s a human in the loop. The phrase means something only when that person can affect the result.

A sound proposal describes the review step in working detail: the reviewer’s role and backup, what they see, the criteria they’re checking, the expected volume and time per item, and the actions available to them (approve, edit, reject, pause). It names the escalation route. And it states what happens if review doesn’t occur, rather than leaving that to a product default.

It also asks something of you: a named owner with time to check output during testing, sign off at acceptance, and run the workflow afterward. On my builds that’s about an hour a week while the work is under way. If the proposal asks nothing of your team, ask who’ll be checking output while it’s being built.

Be wary of a promise that the model won’t make things up. It can, and I won’t promise otherwise on my own builds. What a builder can commit to is a design where unsupported output is detected, constrained, and routed to a person, with acceptance tests that check the routing works.

If it’s missing, you’re looking at one of the clearest reasons to pause: a vendor promising autonomous outcomes without showing how people will review, correct, or stop the system. You Put a Human in the Loop. Can They Say No? lists what the review step needs in writing.

It Should Define Done and Say What’s Left Out

“Done” needs acceptance criteria a builder can test against, an agreed test plan, and a named person on your side who runs the checklist on realistic inputs and either signs off or sends it back with specifics. Look at what the payments are tied to. On my builds, the final portion is billed at accepted handoff.

Then read the exclusions. A sound proposal has a list of what’s left out. When there’s no list, the disagreement about what’s included happens later. Check:

What the base price covers: how many workflows, how many systems, how many environments
Whether testing, documentation, and training sit inside the price
What counts as a defect, for how long it’s fixed at no charge, and what counts as new scope (a new feature, a changed requirement, a system connected afterward)
When the clock starts. Procurement, security review, and account provisioning can all sit before it, and delays there move the schedule.

It Should Leave You Owning It

Ask where the thing will live. The answer you want is your accounts, on your licenses, documented for your owner, with nothing depending on the builder staying involved. Look for an operating guide (what to check, how often, what “wrong” looks like, who to call) and training for the people who’ll run it.

Then the money after launch. License cost is the visible part. Ask who pays for model and platform usage. On my builds it’s billed to your accounts, and I estimate it from testing volume before handoff. Add monitoring, maintenance, and the work that appears when a vendor changes an API. Code you didn’t write and don’t understand is technical debt regardless of what produced it, so ask who the named technical owner will be.

And exit. If you removed this in eighteen months, what would leave with it? If the definitions, history, or business rules live only inside the vendor’s product, set export and retirement requirements before dependence deepens. The Integration Tax Just Got Cheaper has the full set of questions.

What the Proposal Says, and What to Ask

Each line starts with a phrase you’ll find in the proposal, then the questions to put back to whoever wrote it.

“Fully automated” or “no manual work.” How do people review, correct, or stop it? What happens to the items it can’t handle?
“Human in the loop.” Who, by role? What are they looking for, and what can they do when they find a problem?
“Integrates with your systems.” Which systems, through which supported interface, with permission to read or write what? Is any custom connector priced?
“Learns from your data.” Which data, collected under what terms? Does it train a shared model? Is there a data-readiness check before go-live?
Accuracy figures from the demo. Measured on whose data? Can we test it on our own realistic cases, including the exceptions?
“Live in a few weeks.” Does the clock start before or after access, security review, and account provisioning?
“Support included.” What’s a defect and what’s new scope? For how long? Who monitors it after that?
A license fee and a build fee. What will usage cost, who pays it, and what’s the estimate from testing? Who maintains it?
“Hosted on our platform.” If we stopped in eighteen months, what leaves with you? What can we export?

Get a Second Read Before You Sign

Gaps in a proposal aren’t proof of bad faith. Treat them as questions meant to strengthen the work: is this coming in a later phase, or has nobody decided yet? Send them to the builder. Some answers will be “not yet.” “Not yet” saves teams from expensive detours disguised as roadmaps.

If you’d like someone independent to look at it with you, bring it to a Working Session. We read it together on a 90-minute call, and you leave with a named constraint and a first step with an owner. There’s no document review before or after the call, so bring the person who’ll run the workflow. If proposals and vendor claims keep arriving faster than your team can evaluate them, Ongoing AI Advisory is a standing second read: written comments on the material and a decision log that stays with you. I’m independent of vendor incentives, so the read doesn’t depend on which tool you choose.

The proposal tells you what you’re buying. The gaps tell you what you’ll own.

You’re Not AI-Ready. One Workflow Might Be.

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

You have the AI mandate now.

Maybe the founder asked what the company is doing with AI. Maybe a vendor demo made your current workflow look painfully manual. Maybe your team has tried ChatGPT, Copilot, and a few automations, and now everyone has a different idea about what should happen next.

The pressure is real. So is the risk of spending money on a tool that creates another disconnected system, another unclear process, or another pilot nobody owns.

The first AI decision is not which tool to buy. It’s whether you can name the problem well enough to solve it.

The expensive failure happens before anyone compares models. A tool gets bought before the organization has decided what work should change, what information the system can use, who’s accountable for the result, and how anyone will know the workflow got better.

Start With the Decision

A useful AI initiative starts with a business decision or workflow that’s slow, inconsistent, repetitive, or poorly supported today.

Ask:

What decision or task is difficult right now?
Who does it, and who feels the consequence when it goes wrong?
What information does the work depend on?
What would a meaningful improvement look like?
What failure would be unacceptable?

That moves the conversation from “Where can we use AI?” to “Where would assisted judgment improve the work?”

The Tool Is Only One Possible Move

Once the problem is clear, choose the intervention rather than the brand name.

Move	Choose it when	Example
Improve the process	The work is unclear or inconsistent before technology enters it	Standardize an intake, approval, handoff, or exception process
Repair the data	The workflow depends on incomplete, conflicting, or inaccessible information	Resolve source-of-truth conflicts, clean identifiers, document definitions
Automate with rules	The task follows stable conditions and predictable actions	Route an approved request, move records between systems, schedule a notification
Use AI assistance	The work benefits from categorization, summarization, retrieval, drafting, or bounded judgment support	Search an approved knowledge base, classify records, generate a reviewed first draft
Keep it human-led	The task depends on empathy, intuition, human connection, or hands-on assessment	Sensitive service interactions, complex real-world assessment, final consequential judgment

Sometimes the answer is a better intake form, a documented procedure, cleaner reporting definitions, or data repair before anything intelligent can help. That’s the right order of operations, even when it lands smaller than the answer you were promised.

The best answer improves the work while staying proportionate to the risk, maintainable by the team, and defensible to the people responsible for the outcome. That’s often less technical than the option currently on the table.

Don’t Buy Yet If

Pause the purchase conversation when the missing work is diagnostic rather than technical. A new model, platform, agent, or subscription can’t decide what the workflow is for, reconcile competing definitions, establish permission to use data, assign ownership, or create the evidence proving an output improved the work.

So pause if any of these are true:

Nobody can name the workflow, owner, and outcome the tool is meant to improve.
The information the system needs lives in several places and nobody knows which version is current.
The team can’t describe what a bad answer would look like.
A vendor promises autonomous outcomes without showing how people will review, correct, or stop the system.
The plan ends at the demo and doesn’t include training, access, maintenance, cost, or accountability.

Every one of these points to the same next investment: discovery and design, rather than another tool.

If a vendor can’t explain how the system will be evaluated, maintained, corrected, and retired when conditions change, the proposal is incomplete however impressive the demo was.

Readiness Is a Property of One Workflow

AI readiness isn’t a score, a certification, or something an organization either has or lacks.

It’s specific to the use case. A team may be ready to summarize approved internal material or search a well-maintained knowledge base while being nowhere near ready for a workflow that handles sensitive customer information, writes to operational systems, or produces customer-facing recommendations.

The useful question is narrower. What has to be true for this particular workflow to improve the work without creating a problem you can’t explain, test, or manage?

Readiness belongs to the broader system: the workflow, the data and context, the controls, the people, and the organization’s ability to learn from use.

Readiness condition	A credible answer looks like	An unclear answer suggests
Problem	A specific task or decision is slow, repetitive, inconsistent, or poorly supported	The team is shopping for a capability rather than solving a defined problem
Outcome	The team can name what should improve and how it will be observed	The pilot becomes a demo with no viable success condition
Context and data	Sources, definitions, permissions, and limitations are understood	The system produces plausible output from incomplete or unsuitable information
Workflow	Inputs, handoffs, exceptions, downstream actions, and human roles are visible	Automation adds a disconnected process rather than improving the existing one
Ownership	A named person owns the decision, maintenance, monitoring, and escalation	The workflow becomes an orphaned pilot or a responsibility nobody can exercise
Evidence and controls	The system can be tested with realistic examples, boundaries, and pause conditions	The team is relying on a demo or a confident output rather than proof

Six separate questions, with different owners and different fixes. Keep them separate long enough to make the next decision clear.

NIST’s AI Risk Management Framework and its Generative AI Profile offer a useful lifecycle: govern the work, map context and risks, measure performance and impacts, manage the system over time. The framework is voluntary, and AI RMF 1.0 is currently being revised under the White House AI Action Plan, so treat it as a durable discipline rather than a fixed checklist. The discipline is useful even for a small company with no compliance function.

You don’t need a large data team to answer these questions. You do need someone responsible for answering them.

Human Review Needs a Job

“Human in the loop” means something only when the person in the loop can affect the result.

Meaningful oversight requires four things: subject-matter expertise to recognize a weak, misleading, or harmful output; enough access and context to add what the system doesn’t know; authority to override, reject, or change the output; and a documented escalation path when the issue can’t be resolved inside the workflow.

For low-risk work, that may mean initial review, structured feedback, sampling, quality thresholds, or monitoring for defined failure conditions. For a higher-impact use case, the reviewer may need formal decision rights, evidence matched to the consequence, and authority to pause the system.

A person who can’t challenge the output isn’t providing meaningful oversight. They’re being asked to approve a system they don’t control.

Some Work Shouldn’t Be Handed to AI

AI shouldn’t be the final decision-maker when the result materially affects a person’s life, livelihood, safety, rights, health, legal standing, financial standing, employment, or access to essential services. It may support research, preparation, analysis, or a qualified person’s judgment. It doesn’t replace the accountability of the person responsible.

It’s also a poor substitute for work depending on empathy, intuition, human connection, or hands-on assessment of a real situation.

Whether AI can produce an answer was never the question. The question is whether your organization should rely on that answer, in this context, with these people, under these conditions.

A Demo Is Not an Evaluation

A vendor demo shows what a tool can do under prepared conditions. It doesn’t show what the workflow will do with your definitions, exceptions, data quality, permissions, users, change history, and operating constraints.

Before relying on an AI-enabled workflow, define the context it may use, the sources it should prefer, the output it’s allowed to produce, the boundaries it must respect, realistic test cases, known failure conditions, the evidence required for a recommendation, and the conditions requiring human review or a pause.

The aim is an evaluation process that reveals whether the system improves the work, where it fails, and whether your team can catch and correct those failures before they become routine. Perfect prediction was never on offer.

Adoption Is Part of the Evidence

Accuracy, reliability, security, and cost matter. So do adoption, trust, accessibility, and whether the people doing the work experience a real improvement.

A system people route around hasn’t created much value. A system people use enthusiastically but can’t challenge or safely correct isn’t a success either.

Look at three levels. At the task level, did the work become faster, clearer, or less repetitive? At the workflow level, did the end-to-end process improve once review, exceptions, handoffs, maintenance, and rework are counted? At the outcome level, did the change improve a customer experience, a team capability, a decision, or a risk profile?

A small, bounded workflow that earns trust and teaches a company how to operate AI well can be worth more than a technically impressive proof of concept ending in a slide deck.

The Next Useful Move

If your internal AI efforts have stalled and you can’t quite name why, don’t respond by buying another tool.

Start with one workflow. Map what happens now, who does it, what information it depends on, where the exceptions are, what failure would look like, who owns the decision, and what evidence would show the work improved.

Then choose the next move: automate, improve the data, redesign the process, use bounded AI assistance, or leave the task human-led.

That diagnosis is valuable before the spend.

AI earns its place in the workflow when people can understand it, challenge it, and keep it running.

There’s No Such Thing as an AI Visibility Score

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

Somebody is going to try to sell you a number between 0 and 100 this quarter.

It’ll be called an AI Visibility Score, or an AI Presence Index, or something with the word “readiness” in it. It’ll go on a slide. Your CEO will ask why it went from 61 to 58, and you won’t have an answer, because nobody who built the number can tell you either.

I understand the appeal. Your boss asked whether you show up in ChatGPT. A single number is an answer you can give in a meeting. But the number is doing something specific under the hood, and once you see what it’s doing, you can’t unsee it.

One Number Means Averaging Things That Shouldn’t Be Averaged

Every composite visibility score works the same way. It takes unrelated observations, normalizes them onto a shared scale, weights them by a formula the vendor won’t show you, and returns an average.

Say your score is 58. That could mean your content is clear but your pages block the crawlers that feed AI answers. It could mean your pages are perfectly accessible but the claims on them have no supporting evidence, so nothing gets quoted. It could mean you’re represented accurately in one platform and described as a competitor’s subsidiary in another.

Four different problems, four different owners, four different fixes. Averaged into a 58, they’re indistinguishable. The score tells you that something is wrong while actively concealing which thing.

Then there’s the second-order problem. One number turns into a target. The moment a score goes on a dashboard, someone gets measured on moving it, and the fastest way to move a composite is to optimize whichever input is cheapest to game. That’s usually volume of published content, which is how you end up with forty thin pages and a rising score while your representation gets worse.

What You Can Observe

Plenty. Just not as one number.

Observation	What it tells you	What it doesn’t
Brand appearance in a defined prompt set	Whether you appear in the questions you chose to monitor	Whether the mention was accurate, influential, or repeatable
Description accuracy	Whether the answer represents your offer and category correctly	Whether that description drove a visit or a sale
Citation or source link	Whether an answer linked to one of your pages in that instance	Whether anyone saw, trusted, or clicked it
Technical access	Whether relevant crawlers can reach and render priority pages under your policies	Whether a platform will select the page for an answer
Generative-search impressions	Whether eligible pages appeared in Google’s reported generative surfaces	Why the page appeared, or what it was worth
Identifiable AI referrals	What attributable visitors from AI sources do on your site	Total AI influence, including unattributed journeys
Retention and customer feedback	Whether customers get the value the public representation implies	Whether AI exposure caused the outcome

These aren’t ingredients for a score. They’re different kinds of evidence, with different owners, time horizons, and decisions attached.

Two of them need more than a row.

Answer content, on a dated baseline. Define 15 to 25 questions reflecting how a prospective customer discovers, compares, and evaluates what you sell. Include category questions, problem questions, comparison questions, and questions that test whether the platform understands your business at all. Run the same set across the platforms that matter to your audience, in fresh sessions, on a defined cadence. For each response record the platform, date, whether you appeared, how you were described, whether that description was accurate, whether a source link appeared, and which competitors showed up.

One run is a snapshot. Repeated runs are a pattern. This is structured qualitative observation rather than a stable platform metric, and that distinction is a strength rather than a defect.

Whether relevant systems can reach and render your pages. Access is a prerequisite, not a citation guarantee. Crawlers and answer products differ in capability, policy, and rendering behavior, so don’t assume a page visible in your browser is equally usable to every system. Review robots directives, server responses, indexing controls, JavaScript-dependent content, and the specific crawler policies you’ve chosen. The conditions are usually checkable. The right remediation may be quick, or it may require decisions about rendering, infrastructure, security, and crawler policy.

A reasonable first audit

Fifteen questions, two answer platforms, a spreadsheet. Record results in fresh sessions on the same day, then repeat monthly before drawing any conclusion.

Don’t start by monitoring every possible prompt. Start with the questions that would matter if the answer were wrong.

What You Can’t Observe

I’d rather say this plainly than let it get smoothed over in a proposal.

Nobody outside these companies knows the retrieval and ranking logic behind a given answer. Including me. Anyone claiming to have reverse-engineered it is describing correlations they’ve observed, which is a reasonable thing to do and a different thing from knowing.

You can’t guarantee a citation. No content change, schema addition, or structural fix produces a promise that a platform will quote a specific page for a specific question. Any offer including that promise is selling something it can’t deliver.

You can’t compare across platforms on a shared scale. Each retrieves differently, updates on its own cadence, and weights sources by its own logic. A number flattening ChatGPT, AI Overviews, and Perplexity into one index has already lost the information you’d need to act.

Track These Separately, With Owners

If you need to report upward, use a short set of distinct observations with their own trend lines, owners, and decision implications. Question coverage. Description accuracy. Access status. Generative impressions. Referral volume and behavior.

Five numbers that stay five numbers.

The purpose isn’t a new dashboard category. It’s making the next decision legible: whether the issue belongs with content, the technical web team, analytics, product marketing, customer experience, or leadership. A change in question coverage should start a different conversation from a change in description accuracy. That’s exactly why they can’t be averaged.

What the Evidence Supports

Public research helps here when it narrows bad assumptions, not when it claims to expose a platform’s hidden ranking system.

Ahrefs’ analysis of 75,000 brands, published May 2025, measured Spearman correlations against AI Overview brand mentions. Branded web mentions came in at 0.664, branded anchors at 0.527, branded search volume at 0.392. The link metrics that dominated classic SEO trailed: Domain Rating at 0.326, referring domains at 0.295, backlinks at 0.218.

The practical implication is modest and useful: broad, attributable discussion of a brand appears to track more closely with AI Overview mentions than traditional link metrics do in this dataset. It doesn’t establish that producing more mentions causes AI systems to represent a brand more often or more accurately.

The study says so itself. Correlation isn’t causation, and the authors describe the full set of factors as showing “moderate to very weak correlations.” Branded mentions and AI visibility may both be downstream of simply being a well-known company.

Visibility You Can’t Deliver On Is a Different Problem

High visibility in AI answers doesn’t help if the experience underneath can’t sustain the expectation it creates. If AI referrals rise while conversion quality, retention, or sentiment stays flat or falls, investigate whether the description circulating about you is more generous, broader, or simply different from what customers encounter.

That may be a discovery issue. It may also be a positioning, product, onboarding, or expectation-setting issue that AI-mediated discovery is making visible faster.

Net Promoter Score, retention, support themes, and review language won’t explain why an AI answer appeared. They’ll tell you whether the attention it sends is becoming earned demand or amplified mismatch.

The Honest Version

You can build a defensible, dated, repeatable picture of how AI answers represent your company. It takes reading, a defined question set, and the discipline to keep the observations separate.

What you can’t build is a number that means something. And the number is the part everyone wants to buy.

Your AI Traffic Is Measurable Now. The Reports Still Miss a Lot.

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

For about two years, anyone measuring traffic from AI tools was building the instrument themselves. Custom segments. Referrer regex. A spreadsheet of user agent strings that went stale every few weeks.

That changed in 2026, with so little fanfare that plenty of teams are still running the workaround.

Here’s what shipped, what it covers, and where the gaps are. The gaps are the part worth your attention, because a report that looks complete and isn’t is more dangerous than no report at all.

The Platforms Caught Up

Google Search Console added a generative AI performance report in June 2026, covering AI Overviews and AI Mode by page, country, date, and device.

Google Analytics 4 added an AI Assistant default channel group. Traffic from recognized AI assistants now lands in its own bucket instead of getting mixed into Referral. Google has named ChatGPT, Gemini, and Claude as examples and hasn’t published the full recognized list.

If you built custom segments in 2024 or 2025, they’re probably still running alongside the native reporting and probably disagreeing with it.

Start With the Decision

Before you build an AI traffic report, decide what it’s meant to support.

Content discovery? Lead quality? Assisted conversions? Search visibility? Crawl activity? Whether a change in site traffic deserves investigation? Those questions are related. They aren’t interchangeable, and a report that blends them before the decision is defined will look comprehensive while failing to answer what leadership asked.

A content team needs to know which pages get identifiable referrals. A demand team needs qualified-lead rate by channel. A web or security team needs crawler access and request volume. None of those is answerable by one “AI traffic” number.

Which matters because the phrase covers at least four different signals:

Identifiable human visits arriving with an AI product referrer or campaign parameter
Google generative-search appearances reported in Search Console
Human visits plausibly influenced by an AI conversation but unattributable, often sitting inside Direct
Automated crawler and agent activity, which is operationally important and isn’t audience behavior

Don’t combine these because they all involve AI. They carry different evidence, different limits, and different owners.

What Each Signal Can and Can’t Support

Signal or system	Supports	Doesn’t support on its own
GA4 AI Assistant channel	Identifiable sessions from recognized AI referrers	Total AI-influenced traffic, stable coverage over time, causal impact
Custom AI referrer segment	Sessions matching your documented source logic	Complete capture, especially referrerless journeys
Direct traffic	A category of visits with no attributable source	An estimate of AI influence without corroborating evidence
Search Console generative report	Eligible appearances in Google’s generative surfaces	Clicks, sessions, conversions, or source-selection logic
Server, CDN, or bot-management logs	Evidence of automated requests and crawl activity	Human engagement, intent, or conversion
CRM and conversion data	Quality and downstream value of recorded outcomes	Discovery source when attribution is absent

A dashboard should show these boundaries rather than hide them. When the reports disagree, the disagreement is often the finding.

Two limits deserve calling out specifically. Search Console gives you impressions only, with no clicks or position data for generative surfaces and Search Labs excluded. And GA4’s channel catches only what carries a referrer, so assistant traffic arriving without one still lands in Direct. That covers in-app browsers, mobile apps, and anyone copying a link out of a chat into a new tab, which is a lot of people.

Where the Traffic Hides

In one historical analysis across a Drumline client base, the AI-influenced traffic we could identify scattered across three reporting buckets rather than concentrating in one: roughly 45% surfaced as referring-domain traffic, roughly 42% landed in Direct, and roughly 13% appeared inside Organic Search.

That’s not a benchmark to copy. It’s an illustration of the attribution problem: one user behavior can produce different observable traces depending on product, browser, device, referrer handling, and the path between a chat interface and your site.

Volume moved fast once we could see it. AI-driven visits were running at up to 3% of total site traffic, with some clients seeing that segment grow 5.4x across the first half of 2025. Small base, steep curve, and still modest enough that nobody notices without looking deliberately.

Treat any AI traffic number you produce as a floor. It’s the traffic you could identify, not the traffic that happened.

What This Traffic Is Worth

First, confirm the outcome you care about is being recorded reliably. Qualified leads, purchases, subscriptions, applications, demo completions, renewals, whatever matters. Review duplicate events, consent effects, cross-domain journeys, source attribution, and the CRM handoff if the business case depends on it.

A new channel grouping can’t repair a conversion event that’s been double-counting for a year. It can’t fix a broken cross-domain journey. It can’t turn an undefined “lead” into evidence of commercial value. If the foundation is unreliable, fix that before using AI traffic to make an argument.

Then the quality question. You’ll see claims that AI-referred visitors convert at four or five times the rate of other traffic. Those figures trace back to a small number of vendor analyses whose methodology isn’t published. I wouldn’t put them in a business case.

AI products can absorb some informational work before a user reaches you. In some journeys that means an identifiable AI-referred visitor arrives with a narrower question, a more developed comparison set, or more readiness to act. In others, the answer satisfied the user without producing a click at all.

Treat “AI-referred visitors are warmer” as a hypothesis rather than a property of the channel. Segment the human visits you can identify and compare them against relevant baselines: organic, referral, Direct, paid, or comparable landing-page cohorts. Look past a single conversion rate to qualified lead rate, opportunity creation, revenue, return visits, or whatever fits your model. Hold the result loosely. Small volumes, changing referrer behavior, seasonality, and landing-page differences make channel comparisons look more conclusive than they are.

What isn’t in dispute is that the click environment got harder. SparkToro’s analysis of Similarweb clickstream data found 68.01% of US Google searches ended without a click between January and April 2026, up from 60.45% in 2024, with AI Overviews on more than 20% of searches and cutting click-through rates by nearly 60% where they appear. Pew found the same behavioral pattern: people click less when an AI summary is present.

This is also why click-through rate stopped working as a health metric alone. One Drumline client saw impressions double while CTR declined over the same period, which reads as failure on a CTR dashboard and as a visibility win on an impressions one. Both numbers were accurate. One of them described what happened.

Separate the Bots, and Keep Letting Them In

Two jobs that get confused with each other.

The first is access. AI crawlers need to reach your pages for your content to inform AI answers. Check robots.txt against the current crawler landscape, because a lot of sites added blanket blocks during the 2023 scraping panic and never revisited them. Access and training are separate permissions for some platforms: several operators use different user agents for crawling that feeds live answers versus crawling that feeds model training, and you can allow one without the other.

Don’t make that decision in an analytics silo. It touches security, infrastructure, legal, content governance, and brand strategy. Document what’s allowed, what’s blocked, why, who approved it, and when it gets reviewed.

The second job is hygiene. Crawler visits shouldn’t sit inside your human engagement metrics.

Don’t rely on a user-agent list alone for either job. User agents change, can be spoofed, and may be stripped before reaching your analytics platform. Use known agents as one input alongside server logs, CDN or bot-management signals, request patterns, rate anomalies, and current vendor documentation. Keep two views: a human-behavior view that excludes known automated activity, and an operational crawler view for access, crawl load, and site-impact decisions. Don’t mix either into the other.

Five Things Worth Doing This Month

Define the decision before the dashboard. Name whether you’re after identifiable referrals, conversion quality, search appearances, crawl activity, or a specific campaign. Assign an owner and a review cadence.
Validate the measurement foundation. Confirm the conversions and source data are reliable enough to build on. Check consent effects, duplicate events, cross-domain paths, and the CRM handoff.
Reconcile custom segments against the native GA4 channel. Document both definitions. If they disagree, find out whether the gap is referrer coverage, changing source recognition, or an implementation difference, before either number reaches a slide.
Review crawler policy and reporting hygiene. Check robots directives against current platform documentation. Separate automated activity from human engagement reporting while keeping an operational view of crawl requests.
Turn on and trend the Search Console generative report if your property has access. Treat it as an impressions signal for Google’s covered surfaces, not a traffic report.

Do those before you buy a monitoring tool. Most of what a tool would tell you is sitting in platforms you already own.

The Part That Stays True

The measurement caught up enough to be useful and not enough to be complete. Anyone reporting AI traffic as a precise figure is reporting the identifiable portion and calling it the total.

Say the number with the caveat attached. The caveat is what makes the number worth having.

The Browser Didn’t Force Your Hand. Your AI Roadmap Will.

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

For about four years, every data governance conversation in marketing ran on the same engine: third-party cookies are going away, so get your first-party data in order.

It was a useful fiction while it lasted. Deadlines are motivating, and that one had a date on it.

Then Google reversed the deprecation in April 2025 and kept third-party cookies in Chrome. In October 2025 it retired most of the Privacy Sandbox APIs, including Topics, Protected Audience, and Attribution Reporting, citing low adoption. The replacement architecture a generation of measurement roadmaps was built around mostly doesn’t exist.

If your governance program was justified on the cookie deadline, the justification evaporated. The work didn’t.

The Pressure Just Moved

Three things kept going while everyone watched Chrome.

State privacy law stopped being a coastal issue. Roughly twenty states have comprehensive laws in effect in 2026, with Indiana, Kentucky, and Rhode Island coming online January 1. The obligations aren’t uniform. Rhode Island skips universal opt-out mechanisms other states require. Oregon now requires recognition of universal opt-out signals and a full stop on geolocation data sales. You can’t write one policy and call it covered.

The point isn’t that every marketing or AI team becomes its own legal department. It’s that legal, privacy, security, marketing, analytics, product, and technology can’t each hold a partial view of the same data use and assume someone else has the whole picture.

Automated decision-making became a regulated category. California’s rules now require opt-outs when automated decision-making technology replaces or substantially replaces human judgment in significant consumer decisions, plus risk assessments and cybersecurity audits for high-risk processing. Whether a particular workflow triggers an obligation depends on jurisdiction, organization, data, decision, and implementation. The operational lesson is broader: once automation meaningfully shapes consequential outcomes, you need documented accountability and a way to challenge or stop the process.

A fifth of the browser market never waited. Safari has blocked third-party cookies since 2020. Firefox partitions by top-level site. Brave blocks by default.

None of that produces a deadline for a slide. All of it produces the same requirement: know what data you hold, why you hold it, who agreed to it, and whether you can remove it.

That’s governance in its useful form. The visible ownership, definitions, provenance, permissions, quality checks, and escalation paths that let a team use information in ways it can explain and defend. Compliance is one outcome. Better reporting, safer automation, and more dependable AI are others. You don’t need a complete enterprise program to begin. You need enough structure that a person facing a real decision can tell which data to use, what it means, who’s accountable, and what happens when the answer is uncertain.

AI Is What Makes the Gap Expensive

Ungoverned data used to produce bad reporting. Somebody pulled a number, the number was off, an analyst caught it, and the cost was a week of rework.

Ungoverned data now feeds systems that generate outputs at volume, with confidence, to people who won’t check. The failure mode moved from a wrong number in a deck to a wrong answer in a workflow, repeated a few thousand times before anyone notices.

Gartner’s February 2025 research predicted organizations would abandon 60% of AI projects unsupported by AI-ready data through 2026, drawing on a July 2024 survey of 1,203 data management leaders in which 63% either lacked appropriate data management practices for AI or weren’t sure. That window has closed and I haven’t seen a credible retrospective, so treat it as a well-grounded warning rather than a verified outcome.

What I can say from the work: when an AI initiative stalls, the cause is almost never the model. Nobody can say where a field came from, which definition of “customer” a table uses, or whether the records were collected under terms permitting this use.

That’s not a technology problem. That’s a foundation problem.

“AI-ready” doesn’t mean perfectly clean data or a finished architecture. It means the team can answer a short list:

What purpose does this workflow serve, and who owns the outcome?
Which sources and fields does it use, what does each mean, and where did it originate?
Are the data and its intended use permitted under the relevant collection terms, contracts, and policies?
Who can review, correct, pause, or stop it?
How will anyone know when the output is wrong or causing harm?

If those answers are missing, the model doesn’t need better prompting. The workflow isn’t governable yet.

Not Every Workflow Needs the Same Controls

Governance should be proportional to risk. Not every spreadsheet needs a review board, and not every AI experiment needs the controls appropriate for a workflow affecting employment, credit, health, housing, pricing, or eligibility.

Raise the level of review when the data is more sensitive, the decision carries more consequence for a person, an external vendor receives the information, an action is hard to reverse, or an error would be hard to detect before it reaches people.

That isn’t an argument for slowing everything down. It’s how a team moves fast on low-risk work without pretending every use of data is equally safe.

Four Things, in Order

The version of this article I wrote two years ago had seven steps. Seven steps is a way of avoiding the harder work of saying which comes first.

1. Map what you have and where it came from.

Not a full catalog. A working inventory of the systems feeding anything customer-facing, decision-shaping, or connected to an AI workflow. For each critical source, record what it contains, what the important fields mean, where the data originated, the collection or permission basis, who can access it, where it flows next, how long it’s retained, and who is accountable for it.

You’re hunting for three things: places where nobody can answer the provenance question, places where two systems use the same word differently, and datasets that have spread farther than anyone can trace.

The middle one does the most damage. “Customer” may mean a closed-won deal in the CRM, an active subscriber in billing, a user with product activity in the last 30 days, or anyone who ever submitted a form. An AI assistant won’t resolve that disagreement. It’ll reproduce whichever definition was easiest to connect.

If the estate is large, automated governance and auditing tools can inventory tags, collection, access patterns, and implementation drift at a scale that isn’t manual-feasible. Choose based on the systems you operate and the evidence you need, not because a category label promises AI readiness.

2. Name an owner for every consequential decision.

Most governance frameworks fail here. They document policies and leave decision rights implicit, so a real question routes to whoever is available rather than whoever is accountable.

At minimum, document who owns five decisions: approving a new data source, approving a new use of existing data, changing a consequential definition, granting or changing access, and pausing or retiring a workflow producing unreliable output.

Ownership isn’t a mailing list. It means a named person or role has the authority, context, and obligation to decide or to escalate through a known path. Include the authority to say no. A reviewer who can’t stop something is decoration.

3. Make deletion work before you need it.

Deletion requests are the cheapest test of whether governance is real, because the request either gets fulfilled across every system or it doesn’t. Run one against yourself. Pick a record and follow it through the warehouse, the CDP, the email platform, exports, reverse-ETL destinations, analytics tables, audience lists, vendor systems, manual spreadsheets, backups, and any AI or automation workflow that received it or a derivative.

The point isn’t a perfect audit in one afternoon. It’s revealing where your understanding of data flow stops.

Record the result. For every location you can’t reach, note the system owner, the reason, the remediation path, and the next review date. A discovered gap becomes governance work only when someone owns closing it.

4. Write the plan for when it fails.

Governance documents describe the intended state. What’s usually missing is the response when the intended state fails.

Write down what happens when a workflow produces an inaccurate answer, uses the wrong source, exposes information it shouldn’t, applies an outdated definition, or can’t explain how it reached an output. Name who can pause it, who gets told, how fast action is expected, what evidence is preserved, and who communicates with affected teams or customers.

This doesn’t need to be long. It needs to work when people are under pressure and the person who built the workflow is unavailable.

For any AI workflow reaching customers, employees, decisions, or operational systems, add a one-page map before scaling: purpose, owner, input data, source systems, output users, connected vendors, expected action, evaluation method, human decision rights, escalation path, and the conditions requiring a pause. That’s the minimum needed to answer a basic question afterward. What did the system use, what did it produce, who relied on it, and who was watching?

Start Small

If you need a place to begin this month, don’t launch a governance program. Pick one data domain or workflow that already causes rework, disagreement, or uncertainty.

Resolve one definition dispute. Trace one consequential field to its source. Name the person who can approve or halt its use. Test whether one record can be deleted across its known destinations.

The first useful governance artifact is a map, not another tool.

The Argument That Holds Up Now

The cookie deadline was a good forcing function and a bad reason. It made governance a project with an end date, which is the wrong shape for an operating capability.

The reason that holds up is simpler. You’re about to connect systems that generate output at a scale no human reviews line by line. What comes out is bounded by what you can vouch for going in. And the ability to vouch for it is something you either built beforehand or explained the absence of afterward.

Nobody’s imposing a deadline on that one. That’s what makes it easy to defer and expensive to defer.

The Integration Tax Just Got Cheaper

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

Martech stopped growing this year.

The 2026 landscape counts 15,505 solutions, up 121 from last year’s 15,384. That’s 0.79% growth after a decade of double digits. Underneath the flat number there’s enormous churn: about 1,488 products added, about 1,367 removed. Scott Brinker and Frans Riemersma call it martech’s Darwin phase. More than half the departures came from the 2010 to 2019 SaaS wave, and the mid-market is getting squeezed between incumbents bundling everything and AI-native startups unbundling it.

For years the argument for composability was about abundance. Too many options, so build a stack you can rearrange.

That argument is over. The new one is more interesting, and it changes what you should buy.

The Suite Was Always a Bet About Integration Cost

Every all-in-one purchase comes down to the same trade. You accept constraints on capability in exchange for integration you don’t have to build.

That was defensible when integration was expensive. Connecting a best-of-breed testing platform to your CDP and your warehouse and your email tool meant engineering time you didn’t have, a schema mapping exercise nobody wanted to own, and a maintenance burden that appeared every time a vendor changed an API. The suite’s pre-built connections were worth real money because building them yourself cost real money.

So companies bought the suite. Then spent three years discovering which components they’d never have chosen on their own.

What AI Changed

The cost of interfacing between systems dropped, faster than most stack decisions have caught up with.

Schema mapping used to be a discovery project. Now it’s something you draft in an afternoon by pointing a model at both schemas and reviewing what it proposes. Transformation logic that took a week is a first draft in an hour. API documentation that needed a specialist is something a competent marketer can work through with assistance. The glue code between a workflow tool and an unfamiliar endpoint is no longer the bottleneck.

Agent-mediated tool access pushes this further. When systems expose their capabilities in a standardized way a model can call, integration work moves from custom code per connection toward configuration and permission design. That shift is early and uneven across vendors, and it’s real enough to plan around.

Add it up and the suite’s central value proposition weakened. You’re still paying the constraint cost of the all-in-one. You’re getting less in return, because the thing it was saving you from got cheaper.

Cheaper doesn’t mean free, automatic, or low-risk. Production integration still requires authentication, permissions, testing, observability, error handling, change control, documentation, and someone who can support the connection after launch. The point isn’t that integration disappeared as a cost. It’s that integration is less often the dominant reason to accept a tool’s constraints without examining the alternatives.

What AI Didn’t Change

This is the part skipped in the pitch, and the part that determines whether any of it works.

AI can generate the connector. It cannot decide what “customer” means in your business.

It can map a field from one system to another. It can’t tell you the two systems define an active account differently, that one excludes trials and the other doesn’t, and that reconciling them is a business decision somebody has to make and document.

It can write transformation logic quickly. It can’t tell you whether the data going through it was collected under terms permitting this use.

It can connect two tools in an afternoon. It can’t tell you whether connecting them creates a compliance exposure, a circular dependency, or a system nobody can explain in six months.

Every one of those is a definition, a decision, or an ownership question. The same ones as 2019. The cost of the plumbing went down. The cost of getting the semantics wrong went up, because now the wrong semantics propagate through automated workflows at a speed no human reviews.

Choose the Move Before You Choose the Tool

Before treating a suite renewal or a new AI capability as a product decision, decide which kind of decision it is.

Move	Choose it when	Watch for
Buy	A proven product fits a defined workflow and your team can operate it	Paying for capability you won’t use, or letting the product become the only place critical business logic lives
Build	The capability is central to your advantage, requirements are specific, and you can maintain it	Writing custom software for a problem a mature product already solves
Connect	Existing tools can solve it if data, handoffs, permissions, and ownership are reliable	Brittle integrations with no monitoring, error process, or named owner
Skip	The workflow, data, owner, or success measure is unclear	Buying software to avoid diagnosing the problem

The fourth option is underrated. “Not yet” saves teams from expensive detours disguised as roadmaps.

Own the Place Where Data Gets Stitched Together

If integration is cheap and definitions are expensive, the thing worth owning is the layer where definitions live.

One place where your customer data gets unified, where a field means one thing, where identity resolution logic is explicit, and where you can name the source of truth without a meeting. A warehouse, a CDP, a data layer, or a combination, depending on what you already run.

Vendor-neutral doesn’t necessarily mean self-hosted, custom-built, or centralized in one product. It means the organization can access, understand, govern, export, and migrate the definitions and history that make the business intelligible. Identity rules, event definitions, consent states, campaign taxonomy, and critical metrics shouldn’t exist only as opaque configuration inside one application. For one company that coordination layer is governed warehouse data. For another it’s a well-maintained CRM, event pipeline, or integration layer. Architecture should follow the use case and the operating capacity rather than a fashionable reference diagram.

The real unit of composability is the interface. For every connection that matters, document what moves between systems, what each critical field means, which system is authoritative, what triggers the handoff, how errors get detected, who owns it, and what happens when a vendor changes an API or a consent rule changes.

That’s a data contract. It doesn’t need to be elaborate. It needs to exist somewhere other than the memory of the person who built the workflow.

A stack isn’t composable because it has APIs. It’s composable when you can change one part without losing the meaning, history, permissions, and operating knowledge attached to the connection.

Four Risks Worth Naming

Generated connector code still needs an owner. A model writes an integration in an hour. Somebody still has to understand it, test it, and fix it at 11pm when the vendor changes a field. Code you didn’t write and don’t understand is technical debt regardless of what produced it.

The standard isn’t that every marketer explains every line. It’s that the organization has a named technical owner, accessible documentation, test coverage matched to the consequence of failure, monitoring for breakage, and a recovery path. AI-generated code lowers the cost of starting. It can raise the cost of operating if it lets teams create more connections than they can maintain.

Easy connection makes governance harder. When connecting two systems required a ticket and an engineer, the friction was accidental governance. It slowed things down and created a checkpoint. Remove the friction without replacing the checkpoint and nobody can produce an accurate diagram of what’s connected to what.

Replace accidental governance with deliberate, light controls: a named owner before a connection goes live, a documented purpose and data flow, least-privilege access, an approval path for sensitive uses, monitoring for failed handoffs, and a periodic review of whether the connection still serves a real need. The goal isn’t a ticket queue for every low-risk workflow. It’s making sure speed doesn’t outrun your ability to explain, repair, or retire what you connect.

The tool you plug in may be gone next year. Composability is the defense, and it works only if you kept the coordination layer neutral instead of letting the newest tool gradually become the place where definitions live.

Bundling is a real counter-move. Incumbents are folding AI capabilities into the suite at prices that make standalone tools hard to justify. Sometimes the suite is the right answer. Choose it because it wins on merit rather than because rearranging felt too expensive.

License Cost Is the Visible Part

Every additional tool can add authentication, permissions, monitoring, data-quality checks, security review, access management, training, documentation, incident response, vendor-change work, and support.

AI reduces implementation effort. It doesn’t eliminate operating responsibility after launch. A connector that takes an hour to draft can still become an undocumented dependency costing a team days every quarter.

Count the work before declaring a tool cheap. The cost of a stack is the subscription total plus the people required to make it reliable.

How I’d Decide

Question	What a concerning answer reveals	What to do next
If we removed this tool in eighteen months, what leaves with it?	Definitions, identity logic, history, consent states, or reporting logic live inside a vendor you can’t readily replace	Identify which business rules must be extracted, documented, or made portable before renewal
What does this tool do that we couldn’t reach through the layer we already own?	“Integration depth” may be standing in for a capability claim untested against current implementation cost	Compare differentiated capability, total operating cost, and implementation risk against a connection you could maintain
Who owns the connection after it exists?	You reduced build effort by creating an unowned future liability	Assign an owner, support path, monitoring approach, and review cadence before launch
How do we exit if the vendor, pricing, API, or consent rule changes?	The team treats adoption as reversible without having designed for reversibility	Define export, migration, and retirement requirements before dependence deepens

If the honest answer to several of these is “we aren’t sure,” the next investment is discovery and design rather than another subscription.

Start With One Workflow

Don’t begin by diagramming every tool in the company.

Choose one high-value workflow: lead capture to CRM, product event to customer communication, attribution to reporting, or an AI-assisted workflow that reads from and writes to business systems. Map the applications, identifiers, fields, handoffs, permissions, owners, failure modes, and downstream actions. Then ask which parts are portable, which are undocumented, and which would be difficult to replace.

The first useful deliverable usually isn’t a future-state architecture. It’s an honest map of the dependency you already have.

The Short Version

The landscape stopped expanding and started turning over. Integration got cheap. Definitions got expensive.

Own the layer where meaning lives. Rent everything else.

The Question Legal Asks After the Demo

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

Your data science team just showed a model trained on millions of customer records. The accuracy numbers are good. The room is happy.

Then legal asks one question.

“Can we document consent for all the training data?”

I’ve watched a version of this moment more than once, and the tell is always the same. Nobody says no. Somebody says they’ll look into it.

The question may be consent, another documented lawful basis, contractual permission, a retention obligation, a sector restriction, or simply whether the data was ever authorized for this use. The underlying problem doesn’t change: can you show why each material input belongs in the workflow?

The Answer Determines Your Timeline

At Drumline, I worked with a financial services organization that restarted a fraud detection initiative from the beginning. The model worked. What ended it was discovering sensitive data in the training set that nobody could account for.

The second part is what cost them. Removing the data would have been a week. Being unable to reconstruct the lineage meant they couldn’t demonstrate the remaining data was clean either, which made the whole set suspect. Deployment slipped roughly eight months.

The model was never the problem. The inability to answer a provenance question was the problem, and it surfaced at the worst possible moment, which is the only moment it ever surfaces.

What “Privacy-Safe” Means in Practice

Privacy-safe data is a set of properties, not a volume reduction. Five of them, each a question you can either answer or you can’t.

Intentional. You can say why each element is there. Not “it was available.” A reason tied to the question the model answers.

Permissioned. Every individual whose data is in there provided consent or falls under a documented lawful basis, and you can produce the documentation.

Clean. Validated, deduplicated, and checked for the biases that matter to your use case. This one pays for itself independent of regulation, because it’s the difference between a model that behaves in production and one that behaved in testing.

Auditable. You can trace which data trained which model, who accessed it, and when. Reconstructible after the fact, rather than remembered by the person who built it.

Removable. When someone requests deletion, you can find their data across systems and remove it, including from anything derived from it.

The last one is where most organizations discover the gap. Deletion from a warehouse is straightforward. Deletion from a model trained on that warehouse is a different problem with no clean answer, which is why the intentional and permissioned properties matter so much upstream.

Before production use, define what deletion means for your architecture specifically. It may involve removal from the operational source system, an analytics table, a retrieval or vector index, suppression from future use, deletion from vendor logs where contractually available, retention limits, retraining, or a documented exception where complete removal from a trained model isn’t technically feasible.

The standard isn’t a universal delete button. It’s that you can explain accurately what you can remove, what you can’t, how fast you can act, who’s responsible, and how that reality appears in contracts, notices, and internal process.

Make the Use Case Legible First

Before a workflow reaches production, write a one page record for it: purpose, named owner, intended users, model or vendor, source systems, material data categories, data that must be excluded, expected output, decision impact, retention expectation, review method, escalation contact, and the conditions requiring a pause.

The document isn’t the control. It’s the fastest way to discover whether the control, the decision, and the owner exist.

If the team can’t fill in a field, don’t paper over the blank with a policy statement. Treat it as a design question. The missing answer may mean narrower inputs, a different architecture, a different owner, a legal review, or a decision not to proceed yet.

The Vendor Doesn’t Own Your Data Decision

A vendor contract allocates obligations. It can’t replace your understanding of the data flow.

Before connecting customer, employee, or otherwise sensitive information to an AI vendor, ask:

Will our data train a shared model or improve the service for other customers?
Where is it processed and stored, and which subprocessors receive it?
What controls exist for access, retention, deletion, and export?
What logs, audit records, security documentation, and incident-notification commitments can they provide?
What happens if the model, product terms, data-processing terms, or subprocessor list changes?
Can we prevent specific data classes from entering the workflow at all?

If nobody on your side can explain what leaves your systems and where it goes next, the integration isn’t ready just because the authorization screen was easy to complete.

The Regulatory Picture Keeps Moving, and Not Only Forward

Here’s where I’d push back on the usual framing. The standard version says regulation is tightening on a known schedule, so build now or pay later. The schedule part isn’t holding up.

The EU AI Act’s high-risk obligations were postponed. Under the Digital Omnibus agreed in May 2026, standalone Annex III systems moved from August 2026 to December 2, 2027. Embedded Annex I systems moved to August 2028. What didn’t move: the Article 5 prohibitions, in force since February 2025, and the Article 50 transparency obligations on AI-generated content, which still landed August 2, 2026.

Colorado repealed its AI Act before it took effect. In June 2026 the state replaced the CAIA with the narrower Automated Decision-Making Technology Act, dropping the algorithmic discrimination obligations for transparency and documentation requirements. It takes effect January 1, 2027, contingent on rulemaking. The new law avoids the term “AI” entirely.

California’s disclosure rules landed late but landed. SB 942 became the California AI Transparency Act as amended by AB 853, operative August 2, 2026 rather than January 1, with a phased rollout through 2028.

State privacy law kept expanding regardless. Roughly twenty comprehensive laws are in effect, and California’s automated decision-making rules now require opt-outs when those technologies replace human judgment in significant consumer decisions.

Whether any particular requirement applies depends on jurisdiction, organization, system design, data, and decision. This isn’t legal advice. The operational lesson is broader: a team that can trace data, explain its use, document decisions, and stop a problematic workflow adapts better than one building only to the date circled on a compliance calendar.

Direction is consistent. Timing is unreliable in both directions. Build to the obligation rather than the deadline.

Techniques Are Architecture Choices, Not a Program

Before choosing a privacy-enhancing technology, ask a simpler question: should this data enter the workflow at all?

The answer may be to minimize the input set, exclude a sensitive category, use aggregated or de-identified information, narrow the purpose, constrain the outputs, or decide the use case isn’t justified. Privacy engineering is an architecture choice rather than a substitute for purpose limitation.

When it is the right move, several methods are now practical without a research team. Federated learning trains across distributed sources without centralizing data, which is how multi-entity organizations build a shared model without any party seeing another’s records. Differential privacy adds calibrated noise so models learn patterns without exposing individuals. Secure multi-party computation lets organizations contribute to a shared model without revealing inputs to each other. Edge processing keeps data where it originated.

Each creates tradeoffs in accuracy, cost, latency, operational complexity, and governance. Don’t pick one because it sounds privacy-preserving in the abstract. Start with the exposure problem, define the system and the people affected, then decide whether the control is proportionate.

Which also means matching controls to risk generally. A low-risk internal drafting aid using approved reference material doesn’t need the process appropriate for a system influencing eligibility, pricing, credit, employment, or access to services. Raise the controls when the data is more sensitive, the decision carries more consequence, a vendor receives the information, an action is hard to reverse, or an error would be hard to catch before it causes harm.

And know which system you’re governing. “AI system” can mean a trained model, a retrieval-augmented assistant, a classification workflow, an agent that calls other tools, or a vendor feature with customer data attached. A retrieval index creates one deletion problem. A fine-tuned model creates another. An agent that takes actions introduces permissions and logging questions a read-only assistant doesn’t. Don’t borrow a control list from one architecture and assume it answers another.

Three Things Worth Doing

1. Map the data behind one consequential AI system. Pick one workflow reaching customers, employees, material decisions, or sensitive information. Identify what it uses, where every material source came from, the collection or authorization basis, what the data means, where it flows, and whether you can produce documentation. Stop at the first gap you can’t close. That gap is the honest initial scope.

2. Run a deletion request against yourself. Pick one record and trace it through source systems, warehouses, analytics tables, exports, retrieval indexes, vendor tools, audience lists, backups, and known derived uses. Record what can be deleted, what can only be suppressed, who owns each location, and what needs remediation.

3. Decide who can stop a deployment. Define the pause conditions: unsupported provenance, unauthorized data, a harmful output, a security incident, a quality failure, an unresolved complaint. Name the person who can halt it, specify the escalation path, and make sure that authority is real rather than ceremonial.

What I’d Say to the Room

The moment legal asks the question, the answer already exists. You’re either the organization that can produce documentation or the one spending eight months reconstructing it.

Nothing about the model changes that. It was settled upstream, by decisions nobody was in the room for.

You Put a Human in the Loop. Can They Say No?

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

“There’s a human in the loop.”

It’s the sentence that ends the risk conversation. Somebody raises a concern about an AI workflow, somebody else says a human reviews the output, and the room relaxes. It goes into the policy. It goes into the vendor response. It goes into the slide shown to the board.

Then the workflow ships, and the human in the loop turns out to be a person with forty other things to do, looking at a queue of outputs that are right most of the time, with no clear sense of what they’re catching or what happens if they stop the line.

That’s not oversight. That’s a person standing next to a process.

First, Decide Whether Review Is the Right Control

A human review step isn’t automatically the answer to AI risk.

Before designing one, decide what kind of work this is. Some tasks should stay human-led because they depend on empathy, intuition, human connection, or hands-on assessment of a real situation. Some are better handled by rules-based automation because the inputs, conditions, and actions are stable. Some can use bounded AI assistance with testing, monitoring, and feedback rather than item-by-item review.

Review becomes necessary when a person needs to assess an individual output before it creates a consequence, and when that person can realistically improve the decision.

The question isn’t “Where can we add a human?” It’s “What control makes this workflow safer, more reliable, and more useful?”

Europe Wrote Down a Useful Minimum Standard

You don’t need to be subject to the EU AI Act to learn from its human-oversight standard. Article 14 describes the capabilities required for people assigned to oversee high-risk AI systems. Its legal applicability depends on the system, organization, role, and jurisdiction. Its operating logic applies much more broadly.

The person must understand the system’s capacities and limitations and monitor its operation. They must correctly interpret the output. They must remain aware of the tendency toward “automatically relying or over-relying on the output,” which the Act names as automation bias. They must be able to decide, in any particular situation, not to use the system or to “disregard, override or reverse the output.” And they must be able to intervene or interrupt through a stop button or similar procedure.

Read that against your own workflow. The last two are where most implementations come apart, because they require somebody to have given the reviewer real power over a process other people are counting on.

Under the Digital Omnibus agreed in May 2026, the high-risk obligations carrying Article 14 were postponed to December 2027 for standalone systems. The legal timetable may keep moving. The design question doesn’t: if you claim a person is overseeing an AI-enabled process, can that person recognize a problem, act on it, and cause the right response?

Five Conditions, and You Need All Five

Expertise and context. The reviewer has enough subject-matter knowledge to recognize a good, weak, misleading, or harmful output, and enough access to the relevant facts, source material, customer history, or policy to add what the system doesn’t know.

Availability is not a qualification. Someone reviewing generated customer communication needs to understand the product, the customer promise, and the exception policies. Someone reviewing a classification needs to understand the taxonomy, the consequence of error, and the evidence that should outweigh the model’s recommendation.

Time. The volume of outputs and the time per output multiply out to something a human can do at the quality you’re assuming. Write the arithmetic down. Two hundred items a day at thirty seconds each is nearly two hours of continuous close reading, and nobody sustains close reading for two hours.

Volume isn’t only a staffing question. It determines whether the review model is honest. If the workload can’t be reviewed at the quality the control assumes, change the design rather than pretending the queue is covered.

Authority. The reviewer can disregard, edit, reject, override, or reverse an output, and can interrupt the process. Stopping can’t depend on locating someone unavailable, getting discretionary approval after the fact, or persuading the team whose targets depend on the process continuing.

Escalation. The reviewer knows what happens when the issue can’t be resolved at the review step: who receives the exception, what evidence goes with it, how fast it’s expected to move, and whether the system pauses while the question is open.

“Escalate it” is not a process. A working escalation path names the next decision-maker, their authority, the expected response time, and what happens while the decision is pending.

Incentives that permit refusal. If the reviewer is measured on throughput, punished for delay, or expected to defend every stop to a frustrated stakeholder, the authority on paper is authority they may not use. People respond to what gets counted.

The organization should make clear that an appropriate rejection, correction, escalation, or stop is evidence the control is working, rather than evidence the reviewer failed to keep things moving.

A review step missing any of these may still create a signature or an audit trail. It doesn’t reliably create meaningful oversight. That’s worse than no review when it transfers accountability to someone who couldn’t exercise it and tells everyone downstream the output was checked.

How to Spot a Hollow Review Step

Workflow diagrams hide this well. A review step is a box, and every box looks equally load-bearing.

What happens if the reviewer does nothing? If the output proceeds automatically after an interval, you have a timeout with a human decoration on it. That may be a legitimate design. It isn’t a control, and it shouldn’t be described as one.

Who is measured on the thing the reviewer might stop? If it’s the reviewer’s own team, the control has a conflict built in. If it’s a different team with more organizational weight, it has a different one.

When did somebody last reject, change, escalate, or stop an output? A zero rejection rate over six months isn’t proof the review step failed. It’s a reason to investigate. The system may be unusually reliable, the input set narrow, failures caught upstream, or reviewers correcting outputs without recording it. A rate of zero should be explainable.

Can the reviewer explain what they’re looking for? Ask them. If the answer is “anything that looks wrong,” they’ve been given a judgment task without the criteria that make judgment possible.

What evidence can they see? A reviewer can’t meaningfully challenge an output if they only see the output. For consequential work they need the relevant source material, the important context, the reason the case was routed for review, and where appropriate, evidence that could contradict the recommendation.

The Research Is More Mixed Than the Confident Version Suggests

Automation bias, the tendency to over-trust an automated recommendation, is the standard justification for all of this, and it’s real enough that the EU wrote it into legislation.

The empirical picture is less tidy. Alon-Barkat and Busuioc ran three preregistered experiments across 605 and 904 Dutch citizens and 1,345 Dutch civil servants, and found no significant tendency to over-trust algorithmic advice specifically. Around 80% of participants rejected advice, algorithmic or human, when it conflicted with qualitative evidence in front of them. What they did find was selective adherence: participants more readily accepted advice predicting poor performance for individuals with Moroccan-sounding names, and the effect appeared whether the advice came from an algorithm or a person.

Two tentative conclusions. Given clear conflicting evidence and time to consider it, people may push back on machine output more than the automation bias framing predicts. And the bias that showed up wasn’t about trusting the machine. It was about which recommendations felt confirmatory.

So build the review process to surface confirmatory risk, not only obvious model errors. Give reviewers the evidence supporting the recommendation, the evidence that could contradict it, and the reason the case was routed. Where bias or disparate impact is plausible, test outcomes across relevant groups and make sure review criteria don’t simply formalize the assumptions the system was meant to challenge.

The goal isn’t claiming a system or a reviewer is bias-free. It’s identifying foreseeable risks, making assumptions visible, and creating a process capable of catching harm before it becomes routine.

Pick a Review Model on Purpose

Review every output when an individual output can create a material consequence, volume is low enough for competent review, and the reviewer has the evidence and authority to act. This suits more work than organizations assume, and only when the time, expertise, and escalation capacity are real. One hundred percent review doesn’t compensate for a reviewer lacking context or authority.

Sample when volume is high and the purpose is monitoring quality, error patterns, or process health. Define the sample method, frequency, acceptable ranges, and investigation triggers.

Review exceptions only when the system can identify meaningful uncertainty or out-of-scope conditions, and those signals have been tested against real examples. Often the highest-value model when it’s valid. It fails when teams assume a confidence score is meaningful without checking whether low-confidence cases are the ones most likely to need judgment.

Review nothing, log everything, make rollback fast when consequences are low or reversible, detection is timely, and boundaries prevent a bad output from causing material harm before anyone intervenes. Sometimes review isn’t feasible at the volume, and the honest move is investing in detection, logging, thresholds, rate limits, and rollback rather than staffing a ceremonial step.

Review model	Use it when	What it doesn’t do
Review every output	Individual errors create material consequences and volume permits competent review	Compensate for inadequate expertise, evidence, authority, or time
Sample	The goal is measuring quality and error patterns at high volume	Prevent an individual incident before it occurs
Review exceptions	Tested uncertainty or policy signals can identify cases needing judgment	Work if confidence or routing signals are assumed rather than validated
No routine review; log and roll back	Effects are low-risk or reversible, detection is timely, rollback is real	Suit anywhere one error can cause material or irreversible harm

Choose the model before you build the workflow. Don’t call all four “human in the loop.”

Write the Control Into the Specification

The reviewer role, required expertise, and named backup
The information and evidence they need to see, including contradictory signals where appropriate
The criteria they’re checking, specific enough that two qualified reviewers could identify and discuss a disagreement
Expected volume, time budget per item, service-level expectation, and the total workload calculation
The actions available: approve, edit, reject, override, route, pause, or stop
The escalation route, named decision-maker, response expectation, and what happens while the issue is unresolved
What happens if review doesn’t occur, stated explicitly rather than left to a product default
The expected range of corrections, rejections, escalations, and stops, and the trigger for investigating unusual patterns
The logging, audit, evaluation, and feedback process used to improve the workflow over time

A review box in a workflow diagram isn’t a control. These details are what make it one.

The Test

Before the phrase goes into a policy, a vendor response, or a board slide, ask the person named as reviewer four questions.

What are you looking for?
What evidence do you need to decide?
What can you do when you find a problem?
What has your correction, rejection, escalation, or stop rate been this quarter?

If they can’t answer, can’t act, or can’t describe what happens after they raise a concern, you don’t have meaningful human oversight. You have a name in a document.

A Great Solution Design Makes the Next One Easier

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

Most teams can produce a good solution design. It lists the events to track, the fields to capture, and the integrations to wire up. It gets the job done.

Six months later you’re struggling to maintain it. Adding a use case means restructuring something. Your teams argue about what the data means. And nobody can explain how the implementation connects to a business outcome.

Great solution designs feel different. They focus on people as much as platforms, create alignment instead of confusion, and respect the future instead of becoming obsolete.

More precisely, a complete solution design lets a team answer six questions before implementation starts:

What decision or outcome is this meant to improve?
How does the work move from the current process to the intended one?
What data, rules, systems, and definitions does it depend on?
What behavior proves it’s working, and what conditions prove it isn’t?
Who accepts the work at launch, and who owns it after?
How does the team make the next change without rediscovering the meaning of everything already built?

If those answers live in different people’s heads, disconnected decks, tickets, and vendor calls, the design isn’t finished. It’s partially documented.

Start With the Business, Not the Event List

A good design says what to implement. A great one explains why.

Every strong design begins with the problem. What business goal does this support? Who cares, and why? How will you know whether you succeeded? These sound obvious, and skipping them is the difference between tracking activity and building something that informs decisions.

The most effective teams align on a small set of shared KPIs that everything ladders up to. When everyone has a north star, decisions get easier and debates get shorter.

The test I’d apply: pick any event in your specification and ask which decision it informs. If nobody can answer, you’ve found something you’re paying to collect, store, govern, and eventually migrate for no reason.

Apply the same test to every field, rule, integration, audience, report, and automation step. For each, the team should be able to say what decision it supports, which source makes it true, who relies on it, what happens if it’s missing or wrong, and who can approve a change.

An item may still be necessary without mapping to a leadership KPI. A consent state, audit field, error flag, or identifier may exist to protect a workflow, support a legal obligation, or make reconciliation possible. The point isn’t that every field proves revenue impact. It’s that no important element should exist without an explainable purpose.

Expect Six Things Before a Build

You don’t need a hundred-page requirements document. You do need enough shared clarity to build responsibly.

A decision statement. The problem, intended users, desired outcome, constraints, and how success gets measured.
A workflow map. Current process, intended future process, handoffs, exceptions, escalation paths, and the role AI or automation plays, including where a person must review, decide, override, or stop.
A data and systems map. Source systems, inputs, identifiers, definitions, consent or access constraints, transformations, and downstream destinations.
A solution design. Architecture, integration approach, reusable patterns, assumptions, dependencies, and the design choices needing validation.
An acceptance and QA plan. Expected behavior, test cases, source-of-truth checks, failure conditions, acceptance criteria, and who signs off.
An ownership and maintenance plan. Who monitors, updates definitions, approves changes, maintains documentation, pays for the tools, manages incidents, and answers questions after launch.

These aren’t six documents to create independently. They’re six views of one system. A changed definition should show up in the workflow map, the data map, the implementation rules, the QA tests, the reporting logic, and the ownership model. If they contradict each other, the solution isn’t ready.

Balance the Strategic and the Technical

Think of solution design like city planning. A good plan builds roads where you need them today. A great plan creates a grid system that accommodates growth, new neighborhoods, and transportation methods that don’t exist yet.

In practice that usually means:

A phased approach that delivers the smallest reliable capability first, validates the assumptions that matter most, and creates visible value before enthusiasm or sponsorship runs out
Reusable patterns your team understands well enough to apply without asking
A vendor-independent data model or coordination layer that keeps identity, definitions, consent states, critical metrics, and business rules accessible and portable rather than trapped inside one application’s configuration
Privacy by design, with governance considered throughout the pipeline instead of reviewed at the end
A QA and validation plan written at the start rather than bolted on the week before launch

A reusable pattern is more than a naming convention. It includes clear interfaces between systems: what moves, what each critical field means, which system is authoritative, what triggers a handoff, how consent and permissions are handled, how errors get detected, and what happens when an API or a business definition changes. That’s a data contract. It needs to be visible enough that a new team member can understand the connection and a future change doesn’t become an archaeological project.

A future-proof design isn’t one that predicts every requirement. It’s one that establishes stable patterns early enough that the next requirement can be added without restructuring the system or rewriting reporting history.

One of my favorite indicators of great solution design is simple: it reduces future effort. If adding a new event, channel, or integration forces you to restructure half the system, the design was never great to begin with.

That indicator is also the one nobody measures, because the cost of a bad design arrives as small delays rather than one visible failure. The sprint that took three weeks instead of one. The integration that needed a workaround. The report that had to be rebuilt. None of those get attributed to the design.

QA Is Part of the Design

Quality assurance belongs at the beginning, not as a scavenger hunt before launch.

For an analytics, martech, or AI-enabled implementation, “the tag fired” or “the workflow ran” isn’t enough. An event can carry the wrong value, ignore a consent state, omit the identifier required downstream, trigger twice, send data to the wrong destination, or produce an output nobody knows how to review.

Define the source systems, expected values, test cases, thresholds, acceptance criteria, failure conditions, downstream reporting effects, and sign-off roles while the solution is being designed. Decide what evidence proves the work is functioning: source-of-truth reconciliation, expected-value checks, consent-state tests, identity-resolution tests, duplicate-event checks, controlled failure tests, and acceptance by the person accountable for the business outcome.

A clean implementation is one the team can prove is working, rather than one that appears to work in a demo.

Automation and AI shouldn’t erase human judgment from the design either. For each consequential workflow, define where a person reviews, approves, decides, overrides, investigates, or pauses. Identify the exceptions requiring escalation rather than a default automated response. Name the role that can stop the workflow when the output is inaccurate, unsafe, or materially different from what the design intended. The goal isn’t a person in every step. It’s making human responsibility explicit where automation can’t legitimately substitute for judgment.

Write It for the People Who Have to Use It

The best architecture in the world will fail if nobody understands it.

Great designs are written for humans. Clear language, minimal jargon, a glossary when one is needed, and enough context for a marketing manager, an engineer, an analyst, or someone trying to understand the big picture.

Give people an obvious route in. A useful design package usually includes a concise overview for decision-makers, a workflow map for operators, an architecture view for technical teams, a specification for builders, a glossary for shared definitions, a QA plan for testers, a decision log for changes, and an ownership model for after launch. Link them and make the source of truth obvious. A document set nobody can navigate becomes another dependency on the person who created it.

Enablement isn’t a presentation at the end of implementation. Define who needs to learn what, how a new team member gets oriented, where people request a change, and how you verify the team can use the system without calling the original builder for routine interpretation.

Have you watched a solid implementation gather dust because the marketing team didn’t know how to reach the data? Or seen analysts unable to answer a basic question because nobody wrote down what “conversion” means? That’s the adoption gap, and great design closes it.

A tool is only as valuable as your team’s ability to use it without you.

Design for the Version After This One

Launch day isn’t the finish line. It’s the start of the part that lasts longer.

Great designs expect change instead of being broken by it. That means:

A named owner for each critical data domain, technical component, business definition, and prioritization decision
Versioning rules recording what changed, why, who approved it, and which reports, workflows, audiences, or customer experiences it affects
A change process distinguishing low-risk adjustments from changes requiring technical, privacy, security, or business review
Monitoring criteria for quality, adoption, performance, cost, governance, and known failure conditions
A prioritized backlog based on business need and evidence rather than whichever request arrives loudest
A retirement path for integrations, reports, automations, and vendors that no longer serve the outcome

The maintenance plan isn’t an appendix. It’s the design’s proof that the organization expects the system to live past launch.

If nobody owns the next version, every future request becomes an exception. If changes aren’t recorded, reporting history becomes unreliable. If the team can’t retire old logic, the stack accumulates dependencies nobody feels responsible for.

Moving From Good to Great

Most organizations can already produce a good solution design: a diagram, a field list, a set of requirements, and a path to launch.

Great solution design gives every consequential component a reason to exist, connects business outcomes to workflows and data, defines the interfaces that make change survivable, states how the work will be tested, assigns the people who own it, and leaves documentation the next person can use.

It isn’t more diagrams or a longer document. It’s a shared operating model for building, validating, running, and improving the system.

Great design requires strategic vision and tactical precision together. It asks teams to think like architects, test like operators, and explain the work clearly enough that the people who inherit it can make good decisions without starting over.

AI Makes the First Draft Cheaper. Trust Still Costs Money.

Allison Marx — Thu, 17 Sep 2026 12:00:00 GMT

There’s a question circulating in procurement meetings and executive briefings right now. You may have fielded it yourself.

“If your team is using AI to do the work faster, why aren’t your fees going down?”

It’s a fair question.

Sometimes the answer should be yes. If the work is repetitive, low-risk, clearly specified, and easy to check, AI can reduce delivery cost, and a fair provider should redesign that workflow and share the gain through lower cost, faster turnaround, expanded scope, or an agreed combination.

But most strategic, analytics, data, and systems work isn’t a fixed task with a known answer. The visible deliverable may arrive faster. The work required to decide whether it solves the right problem, uses the right data, survives real conditions, and can be defended to the person accountable for the outcome doesn’t disappear.

AI lowers the cost of making a first draft. It doesn’t lower the cost of being accountable for the answer.

AI Saves Time. The Time Doesn’t Go Where You’d Expect.

The savings are real and measured.

Employees with access to enterprise AI tools save 40 to 60 minutes per day, according to Fortune’s reporting on Goldman Sachs data. The Federal Reserve Bank of St. Louis found AI users saved roughly 2.2 work hours per week. An MIT experiment by Noy and Zhang found that for knowledge workers, ChatGPT cut task completion time by roughly 40% while raising output quality by 18%.

These studies measure early adoption periods. Teams who’ve embedded AI deeply since then are plausibly saving more, though I haven’t seen a study confirming it at the same rigor, so treat that as expectation rather than finding.

The important question is what happens to the capacity AI creates.

Time savings don’t automatically become fewer labor hours, lower total cost, or better outcomes. Organizations may reinvest capacity in more work, broader scope, higher quality, experimentation, rework, validation, a faster pace, or demand that was previously deferred. The destination of the savings is a management decision, not an automatic outcome.

A UC Berkeley Haas study found that AI intensifies work rather than reducing it: people expanded scope, accelerated pace, and extended hours voluntarily, because AI makes starting a new task nearly frictionless. Workday’s research found that while most employees reported saving one to seven hours per week, a substantial share of those savings went straight back into rework.

Faster output can create more work to review, compare, correct, integrate, and govern. Whether that oversight is proportionate depends on the task, the consequence of error, the reliability of the inputs, and the controls around the workflow.

Faster Tasks Are Not Automatically Better Outcomes

Most AI productivity claims operate at the task level: how long it took to write a draft, summarize a document, produce a first-pass analysis, or generate code.

That’s useful evidence. It’s not the same as a better workflow or a better business outcome.

Level	The question to ask
Task	Did AI reduce the time or improve the quality of a defined activity?
Workflow	Did the full process improve after review, exceptions, handoffs, rework, and maintenance were included?
Business outcome	Did the changed workflow improve a decision, customer outcome, revenue measure, risk profile, or sustainable capacity?

A task can get faster while the workflow becomes harder to manage. A workflow can get faster without improving the decision it supports. “We saved time” is the beginning of the analysis, not the conclusion.

The Confidence Premium

AI makes it cheaper to produce plausible outputs. It doesn’t make those outputs trustworthy by default.

The work required to turn a plausible draft into a decision-ready result is the confidence premium:

Validating source data, definitions, and business rules
Checking whether the recommendation fits the organization’s real constraints
Testing technical logic, failure states, and edge cases
Evaluating assumptions, uncertainty, and alternative explanations where they matter
Making privacy, security, accessibility, and governance decisions explicit
Explaining tradeoffs so the accountable person can make the call
Documenting the result so someone else can maintain, audit, challenge, or improve it

The confidence premium isn’t a justification for invisible busywork. It’s the work required to turn a fast first draft into something a person can responsibly act on.

The more consequential the decision, the less appropriate it is to treat first-pass speed as evidence of finished work.

When a Lower Cost Is Fair

AI can and should make some work cheaper.

That’s most likely when the scope is fixed, the output is standardized, the source material is approved or well governed, error consequences are low, and quality can be checked objectively or through defined spot checks.

Converting approved source material into a known reporting format. Producing first-pass documentation from reliable implementation metadata. Classifying well-labeled low-risk records. Drafting routine variations. Generating a standard transformation with established tests.

For work like that, lower cycle time and lower cost are the point, and the client should see the benefit.

Complex advisory work has a different value equation. Saved production time is often best reinvested in deeper validation, scenario testing, documentation, and decision support. That doesn’t mean charging for invisible activity. It means agreeing on the standard of confidence the work requires and making the work needed to reach it explicit.

Three Myths Worth Busting

“AI can handle the strategy now.”

AI is good at executing a clearly defined task. It’s poor at identifying which task is worth doing.

Strategy isn’t the absence of execution work. It’s the selection and sequencing of problems under real constraints: objectives, budgets, data availability, customer impact, organizational readiness, risk tolerance, ownership, and the opportunity cost of not doing something else. AI can help generate options. It can’t make those tradeoffs legitimate on the organization’s behalf.

Take data science work. AI can generate code, suggest modeling approaches, summarize research, or produce a first-pass analysis. It cannot decide whether your business question calls for a predictive model, a segmentation study, an incrementality test, or a simple descriptive cut.

Sometimes the smarter solution is the less complex one. A well-scoped regression model, cleanly explained and tied to a decision, creates more value than an impressive black-box model nobody trusts or knows how to operate. The right method is the one producing enough reliable evidence for the decision at hand, rather than the one producing the most impressive-looking output.

I regularly see organizations paralyzed by what I’d call the Paradox of Possibility: powerful tools, and no clear sense of which problem to solve first. A chatbot doesn’t close that gap. It gets closed by people who can translate statistical patterns into business decisions, tell correlation from causation, and navigate the dynamics that determine whether anyone adopts the result.

“More data input means better AI output.”

Not automatically. More data helps when it’s relevant, clean, connected, and understood. Otherwise you’ve handed the model a bigger junk drawer.

A model or workflow also needs context designed for the actual question: definitions, known exceptions, current business rules, source-system limitations, consent and access constraints, change history, and the boundaries of what the system may infer or do. A model can’t reliably retrieve context that was never captured, resolve a conflict the organization never settled, or recognize a critical exception nobody made visible.

“More data” and “better context” are not the same strategy.

Gartner’s February 2025 research put the mechanism plainly: it predicted organizations would abandon 60% of AI projects unsupported by AI-ready data through 2026, based on a survey of 1,203 data management leaders in which 63% either lacked appropriate data management practices for AI or didn’t know whether they had them. You’ll see higher failure rates quoted elsewhere, often from contributor posts or vendor material with no published methodology. I’d stick with the number that shows its work.

In client work, the time saved through drafting and code generation should be visible in what becomes possible next: more careful validation, more scenario coverage, better documentation, clearer tradeoffs, and stronger enablement. The goal isn’t faster deliverables for their own sake. It’s a result that can be trusted for the decision it supports.

“My analytics platform has AI built in, so why do I need a consultant?”

Embedded AI in tools like Adobe, Google, and Amplitude is built for the median use case. It can surface “conversions are down.” It can’t surface “conversions are down because the new consent banner was applied to your checkout flow.” Worse, these systems often won’t flag when they can’t answer your question, which creates real risk when decisions get made on a confident-sounding recommendation.

These systems generate or rank plausible outputs from the context available to them. That can be extremely useful. It isn’t the same as verifying that the output reflects your organization’s actual definitions, current implementation state, data lineage, or causal reality. A novice accepts the output. Someone who knows the implementation spots the flaw immediately.

So how do you help these tools give better answers? For consequential use cases, define the context the system may use, the source-of-truth references it should prefer, the output it’s allowed to produce, the evidence required for a recommendation, the conditions requiring human review, and the evaluation method that will reveal whether the workflow is improving or degrading.

Most organizations don’t have that yet. Gallup’s February 2026 survey of 23,717 US employees found half of employed US adults now use AI at work at least occasionally, with 13% using it daily, while only 41% reported that their organization has integrated AI tools at all. Individual adoption is running well ahead of organizational design.

What Changes

AI changes where expert hours go, not whether they’re needed.

AI can accelerate	Still requires accountable judgment
First drafts of SQL, code, requirements, analysis, research summaries, and documentation	Whether the organization is solving the right problem
Candidate strategies, model approaches, scenarios, and workflow options	Which method fits the decision and which assumptions must hold
Schema mappings, transformations, and connector logic	Whether definitions, permissions, identities, and source logic are correct
Anomaly summaries and recommended explanations	Whether the explanation reflects actual change history and proprietary context
Routine formatting, classification, and production tasks	Testing, approval, exception handling, maintenance, and response when output is wrong

The commercial question isn’t whether AI made one artifact faster. It’s whether someone still needs to be responsible for the decision the artifact supports. In consequential work, the answer is yes.

The capacity AI creates can be used in different ways: lower cost, faster turnaround, broader exploration, deeper validation, stronger documentation, more experimentation, or an agreed combination.

For complex work, the most valuable use is usually a higher standard of confidence. More scenarios considered, more assumptions tested, better evidence documented, fewer preventable surprises after launch.

Make that choice explicit. Don’t treat AI savings as a reason to hide reduced effort, or as an excuse to expand scope without agreement.

That’s how AI changes the value equation: less time producing a first draft, more deliberate investment in deciding whether the result is worth acting on.