Why AI Answers Change Every Time You Ask (and How to Measure Visibility Anyway)

Sep 15, 2026 · 8 min read

If you work in marketing, you've probably seen some version of this. Someone drops a screenshot in Slack: ChatGPT recommended you first for your main category. Somebody else tries the same prompt and you're not there. A third person gets you in fourth place, behind a competitor nobody on the team has heard of.

They're all telling the truth. Each of them got a real answer. But one answer from an AI assistant is a sample, and you can't run a marketing program on samples.

Google rankings taught us to think differently. If you were third on Tuesday, you were probably third on Wednesday, so a screenshot meant something. AI answers move around much more, and if you keep reading them like rankings you'll end up reacting to noise.

Why you get a different answer each time

There are a few reasons, and it helps to keep them apart because each one calls for a different response.

The first is plain randomness. Language models pick each next word from a set of likely options, with a bit of chance built in. Run a prompt twice and the model can wander down slightly different paths, which is sometimes enough to swap out the brand in second place. That's how these systems are designed to work.

Then there's retrieval. When an assistant searches the web before answering, it doesn't get the exact same pages every time. If the main thing tying you to a category is one review roundup that only gets pulled in now and then, you'll only show up now and then.

Context matters too. Memory, location, earlier messages in the chat and the exact wording of the question all shape the answer. Someone in Berlin who has just asked about pricing won't get the same list as someone in Chicago starting a fresh chat.

And the providers keep changing things. New model versions roll out, defaults change, search integrations get updated. This happens less often, but when it does, a whole category can reshuffle overnight without you having touched anything.

The first three cause the day-to-day wobble. Updates cause sudden jumps. You'll want to be able to tell which one you're looking at.

Habits from SEO that stop working

The most obvious casualty is the idea of a rank. "We're #2 in ChatGPT for best CRM for startups" sounds precise, but it describes one answer. Say you appear second in four out of ten runs, first in one, and not at all in the other five. What's your rank? There isn't really one. Quote the best run and you overstate progress; quote the latest and you might be raising an alarm over nothing.

Spot checks have the same weakness. Checking a few prompts by hand before a quarterly review tells you what one person saw, on one model, at one moment. It can't tell you whether last month's content push did anything, because the answers might have shifted on their own.

Quick before-and-after tests mostly don't hold up either. You publish a new comparison page, check the next morning, and there you are in the answer. Maybe the page helped. Maybe you'd have shown up that morning anyway. Without a baseline, you honestly can't say.

Measuring it properly

A more useful question than "where do we rank?" is "how often do we show up, and how prominently?"

Start by running each prompt several times on each assistant, every reporting period. One run shows what can happen; ten show what usually happens. The exact number matters less than sticking with it, so this month can be compared with last month.

From those runs you can work out your appearance rate, meaning the share of answers that mention you at all. It's the nearest thing AI search has to a headline visibility number, and it's far steadier than any single answer.

Showing up is only part of it, though. Being the first recommendation is worth a lot more than a mention tucked into the last sentence. So track where you sit among the brands named, and whether you're actually recommended, just listed, or cited as a source.

Then put your numbers next to your competitors'. A flat appearance rate could be good news if everyone else dropped, or bad news if they all climbed.

With numbers like these, the screenshot debates mostly go away. "We showed up in 62% of runs this month, up from 48%, and we were named first in about a third of those" is hard to argue with using one screenshot.

Choosing the prompts

All of this depends on tracking the right questions. Your prompt set, the list of prompts you monitor, is probably the most important decision you'll make here, and it's easy to get wrong.

Try to cover the different ways buyers actually ask, since brands are often strong in one and missing from another. Broad category questions ("best project management tools for agencies") show whether you make the shortlist. Problem-first questions ("how do I stop missing client deadlines?") show whether you come up before the buyer even knows what they're shopping for. Head-to-head comparisons show how you're described next to named rivals. Questions with specific requirements ("cheapest PM tool with time tracking and SSO") test whether the model knows what you're good at. And direct questions about your brand show what the model believes about you, which isn't always accurate.

A few practical tips. Write prompts the way customers talk; sales call notes, support tickets and Reddit threads are better sources than your keyword list. Include questions you currently lose, or you'll only be measuring the places where you already look good. Keep most of the set fixed over time so the trend line means something, and add new prompts alongside the old ones rather than swapping them out. If you sell in several markets or languages, track them separately, because the shortlists can look completely different.

What to do with the results

Once you've got repeated runs across a decent prompt set, the variation itself starts telling you things. How often you appear shows how strong you are on a question. How consistently you appear hints at why.

If you show up in nearly every run, the model has a settled view of you for that question. The job there is maintenance: keep the pages and listings behind that view up to date, and watch the direct brand questions in case the facts start drifting.

If you show up in some runs but not others, you're on the edge of the shortlist, and that's usually where work pays off quickest. Often it means you're relying on one or two sources that only get retrieved some of the time. Look at what the answers cite when you do appear, then try to get into more of those places.

If you rarely or never appear, the model either doesn't know you in this context or doesn't connect you with this need. That's a content and authority problem, and it takes longer to fix.

If lots of prompts move at once, check for a model or search update before you start rewriting pages. If your competitors moved too, it's probably the platform. If only you dropped, take it seriously.

And when you test a change, give it a fair chance. Get a baseline from repeated runs, make the change, then compare over a similar stretch of time.

Where to go from here

AI answers will keep varying from one ask to the next. You can't stop that, but you can raise how often you get named across the questions your buyers really ask.

So drop the one-off checks. Pick a stable set of prompts, run them repeatedly, track how often and how prominently you appear, and learn to tell normal fluctuation from a real shift. Once you start treating visibility as a probability, it becomes something you can actually manage.

AnswerRank handles this tracking for you. It runs your prompts repeatedly across the major AI assistants and reports your appearance rate, prominence and share of answer over time.