More AI Traffic, or a Different Way of Counting?

Imagine this line in a team update: "Our AI visibility went up forty percent." It's an appealing sentence. But forty percent of what? Out of how many attempts, compared with which period, and measured how? If nobody can answer, the number is decoration. So today's question is this: what is each of your numbers actually counting, and what is it allowed to tell you?

This is an educational series from Verity Score for Shopify merchants. Last time, we built an eligibility matrix, market by market, before building anything. Today, we turn to measurement. Not to find the one number that proves AI is working for your store. No single number in our observation protocol establishes that conclusion. Instead, we'll set up measurements that are honest about what they count, and about what they can't.

We're still following the fictional luggage team. The prompts, the reports, the visits and every figure in this story are invented for the example, including that forty percent. No real store data or account is involved.

After several weeks of work on product data and answers, the founder wants to know whether it's paying off. Three people bring three numbers.

The marketing lead has been running a set of prompts in ChatGPT every week, as we described in episode two. They say the backpack now appears more often than before. That's where the forty percent comes from. But when the team looks closer, the prompt set changed halfway through. Two prompts were dropped, three were added, and some weeks were run in a new conversation, others weren't.

The operations lead opens the store's reports. They see more sessions coming from referrers that look like AI services, and more direct visits too. They want to say the work is driving traffic.

The founder looks at orders. Orders are roughly flat.

There's a fourth person, too, a freelancer who suggests a quick fix: run the same prompts through several different assistants and rank them, to see which one "sends" the most customers. It sounds rigorous. But a handful of prompts from one team, in one country, says nothing reliable about how different platforms behave for real shoppers. The team can compare the outputs in its recorded sample. It cannot turn that comparison into a ranking of which platform sends the most actual customers.

Three numbers, three different things counted, and a strong temptation to link them into a story: more mentions, more visits, more sales soon. Let's take that story apart, gently.

Start with the prompts. These are queries the team itself sends. They aren't shoppers. They're tests. OpenAI's help article on shopping in ChatGPT, which we reviewed on October twenty-second, twenty twenty-six, says that product selection considers the question, the context, and first-party and third-party metadata, and that not every available product will necessarily be shown. So if the questions change, or the conversation context changes, the results can change for reasons unrelated to your store. A stable prompt set and documented conditions improve comparability over time; they do not hold the model or all context constant.

And the denominator matters. "Appears more often" means nothing without "out of how many runs." If the team runs, say, ten prompts three times each, that's thirty runs, and every share should be stated as a count out of thirty, in that week, under those conditions. Those numbers are hypothetical, chosen for this example. The important part is that the denominator is written next to the result.

One more thing about prompts. If you run them through a developer interface rather than the consumer app a shopper would use, don't assume the results are equivalent. Record which access you used, and keep those observations separate and document the differences.

Now the store reports. Shopify's documentation on acquisition reports says that these reports are about visitors and sessions on the online store. They don't show converted sales or order counts. There's a separate report for sales by referrer. So when the operations lead sees more sessions from a referrer, that's a fact about sessions. It isn't a fact about orders. Those are different quantities from different reports. A count needs its unit and window; a rate also needs a clearly defined denominator.

The same documentation flags a change in how sessions were measured, between the twenty-first and the twenty-third of September, twenty twenty-six. So if the team compares a window before that change with a window after it, a break in the numbers may come from the change in measurement, not from anything the team did. That doesn't make the numbers useless. It means a comparison across that date needs a note, or a different window.

Then there's the question of who, or what, those sessions are. Shopify's documentation on bot filtering says that it classifies traffic as human or bot, and that this classification is conservative. It isn't a perfect identification. A filter is available in compatible reports. So when you compare two periods, compare them with the same filter setting, and note the setting. Some automated traffic may be counted as human, and that is a classification limit to keep visible, rather than treating every human-labelled session as verified human behaviour.

That's why the team must not merge its own prompt tests with store session counts as if they measured the same activity. A query you typed into an assistant isn't a visit. An automated request does not by itself establish a human shopping session or a purchase. Its classification alone does not establish its commercial value either. And a shopper's session isn't an order.

What about Google? Google's documentation on AI features in Search says that their performance is included in the Web traffic reported in Search Console. So you can see Search traffic there, but not a separate, universal measure of AI visibility. If someone offers you that measure for Google, ask what it counts.

And the visits whose origin is unclear? They may have several origins, including shared links or typed addresses; an unknown origin cannot be assigned to an assistant by guesswork. Reports can't always tell them apart. Leave that uncertainty visible, rather than assigning those visits to the channel you hope worked.

The team decides to stop reporting a single number. Instead, they write down their denominators, and they keep three measurements separate. That's your action this week: define what each measurement counts, add denominators for rates, and separate your own queries, automated activity and human-labelled sessions.

Here's how. Make a short measurement sheet with one row per measurement.

Row one: your prompt observations. Count the runs under recorded conditions. To report a share of appearances, use those runs as the denominator. Freeze the prompt set, and if you change it, start a new series rather than continuing the old one.

Row two: sessions. Record the count of human-labelled sessions in the online store, with the bot filter setting recorded, in a stated date window, with a note if that window crosses the late September measurement change.

Row three: orders. Take these from the sales report, not from the acquisition report. Write down which report and which window.

Then add one column that matters most: what this measurement can't tell you. For prompts: what shoppers see. For sessions: whether they bought. For orders: why they happened.

Add a fourth row if you can: visits of unknown origin. Don't fold them into any channel. Give them their own line, with their own count in the same window, so that everyone reading the sheet sees how much of the picture is simply unexplained. If that line is large, check the attribution configuration and available consent information before interpreting it. Keep the unknown origin visible instead of assigning it to a channel by guesswork.

And decide in advance how often you'll update the sheet. For a small team, once a month may be enough. That's a suggestion, not a standard. What matters is that the rhythm is fixed before you look at the numbers, so you're not tempted to check again whenever you hope something has changed.

For our team, the result is less exciting than forty percent, and much more useful. The founder now knows that the prompt series has to restart from a stable set, that sessions from AI-looking referrers rose in the window they chose, with the filter on, and that orders didn't move in the same window. None of that proves the work failed, or that it succeeded. It says what was counted. The full template is in the show notes.

So: every number needs a clear definition and time window, and every rate needs a denominator. Your own queries aren't visits, bot traffic isn't shoppers, sessions aren't orders, and a change in measurement isn't a result. Measurement can show you what moved. These observations alone do not establish why it moved. Next time, in our final episode, we'll turn everything from this season into a routine: a thirty-day plan for a small team, with evidence of completion and a way back if something goes wrong.

Sources and review dates are in the show notes. This episode is read by a synthetic voice.