Manual testing: a healthy first reflex
Opening ChatGPT, Gemini or Perplexity and asking "which brand do you recommend for…" is the simplest way to discover what AI says about you. It is free, immediate, and gives a first intuition: are you cited, recommended, or absent? As a first contact with the topic, it is a sound approach.
The problem is not manual testing itself, but what you conclude from it. A single answer is not a measurement.
Limit 1: response variance
Language models are stochastic: the same question gets a different answer from one run to the next. Your brand may appear one time out of three without anything changing in your actual visibility. Drawing conclusions from a single run means confusing the luck of a draw with a trend.
A reliable measurement requires replaying each query many times — at least ten — and aggregating results into rates: visibility rate, recommendation rate, share of voice. That aggregate, not the individual answer, is what can be tracked over time.
Limit 2: too narrow a sample
Testing two or three questions covers a tiny fraction of real conversations. Your customers do not phrase their needs the way you do: they arrive with different contexts, constraints and levels of maturity. A useful sample covers varied personas and scenarios representative of the buying journey, not just the obvious query.
Limit 3: no multi-turn
The recommendation is rarely settled in the first answer. A real user refines, compares, objects, asks for alternatives. It is often at the third or fourth exchange that the AI switches from one brand to another. Manual testing almost always stops at the first answer — precisely where the decision has not been made yet.
Limit 4: your history skews the measurement
When you test from your own account, the AI factors in your conversation history and preferences. You are not measuring what an unknown prospect would see: you are measuring what the AI tells you, specifically. A reliable measurement starts from a fresh conversation, with no history.
When a tool becomes useful
A measurement tool becomes relevant when you want to move from intuition to steering: tracking your visibility month after month, comparing personas and markets, detecting the inflection points where the recommendation shifts, and verifying that your actions have a measurable effect. Manual testing remains useful for exploring; it is not enough for steering.
Frequently asked questions
Can I assess my AI visibility for free?
Yes, by testing a few queries by hand. It is a good starting point, but a single answer is not a measurement: AI answers differ on every run, and your conversation history influences the responses.
How many times should a query be replayed to be reliable?
At least ten runs per query, aggregated into a visibility rate. Below that, the models' natural variance makes conclusions fragile.
Why test in multi-turn conversation rather than a single question?
Because the recommendation forms over the exchanges: comparisons, objections, alternatives. The first answer is rarely the one that decides.