How AI Search Actually Works: RAG, Embeddings & Vector Search Explained

AI answer engines are retrieval systems with a language model attached at the end. Here is how RAG, embeddings and vector search really decide which business gets recommended - and what Google says you can safely ignore.

Every marketer in South Africa is now selling “AI optimisation”. Very few can tell you how AI search actually retrieves the answer it gives. That matters, because once you understand the plumbing, most of the advice you’re being sold falls apart — and what’s left is refreshingly simple.

Here’s the sentence that explains almost everything:

An AI answer engine is a retrieval system with a language model bolted on the end. The model isn’t the gatekeeper. The index is.

ChatGPT doesn’t “know” about your business and decide to mention you. It runs a search, reads what comes back, and writes an answer from those pages. If your page wasn’t retrieved, no amount of clever writing gets you named. Everything you can control happens before the AI ever starts writing.

RAG: what happens between your question and the answer

The technique behind almost every AI answer engine is retrieval-augmented generation, or RAG. Google names it explicitly in its own documentation, and calls it “grounding”:

“A technique… used to improve the quality, accuracy, and freshness of AI responses by relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index.”

Read that again, because it’s the whole game: relying on our core Search ranking systems. Google’s AI features aren’t a separate machine that judges your site by new rules. They sit on top of the search index you’ve been trying to rank in all along.

The pipeline runs roughly like this:

  1. Understand the question — and usually break it into several smaller ones.
  2. Retrieve candidate passages from an index.
  3. Rank them, keeping the best handful.
  4. Generate an answer using only what was retrieved.
  5. Cite the pages that support each statement.

Notice where the language model appears: step 4, near the end. Steps 1–3 are search. That’s why we tell clients that GEO and SEO aren’t rival disciplines — the AI is reading search results.

Embeddings, in plain English

An embedding turns a piece of text into a long list of numbers that represents its meaning. Google’s own definition: a vector where “the distance between points in an embedding space represents the relative similarity between the corresponding items.”

Picture a giant map. Every passage on the web gets a pin. Passages about the same idea land near each other — even when they share no words at all. “Emergency plumber Cape Town” and “burst geyser, need someone tonight” sit close together on that map, despite having almost nothing in common lexically.

Vector search is simply: convert the question into a pin, then grab whatever’s nearest.

This is why keyword stuffing is dead as a strategy. Google says so directly: AI systems “can understand synonyms and general meanings,” so “you don’t have to worry that you don’t have enough ‘long-tail’ keywords or haven’t captured every variation.”

But keywords aren’t entirely dead — and here’s the nuance nobody explains

Serious retrieval systems don’t use vectors alone. They run hybrid retrieval: semantic search for meaning, plus old-fashioned keyword matching (BM25) for exact strings, then merge the results.

Anthropic’s engineering team gives the clearest example of why. Search for error code “TS-999” and an embedding model “might find content about error codes in general, but could miss the exact TS-999 match.” Meaning-based search is bad at exact identifiers. Their conclusion is blunt: “Embeddings+BM25 is better than embeddings on their own.”

What that means for a South African business: the exact names of things still matter enormously. “SARS”, “CIPC”, “B-BBEE Level 4”, “POPIA”, “load shedding Stage 6”, your suburb, your product’s model number. Semantic search understands your topic. Lexical search catches your specifics. You want both working for you.

Query fan-out: why you’re not ranking for the question you think

This is the most useful thing on this page. Google’s AI Mode doesn’t just search what the user typed. It uses query fan-out — issuing multiple related searches at once across subtopics.

Google’s own example: ask “how to fix a lawn that’s full of weeds”, and behind the scenes it may also search “best herbicides for lawns”, “remove weeds without chemicals”, and “how to prevent weeds in lawn.”

Your page never had to rank for the original question. It had to rank for one of the invisible sub-questions the AI invented.

Now — the trap. The obvious reaction is to build a page for every possible fan-out query. Google explicitly warns against exactly that:

“While it might be tempting to create separate content for every possible variation of how people might search… doing so primarily to manipulate rankings or generative AI responses in Google Search violates Google’s scaled content abuse spam policy.”

The move isn’t more pages. It’s fewer, deeper pages that genuinely answer the whole cluster of related questions.

Chunking: your page gets read in pieces

Retrieval systems don’t handle whole documents — they work in chunks, typically a few hundred words. A chunk gets retrieved on its own merits, stripped of everything around it.

Anthropic calls the resulting problem the “context conundrum”. Take a chunk that reads: “The company’s revenue grew by 3% over the previous quarter.” Useless. Which company? Which quarter? It answers nothing on its own, so it doesn’t get retrieved — or worse, gets retrieved and can’t support a citation.

Now, an important honesty note, because plenty of agencies get this backwards: Google explicitly says you should not pre-chunk your content.

“There’s no requirement to break your content into tiny pieces for AI to better understand it. Google systems are able to understand the nuance of multiple topics on a page.”

Chunking is something the engine does to your page. It’s not a task on your to-do list. But the underlying lesson still stands, and it’s just good writing: each section should make sense to someone who lands on it cold. Name the subject. Don’t lean on the H1 three screens up. Write sections that can survive being read alone — because they will be.

Every engine is a different machine

“Optimise for AI” implies one target. There isn’t one.

  • Google (AI Overviews / AI Mode) — the best documented. Runs on the normal Search index, ranking systems and Knowledge Graph. To be cited, a page “must be indexed and eligible to be shown in Google Search with a snippet.” That’s a hard gate.
  • ChatGPT — OpenAI runs its own search crawler, OAI-SearchBot, and blocking it removes you from ChatGPT’s search answers entirely. Beyond that, OpenAI has never published how its index works. Anyone who tells you confidently that “ChatGPT uses Bing” in 2026 is guessing.
  • Perplexity — PerplexityBot handles search surfacing. Perplexity’s citations overlap Google’s top 10 far more than other assistants (28.6% vs roughly 8%), suggesting it leans on conventional ranking signals.
  • Gemini — no separate crawler. It’s fed by normal Googlebot.

How different are they really? 86% of the most-cited sources aren’t shared between ChatGPT, Perplexity and Google AI Overviews. Of the top 50 sources, only 7 overlap. Winning in one tells you very little about the others.

The number that should calm you down

If you run a local business in South Africa, here’s some perspective the hype cycle won’t give you: AI Overviews appear on only 7.9% of local searches, against a 20.5% baseline across all searches. On shopping queries it’s 3.2%.

Someone searching “plumber near me” in Bellville is overwhelmingly still getting a normal map pack and normal results. Your Google Business Profile is doing the work. AI search is worth preparing for — it is not currently eating your local leads.

So what actually gets you cited?

Everything that matters sits upstream of the language model:

  1. Be indexed and snippet-eligible. A stray nosnippet tag silently disqualifies you from Google’s AI features. Check this first.
  2. Don’t block the retrieval bots — in robots.txt and at your CDN or firewall. This is the single most common own-goal we find on South African sites, and owners are usually unaware.
  3. Rank. 76.1% of pages cited in AI Overviews already sit in Google’s top 10. Though be honest about the ceiling: ranking #1 makes citation likelier, but it’s still roughly a coin flip.
  4. Stay fresh. Across 17 million citations on 7 platforms, AI-cited pages averaged 25.7% newer than standard organic results.
  5. Publish something only you could publish. Google’s guidance is explicit that a first-hand account beats “a summary of existing content” that “could easily be produced by a generative AI model.” An AI won’t cite your page for saying what every other page says.
  6. Be an unmistakable entity. Consistent name, address and phone number; structured data that matches your visible text; a properly maintained Google Business Profile.
  7. Name your specifics. Hybrid retrieval means exact terms — suburbs, regulations, product names — still get caught by the lexical half of the system.

And what doesn’t

Straight from Google’s own AI optimisation guide, published 10 July 2026 — worth quoting because it contradicts a lot of what’s being sold right now:

  • llms.txt: “Google Search itself doesn’t use them.” We wrote about that honestly here.
  • Special AI schema: “Structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add.” (Do structured data for rich results — just don’t let anyone sell it to you as an AI hack.)
  • Writing “for AI”: “You don’t need to write in a specific way just for generative AI search.”
  • Buying mentions: “seeking inauthentic ‘mentions’ across the web isn’t as helpful as it might seem.”

Google’s own summary of the discipline: “optimizing for generative AI search is optimizing for the search experience, and thus still SEO.”

The honest conclusion

AI search is a genuine shift in how people find businesses. But it isn’t a new rulebook — it’s the old rulebook, read by a machine that summarises instead of listing. The businesses getting recommended by AI aren’t the ones who bought the newest trick. They’re the ones who are easy to retrieve, easy to verify, and worth citing.

That’s slower and less exciting than a magic file. It’s also the only thing the evidence supports.

Where you stand right now

Our free website scan checks the signals that actually drive retrieval — crawlability, indexability, structured data, speed, entity clarity — and shows you the raw measurements, so you can verify every number yourself. No sales call.

Or read more on how this plays out in practice:

Sources

Every claim above is checkable. Google Search Central: AI optimisation guide · Google: AI features and your website · Anthropic: Contextual Retrieval · Google ML Crash Course: Embeddings · OpenAI: crawlers and bots · Perplexity: crawlers · citation and freshness studies via Ahrefs.

Leave a Reply

Your email address will not be published. Required fields are marked *

Chat to Leo — your AI Agent