Marauders — an offline-first AR guide, grounded without a vector database

Marauders is an augmented-reality tour guide that works with the phone in airplane mode. Point it at a checkpoint — the gate at the Taj Mahal, a doorway on a venue walk — and it recognises the target, runs the narration, and answers questions about the thing in front of you. I wrote the backend. The decision worth writing up is not the AR: a system whose entire job is answering questions has no vector database in it, and did not need one.

Monuments are exactly where mobile networks stop working

Thick stone, dense crowds, a cell tower provisioned for a smaller decade. Any design that streams narration fails at the precise moment the visitor is standing in front of the thing they came to see — which is also the moment the product is supposed to be worth having.

So the rule was set before any code: every visitor-facing asset — narration audio, images, AR reference targets, checkpoint copy, in all four languages — is baked into one archive at build time. The app downloads that archive once, over whatever network exists at the hotel, and then the tour runs with zero server calls. Only open-ended voice questions reach the backend, and when they cannot reach it, the app still answers.

Why not just use a vector store?

Retrieval exists to select a small relevant subset from a corpus too large to fit in a prompt. That is the whole job. Every other property people attribute to a vector database — freshness, citations, grounding — is available by other means; selection under size pressure is the only thing that requires it.

At a checkpoint there is no size pressure. The corpus is the facts for that one checkpoint: a hand-written pack of a few dozen assertions. It fits in a system prompt with room left over.

Reaching for RAG here is correct reasoning applied to an assumption nobody stopped to restate — that the corpus is too big. At a checkpoint the assumption is simply false, and everything built on top of it inherits the error.

Retrieval is a fetching step. It earns its place when the fetching is hard, and at a checkpoint the fetching is finished before the question is even asked.

The camera is a better retriever than embedding similarity

An index on top of a few dozen facts buys nothing and costs plenty — an embedding model, a vector store to operate, an extra hop on the hot path, and a new failure mode where the retriever hands the model the wrong passage and the model answers it fluently and wrongly.

The camera already knows which checkpoint it is looking at, and that is a better retriever than embedding similarity, because it is a fact rather than an estimate.

So the request carries the checkpoint id and the visitor's language, and the backend assembles the prompt from exactly that checkpoint's facts and nothing else.

def ask(checkpoint_id: str, question: Audio | str, lang: str) -> Answer:
    facts  = store.facts_for(checkpoint_id, lang)   # curated, small, authored
    system = build_prompt(facts, lang, refuse_outside=True)
    text   = whisper.transcribe(question) if question.is_audio else question
    reply  = gpt.respond(system=system, user=text)
    return speech.synthesize(reply, lang)

There is no branch in that function where a retriever can be wrong, because there is no retriever. The selection happened in the physical world when the visitor pointed a camera at a gate.

What would actually make me build the index

The easy version of this answer is "when the corpus gets big", and it is wrong. A property with hundreds of checkpoints does not need semantic search; it needs a filter. WHERE checkpoint_id IN (…) against SQLite scales a long way past anything a monument will produce, and full-text search over a few thousand curated assertions is cheap and exact. Size alone never forced this decision.

What forces it is a change in the shape of the question. Every question Marauders answers is scoped by an attribute the camera already resolved. The moment a visitor asks something with no attribute to filter on — "which part of this was rebuilt after the fire", where that fact sits in nobody's checkpoint pack in those words — a filter has nothing to match, and you need proximity in meaning instead. That is the line. Corpus size is only the thing that usually walks you towards it.

I have built the other side. A Pinecone-backed retrieval system over Canadian immigration policy, where a question genuinely cannot be resolved to a filter — the applicant's situation has to be matched against policy that never names it — and selection is the entire problem. Same author, opposite decision, and the difference is not how much text there was.

I should be honest about how well I actually understand that line, because I am still working it out. I have built one system on each side of it and I can describe the difference cleanly enough afterwards. What I could not do is stand in front of a new corpus and tell you in advance which side it falls on. That is the part I am still learning, and it is why the experiment is on the list rather than the answer being in this paragraph: hold the model fixed, grow the pack, and find where grounded accuracy falls faster than a retriever's own error rate introduces new failures.

There is a pattern here I keep running into. The reflex reaches for the newer and more capable component, and the question that actually decides it is what the older, duller one is already doing well enough. A filter is not an exciting answer. It was the right one, and I have started checking for it first.

It is also worth saying that the clock imposed its own answer. Sixteen to eighteen hours of build time is not enough to stand up an embedding pipeline, a vector store and the operational surface underneath them and have any of it be trustworthy by the demo. The decision was correct on the merits. It was also the only decision the clock permitted, and I would rather say that than pretend the constraint played no part.

Refusal is the feature

A small hand-written fact bank means the model will constantly be asked things it does not know. The guardrail matters more than the coverage does.

Refusing all three adversarial cases is not the same as being safe. It is evidence that the refusal path exists and is wired to the prompt rather than to a hope.

What happens when the answer needs the network?

Three answers, in falling order of confidence, and the app walks down the list rather than failing.

What shipped at tier one that weekend is the protocol boundary and the routing behind it, with compatible stubs standing in for the on-device model — the iOS 27 SDK surface was not fully available. The seam is real and the degradation logic runs; the implementation drops in behind an interface that already exists.

A factual correction ships as a rebuild, not a release

SQLite is the authoritative store. Content is edited in an admin panel — the Content Studio — that writes to SQLite, exports YAML, and hands off to a builder that compiles one archive per property.

Everything on the device side of that network check cannot be taken down by a dead cell tower.

GET  /packages/{monumentId}.zip   # no auth, all languages, one file
POST /admin/rebuild               # X-App-Key; recompiles from SQLite
GET  /health                      # liveness + content version

Two things follow from shipping content as a build artifact. Someone fixes a wrong date in the admin panel, hits rebuild, and the next package download carries the correction — no redeploy, no App Store review. That matters more than it sounds, because the content bank was written in a hurry and some of it is known to be wrong. And one archive carries every language: the alternative halves the download and doubles the version matrix, which is the thing that breaks at 3am on demo day. Older packages keep decoding on newer app builds because the endpoint shape never changed.

The room rewards the demo, not the discipline

Kept: content as a build artifact; the refusal-first prompt; rebuild without redeploy; one archive for all four languages; a shared app key instead of a login the panel did not need for a weekend.

Cut: per-language packages; any vector store; every idea that required a second service to operate.

Missed: the AR lock-on takes a beat to acquire. Invisible in real use, very visible in ninety seconds on a stage — image tracking against a photographed target under stage lighting is harder than tracking the real thing. The next hour of work belongs on a pre-warm and a visible searching state, not on any feature.

The build did not place in the top ten. That is the honest reading of the tradeoff: a guide whose distinguishing property is that it declines to invent history does not look like much from six feet away.

Built at SwiftDidLoad, Eternal's iOS hackathon, on 18 and 19 July — three of us, sixteen to eighteen hours of build time. Gitansh Kapoor built the AR, the optimisation and the frontend-to-API integration that held the two halves together; Kartik Masiwal built the core iOS experience and the chatbot surface. Mine was the backend end to end — the FastAPI service, the SQLite content pipeline, the package builder, the Content Studio, the Azure plumbing — and the iOS integration behind it. I had not written Swift before that weekend and I was not on the frontend UI.

Both halves are public under MIT, architecture notes and API flow included, at github.com/kritish08/marauders. The part worth breaking first is the refusal prompt: rebuild the package with a fact removed and see how long the boundary holds.

Appendix: method

The refusal check. Six cases is a smoke test, not an eval suite. Three in-pack and three adversarial, run by hand on one physical device shortly before the demo, on the content bank as it stood that evening. It proves the refusal path exists and is connected; it comes nowhere near characterising it. The error runs optimistic — cases I wrote myself, against a prompt I wrote myself, and I would expect a stranger with an hour to find a hole.

What was not measured. Package size, rebuild time, backend cold start, and the AR lock-on delay in milliseconds. None were instrumented; the lock-on delay is described from watching it, not from a timer. The retrieval crossover point is also unmeasured, and is the experiment I would run first.

Sample and hardware. One physical iPhone, one venue, one weekend, 18–19 July. Zero server calls during a tour is a property of the design rather than a measurement across many tours.

The tier-one stub. Tier one routes to compatible stubs rather than to Apple Foundation Models. Everything described about routing and degradation ran; the on-device inference itself did not.

Counter-argument worth taking seriously. A vector store would have bought a single content path for both scoped and cross-checkpoint questions, and a property with hundreds of checkpoints will want that. The decision here is scoped to a corpus that fits, not to retrieval as a technique.

Two things the repository still says wrong. The README describes a 48-hour hackathon; the build was sixteen to eighteen hours across the two days. It also calls the project a personal one, which undersells two collaborators. Both predate this write-up and both will be corrected.