Similarity Caching: When Can Two Questions Share an Answer?

Cache hit, cache miss, and now a similarity hit. What changes when we stop asking for an exact match?

Two people can ask the same question with different words. Reusing a suitable answer could save work, especially when producing it is expensive. The difficult part is deciding whether "close" really means the answer still fits.

Let's build that distinction into a small help assistant, starting with the similarity calculation and ending with a reuse decision we can defend.

Exact identity and nearby meaning are different

An exact cache checks a defined key. If two prompts produce different keys, it won't reuse a result unless the application's normalization or key design makes them equivalent.

A similarity search looks for nearby candidates. For music search, "Kesariya" and "Kesariya acoustic version" may be closely related, but the listener may specifically want one recording. Returning something related can be useful for discovery while still being wrong for exact selection.

The same distinction matters in an answer cache. "How do I enable dark mode?" and "Where is the dark-theme switch?" may need the same instructions. "How do I disable dark mode?" may need the opposite action despite sharing most of the wording.

Represent a question as numbers

An embedding model converts text into a numeric representation, commonly called a vector. Nearby representations can help retrieve related questions.

The model, its version and the input affect those numbers. We can't assign a universal similarity score to two titles or assume every paraphrase always gets the same result.

Cosine similarity compares vector directions. For nonzero vectors, it ranges from -1 to 1; it doesn't itself measure whether two answers are interchangeable. A value of 1 means aligned directions under this calculation, not proof of identical meaning.

Make the calculation visible

Here's a short implementation using deliberately invented two-dimensional vectors:

from math import sqrt

def cosine(left, right):
    if len(left) != len(right):
        raise ValueError("Dimensions must match")
    a = sqrt(sum(value * value for value in left))
    b = sqrt(sum(value * value for value in right))
    if a == 0 or b == 0:
        raise ValueError("Nonzero vectors required")
    return sum(x * y for x, y in zip(left, right)) / (a * b)

assert cosine([1, 0], [1, 0]) == 1
assert cosine([1, 0], [0, 1]) == 0
assert abs(cosine([1, 0], [0.8, 0.6]) - 0.8) < 1e-9

These aren't measured embeddings for the example questions. They show what the calculation does when directions align, differ by a right angle or lie somewhere between.

A cache library may expose cosine distance instead of similarity, with a different threshold comparison. Check the documented metric before deciding that "above 0.9" or "below 0.1" means the policy you intended.

Retrieve candidates inside the right scope

Suppose our help assistant answers questions about version 3 of an app's theme settings. An answer for version 1 may point to a menu that no longer exists.

Restrict candidates to the appropriate documentation, language and product version. Include account or permission context when the answer depends on it, or exclude private answers from shared reuse altogether.

RedisVL's semantic-cache reference provides filterable fields and a distance threshold as practical mechanisms. Configuring a field only helps if the retrieval actually applies it. A high score cannot repair an answer drawn from the wrong version.

Check suitability before returning the answer

A similarity score is evidence for considering a candidate. The application still needs a policy for whether its answer is safe to reuse.

For this narrow assistant, we might use approved question groups with stable instructions. A question assigned to "enable dark mode" can reuse that group's current answer; a disable request belongs elsewhere. More open-ended systems need their own validated decision process and conservative handling when uncertain.

Which small change in this question would require a different answer?

Negation, a product version or a private account reference can change the required answer while leaving much of the sentence familiar. Test those near matches deliberately.

Measure wrong reuse, not just hits

Build an evaluation set with genuine paraphrases, opposite requests, obsolete instructions and different access contexts. Check the final returned answer, not only whether retrieval found a close candidate.

A higher hit rate isn't an improvement if it increases wrong reuse. Track that error alongside latency, model work saved and the cost of embedding, search and suitability checks. On a miss, those checks add work before normal answering begins.

The original post points to Ryo Nakamura and Noriaki Kamiyama's Analysis of Similarity Caching on General Cache Networks. This help-assistant example is our separate application of the broader idea, not a claim about that paper's implementation.

Start with answers that genuinely survive rewording

A narrow public help topic is a useful place to test reuse because we can define the correct instruction and version. We can then expand only where evaluation shows the policy still works.

A similarity hit should mean more than finding familiar words. It should mean we saved the expensive work while preserving the answer the reader needed. Start with that requirement, and let the score help the decision rather than make it alone.