Rerank — Turning Recall into Selection

Chapter 3 sketched the three-stage shape of selection (filter / rerank / LLM-judge). This chapter pulls the middle stage apart.

Why retrieval uses bi-encoder and rerank uses cross-encoder

Both encoders answer "how relevant are query and doc," differently —

Bi-encoder: query and doc go through the encoder separately, producing two independent embeddings, then cosine similarity.

Query   ──→ Encoder ──→ q_vec  ┐
                                 ├─→ cosine(q_vec, d_vec) → score
Doc     ──→ Encoder ──→ d_vec  ┘

Win: doc embeddings are computed offline and stored in a vector DB. Querying a 10M-doc corpus returns in milliseconds — only the query vector runs at runtime.

Cost: no cross-attention, so the model can't see specific match points. Query "Is Australia's GST 10%?" vs doc "Australia's consumption tax (GST) is currently 10%." — bi-encoder knows they're similar but can't tell whether the similarity is on "tax rate" or "country."

Cross-encoder: query and doc are concatenated through the encoder together, outputting a 0-1 score.

[Query] [SEP] [Doc] ──→ Encoder ──→ score

Win: cross-attention sees token-to-token relationships. Precision lands 30-50% above bi-encoder (Sentence-Transformers comparison).

Cost: every query × doc requires a forward pass. 1 × 10M = 10M forwards. Can't be the first-stage filter — only the rerank.

The standard two-stage selection pipeline

Stage A (Retrieval, bi-encoder)
  10M docs → 提前 embed → 向量库
  Query → embed → ANN top-50
  ~50ms, 召回率 90%+

Stage B (Rerank, cross-encoder)
  Top-50 + Query → cross-encoder 逐对打分 → top-10
  ~200-500ms（batch + GPU）

Stage C (LLM-as-judge, optional)
  Top-10 → 便宜 LLM（Haiku/4o-mini）yes/no → top-3
  ~1-2s, $0.001/query

bi-encoder top-N caps the rerank ceiling. N=50 is the common compromise.

Mainstream rerank options (2026-04)

NDCG@10 from vendor announcements + BEIR benchmark third-party comparisons. Corpora vary ±10% — run your own eval.

• < 10K queries/mo → Cohere/Voyage

• 100K queries/mo → BGE self-host (cost crushes managed)

• All-in Anthropic → Contextual Retrieval
• Chinese-heavy → BGE / Cohere

Rerank code (cross-encoder, BGE)

# tested: 2026-04-26 · sentence-transformers@3.0.1
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")  # 首次 ~2GB

def rerank_top_k(query: str, docs: list[str], top_k: int = 10):
    """50 candidates → top_k. GPU batch_size=32 单次 ~200ms."""
    pairs = [(query, doc) for doc in docs]
    scores = reranker.predict(pairs, batch_size=32)
    ranked = sorted(zip(scores, docs), key=lambda x: x[0], reverse=True)
    return ranked[:top_k]

candidates = vector_db.search(query_embedding, top_k=50)
top10 = rerank_top_k(query, [c.text for c in candidates], top_k=10)

Without GPU batching, 50 pairs takes 5 seconds — production needs GPU or managed API.

LLM-as-judge — final filter on top of rerank

Rerank misses one class: topically relevant but doesn't actually answer. Query "Australia GST registration threshold" — rerank ranks "History of Australia's GST" #3. Both about GST, both score high, but doc never answers the threshold question.

Cheap model decides directly:

# tested: 2026-04-26 · anthropic@0.40.0
def judge(query: str, doc: str) -> bool:
    """Haiku 判断 doc 能否回答 query. ~$0.0001/call."""
    prompt = f"""你是检索质量评估员。
Query: {query}
Doc: {doc}

这个 Doc 能直接回答 Query 吗？只回答 yes 或 no。"""
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        messages=[{"role": "user", "content": prompt}]
    )
    return resp.content[0].text.strip().lower().startswith("yes")

Cost: 10 candidates × $0.0001 = $0.001/query. 10K/day costs $10.

Skip when: realtime chat (users feel +1-2s latency), or retrieval is already stable (eval doesn't show judge gains).

Real JR case: implicit rerank inside daily-jobs

daily-jobs "Stage 2: Rerank" isn't a cross-encoder model — it's structured tag + business rule lightweight rerank:

# tested: 2026-04-26 · routine: Daily Jobs - AI Engineer Bootcamp
# 18 候选 job → 3 final pick

Step 1: 打 tier 标签
  - aspirational  → Mid-Senior + brand in {Apple, Google, Canva, Atlassian, ...}
  - actionable    → Junior/Graduate + brand in {AU Big 4 banks, top tech}
  - special       → title 含 {"Intern", "Graduate Program", "2026-27 Start"}

Step 2: tier 内 rerank
  - 优先 posted < 24h
  - 优先 title 含 {"AI Engineer", "ML Engineer", "LLM Engineer"}
  - 优先 location 在 Sydney/Melbourne

Step 3: 每 tier 选 1 → 3 final picks

"Business rule as rerank" — with clean ranking signals (tier, freshness, keywords), rules beat cross-encoder on explainability. But serious semantic search ("FastAPI vs Django for AI backend?") defeats rules — needs cross-encoder.

Managed API vs Self-host — Trade-off

JR experience: first RAG project went on Cohere; once shipped, monthly bill + compliance decided BGE migration. Validate on managed first, optimize cost on self-host second.

Takeaway

Bi-encoder solves "find the candidates," cross-encoder solves "rank them precisely." The two stages stack to 90% recall + 80%+ precision. LLM-as-judge is the optional third stage, catching "relevant but doesn't answer." Production RAG pick: < 50K/month go managed (Cohere), > 100K/month go BGE self-host.

---

References

1. Reimers, N. & Gurevych, I. Sentence-Transformers: Cross-Encoders documentation — bi vs cross encoder 工程对比.
2. BGE. BAAI bge-reranker-v2-m3 model card — 中英双语 rerank 模型.
3. Cohere. Rerank API documentation — managed rerank API.
4. Anthropic. (2024-09-19). Introducing Contextual Retrieval — Anthropic 自家 RAG 改进方案.
5. Thakur, N. et al. BEIR benchmark — retrieval / rerank 第三方评测基准.
6. Voyage AI. Rerank-2 API docs — token 计费 rerank.

Production case: JR Academy omni-report Daily Jobs routines — 用业务规则（tier + freshness + keyword）做 rerank 的轻量模式.