Open domain Q&A

open-domain question answering prompt example

#TL;DR（中文）

code
Open-domain Q&A
的特点是：没有给 evidence/context，模型很容易
code
hallucination
。
适合用来测试：模型的知识覆盖、回答风格、以及是否能在不确定时说 “I don’t know”。
生产场景建议：优先加
code
RAG
/citations/约束输出格式，并用
code
evaluation
做回归测试。

#Background

The following prompt tests an LLM's capabilities to answer open-domain questions, which involves answering factual questions without any evidence provided.

⚠️ Note: due to the challenging nature of the task, LLMs are likely to hallucinate when they have no knowledge regarding the question.

#How to Apply（中文）

你可以把这个 prompt 当成一个最小“回答风格 + 不确定性处理”的测试用例：

观察模型是否会编造（
code
hallucination
），还是会在不确定时明确说 “I don’t know”
把问题替换成你业务里的典型 open-domain 问题（但注意：没有 evidence 时，正确性无法保证）
对同一问题跑多次，比较一致性（temperature 变动会放大差异）

#How to Iterate（中文）

常见迭代方向（不改 code block，只改外部说明/提示语）：

增加输出 schema（例如要求
code
Answer
/
code
Confidence
/
code
Assumptions
）
加上 “no fabrication” 规则（不知道就说 “I don’t know”）
要求给 citations（如果你同时接入了 search /
code
RAG
）
加
code
self-check
：回答前先列出需要验证的关键事实

#Self-check rubric（中文）

是否在无 knowledge 时仍然硬答（高风险
code
hallucination
）？
是否能清晰区分：确定事实 vs 推测（assumptions）？
输出是否遵守格式？是否出现与问题无关的内容？

#Practice（中文）

练习：把

code

Human

的问题替换成你工作中经常遇到的 5 个 open-domain 问题，并为每个问题写一个判分标准：

正确性（如果你能验证）
是否能在不确定时说 “I don’t know”
是否避免编造具体细节（地点/数字/时间）

#Prompt

markdown
In this conversation between a human and the AI, the AI is helpful and friendly, and when it does not know the answer it says "I don’t know".

AI: Hi, how can I help you?
Human: Can I get McDonalds at the SeaTac airport?

#Code / API

#OpenAI (Python)

python
from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": 'In this conversation between a human and the AI, the AI is helpful and friendly, and when it does not know the answer it says "I don’t know".\n\nAI: Hi, how can I help you?\nHuman: Can I get McDonalds at the SeaTac airport?',
        }
    ],
    temperature=1,
    max_tokens=250,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0,
)

#Fireworks (Python)

python
import fireworks.client

fireworks.client.api_key = "<FIREWORKS_API_KEY>"

completion = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[
        {
            "role": "user",
            "content": 'In this conversation between a human and the AI, the AI is helpful and friendly, and when it does not know the answer it says "I don’t know".\n\nAI: Hi, how can I help you?\nHuman: Can I get McDonalds at the SeaTac airport?',
        }
    ],
    stop=["<|im_start|>", "<|im_end|>", "<|endoftext|>"],
    stream=True,
    n=1,
    top_p=1,
    top_k=40,
    presence_penalty=0,
    frequency_penalty=0,
    prompt_truncate_len=1024,
    context_length_exceeded_behavior="truncate",
    temperature=0.9,
    max_tokens=4000,
)

#Reference

Sparks of Artificial General Intelligence: Early experiments with GPT-4 (16 March 2023)

Prompt 大师