TL;DR（中文）

• Closed-domain Q&A 的核心是：回答必须严格限制在给定 facts/context 内，不能“自由发挥”。
• 最常见问题是：模型会补全不存在的信息（hallucination），尤其是专业领域（medical/legal/finance）。
• 落地建议：强制 “use exclusively the information above”，并配合 evaluation（对每条 claim 做对照）。

Background

The following prompt tests an LLM's capabilities to answer closed-domain questions which involves answering questions within a specific topic or domain.

⚠️ Note: due to the challenging nature of the task, LLMs are likely to hallucinate when they have no knowledge regarding the question.

How to Apply（中文）

这个示例可以直接迁移到“基于资料改写/生成文档”的场景：

• 输入是 facts（bullet list / 表格 / JSON）
• 输出是 structured note（medical note、case summary、policy memo 等）
• 要求 “只用给定信息”，避免补充推测性内容

How to Iterate（中文）

建议的迭代方式（保持 code block 英文不动）：

1. 明确输出模板（sections + 字段），减少模型自由发挥空间
2. 要求引用：输出每一段都标注对应的 fact 行号/字段名（如果输入可编号）
3. 加 self-check：输出前先列 “claims list”，再逐条对照输入
4. 加 “unknown policy”：遇到缺失信息，输出 “Unknown” 或提出 clarifying questions

Self-check rubric（中文）

• 是否出现输入中不存在的具体信息（诊断、BMI、并发症、治疗建议等）？
• 是否把事实与推测混在一起？是否能明确标注 assumption？
• 是否遵守 “exclusively the information above”？

Practice（中文）

练习：用你自己的工作资料做一次 closed-domain 改写。

1. 准备一份 facts（10-20 条，建议编号）
2. 让模型输出一个结构化 note
3. 再用 truthfulness rubric 做 claim-by-claim 复核（可配合 /prompt-truthfulness-identify-hallucination）

Prompt

Patient’s facts:

-   20 year old female
-   with a history of anerxia nervosa and depression
-   blood pressure 100/50, pulse 50, height 5’5’’
-   referred by her nutrionist but is in denial of her illness
-   reports eating fine but is severely underweight

Please rewrite the data above into a medical note, using exclusively the information above.

Code / API

OpenAI (Python)

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nPlease rewrite the data above into a medical note, using exclusively the information above.",
        }
    ],
    temperature=1,
    max_tokens=500,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0,
)

Fireworks (Python)

import fireworks.client

fireworks.client.api_key = "<FIREWORKS_API_KEY>"

completion = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nPlease rewrite the data above into a medical note, using exclusively the information above.",
        }
    ],
    stop=["<|im_start|>", "<|im_end|>", "<|endoftext|>"],
    stream=True,
    n=1,
    top_p=1,
    top_k=40,
    presence_penalty=0,
    frequency_penalty=0,
    prompt_truncate_len=1024,
    context_length_exceeded_behavior="truncate",
    temperature=0.9,
    max_tokens=4000,
)

Reference

• Sparks of Artificial General Intelligence: Early experiments with GPT-4 (13 April 2023)