Science Q&A

TL;DR（中文）

• Science Q&A 的关键不是“背知识”，而是：能否从给定 context 中做正确抽取与定位。
• 适合用来测试：阅读理解、信息抽取、以及在不确定时输出 “Unsure about answer”。
• 落地时建议：把 context 切片、限制输出长度、并加 evidence 字段或引用片段。

Background

The following prompt tests an LLM's capabilities to perform science question answering.

How to Apply（中文）

这个 prompt 是一个典型的 “answer based on context” 模式：

• 把你的业务资料（FAQ、policy、product spec、research notes）作为 Context
• 把用户问题作为 Question
• 让模型只基于 Context 回答，缺信息就输出 “Unsure about answer”

How to Iterate（中文）

1. 输出格式结构化：要求 Answer + Evidence（从 Context 复制关键句）
2. 控制回答长度：短回答更容易对齐输入证据
3. 加 abstain 规则：不确定就 “Unsure about answer”，不要猜
4. 多问题批处理：对同一 Context 一次问多个 Question，用 evaluation 比较一致性

Self-check rubric（中文）

• Answer 是否能在 Context 中找到对应证据？
• 是否出现 Context 之外的信息（潜在 hallucination）？
• 是否正确执行 abstain 规则（不确定就 “Unsure about answer”）？

Practice（中文）

练习：用一段 200-400 字的内部文档当 Context，写 5 个问题覆盖：

• 事实抽取（who/what/when）
• 因果/目的（why）
• 数值/条件（how many/under what condition）
• 反事实（如果缺证据，必须 “Unsure about answer”）

Prompt

Answer the question based on the context below. Keep the answer short and concise. Respond "Unsure about answer" if not sure about the answer.

Context: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.

Question: What was OKT3 originally sourced from?
Answer:

Code / API

OpenAI (Python)

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": "Answer the question based on the context below. Keep the answer short and concise. Respond \"Unsure about answer\" if not sure about the answer.\n\nContext: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.\n\nQuestion: What was OKT3 originally sourced from?\nAnswer:",
        }
    ],
    temperature=1,
    max_tokens=250,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0,
)

Fireworks (Python)

import fireworks.client

fireworks.client.api_key = "<FIREWORKS_API_KEY>"

completion = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Answer the question based on the context below. Keep the answer short and concise. Respond \"Unsure about answer\" if not sure about the answer.\n\nContext: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.\n\nQuestion: What was OKT3 originally sourced from?\nAnswer:",
        }
    ],
    stop=["<|im_start|>", "<|im_end|>", "<|endoftext|>"],
    stream=True,
    n=1,
    top_p=1,
    top_k=40,
    presence_penalty=0,
    frequency_penalty=0,
    prompt_truncate_len=1024,
    context_length_exceeded_behavior="truncate",
    temperature=0.9,
    max_tokens=4000,
)