Science Q&A

TL;DR

• The key to Science Q&A isn't "memorizing facts" -- it's whether the model can do correct extraction and localization from a given context.
• Good for testing: reading comprehension, information extraction, and outputting "Unsure about answer" when uncertain.
• Production tip: chunk the context, limit output length, and add an evidence field or citation snippet.

Background

The following prompt tests an LLM's capabilities to perform science question answering.

How to Apply

This prompt is a classic "answer based on context" pattern:

• Use your business materials (FAQ, policy, product spec, research notes) as Context
• Use the user question as Question
• Have the model answer based only on Context, and output "Unsure about answer" if info is missing

How to Iterate

1. Structured output format: require Answer + Evidence (copy key sentence from Context)
2. Control answer length: shorter answers are easier to align with input evidence
3. Add abstain rule: if unsure, say "Unsure about answer" -- don't guess
4. Batch processing: ask multiple Questions against the same Context, use evaluation to compare consistency

Self-check Rubric

• Can the Answer find supporting evidence in the Context?
• Is there any information from outside the Context (potential hallucination)?
• Does it correctly execute the abstain rule (says "Unsure about answer" when uncertain)?

Practice

Exercise: use a 200-400 word internal document as Context, and write 5 questions covering:

• Fact extraction (who/what/when)
• Cause/purpose (why)
• Numeric/conditional (how many/under what condition)
• Counterfactual (if evidence is missing, must say "Unsure about answer")

Prompt

Answer the question based on the context below. Keep the answer short and concise. Respond "Unsure about answer" if not sure about the answer.

Context: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.

Question: What was OKT3 originally sourced from?
Answer:

Code / API

OpenAI (Python)

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": "Answer the question based on the context below. Keep the answer short and concise. Respond \"Unsure about answer\" if not sure about the answer.\n\nContext: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.\n\nQuestion: What was OKT3 originally sourced from?\nAnswer:",
        }
    ],
    temperature=1,
    max_tokens=250,
    top_p=1,
    frequency_penalty=0,
    presence_penalty=0,
)

Fireworks (Python)

import fireworks.client

fireworks.client.api_key = "<FIREWORKS_API_KEY>"

completion = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Answer the question based on the context below. Keep the answer short and concise. Respond \"Unsure about answer\" if not sure about the answer.\n\nContext: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.\n\nQuestion: What was OKT3 originally sourced from?\nAnswer:",
        }
    ],
    stop=["<|im_start|>", "<|im_end|>", "<|endoftext|>"],
    stream=True,
    n=1,
    top_p=1,
    top_k=40,
    presence_penalty=0,
    frequency_penalty=0,
    prompt_truncate_len=1024,
    context_length_exceeded_behavior="truncate",
    temperature=0.9,
    max_tokens=4000,
)