Generated Knowledge Prompting
Generate knowledge first, then answer (boosts commonsense / knowledge-intensive tasks)
Image source: Liu et al. 2022
LLMs keep getting better, and one popular technique is incorporating knowledge or information to help the model make more accurate predictions.
But here's the question: can the model also generate knowledge before making a prediction? That's what Liu et al. 2022 tried -- generate knowledge to use as part of the prompt. Specifically, how much does this help with tasks like commonsense reasoning?
Let's try a simple prompt:
Prompt:
Part of playing golf is trying to get a higher score than other people. Yes or no?
Output:
Yes.
Wrong. This exposes an LLM limitation -- tasks that need real-world knowledge trip it up. So how do we fix this with generated knowledge?
First, we generate some "knowledge":
Prompt:
Input: Greece is larger than Mexico.
Knowledge: Greece has an area of about 131,957 square kilometers, while Mexico has an area of about 1,964,375 square kilometers, making Mexico much larger than Greece.
Input: Glasses always fog up.
Knowledge: Glasses fog up when water vapor from your breath, sweat, or surrounding humidity condenses on a cooler lens surface, forming tiny droplets that look like a thin film.
Input: Fish can think.
Knowledge: Fish can show learning and memory abilities. Some fish demonstrate problem-solving and can retain information over time, which can help them navigate environments and social interactions.
Input: A common effect of smoking many cigarettes over a lifetime is a higher-than-normal chance of developing lung cancer.
Knowledge: Smoking increases the risk of lung cancer compared to never-smokers. The risk generally rises with greater smoking intensity and duration.
Input: A rock and a pebble are the same size.
Knowledge: In sedimentology (Udden-Wentworth scale), a pebble is typically 4 to 64 mm in diameter. Larger clasts (e.g., cobbles) are 64 to 256 mm, while smaller grains are below 4 mm.
Input: Part of playing golf is trying to get a higher score than other people.
Knowledge:
Knowledge 1:
The goal of golf is to complete a set of holes in as few strokes as possible. A round typically includes 18 holes. Each stroke counts toward your total, and the player with the lowest total number of strokes wins.
Knowledge 2:
Golf is a precision club-and-ball sport where players use various clubs to hit a ball into a series of holes on a course in as few strokes as possible. The objective is to finish with the lowest score, where score is the total number of strokes across all holes.
We used the prompt from Liu et al. 2022.
Next step: integrate the knowledge and get a prediction. I reformatted the question as QA format to guide the answer.
Prompt:
Question: Part of playing golf is trying to get a higher score than other people. Yes or no?
Knowledge: The goal of golf is to complete a set of holes in as few strokes as possible. A round typically includes 18 holes. Each stroke counts toward your total, and the player with the lowest total number of strokes wins.
Explanation and answer:
Answer 1 (very high confidence):
No. In golf, the objective is not to get a higher score. Instead, players aim to take as few strokes as possible. The lowest total number of strokes determines the winner.
Question: Part of playing golf is trying to get a higher score than other people. Yes or no?
Knowledge: Golf is a precision club-and-ball sport where players use various clubs to hit a ball into a series of holes on a course in as few strokes as possible. The objective is to finish with the lowest score, where score is the total number of strokes across all holes.
Explanation and answer:
Answer 2 (much lower confidence):
No. In golf, players try to take the fewest strokes possible. The winner is determined by the lowest total number of strokes, not a higher score.
Something interesting happened here. The first answer was very confident, while the second was less so. I've simplified the process for demonstration purposes -- there are more nuances when arriving at the final answer. Check the paper for the full details.
📚 Related resources
❓ Common questions
Open a question to review the practical answer.
What is Generated Knowledge prompting?
Liu et al. 2022 — get the LLM to generate relevant knowledge before answering, and feed that knowledge back as part of the prompt. Two steps: few-shot the model to produce knowledge statements about the input, then re-prompt with the generated knowledge plus the original question. Built for commonsense reasoning tasks where the model's world knowledge is patchy.
Is Generated Knowledge the same thing as RAG?
No. RAG fetches real documents from an external store (vector DB, Wikipedia); Generated Knowledge has the LLM produce its own knowledge — really just unlocking the model's parametric memory. RAG is more hallucination-resistant (sources are traceable); Generated Knowledge needs no infrastructure but can fabricate facts. They compose: pull real evidence with RAG, then expand with the model.
When should I use Generated Knowledge?
Commonsense questions where you need a sliver of world knowledge but don't require strictly cited facts. The paper's example: "Is part of golf trying to score higher than others?" Asked directly the LLM says Yes (wrong); pre-generate "golf rule: fewer strokes is better" and it self-corrects. Best returns are on small-to-mid models — large models already encode most commonsense, so the lift shrinks.
How does the confidence of generated knowledge affect the answer?
A key observation from the paper: feeding the same question two different pieces of generated knowledge produces noticeably different confidence in the final answer — sharper, more focused knowledge ("fewest strokes wins") lets the model commit; vaguer knowledge leaves it hedging. Final decisions should aggregate over multiple knowledge samples, not bet on a single draft.
How does Generated Knowledge combine with CoT?
Standard recipe: generate knowledge → splice it into the prompt → let the model run CoT reasoning. Beats plain CoT because the knowledge layer is now explicit and the model can quote it as a premise instead of assuming. The inverse also works — run CoT first to spot which step lacks information, then targetedly generate just that piece of knowledge in a second pass.