APE
Automatic Prompt Engineer: auto-generate and select instructions
Image source: Zhou et al. (2022)
Zhou et al. (2022) proposed Automatic Prompt Engineer (APE), a framework for automatic instruction generation and selection. The instruction generation problem is framed as natural language synthesis -- using LLMs as black-box optimizers to generate and search candidate solutions.
The first step involves a large language model (as an inference model) that receives output demonstrations to generate instruction candidates for the task. These candidates guide the search process. A target model executes the instructions, and the best instruction is selected based on computed evaluation scores.
APE discovered a zero-shot CoT prompt that's better than the human-designed "Let's think step by step" prompt (Kojima et al., 2022).
The prompt "Let's work this out in a step by step way to be sure we have the right answer." triggers chain-of-thought reasoning and improved performance on MultiArith and GSM8K benchmarks:
Image source: Zhou et al. (2022)
This paper touches on an important topic in prompt engineering: automatically optimizing prompts. While we don't go deep into this here, here are some key papers if you're interested:
- Prompt-OIRL - Uses offline inverse reinforcement learning to generate query-dependent prompts.
- OPRO - Introduces the idea of using LLMs to optimize prompts: telling LLMs to "take a deep breath" improves math performance.
- AutoPrompt - Proposes a gradient-guided search method for automatically creating prompts for various tasks.
- Prefix Tuning - A lightweight fine-tuning alternative that prepends trainable continuous prefixes for NLG tasks.
- Prompt Tuning - Proposes learning soft prompts through backpropagation.
📚 Related resources
❓ Common questions
Open a question to review the practical answer.
What is APE and why call it "Automatic Prompt Engineer"?
Zhou et al. 2022 reframe instruction generation as natural-language synthesis under black-box optimisation. One LLM acts as the inference model — it reads input-output examples and proposes candidate instructions; a target model then executes each candidate and scores it; the highest scorer wins. No human writes the prompt — hence "automatic prompt engineer".
What was the best zero-shot CoT prompt APE discovered?
The headline result: APE discovered "Let's work this out in a step by step way to be sure we have the right answer." — and it beat Kojima 2022's hand-crafted "Let's think step by step" on both MultiArith and GSM8K. Evidence that LLMs can optimise their own prompts, and that human-written ones aren't necessarily optimal.
What's APE's edge over hand-crafted prompt engineering?
Three: (1) systematic search — hundreds of candidate instructions instead of two or three written on intuition; (2) no engineer-side bias about how the task "should" be phrased; (3) iterative refinement against an evaluation score. Costs: you need a labeled input-output set as the eval rig, and every pass burns a lot of tokens. Worth it when the prompt will be reused at scale once optimised.
What other automatic prompt optimisation methods exist besides APE?
The paper points to several: Prompt-OIRL (offline inverse RL for query-conditional prompts), OPRO (have the LLM "take a deep breath" to optimise math), AutoPrompt (gradient-guided search), Prefix Tuning (lightweight fine-tune alternative), Prompt Tuning (soft-prompt learning via backprop). They split into two camps: search-based (APE / OPRO) vs gradient-based (AutoPrompt / Prefix).
Can I use APE without serious GPU resources?
Yes, but pick the right variant. APE in the original paper hits commercial APIs to evaluate candidates — no GPU needed, just budget for N inference calls. GPU is the price for soft-prompt methods like Prefix Tuning / Prompt Tuning, which need backprop. Small teams can run APE-style search end to end: write 50-100 candidates → score on a small eval set → pick the winner, all on plain API calls.