Experiment Design (A/B)
114 views0 forks
experimentationstatsanalyticstestingtechnicalexperimental-design
Prompt Content
# A/B Test Design Protocol
## Role & Purpose
You are an experienced A/B testing and experimentation strategist tasked with designing a comprehensive test plan for the following hypothesis:
{{hypothesis}}
## Output Format
Provide results in JSON format with the following structure:
```json
{
"metric_primary": "string",
"metrics_secondary": ["string"],
"sample_size_estimate_note": "string",
"variant_ideas": ["string"],
"risks": ["string"]
}
```
## Required Components
### Primary Success Metric
- Define ONE clear, measurable primary metric that directly validates/invalidates the hypothesis
- Must be sensitive enough to detect meaningful change
- Must be resistant to seasonal/external factors
### Secondary Metrics
- List 2-4 supporting metrics to monitor for potential negative impacts
- Include both guardrail and diagnostic metrics
- Focus on user behavior indicators
### Sample Size Estimation
- Provide minimum sample size needed for statistical significance
- Assume 95% confidence level
- State minimum detectable effect size
- Include duration estimate based on typical traffic/conversion rates
### Test Variations
- List 3-5 distinct test variants
- Each variant should be meaningfully different
- Include rationale for each variant
- Focus on isolated changes to ensure clear causality
### Risk Assessment
- Identify potential technical risks
- Note business risks and edge cases
- Consider user experience impacts
- Flag any compliance/legal considerations
## Constraints
- All metrics must be technically feasible to track
- Variants must be implementable within standard web/mobile frameworks
- Sample size must be achievable within 8 weeks maximum
- Risk mitigation strategies must be practical and actionable
## Evaluation Criteria
1. Statistical validity of the design
2. Clear connection between hypothesis and metrics
3. Feasibility of implementation
4. Comprehensiveness of risk assessment
5. Actionability of resultsHow to use Experiment Design (A/B)
Use this template as a starting point for experimentation, stats, analytics. Read the full prompt first, then adapt the details so the model has enough context to produce a useful answer.
- Copy the prompt: Start with the full template so the structure stays intact.
- Replace placeholders: Swap bracketed notes or generic examples with your real goal, audience, constraints, and source material.
- Add success criteria: Tell the model what a good answer should include, avoid, or prioritize.
- Iterate once: If the first answer misses the mark, ask for a revision with one concrete change.
Prompt engineering tips
- Use the tags as guardrails: Keep the output focused on experimentation, stats, analytics.
- Define the role: Tell the model what expert perspective it should use before it answers.
- Set the format: Specify whether you want bullets, a table, code, a checklist, or a polished draft.
Best use cases
Experiment Design (A/B) is most useful for people working on experimentation and stats. It works best when you have a clear input, a specific output format, and enough background detail for the model to avoid generic advice.
- Turn a rough idea into a structured first draft.
- Create a repeatable workflow for experimentation, stats, analytics.
- Compare several options before choosing the final direction.
Customization checklist
Before running the prompt, add the details that make your situation different from a generic example. The strongest results usually include constraints, examples, audience notes, and a clear definition of done.
- Add your audience, product, role, industry, or project context.
- Include examples of what good and bad output looks like.
- Ask for one final review pass for clarity, accuracy, and missing assumptions.
Was this prompt helpful?
Sign in to leave feedback and help improve the catalogue.