Challenge 04 · Self-Consistency Prompting
Self-Consistency Panel 🔴
✓
Zero-Shot
✓
Few-Shot
✓
CoT
4
Self-Con.
5
ToT
What is Self-Consistency Prompting?
Self-consistency prompting runs the same question through multiple independent reasoning paths and selects the most consistent answer via majority vote. Instead of trusting a single model response, you sample diverse reasoning paths and let consensus determine the final answer. This reduces bias and increases reliability — especially for complex judgement tasks.
Scenario
A hiring manager wants an AI hiring advisor that doesn't rely on a single opinion.
By consulting multiple reasoning paths and taking the consensus,
the system avoids bias and gives more reliable recommendations.
Your Task
Build a PartyRock app called "The Hiring Panel" that:
- Takes a candidate profile + job description as input
- Runs it through 3 separate AI "panelist" widgets — each evaluating independently with a different persona
- A 4th "Consensus" widget reads all 3 evaluations and produces the final recommendation based on majority agreement
Rules
- All 3 panelist widgets must receive the same candidate input
- Each panelist must have a distinct evaluation persona in their system prompt
- The 4th consensus widget must explicitly look for majority agreement across all 3
- Suggested personas (use your own variations):
🔧
Technical
Evaluator
Technical
Evaluator
🤝
Culture-Fit
Evaluator
Culture-Fit
Evaluator
⚠️
Risk & Red-Flag
Evaluator
Risk & Red-Flag
Evaluator
Test Candidate (Use This for Testing)
Candidate Profile
5 years Python experience
Led a team of 3 engineers
No formal degree
Switched jobs 4 times in 6 years
Excellent GitHub portfolio with 3 shipped open-source projects
Target Role
Senior Software Engineer at a Series B startup (~50 employees)
How to Build This on PartyRock — Step by Step (5 Widgets)
STEP 1 — Create the app
1. Go to partyrock.aws → "Build your own app"
2. Describe it as: "A hiring panel with multiple evaluators and a consensus advisor"
3. Clear auto-generated widgets and build manually
STEP 2 — Add the input widget
"+ Add widget" → "User Input" → label: "Candidate profile + job description"
STEP 3 — Add 3 Panelist AI widgets
Add 3 separate "AI Generation" widgets:
• Widget 1 — System Prompt: "You are a Technical Skills Evaluator. Assess..."
• Widget 2 — System Prompt: "You are a Culture-Fit Evaluator. Assess..."
• Widget 3 — System Prompt: "You are a Risk and Red-Flag Evaluator. Assess..."
Each connects its User Prompt to the same input widget (@input)
Give each widget a clear label (Panelist 1, 2, 3)
STEP 4 — Add the Consensus widget
Add a 4th "AI Generation" widget — "Consensus Advisor"
System Prompt: instruct it to read all 3 panelist outputs and find majority agreement
User Prompt: connect it to all 3 panelist widgets (@panelist1, @panelist2, @panelist3)
It should output a final HIRE / NO HIRE recommendation with reasoning
STEP 5 — Set Model, Temperature & Top P
Consider: slightly higher temp on panelists creates useful diversity in reasoning paths.
Lower temp on the consensus widget keeps the synthesis focused.
⚠️ Values auto-read from JSON. The scores shown are for the first AI widget found.
STEP 6 — Test & Export
Test with the candidate profile provided. Verify 3 independent evaluations appear.
Mac: Cmd+K | Windows: Ctrl+K → exports JSON
Copy share URL and submit both below.
💡 Your model, temperature, and Top P are extracted from the JSON automatically.
Scoring Breakdown (150 pts)
| Criterion | Points |
|---|---|
| Correct self-consistency structure (3 paths + consensus) | 25 |
| Each panelist has a distinct, well-justified persona | 25 |
| Consensus widget correctly synthesizes all 3 evaluations | 20 |
| Realistic and interesting use case | 15 |
| Model choice + written justification | 25 |
| Temperature value + written justification | 20 |
| Top P value + written justification | 20 |
| Total | 150 |
Submit Your Entry