Learn how to move beyond agency-driven conversion testing and build a scalable, org-wide experimentation capability — from your first in-house test to a full experimentation platform.
Taught by an experimentation leader with 20+ years and 6,000+ experiments.
Experimentation experience across
The problem with CRO
Most companies start by hiring a CRO agency. It feels like progress. But the testing stops the moment the retainer does.
Agency CRO is rented capability, not built capability. When the retainer ends, so does the testing.
Your customer learnings live in their Notion docs and decks, not in your organisation's memory.
CRO focuses on conversion. It rarely drives product decisions, pricing, UX or operations.
Teams don't learn to think experimentally. The decision-making culture doesn't change.
A CRO agency can't embed experimentation into product, marketing and leadership simultaneously.
Each test is a one-off. You're not building a library of insight that makes the next test smarter.
The shift
Experimentation isn't a bigger CRO programme. It's a different operating model — one where testing is how your whole organisation makes decisions.
Before · CRO agency model
After · Experimentation platform
Most organisations are somewhere on this path. This programme takes you from wherever you are to a self-sustaining platform.
Outsourced testing, conversion focus, no internal ownership
First internal experiments, basic tools, ad-hoc methodology
Hypothesis backlog, clear process, documented learnings
Self-service, cross-team, systematic insight generation
Embedded in product decisions, leadership and culture
The curriculum
Eight practical modules covering everything from research and statistics to culture-building and AI-powered test design.
Why the shift matters, what changes, and how to make the case internally for a programme.
Build a fact-based test backlog using analytics, user research and competitive analysis.
What makes a valid experiment: sample sizes, significance, guardrail metrics and avoiding false positives.
Tools, workflows, roles, reporting cadence and how to run your first three tests without an agency.
How to get product, marketing and leadership bought in and experimenting alongside you.
How to turn test results into durable organisational knowledge that compounds over time.
Tooling, governance, templates and workflows that let teams run tests without a central gatekeeper.
Use AI at every stage: hypothesis generation, test analysis, insight synthesis and roadmap prioritisation.
8 weeks, live
One live session a week, each ending with something you've actually built — a hypothesis, a test design, a process, or a piece of your programme.
The difference between CRO and experimentation, why it matters, and how to map your current state.
Build a fact-based hypothesis backlog using analytics, heuristics, session data and user research.
How to design valid experiments: sample size, significance thresholds, guardrails and avoiding common traps.
Tools, roles, cadence and the exact workflow for running your first in-house tests without an agency.
How to bring product, marketing and leadership into the programme — and make testing how everyone decides.
Turn results into durable insight: how to document, share and compound organisational learnings.
Governance, tooling templates and the workflows that let teams experiment without a central bottleneck.
AI at every stage of the testing cycle — an unfair advantage for teams who get there first.
Who it's for
Whether you're just running your first A/B tests or trying to build an org-wide programme, this is for digital teams who want to own their own capability.
Ready to move beyond agency work and build a programme you own and can scale internally.
Want to embed experimentation into product decisions and build a data-driven development culture.
Moving off agency retainers and building in-house testing capability for the first time.
Supporting a testing programme and want to build the methodology and infrastructure to do it properly.
Responsible for building a testing culture and want a proven framework to do it systematically.
At any stage — from first test to established programme — who want to level up their experimentation maturity.
Experimentation leader · Analytics practitioner · AI builder
Who's teaching this
Olly Paton has spent over 20 years in digital performance, analytics and experimentation. He has led and contributed to experimentation programmes at Expedia, Hotels.com, Trainline, Hastings Insurance and Skyscanner — from small in-house programmes to scaled, org-wide platforms.
He has designed and run over 6,000 experiments across ecommerce, travel, insurance and finance. He spoke at Speero's Experimentation Circus on AI and experimentation, and is the creator of Conform, an A/B testing and experimentation platform built for digital teams.
This programme is built directly from his experience of building and scaling experimentation programmes — what works, what doesn't, and how to avoid the mistakes that slow most programmes down.
What you walk away with
By the end of eight weeks, you'll have built the foundations of a real in-house experimentation programme — not just learned the theory.
A structured plan to move your organisation from its current state to a self-service platform.
A prioritised, evidence-based backlog of tests ready to run — built on real research, not gut feel.
The knowledge to design valid tests, read results correctly and avoid the most common mistakes.
A framework for getting cross-team buy-in and making experimentation how your organisation decides.
Templates, governance and tooling recommendations to let teams run tests without a central bottleneck.
A repeatable workflow using AI for hypothesis generation, analysis and insight synthesis.
A system for capturing and compounding test insights so every experiment makes the next one smarter.
Leave with three fully designed, ready-to-run experiments for your own product or site.
Founding cohort
The founding cohort gets the most access, the lowest price it will ever be, and direct input into the curriculum.
Founding member access · lifetime content included
Waitlist members get first access to spaces and founding pricing.
Join the waitlist
Tell us about your organisation and where you are on the experimentation journey. Waitlist members hear first when founding spaces and pricing open.
Thanks — you'll be first to hear when founding-cohort spaces and pricing open. Keep an eye on your inbox.
Prefer to talk first?
Book a strategy call and we'll talk through where your organisation is on the experimentation journey and what it would take to build a programme that scales — cohort or not.
Also at the Academy: Managing Ecommerce With AI →