Outcome A
Run an A/B test
ConditionThere are decisions, data and costly mistakes that justify an experiment.
An A/B test does not find the right solution. It shows whether one specific change worked.
We start by identifying the point of loss, its cause and the decision. We test only where the cost of error is high, the result affects the next action and the data supports a fair answer.
- 01Problem
- 02Evidence
- 03Cause
- 04Solution
- 05Testing method
05 · Testing methodAn A/B test is only one of the options at this stage, not the beginning of the work.
Sequence
“Let’s test” is not an answer to the question of what needs to change.
Analytics shows where users drop off. Research and expertise help explain why. Only then do we formulate specific decisions whose effects can be tested.
- 01DataWhere exactly do users drop off
- 02Behavioural problemwhich action a person does not take
- 03CauseWhy is the decision not made
- 04SolutionA specific product change
- 05Testing methodA test or other evidence
- 06SolutionWe implement, reject or verify in another way
Weak hypothesis
“Let’s make the button red — perhaps there will be more clicks.”
Working hypothesis — example structure
“Users do not continue checkout because they cannot see the full cost before the next step. If we show it earlier, more people will complete orders.”
If we cannot explain how a change will influence user decisions, we are testing a hypothesis—not randomness.
Decision tree
test needed not each changesand
-
Questions 01
Is there a specific decision to test?
No → first analytics, UX-research or audit.
Yes → next
-
Questions 02
Will the test result change our decision?
No → test not needed.
Yes → next
-
Questions 03
Is the cost of a mistake or scaling high?
No → consider a controlled implementation.
Yes → next
-
Questions 04
Is there enough traffic and enough target events?
No → choose another testing method.
Yes → next
-
Questions 05
Can we measure the result correctly?
No → fix analytics first.
Yes → plan experiment
Outcome B
Implement in a controlled way
ConditionThe risk is moderate or traffic is too low for an honest test — we record the constraint in the conclusion.
Outcome C
Start with research or data
Conditionthere is no causal model or trust to analytics.
to launch
We first determine whether a test can answer the question. Then design variants
- 01Primary metric
- 02Guardrail metrics
- 03Baseline
- 04Minimum effect worth changing the product for
- 05Required sample size
- 06Minimum duration
- 07Complete business cycles
- 08Segments
- 09Technical correctness of allocation
- 10Stopping rule
- 11Next action for each possible result
If the test cannot reach enough data within an acceptable business timeframe, It does not become more reliable simply because it runs longer.
Result
the test has more than two possible outcomes: not only “won” or “lost”
01 – Outcome
The decision was confirmed
the effect is convincing enough, guardrail metrics did not worsen and scaling makes business sense.
Next actionWe implement
02 – Outcome
The decision was not confirmed
We do not treat it as proven. We review the causal model or form another hypothesis.
Next actionWe return to reasons
03 – Outcome
data notenough
We do not call random fluctuation a win. We document constraints and decide whether to continue, change the testing method or stop.
Next actionThat is also a result
The primary metric must be as close as possible to the business result. We use an intermediate metric only when its link to the next valuable action is clear.
more clicks – not win, if they do not change the result for which the page or product exists.
boundary
Sometimes the most valuable A/B-test decision is not run it
A before/after comparison is not an equivalent replacement for an experiment. If we use a controlled implementation, we clearly mark the causal-conclusion limitations.
We do not create an experiment just to fill a monthly report.
- Not we launch test, if
- 01there is no specific hypothesis
- 02The result will not affect decisions
- 03There is not enough traffic or data
- 04Analytics does not support trust in the data
- 05Too many external conditions change at once
- 06the test will cost more than a controlled implementation
- 07The problem is an obvious defect, error or violation of a basic scenario
- 08The team is not ready to implement any possible result
in within CRO
What the client gets from each experiment
(01) Solution
An evidence-based decision.
Problem description, causal model, data and a specific change to test.
- Problem
- Causal model
- Data
- Change
(02) Calculation
Calculation before launch.
An estimate of sample size, duration and the test’s ability to detect an effect meaningful to the business.
- Sample size
- Duration
- Minimum effect
- Stopping rule
(03) Implementation
Experiment implementation.
Variant design, technical specification, test data and launch correctness control.
- Variant design
- Specification
- Verification data
- Allocation control
(04) Conclusion
Solution after result.
Interpretation without cherry-picked metrics, an implementation recommendation and documentation of new behavioral knowledge.
- Interpretation
- Guardrail metrics
- Recommendation
- New knowledge
Format work
A/B-testing is not sold separately from the challenge it must solve
A/B tests are part of CRO for eCommerce or digital products. Diagnosis comes before the experiment; afterward, a decision is made and implemented.
Launch pointOne-off
from $2 500
Full website audit: loss points, causes and priorities.
- Decision-chain analysis
- Loss points with evidence
- Priority change plan
- Recommended testing method for each change
Format workdepends from challenges
in stock CRO
Experiment cost depends on hypothesis complexity, number of variants, design needs, implementation complexity, experiment systems, analytics, control duration and the number of platforms and segments.
Check, or needed testPlace in CRO
a test is one step in the cycle, and not single process
- 01DiagnosisWhere and how much is lost
- 02SolutionWhat exactly change and why
- 03Choice of testing methodA/B test · UX research · prototype · controlled launch · pre/post analysis with constraints · metric observation
- 04ImplementationThe change reaches the product
- 05MeasurementWhat changed in behavior and money
- 06Next decisionThe cycle repeats with new knowledge
Our goal is not to run more tests. Our goal is to get enough evidence for the right decision faster.
Questions
What people usually ask about experiments
There is no universal number. The required scope depends on baseline conversion, the minimum important effect, number of variants and acceptable error; it is calculated before launch.
Every test uses traffic and team time and delays implementation. If the result will not change a decision or the test cannot detect an important effect, it creates no value.
Sometimes this is a useful reference, but the comparison does not isolate the effect of a change from seasonality, advertising, audience mix and other factors. Its evidence level is therefore lower than that of a controlled experiment.
It should be as close as possible to the business result. In some products revenue appears too late, so a confirmed intermediate metric is used with guardrail metrics.
This is not a guarantee of business benefit or proof of absolute truth. It means that under the chosen model the observed difference is harder to explain by chance. Effect size, data quality and practical value still require separate assessment.
Use another way to obtain evidence: UX research, prototype testing, a controlled launch, behavioral analysis or post-implementation observation with clear limitations.
No. We oppose testing without a question, mathematics and a next decision. When an experiment is the best way to reduce risk, we use it.
Start
We first define not what to test, but and which decisions you need to make
A 30-minute online meeting with the founder. We discuss the challenge, available data, traffic and the cost of mistakes. After the conversation, we suggest the next step: an A/B test, audit, CRO or another testing method.