All Resources

R-28

Intelligence (AI)

AI Hyper-Personalization: Increasing E-commerce Revenue by 40%

Where the forty per cent comes from, which surfaces actually return anything, the data work that gates all of it, and how to measure it honestly.

PAR2 Labs

March 18, 2024

12 min

AI Hyper-Personalization: Increasing E-commerce Revenue by 40%

The forty per cent is real and it is almost never yours. It comes from vendor case studies measuring uplift on a surface rather than on a business, from tests run without a holdout, and from the handful of retailers with the catalogue depth and traffic volume to make a recommender earn its keep. That does not make personalisation a bad investment — it makes the headline a bad plan. Here is where the number comes from, which of the surfaces actually returns anything, the unglamorous data work that gates all of it, and how to measure the result in a form you would be willing to defend in front of a finance director.

Watch the explainer

5:07

Narrated, with every diagram built on screen: which surfaces pay, recommendations, the data work and measuring with a holdout.

01

Where the number comes from

Almost every headline personalisation figure is measured on the surface rather than on the business. Click-through on a carousel. Revenue per visit for sessions that saw the widget. Conversion rate among users who engaged with a recommendation. Each is a real measurement of something, and none of them is a measurement of what personalisation added.

The problem is selection, and it is severe. Sessions that saw the recommendation widget are sessions that scrolled far enough to reach it, which are sessions that were already going better than average. Users who clicked a recommendation are users who were already browsing with intent. Comparing them to everyone else measures how engaged people differ from disengaged people, and attributes the entire difference to the model.

Then there is the phrase revenue influenced by personalisation, which is the most quoted figure in the category and an attribution artefact rather than a result. It counts every order in which a personalised surface appeared anywhere in the journey. On a site where personalisation is switched on everywhere, that number converges on total revenue, and it has been presented to more boards than any other statistic in retail technology.

02

The surfaces, ranked by what they actually return

Personalisation is not one thing, and the five things it usually means have wildly different returns. The order below is close to the reverse of the order most programmes attempt them in.

SurfaceWhat it movesWhy it lands where it does
Site searchConversion of the highest-intent sessions in the entire funnel.People who search convert at a multiple of people who browse, and a null result on a product you stock is pure lost revenue with a measurable rate attached. This is usually the largest single win available and the least glamorous.
Category ranking and merchandisingConversion across the whole browse path, compounding.It touches every session rather than a widget some sessions reach. Ranking the same catalogue better is worth more than adding a shelf.
RecommendationsAverage order value, and genuine discovery in a deep catalogue.Real, but modest alone and heavily dependent on placement. The carousel sits late in the journey and is easy to ignore. In a shallow catalogue there is very little to recommend.
Lifecycle messagingReactivation and repeat rate.Strong per send and bounded in aggregate by list size and how often you are willing to send. It is a retention lever, not a growth one.
PricingMargin, and reputational exposure.Personalised promotion is ordinary commerce. Personalised pricing — a different price for the same product at the same moment — is a different category of decision, and we usually argue against it.

← Swipe for the rest of the table →

← Drag for the rest of the table →

03

Search is the highest-intent surface and the most neglected

Start by measuring the null-result rate: the share of searches returning nothing, on a catalogue that contains the thing. In most retailers nobody has looked, and the figure is uncomfortable. Every one of those is a customer who told you precisely what they wanted to buy and was told you do not have it.

The causes are almost always mundane. Attribute coverage is thin, so colour, material and size are in the product photography and not in the data. Synonyms are absent, so trainers finds nothing on a catalogue full of sneakers. Typo tolerance is off or too strict. Query understanding cannot tell a brand from a category from an attribute, so a three-word query is treated as one string.

Fix retrieval before ranking, always. Personalisation on search is re-ranking a correct result set towards this particular shopper; it cannot rescue a result set that was wrong. A personalised ordering of twenty products that do not match the query is still twenty products that do not match the query, and the customer leaves either way.

04

Recommendations: the carousel is not the model

Teams buy or build a recommender and then place it wherever there is room on the page. The model is rarely the constraint. What separates a recommender that earns money from one that decorates is the filtering layer between candidate generation and display, and that layer is business logic rather than machine learning.

WHERE RECOMMENDATIONS ACTUALLY FAIL01Candidatescollab + content02Filterstock · owned03Rankrelevance, margin04Rulesexclusions05Placeat high intent06Attributeagainst a holdout

Fig 1 — the hot step is the one with no model in it. Most visible failures are filter failures.

The filter failures customers actually notice

01

Recommending something out of stock, or unavailable in their region, size or currency.

02

Recommending the item they bought last week, which reads as the shop having no memory at all.

03

Recommending the item they just returned — the strongest negative signal you will ever receive, and the one most often not wired in.

04

Recommending an accessory for a product they do not own, because the relationship was learned from co-purchase without direction.

05

Recommending across a boundary that matters: an adult product on a children's category page, a restricted item to an unverified account.

05

Cold start is not an edge case, it is most of your traffic

Collaborative filtering needs history, and on a typical retail site the majority of sessions are anonymous, first-time, or too short to have produced any. Treating that as the fallback path gets the proportions exactly backwards: it is the main path, and it deserves the design attention.

What works there is content-based: product attributes, category structure, the referring query or campaign, and well-constructed popularity priors scoped to the current context rather than the whole site. Popularity within this category, this week, for this price band beats a global bestseller list by a wide margin and costs almost nothing to build.

Which points back at the catalogue again. Content-based recommendation is only as good as your attributes, and attributes are the thing nobody wants to fund.

A personalised ranking of the wrong twenty products is still the wrong twenty products.

06

The data work that gates all of it

Six months of a personalisation programme typically goes here, and that is the right place for it to go. Every model above degrades to noise without these five, and no vendor demonstration will tell you so.

What has to be true before a model helps

01

An event taxonomy that is stable, documented and actually implemented on every surface — view, add to cart, purchase, return — with product identifiers that match the catalogue exactly.

02

Identity stitching across device and login state, with an explicit written rule for what happens to an anonymous session that later signs in.

03

A catalogue with attributes genuinely populated. This single factor determines more of your search and recommendation quality than the choice of algorithm.

04

Inventory, price and eligibility readable in real time by the same service that ranks. A nightly export guarantees the out-of-stock recommendation.

05

A returns signal, wired in as a negative. Without it you will optimise towards the products people send back, and the revenue chart will look excellent.

07

Personalised pricing, and why we usually argue against it

There is an important distinction that gets blurred, often deliberately. Personalised promotion is showing different offers to different people at a published price — ordinary commerce, understood by customers, and the mechanism behind every loyalty scheme. Personalised pricing is showing a different price for the same product to two people at the same moment based on what the model believes they will pay.

The second carries exposure that has very little to do with the mathematics. Consumer protection regulators in several jurisdictions have taken an active interest. The optics are poor in a way that does not survive explanation, because the honest explanation is that we charged you more because we thought you would pay it. And a screenshot of two prices side by side travels further and faster than any amount of uplift.

The alternatives capture most of the value with none of the exposure: segment or market-level dynamic pricing, targeted promotions against a published price, and margin-aware ranking — putting the better-margin item first among genuinely relevant results, which is merchandising and has been for a century.

08

Measure with a holdout, or do not quote a number

This is the part that determines whether you end up with a result or an anecdote, and it costs a small, permanent slice of revenue to do properly. Pay it.

01

Hold back a permanent global control

  • GET

    A baseline that survives seasonality

A fixed random share of users — one to five per cent is usually enough — who never see personalisation on any surface, for the life of the programme. Not a test that ends. A control that persists.

Why: Without it you cannot separate the model from the season, the campaign, the range change and the competitor who raised prices. Every year-on-year comparison is otherwise a guess with a decimal point.

02

Randomise on the user, never the session

    Assign at the user or device identity level and keep the assignment stable. Session-level randomisation puts the same person on both sides of the experiment within a week.

    Why: A contaminated control does not produce a smaller effect. It produces a wrong one, and usually in the flattering direction.

    03

    Name the primary metric before you start

    • USE

      Revenue per user, not widget CTR

    One primary metric, chosen in advance, measured across all users in the arm rather than only those who saw the surface. Secondary metrics are for diagnosis, not for declaring victory.

    Why: With enough secondary metrics something always wins. Choosing afterwards is how a flat result becomes a case study.

    04

    Run to a pre-registered horizon

      Decide the duration or the sample size in advance and do not stop early because the curve looks good. Cover a full weekly cycle at minimum, and a full promotional cycle if the category has one.

      Why: Stopping when it looks good is arguably the single largest source of inflated personalisation numbers in the industry.

      05

      Report incremental, not influenced

      • GET

        A number that survives questioning

      State the effect as the difference between the treated population and the held-out one, with a confidence interval. If you cannot compute that, say so and use the word influenced, and keep it out of anything that looks like a result.

      Why: The distinction between incremental and influenced is the distinction between a measurement and a marketing claim.

      09

      A realistic first year

      The sequencing below is close to what we would propose for a mid-sized retailer, and it is deliberately back-loaded on the things vendors demonstrate first.

      QuarterWhat shipsWhat it is actually worthWhat it unblocks
      OneEvent taxonomy, identity stitching, catalogue attribute backfill, the permanent holdout.Nothing visible. This is the quarter that gets cut, and cutting it is why programmes fail in quarter three.Everything after it, and any number you later want to defend.
      TwoSearch retrieval: synonyms, attributes, typo handling, null-result elimination.Typically the largest single conversion movement of the year.Ranking work, because ranking a correct result set is now a meaningful exercise.
      ThreeCategory ranking and personalised re-ranking on search. Recommendations with the filter layer built properly.Broad, compounding, and measurable against the holdout established in quarter one.Average order value work, and a genuine cold-start path.
      FourLifecycle messaging driven by the same signals. Margin-aware ranking. Measurement maturity.Retention and margin rather than headline conversion.The second year, which is where personalisation programmes either compound or stall.

      ← Swipe for the rest of the table →

      ← Drag for the rest of the table →

      10

      The failure modes that show up later

      Three of these only appear after six months, which is why they survive the pilot and damage the programme.

      What goes wrong once it is working

      01

      Popularity bias eats the long tail. The model promotes what already sells, sales concentrate, and the catalogue quietly narrows until discovery stops working and the buying team notices before the data team does.

      02

      Cannibalised discovery counted as uplift. The recommendation takes credit for a purchase the customer would have made through search. Only the holdout separates the two.

      03

      The filter bubble closes. A shopper who bought one thing for a child is shown children's products for a year. Diversity has to be an explicit objective in ranking, not a hoped-for emergent property.

      04

      Returns-blind optimisation. Revenue rises, net revenue does not, and it takes a quarter for the returns data to catch up with the celebration.

      11

      What to tell the board instead

      Not forty per cent. A realistic programme, sequenced as above and measured against a genuine holdout, tends to produce a mid-single-digit improvement in revenue per user in the first year, concentrated in search, with the compounding arriving in the second — and that is a good outcome, obtained honestly, on a number you can put your name to.

      The forty per cent is available too. It is just measured on a carousel.

      Worth reading properly

      GA4 — ecommerce event reference

      A serviceable starting taxonomy, and the one your analytics already expects.

      The Netflix Recommender System: Algorithms, Business Value, and Innovation

      The honest account of where the value sits, from the team with the most data on the question.

      schema.org — Product

      A useful checklist for catalogue attribute coverage, and it earns you rich results as a side effect.

      Intelligence (AI) at PAR2 LABS

      Where this work sits, and how an engagement starts.

      Key Takeaways

      01

      Headline personalisation figures measure the surface, not the business. Sessions that reached the carousel were already going better than average.

      02

      Search is the highest-intent surface and usually the largest single win. Measure the null-result rate first; it will be worse than anyone expects.

      03

      Fix retrieval before ranking. Personalising the order of the wrong result set changes nothing.

      04

      Most recommendation failures are filter failures — out of stock, already bought, just returned — and there is no model involved in any of them.

      05

      Cold start is the majority of traffic, not an edge case. Content features and context-scoped popularity carry it, and both depend on catalogue attributes.

      06

      Keep a permanent global holdout, randomise on the user, name the primary metric in advance, and report incremental rather than influenced.


      PAR2 Labs · Intelligence (AI)

      Work With Us

      Have a problem worth solving?