Introducing the eCommerce AI Lab

Today we're officially opening a new chapter of Commerce-UI, and we're calling it the eCommerce AI Lab. It's a full-time team of AI engineers (or "forward deployed engineers," if you want the fancy Silicon Valley term) who work on nothing but eCommerce problems.

Every single eCommerce business we talk to wants to apply AI somehow. It's not exactly surprising given how powerful AI has become. It can already close 90%+ support tickets, rewrite multi-million lines-of-code projects in a matter of weeks or solve maths problems unsolved by humans for 100 years. Naysayers will say a lot of interest is driven by pure hype. It might be partially true. For sure, not all applications will bring value. However, we believe that the ones that do will bring gains of absolutely massive scale.

The problem we see is that successful application of AI to companies is actually hard. Of course it's easy to spin up a prototype or set up Claude to do a simple automation. However, building agents that work reliably, in a way you can actually measure, and that get better over time, is a completely different thing. It's an engineering problem that requires building reliable infrastructure, harness and evaluation, all within a process involving deep collaboration with domain experts. The people who do this for a living are called AI engineers.

So where does AI actually make a difference in eCommerce? Nobody can see all the angles yet, but here are the ones we're already seeing in practice:

1. People expect a conversational interface

Today, when users want to find something, they dig through long lists of filters, use inaccurate search or read a ton of blog posts. It's hard to imagine they won't prefer to just ask, the way they already do in ChatGPT:

"i've been fighting acne for years, tried salicylic acid, didn't help, then benzoyl peroxide, turned out it irritates my skin, what do i do".

Conversation is becoming a major channel between brand and customer, and every brand will have to answer questions like this one correctly, in its own voice.

This also has a consequence that is rarely talked about: the user intent gets much richer. Today you guess what customers want from journeys and clicks. Here the customer tells you their exact problem, in their own words. It's first-hand knowledge brands have never had before.

2. 1B people can discover your brand via ChatGPT

ChatGPT has surpassed 1B monthly active users and a growing share of them use it to decide what to buy:

A new sales channel is forming in front of our eyes, and the early data is very telling. From Shopify's Q1 2026 report:

"According to Shopify's Q1 2026 commerce data, shoppers arriving from AI search are more valuable than those arriving from organic search:

They arrive with higher purchase intent. More than half of AI-referred sessions start on product pages, compared to 20% for organic search.

They convert more often. AI-referred sessions convert at nearly 50% higher rates than organic search.

They spend more when they do. Average order values from AI-referred sessions are 14% higher than organic search."

We're certain AEO (answer engine optimization) is going to be a huge deal. In many aspects it's similar to SEO, but there are also major differences: as we saw above, intent in chat is far more detailed. Nobody types "i've been fighting acne for years, tried salicylic acid, didn't help…" into a Google search box, but that's a completely normal ChatGPT message. If you sell niche products, this is enormous: the people with exactly the need you cover can now find you faster than any channel has ever allowed.

ChatGPT and other agents will only find you if they can understand your store. Your products, collections and policies need to be perfectly legible to a machine. Even though it's a very new field, the proper measurement methods are already showing up.

3. eCommerce management will be semi-automated

Every brand sits on tons of analytics data today, but making sense of it (getting insights, finding potential experiments) is an expensive task: you need to recruit analysts (if you can even find them), wait weeks for reports, and quite often the results turn out inconclusive.

But this is the kind of work that AI actually excels at! Digging through data, spotting patterns, forming hypotheses, proposing experiments. We believe every brand should have a tailor-made copilot for eCommerce management connected to its analytics sources and its storefront, so you can simply ask:

"why did conversion drop last week?"

…and get a real answer, with the data to back it up. Or even better: your agent could notice the drop before you do and come back with a proposed experiment to fix it.

Human taste and judgement are still crucial here, but we're past the moment where humans must do everything manually.

4. Internal processes automation

Every manual and repeatable commerce process should be looked at through the lens of potential automation. Logistics, finance, manufacturing, etc. Every business is different so it's hard to point at universal use cases, but here are a few examples from our own conversations:

1. Retail compliance for B2B brands. If you sell through hundreds of retailers, someone has to check whether they actually present your products the right way: current lineup, correct imagery, correct claims. Today that's a person clicking through websites one by one. An agent with a browser can sweep all of them every week and report back only the violations.
2. Product data enrichment. Thousands of SKUs with missing attributes, inconsistent descriptions, untranslated copy, empty alt texts. An agent can clean up and enrich the entire catalogue.
3. Content production. Product descriptions, ad creative variants, localized campaign assets. The agent produces at scale, a human curates (similar to the copilot above).

This category seems the least "fancy", but paradoxically, this is probably where the simplest and fastest ROI lives.

5. Image/video generation

It's a wide category, mostly about visual content: product imagery, lifestyle shots, video ads, etc. Those are things that used to require a photoshoot and a budget to match, and now increasingly don't.

In fashion there's also the promise of virtual try-ons. I'm personally not convinced by this one yet, but plenty of smart folks are, so I decided to put it here anyway.

Summary

Some of this you can already buy off the shelf. Image generation is the best example, with plenty of mature products around. But the closer you get to your products, your data and your processes, the less an off-the-shelf SaaS can do: a conversational storefront, a management copilot or an internal automation only works when it's built around your specific business. That's the part that takes AI engineering know-how, and that's exactly what our AI Lab is for.

In this part I'd like to briefly let you peek under the hood to see what we actually do. AI engineering is, well, engineering, so this part is a bit technical. But even if you're not technical, please do not get discouraged, since it shows where the difference between a demo and a product lives.

In our experience, building reliable agents comes down to roughly five things:

• domain knowledge
• evals
• harness
• UX
• infrastructure

It starts with domain knowledge and deep collaboration

Whether you want to automate an internal process or put an agent in front of your customers, an AI engineer must understand your business. It sounds almost like a cliché: every consultant must do this, right? But in the case of AI, the depth of such understanding is much higher than in other processes, like, for example, rebuilding the website.

Let me explain this by example. A couple of months ago I was building an AI shopping assistant for a $1B+ skincare brand. If I were to build a website, I'd have to understand the problems and vibes of the brand, but I could be totally successful without a deep understanding of products, interactions, conflicts, user problems, etc. In the end it's the client who populates the content, analyses data, and answers customer questions. For building an AI agent, it's not enough.

When AI engineers build an agent, they constantly ask it questions and look at the results. But in the skincare case, how can I, as an AI engineer, say whether it answers well? It's impossible without learning at least the basics of skincare! So I spent a week reading about retinoids, acids, squalane and all, and talked for hours with domain experts from the company. Of course my basic knowledge won't replace the experts, but if I had to wait for feedback with every iteration, that would be completely unsustainable.

The right process to obtain the knowledge and collaborate with domain experts is the key. It consists of 2 simple parts:

1. Initial workshops, which are an audit of how work moves through your organization, the company policies, knowledge, etc.
2. Regular sessions where your domain experts review the agent's work, point out what's wrong, and we turn those corrections into engineering.

The good news: to some extent this is how we've always worked. Our retainer model at Commerce-UI is built around long-term relationships and communication, and the AI Lab inherits that model.

Evals / Benchmarks

The most important question in AI is how do you even know that your agent actually works? When you built an eCommerce theme, you clicked around and you roughly knew it was all okay. You could get an unexpected bug here and there, but overall you knew whether it worked or not.

Agents are different. They can be talked to in an infinite number of ways, with an infinite number of user contexts. Even an identical question asked twice might result in a different answer. This is exactly what researchers mean when they say it's "non-deterministic" or "stochastic". It's a statistical system and randomness is normal. You can't possibly test all the scenarios and even if you did, you'll have to do it again after each change of model or harness.

A huge part of what an AI engineer does is taming this randomness. And it's almost always done via building evaluation systems, aka evals.

You might have noticed that when big labs (OpenAI, Anthropic or Google) announce new models on social media they usually show a table with benchmarks:

Each row in this table represents a "benchmark", which is basically a synonym for "eval". An eval is simply a smartly designed "exam" for the model. It checks how a model performs in a certain domain.

The best example is the 4th benchmark in the table above, called τ2-bench. It's very close to eCommerce problems since it checks how an agent performs in real customer support tasks. Here's a brief explanation of how it works:

1. The authors created an "artificial" retail company, with products, orders, clients, etc. (similar to what you have in Shopify) and "tools", via which the agent can interact with the company (find_order, get_products, create_return etc.)
2. It comes with artificial user issues. For example, user Bob might want to return a product because it had some fault, but it's after the date returns are accepted. Each issue is called "a sample", and there are hundreds of them in the benchmark.
3. Each sample also comes with strict verification criteria: the benchmark must know whether the issue was resolved correctly. So for example, in the case of our Bob, the return shouldn't be accepted. If, after the conversation, the return shows up in the system, it's a fail.
4. τ2-bench runs all the samples against a model. It basically simulates how Bob would talk. In the end it checks the result.

With this approach τ2-bench can now take any model and test how good it is at resolving customer support issues. As you can see above, Claude Opus 4.6 solves the Retail benchmark in 91.9% of use cases.

Now think for a moment: what if you didn't have such evals? Would you, after every agent update, run hundreds of conversations with your agent to see if it's okay? How would you know which model, out of ~100 available, performs the best? Could you confidently go to production with a cheaper model? The answer to all those questions is "no", because no human has so much time.

You need your business-specific evals

The problem with benchmarks like τ2-bench is that they're "general purpose", but they don't check how your agent solves your business issues. That's why you almost always need your own business-specific evals to actually know if your agent works.

This is one of the most important jobs we do at the eCommerce AI Lab.

Harness is more important than the model

When you build an AI agent, it actually has 2 parts: model and harness. The model is 'the brain', the harness is 'the body' (eyes, ears, muscles). A lot of discussion revolves around "which model is the best for x or y?", but actually it's not the most important question to ask.

Every model runs "within an environment". In order to do anything productive, it must have access to your data and your systems. This environment is called a "harness" and how it's built is as important as (or arguably more important than) the model. Here's what we usually think of when building harnesses:

1. the model must have all the necessary business data exposed to actually make informed decisions
2. the data should be clean and not have contradictions
3. the surface area should be as minimal as possible. The more the agent has to browse, the faster it will overload the context (which makes its reasoning worse)
4. sometimes an agent might have specialised "proxy" tools that allow it to make decisions much faster and cheaper
5. the agent might have access to historical conversations
6. often you should extend agents with a browser and other standard computer tools

A big part of our job is to help our clients build harnesses in the right way.

UI/UX

At Commerce-UI we've always treated UI/UX as an absolutely fundamental thing, and it's surprising how often this part is missed in agent implementations. The interface via which you interact with an agent matters. Here's a list of potential things we often do:

1. Is it just chat, or can it display inline widgets?
2. Is it full screen, or is it a small "support widget" at the bottom-right corner of the screen?
3. Can your chatbot display full-screen videos / images, embed products next to the chat?
4. Should an agent direct the user to site links, or embed information inside?
5. Should users have access to historical sessions?
6. Does it just "feel good"? Does streaming work? Can you reload the browser while the AI is generating a response and still see the generation?
7. How to make the hand-off from AI to a human properly?

Building great UX for a simple chatbot is surprisingly challenging, and we're in a unique position to have a company that combines both UI excellence and AI expertise.

Infrastructure

The last part is infrastructure. Your agent code must run somewhere. Your agents increasingly need access to a browser or a computer, and this needs to be absolutely secure. Agents touch your company data, so it's super important to make sure they don't violate the company security rules (for example: which Slack channels does your internal agent have access to?).

Infrastructure has become simpler with the advent of a new generation of AI frameworks like Eve from Vercel or Flue from Cloudflare, but it still requires a lot of engineering skill to get every detail right.

Every eCommerce company is already, to some extent, a technology company. We believe each of them will have to become an AI company too, and the AI Lab is our engine for getting you there.

If any of the areas above sounds like your business, let's talk. We usually start with an audit: we map your processes, find the places where AI brings real ROI, and pick the first project. From there, we work the way we always have: embedded, long-term, side by side with your team.

Reach us at hello@commerce-ui.com.