Password protected

All case studies share one password — please contact me for access.

Back
Home
Home
Home
Home

Growing an AI chat app with A/B tests 
and a research‑led redesign

Growing an AI chat app with A/B tests and a research‑led redesign

Role

Sole Product Designer

Sole Product Designer

Platform

iOS (iPhone)

iOS (iPhone)

Timeline

2025–2026

2025–2026

Chat AI Plus is an AI assistant app built on OpenAI’s GPT models. I designed its MVP in 2023 and evolved it as the market changed. Today, users chat with it, generate images, ask about photos, and summarize files, YouTube videos, and web pages.

Chat AI Plus is an AI assistant app built on OpenAI’s GPT models. I designed its MVP in 2023 and evolved it as the market changed. Today, users chat with it, generate images, ask about photos, and summarize files, YouTube videos, and web pages.

This case study covers onboarding and paywall A/B tests, and the 2026 redesign. After auditing the app, I initiated and ran usability testing, then reworked the UX/UI and core features accordingly.

This case study covers onboarding and paywall A/B tests, and the 2026 redesign. After auditing the app, I initiated and ran usability testing, then reworked the UX/UI and core features accordingly.

+52%

+52%

+52%

purchase conversion

Side-by-side plans: +21% projected 

1-year ARPU in an A/B test.

+45%

+45%

+45%

ARPU

Personalized onboarding: +35% purchase conversion in an A/B test.

+18%

+18%

+18%

MRR

Redesign release: −38% refund rate, March vs. June 2026.

The challenge

By October 2025, Chat AI Plus was losing ground. Purchase conversion and revenue per user had dropped, and refunds had risen. Support tickets, App Store reviews, and my review of competing apps showed why.

To convert more users and keep them, the app had to be:

Worth paying for

Other AI apps offered more features for the same price. Ours had to close the gap.

Easy to use

The UX had aged. Existing features needed to be simpler and more capable.

Modern

The UI looked dated. Users needed a product that felt as modern as the AI inside it.

Revenue

Giving users the confidence to subscribe

A full redesign was a bigger bet that called for deeper research, so we started where users decided whether to subscribe. Across three A/B tests, I redesigned the onboarding, the paywall, and the screen after it.

Personalizing before the paywall

Our onboarding showed a few feature screens, then the paywall. I expected setting up a personal assistant to make the app feel tailored, so more users would subscribe. The trade-off: a longer onboarding could raise drop-off.

I added questions about goals, interests, AI experience, and name, then a screen showing the assistant being built from their answers. These choices ordered the app’s prompt categories, and the chat addressed users by name.

Research insight:

Competing AI apps and web funnels often personalized the experience before asking users to pay.

Onboarding setup

Onboarding setup

A/B test · Personalized onboarding

Setting up a personal assistant made users more willing to subscribe

+35%

purchase conversion

+45%

ARPU

+43%

trial-to-paid

Drop-off rose, but the gain at the paywall outweighed it. Refunds didn’t rise.

Showing plans side by side

Our paywall initially showed only the yearly plan. A free trial toggle swapped it for weekly. After the yearly price rose, its refunds doubled, so the toggle was turned on by default to lead with the lower-risk weekly plan (variant A).

Refunds dropped as expected, but so did purchases. Many users started a trial instead of buying right away, and not enough of them paid to make up for it.

Data insight:

On variant A, the weekly plan with the trial grew from about 30% to 87% of subscriptions, so fewer buyers paid for a full year upfront and ARPU fell.

To lift conversion and ARPU, I put both plans on one screen so users could easily compare them and pick the one that fit. I also preselected the yearly plan and made the price per week the main number on both, so yearly looked affordable and clearly cheaper.

The risk was that refunds would rise again if more users chose the yearly plan. But we believed choosing yearly with the weekly plan in view would be a more deliberate decision. Those unsure about a year could pick weekly instead, with an optional 3-day free trial.

Winner

Winner

A · Weekly-first paywall

Weekly-first paywall

Yearly plan behind the toggle

B · Side-by-side plans

Side-by-side plans

Winner

Both plans in view, trial optional

A/B test · Side-by-side plans

Making plans easy to compare brought more buyers and revenue

+52%

purchase conversion

purchase
conversion

+21%

projected 1-year ARPU

85% of buyers chose the yearly plan, yet yearly refunds were more than a third below their earlier peak.

Making the trial clear

Our paywall had the free trial off by default. We suspected that some users who weren’t ready to pay closed the paywall because they missed the trial or didn’t fully understand how it worked.

I designed a screen that opened right after users closed the paywall. Its headline spoke to users who weren’t sure yet, and the screen offered the free trial as the only choice. A timeline showed how the trial would go, so users knew they could cancel before being charged.

Research insight:

Many subscription apps follow a closed paywall with a free trial: closing it is often a no to the offer, not the app, so a trial is a lower-commitment way in.

Trial screen after closing the paywall

Trial screen after closing the paywall

A/B test · Trial screen after closing the paywall

A trial offer won back users who’d closed the paywall

+31%

trial starts

+28%

purchase conversion

+13%

ARPU

trial starts

+31%

purchase conversion

+28%

ARPU

+13%

Research

Seeing how people really use the app

Our A/B tests raised conversion and revenue per user at the paywall, but couldn’t fix the product itself. In early 2026, overall conversion dropped by more than a third, and the App Store rating fell. Complaints kept reaching support, but they rarely said where exactly users got stuck.

We started discussing a redesign and feature updates. Before we committed, I initiated usability testing to see how people dealt with the problems from my UX audit and where else they struggled. The findings set the direction for the redesign and convinced the team to invest in it.

We started discussing a redesign and feature updates. Before we committed, I initiated usability testing to see how people dealt with the problems from my UX audit and where else they struggled. The findings set the direction for the redesign and convinced the team to invest in it.

Usability testing

I ran the study end to end, from planning and recruiting to moderating 6 online sessions, analysis, and the final report. Participants matched our target audience: they used AI chat apps from weekly to many times a day, for work, everyday tasks, and creative projects. None had used ours, so they saw it the way new users do.

Session structure

01

01

Warm-up

Participants’ AI chat habits, what they value, and what frustrates them.

02

02

First look

Unguided exploration of our interface, capturing first impressions.

03

03

Tasks

Scenario-based tasks observing how participants used key features.

04

04

Debrief

Overall impressions, pain points,
and priority improvements.

01

Warm-up

Participants’ AI chat habits, what they value, and what frustrates them.

02

First look

Unguided exploration of our interface, capturing first impressions.

03

Tasks

Scenario-based tasks observing how participants used key features.

04

Debrief

Overall impressions, pain points, and priority improvements.

Overall impressions

Half of the participants called ChatGPT the ideal AI chat for its simplicity. Ours didn’t feel that simple:

Usability testing · 6 participants

Didn’t read as a chat

5/6

5/6

The main screen had too many elements and a button instead of an input. Some saw a set of separate tools and couldn’t tell what the app was mainly for.

Hard to navigate

5/6

5/6

Most mistook the New Chat icon for an edit button, and moving between the chat, History, and the main screen was confusing.

Felt cluttered

4/6

4/6

Colors and emoji crowded the interface and made it hard to focus. Some said they also made the app look less credible.

Hard to rely on daily

3/6

3/6

Simple text requests went fine, but more complex tasks took effort or weren’t possible, so the app felt too limited.

Deciding what to change

Participants finished almost every task, but often struggled. After analyzing each session in detail, I identified 17 problems recurring across participants and ranked them by frequency. The top ones shaped 8 key recommendations, which set the redesign’s priorities.

My competitor review backed up these findings, and I also suggested fixes for less frequent problems worth solving. I presented the results to the product team, and together we defined what went into the redesign.

Redesign

Rebuilding the app from what I learned

I redesigned the whole app with a clean, modern UI, simplified the navigation, and cut extra steps from common tasks. I also added web search for up-to-date answers.

Making the app read as a chat with extra features

As testing showed, the main screen hid the app’s main action, typing a request, among tools, colors, and emoji. So I made these changes:

Chat-first screen. An input with an animated placeholder, familiar from any chat, replaced the Start AI Chat button, and the AI Tools title became Chat AI Plus. Usage data showed many users opened the app just to ask by voice, and the input’s mic let them do it from the main screen.

Chat-first screen. An input with an animated placeholder, familiar from any chat, replaced the Start AI Chat button, and the AI Tools title became Chat AI Plus. Usage data showed many users opened the app just to ask by voice, and the input’s mic let them do it from the main screen.

Cleaner layout. Rather than a chat-only screen, I kept feature cards to make the app’s value visible. Cards hidden in the carousel were rarely opened, so I fit them all on one screen, made the core ones larger, and cut colors to reduce cognitive load.

Cleaner layout. Rather than a chat-only screen, I kept feature cards to make the app’s value visible. Cards hidden in the carousel were rarely opened, so I fit them all on one screen, made the core ones larger, and cut colors to reduce cognitive load.

Tab bar navigation. Chat Prompts and History moved into a tab bar, so users could switch between sections without going back to the main screen.

Tab bar navigation. Chat Prompts and History moved into a tab bar, so users could switch between sections without going back to the main screen.

Research insight:

4 of 6 participants named an input on the main screen as their top improvement.

Success metrics:

Chat starts, time to first request, and feature card taps, compared with the old main screen.

Old main screen

Start AI Chat button, features in a carousel

New main screen

Chat input, all features on one screen

Old main screen

New main screen

Showing users what they could ask

On the old main screen, Chat Prompts were emoji tiles that hid the prompts, and usage data showed they were rarely tapped. A missed chance: suggestions help unsure users start chatting, and engaged ones are more likely to subscribe. So I made these changes:

Two new entry points. Chat Prompts became a Suggestions tab, with each category’s prompts in plain view. I also added a Suggestions button to every new chat that opened them in a sheet, right when users were deciding what to ask. I expected both to drive more requests.

Two new entry points. Chat Prompts became a Suggestions tab, with each category’s prompts in plain view. I also added a Suggestions button to every new chat that opened them in a sheet, right when users were deciding what to ask. I expected both to drive more requests.

Live topics. We added a Popular category with real-time topics, from news to sports. Its answers used the new web search, introducing users to the feature.

Live topics. We added a Popular category with real-time topics, from news to sports. Its answers used the new web search, introducing users to the feature.

Ordered by interest. After Popular, categories followed the interests users picked in onboarding, making suggestions more personal.

Ordered by interest. After Popular, categories followed the interests users picked in onboarding, making suggestions more personal.

Research insight:

3 of 6 participants dismissed Chat Prompts, mostly because they didn’t see the tiles as prompts at all.

Success metrics:

Requests sent from suggestions compared with the old Chat Prompts, and web search use from Popular.

New Suggestions: from the in-chat button to a sent request

New Suggestions: from the in-chat button to a sent request

Making the Add menu clearer

In the old Add menu, opened with the + button in the chat, every option had an equally bright card, and Image Generation came first. Uploads, among the most used actions per usage data, took extra effort to spot, making a simple task felt harder than it should. So I made these changes:

Uploads first. Camera, Photos, and Files moved into one row above the AI tools, so the most common actions were quick to reach.

Uploads first. Camera, Photos, and Files moved into one row above the AI tools, so the most common actions were quick to reach.

Clear options. All options got simpler icons, and AI tools got one-line descriptions. I expected this to reduce cognitive load and make the right option easier to choose.

Clear options. All options got simpler icons, and AI tools got one-line descriptions. I expected this to reduce cognitive load and make the right option easier to choose.

Research insight:

4 of 6 participants added photos through the Add menu rather than the main screen, yet 3 of 6 found the old menu overloaded.

Success metrics:

Time to attach a photo or file for new users, and the share of AI tool opens that led to a request, compared with the old Add menu.

Old Add menu

Bright cards, Image Generation first

New Add menu

Uploads first, AI tools with descriptions

Old Add menu

New Add menu

Letting users refine AI images

At first, our model could create images in different styles, but only from scratch, so asking to change a detail produced a completely new image. Newer models could edit images through conversation, exactly what users asked support for and tried in testing. So I made this change:

Edits in context. I pushed for keeping the image in context, so changing a detail, like a color, left the rest intact. Users could restyle it without retyping the prompt or edit their own photos. I expected images users loved to be shared more, and each share linked back to the app as a free growth channel.

Edits in context. I pushed for keeping the image in context, so changing a detail, like a color, left the rest intact. Users could restyle it without retyping the prompt or edit their own photos. I expected images users loved to be shared more, and each share linked back to the app as a free growth channel.

Research insight:

5 of 6 participants wanted to keep improving the same image, and 3 of 6 were frustrated at retyping the prompt to try another style.

Success metrics:

Images saved or shared and installs from shared ones, compared with the old version, as well as edits per image.

New image generation: the same image, edited and restyled through conversation

New image generation: the same image, edited and restyled through conversation

Helping users find past results

Usage data showed users often opened History, but past results were hard to find: search matched only chat names, and generated images could only be found inside chats. So I made these changes:

Search by content. Search looked inside messages, so users could find a chat by anything they remembered. Each result showed the matching line, making the right chat easy to spot without opening each one.

Search by content. Search looked inside messages, so users could find a chat by anything they remembered. Each result showed the matching line, making the right chat easy to spot without opening each one.

Separate Images tab. History showed all generated images in one place, so users no longer had to dig through chats.

Separate Images tab. History showed all generated images in one place, so users no longer had to dig through chats.

Research insight:

4 of 6 participants pointed to issues in the old History. One called it “a mess.”

Success metrics:

Chats opened from search compared with the old History, and images opened from the Images tab.

New History: Chat tab, search by content, and Images tab

New History: Chat tab, search by content, and Images tab

Also shipped

Smaller fixes based on my UX audit, usability testing and competitor review:

Chat. A New Chat icon with a chat bubble and a plus instead of a pencil, voice input that could add to text already typed, and more relevant prompts in Suggestions.

Chat. A New Chat icon with a chat bubble and a plus instead of a pencil, voice input that could add to text already typed, and more relevant prompts in Suggestions.

Answers. More detailed by default, streamed word by word, with follow-up questions only where they helped.

Answers. More detailed by default, streamed word by word, with follow-up questions only where they helped.

Images. A banner with sample images in different styles and a Try button, and a more compact style panel.

Images. A banner with sample images in different styles and a Try button, and a more compact style panel.

Tools. AI Scanner renamed Text Scanner with a short explanation, a native iOS crop, photos that could be sent without text, and a fix for a YouTube link bug caught in testing.

Tools. AI Scanner renamed Text Scanner with a short explanation, a native iOS crop, photos that could be sent without text, and a fix for a YouTube link bug caught in testing.

Guiding existing users through the changes

The redesign fixed real problems, but also changed where existing users expected to find things. People often resist even a better interface at first, a reaction known as change aversion.

I didn’t want subscribers to cancel because they felt lost, so I designed a short tour for existing users only, with an update screen and 3 tips on the chat input, Suggestions and History. Users could skip the tour, and it stayed in Settings to replay anytime.

Quick tour for existing users: an update screen and 3 tips

Quick tour for existing users: an update screen and 3 tips

Results after release

The redesign went out to all users, alongside other updates, including a model upgrade and pricing changes.

Before vs. after the redesign · March vs. June 2026

Users paid more, asked for fewer refunds, and rated the app higher

+18%

MRR

−38%

refund rate

4.46 → 4.77

App Store rating

Refunds went from well above the AI-app median to around it.

From the first A/B test in October 2025 to June 2026, MRR grew by over 30%.

What I’d do next

Test free tries of premium features

Free users got only 3 text requests and couldn’t get results from other features. Even a trial meant committing to a subscription. Seeing a feature in action first could show the app fits their needs and make that step easier. I’d A/B test one free try per feature to see whether the gain in ARPU outweighs the cost of free generations.

Turn cancellation reasons into next steps

The redesign fixed what we expected to matter most, but some subscribers still canceled. When someone turns off renewal, I’d ask why in the app, with options like price or a missing feature, plus a free-text option. The answers would show what to prioritize next, and could also help tailor a win-back offer to each person’s reason.

Next case study

Building an NCLEX exam prep app from MVP to ExamReady’s flagship product

Open to what’s next

Let’s talk

Based in Kyiv, Ukraine.

Open to remote, hybrid, and on-site opportunities.

© 2026 Olha Tahirdzhanova. All rights reserved.

Password protected

All case studies share one password — please contact me for access.