Ten strangers, ten phones, one evening

On a Monday evening in September we ordered ten usability sessions from Userbrain: people in the US who had never written code, on their own phones, one of our five home pages each. The task list was short. Read the page and say what you think it is for. Make an app you would send to a friend. Change one thing about it. Send the link.

Nobody from Vibes was in the room, because there was no room. Each person recorded their screen and talked through what they expected as they went, for six to sixteen minutes, and by the next morning we had ten videos and ten transcripts. Three of us make this product. We had never watched ten strangers use it in a row.

What they made

The good part first, because it is the part that was easy to miss while watching the tape for problems.

A personal money ledger, asked for as "personalized for me", with transactions being added a minute later. A trivia night for game night. A family recipe book where each relative manages their own recipes and photos. A cabin-weekend planner for twenty-two sorority sisters, checked to make sure nobody forgets the marshmallows. A festival schedule where the stage and the time had to be readable at a glance. A lawn-care calendar for middle Tennessee, keyed to a zip code. A private photo vault.

Every one of those came from a single sentence typed with a thumb, and most of them were on screen and working inside two minutes of pressing send. One person spent eighty-odd seconds composing that sentence out loud, revising it as they went, the way you compose a text you care about.

One thing worth saying because no tester did it on camera, and people do it all the time: they attach a screenshot. A spreadsheet they are tired of. A paper form. A whiteboard. The app they wish were slightly different. The message is "make this", and the picture does the describing. In the first ten days of this month, ten builder conversations carried a picture in the message. If you have a thing you want to stop doing by hand, photographing it is a perfectly good first prompt.

What broke

In order of how much it hurt.

Eight of the nine people who made an account lost their prompt. Type an idea, press send, meet the sign-in card, tap Sign up, verify an email, and land on an empty home page. Some retyped a shorter version. Two picked a different idea entirely, so the app they built was not the app they had wanted. One said, more or less, that it was crazy that everything they had typed was gone. The ninth person kept their prompt because Sign in with Apple failed for them and dropped them onto a different sign-up path that happened to remember where they came from.

On iPhones, tapping an idea card looked dead. The card filled the composer three screens up and iOS did not scroll to it. Two people tapped cards, one of them three different ones, and said nothing was happening. One found their prompt sitting in the box minutes later, from a different door, and realised the first tap had worked. Every Android tester saw the fill instantly.

Two people finished holding a working app and told the study they had failed. The app was on screen, saved and playable. The button said the app was ready and pointed somewhere, which reads as a step you still have to do, not as an answer. On a phone the chat fills the screen while the app builds, so the moment the chat slides away, the app arrives with no introduction.

Two people ended up inside an example game and believed they had built it. After losing their prompt they tapped Games on the home page, landed in an example word game we made, played it happily, and reported that they had made a word game. Our server has zero turns from either of them.

The finished app's link could not be tapped. The card at the end of the build said "Send this link" over the address, as text. One person tapped it a few times. Another hunted for a while and found the real control behind an icon in the bottom bar. A third never found it and asked, sensibly, for a share button.

The back button led to a ghost. Your first app builds before you have picked a name, at a temporary address. Two people finished, pressed back the way you reach for a light switch, and landed on that temporary address, which faithfully asked them to pick a name for an app that was already saved.

None of that is on the home pages we were testing. The pages read well. Nearly everyone could say what the page was for, in their own words, within a minute or two. The product underneath the pages ate the intent the pages produced.

What we changed

All of this shipped within the week.

• Sign-up from the composer now returns you to the prompt you typed, and it sends itself.
• Tapping an idea card on an iPhone focuses the composer inside the tap, and scrolls it into view on purpose.
• On a phone, the first build now ends with a small card over the app: This is your app, its name, and Tap the chat to change it. It stays until you press Got it. Once per app, ever.
• The example-app card now says "not yours yet" in its title and carries a button, Make my own app, instead of describing where a home button is.
• The address on the build card is the link, with a copy control beside it. The three cards for three versions of one app no longer print the same address three times.
• The temporary address checks whether the app it stood in for is finished, and takes you there.

Each fix carries a test, and the ones for the tap and the completion card run at phone width, because at desktop width neither problem exists.

What travels

We build an app builder, and some of what we learned is only about that. The rest is about putting software in front of strangers, and it would have been true for any product.

You do not need a lab. Ten phones and one evening. The people were real, the phones were theirs, and the narration was the kind you only get from someone who has never been told how the product is supposed to work. The whole thing cost less than a single hour of the three of us arguing about what a stranger would probably do.

Line the video up with your own logs. Each recording's clock is arbitrary. We matched the spinner on screen when an account gets created to the row our server wrote at that moment, and from there every "nothing is happening" on tape could be checked against what the server was actually doing in that second. The tap that looked dead had worked. The build that looked failed had finished.

Never make signing up a task. Always time it. Signing up is not the product, but the clock keeps running while it happens, and a person who spent two minutes on it arrives at your product slightly less patient than the one who did not.

Listen for the question asked twice. When someone asks "how do I see it?" a second time, your completion signal has failed, however clearly you think the screen says it. The thing they were looking for was in front of them both times.

The person who succeeded and said they failed is your most important tape. Success you did not communicate is failure to the person who had it, and it will never show up in your metrics, because by every number you keep, it went fine.

Ten is not a rate. We say "eight of nine", never a percentage. Ten sessions cannot tell you which of five home pages is best. They can tell you, loudly, what is broken under all of them.

We fixed the six things above, and we are running the same five pages past the same instrument again, to find out what the pages themselves have to say now that the floor under them holds.