Can you test an AI startup before building the software?
Yes. A concierge minimum viable product delivers a narrow customer result through a manual or partly manual process. It helps you test whether people want the result, can provide the required inputs, and find the output useful. It does not establish that the eventual software will be reliable, economical, or scalable.
This approach is especially useful when you can deliver the proposed result yourself but still have questions about the workflow. Before building an autonomous agent, learn which decisions require judgment, which inputs are routinely missing, and which exceptions make the work difficult.
Use this guide when: you have a reachable customer group and a specific task that can be completed safely by a person. Open the free concierge pilot worksheet to define the test. No signup is required.
Choose a result a customer can recognize
“An AI assistant for small businesses” is too broad for a useful pilot. Choose a task with a clear start and finish: turn approved product information into a draft comparison page, organize incoming support requests for a human reviewer, or prepare a weekly summary from a defined set of reports.
Write down who receives the output, what they do with it, and what would make it unusable. If the customer only looks at the result and says it is interesting, you may have tested a demonstration. If they use it in their normal process and ask for the next delivery, you have stronger evidence about ongoing usefulness.
Paul Graham’s Do Things That Don’t Scale discusses delivering manually before automating. The pilot structure below is StartupCoach’s own practical application, including explicit disclosure and limits appropriate to testing AI-assisted work.
Describe the service honestly
Tell participants which work is manual, where AI is involved, what a person reviews, and what the pilot cannot do. Do not present a human-produced result as autonomous software performance. A pilot can offer excellent service while still being experimental.
Illustrative service description: “We will prepare a weekly draft of your product FAQ updates from the documents you approve. AI assists with the first draft. A person checks each answer against the supplied sources. Your team approves all changes before publication. The pilot covers one product line for two weeks.”
This makes the deliverable and approval responsibility understandable. It also prevents the test from accidentally becoming a promise to monitor every product, edit the live site, and handle customer support indefinitely.
Set up a five-part pilot
| Part | What to decide | Example for a FAQ service |
|---|---|---|
| Input | Required information and permission | Approved product documents and recurring question themes |
| Delivery | Exact output and timing | One reviewed draft each week for two weeks |
| Review | Quality checks and approval owner | Each factual claim linked to an approved source |
| Boundaries | Exclusions and exception handling | No publishing; missing facts returned as questions |
| Decision | What the review meeting will resolve | Continue, change scope, automate one step, or stop |
Decide whether the pilot is paid. Record the price, scope, and any refund or cancellation terms in the offer. Free participation can teach you about use; it does not establish willingness to pay. A paid manual service also does not establish willingness to buy a self-serve product with a different experience.
Capture a baseline before improving anything
Ask how the task works today. Record its frequency, people involved, elapsed time, active work time, and the errors or delays the customer actually notices. Distinguish observed records from rough estimates.
For the fictional FAQ pilot, the current process might require a staff member to gather questions, check product documents, draft changes, and obtain approval. Time the same boundaries during the pilot. Comparing an AI drafting step with an entire previous workflow would exaggerate the improvement.
Agree a small set of success conditions with the customer. These could include usable answers, review effort, delivery timeliness, and whether the team wants another cycle. Choose conditions that reflect the task rather than an arbitrary accuracy percentage detached from the consequences of errors.
Keep a delivery log, including the awkward parts
For each request, record the input quality, human time, tool usage, corrections, exceptions, and customer action. Include time spent chasing missing information and fixing the output. Those activities can dominate the economics even when generation itself is quick.
Use sample or de-identified material where possible. Obtain permission for the actual data and tools used, restrict access, and agree how long pilot materials will be kept. Avoid tasks where an unreviewed error could create a serious consequence. Route uncertain outputs to a qualified person rather than silently guessing.
Keep the log small enough to maintain. The aim is to discover which work repeats and which work depends on context. A beautifully automated process is premature if the input requirements change with every customer.
Separate customer evidence from automation evidence
Fictional result: three pilot customers receive two deliveries each. Two use both drafts and request another cycle. One uses neither because internal approval is too difficult. The team spends an average of 50 minutes per delivery, including review.
That result suggests a potentially useful service for some customers and an approval obstacle for another. It does not prove product-market fit or a profitable automated business. The next test might simplify approval and automate source collection, while retaining human factual review.
Before automating, replay representative past inputs through the proposed system and compare results with the reviewed outputs. Include messy inputs, missing information, and cases the manual process declined. Preserve a way to stop, correct, or complete the task manually when automation fails.
Decide what to build after the pilot
| What you learned | A useful next move |
|---|---|
| Customers value the output and the process repeats | Automate a bounded, well-understood step |
| Customers value the output but each case is unique | Consider a service model or narrow the audience |
| Customers like the draft but never use it | Investigate the workflow, approval, and urgency |
| Review effort erases the benefit | Change the task or improve inputs before expanding |
| Customers will not supply necessary inputs | Reconsider onboarding or the problem selection |
Set the next review date and keep an explicit stop condition. Sunk effort is not evidence that a product deserves further investment.
Questions founders ask
Is a concierge MVP the same as a prototype?
A prototype usually helps evaluate a proposed experience. A concierge MVP delivers a real result through a manual service. You can use both: a prototype to explore the interface and a pilot to understand delivery.
How many pilot customers do I need?
Choose a number you can support and learn from. A handful may reveal workflow problems, but it cannot establish broad market demand. Seek variation deliberately and keep your conclusions proportional to the evidence.
When should I hire an engineer?
When a technical question or implementation need is clear enough to scope. The forward deployed hiring guide can help you distinguish a design problem from an engineering problem.
Turn the idea into a bounded test
Open the concierge pilot worksheet. Write the result, required inputs, review owner, and decision date. Use the first-offer pricing guide to check delivery costs before promising ongoing service.
AI workflow scorecard
Get the worksheet now, plus the occasional practical next step by email. No spam, unsubscribe any time.
Need a little help applying this? Explore coaching and implementation.