What is an experiment in product development?
The Harvard Chan AI Glossary
In our classes, we often pause to explain the ideas behind AI and the language used to describe it. These posts grew out of those conversations. They’re for anyone who would like to understand AI better, with no technical background assumed.
In product development, an experiment is a deliberately limited attempt to answer an uncertain question. It connects a prediction with something people can try and observations that inform a decision. The object is learning, not merely producing an attractive demonstration.
Suppose the question is whether showing source passages beside an AI summary helps readers identify errors. A prediction might be that readers notice more unsupported claims when the evidence is visible. The team can build a small version, give participants a realistic task and examine what they actually do.
The artifact is the thing created for the experiment: perhaps a sketch, a clickable prototype or a functioning piece of software. A prototype is a working draft used to investigate an idea. The artifact only needs enough detail to answer the question, provided its limitations are clear to the participants and reviewers.
Decide what to observe before looking at the result. Relevant observations might include errors noticed, time taken, requests for help and whether people understood what the system had done. A person saying “I like it” answers a different question from demonstrating that they can use it successfully.
Also specify limits. Who will take part? What information may be used? Which actions are excluded? When will the experiment stop? A bounded experiment need not expose real customers or sensitive records simply to make the demonstration feel authentic.
The strength of the conclusion depends on the design. An informal test can reveal confusion or technical failure without establishing that the intervention caused an improvement. A causal claim—saying that the change itself produced the effect—needs an appropriate comparison and attention to other explanations. A small successful demonstration should not quietly become a claim of effectiveness everywhere.
A clinic could test whether a source-linked handoff brief helps coordinators identify the next administrative step more quickly than reconstructing a case from the underlying records. Begin with fictional or appropriately authorized cases, specify how correct decisions and time will be measured, and include missing or conflicting information. A result that looks faster but omits important blockers would challenge the idea. Decide beforehand whether the evidence would justify revising the brief, trying a limited pilot or stopping. Building the brief is part of the investigation, not its conclusion.
If you would like to find out more about AI and improve your leadership skills, consider the AI in Health Care Certificate of Specialization from the Advanced Learning Academy at Harvard T.H. Chan School of Public Health. You can take an individual course or work toward the full certificate.
AI is a fast-moving field. This explanation reflects our class discussions and related material at the time of writing. Terminology, capabilities and tools may change.
For suggested additions or corrections, please email me at tpanch@hsph.harvard.edu.
Dr. Trishan Panch
Program Director, AI in Health Care
Harvard T.H. Chan School of Public Health — Advanced Learning Academy