Describes the procedure used for a pilot of NIST’s Assessing Risks and Impacts of AI (ARIA) evaluation: ARIA 0.1. The pilot evaluation demonstrates the feasibility of a new approach to evaluation of AI systems: combining data from expert annotators and human testers, illustrated by a transparent measurement tool.