Between NCAP, FMVSS, and other test procedures, there is no lack of structured ways to assess ADAS/ADS performance. However, crafting a test approach that is repeatable and reproducible while capturing the broad and uncontrolled environment we drive in every day can be challenging. Leveraging both their unique facilities and decades of collecting naturalistic driving data, the Virginia Tech Transportation Institute (VTTI) has been developing test methods to assess vehicle automation system and subsystem performance in the varied and challenging conditions encountered in everyday driving. Based on our work, we’ll explore possible answers to a few relevant questions including: - What might we be missing by testing in ideal conditions and does it matter? - What important conditions might we be excluding and how could this change expectations for how the system behaves? - How can we introduce these conditions and characterize their effect?