The Great Lakes create complex meteorological conditions that influence air quality throughout the region. Lake-breeze circulation, lake-induced low level jets, shoreline boundary layer processes, and photochemistry at the land-water interface affect the magnitude and timing of regional ground-level ozone episodes during the summer months. We simulated nine different Weather Research Forecasting Model (WRF) sensitivity runs for a two-week period in June 2016 during which high surface ozone concentrations were observed in the Lake Michigan region. The WRF simulations tested various combinations of physical options, forcing data, sea surface temperature integration, and nudging options for three nested modeling domains. Given the multiple model simulations, we needed a way to select the best WRF configuration for simulating the key atmospheric physical processes that drive ground level ozone formation in the region. We developed a new diagnostic approach for identifying the best WRF model configuration for our application. The approach uses statistical significance testing for comparing multiple model simulations for different periods in the diurnal cycle. We will present how this diagnostic approach is used for understanding the differences between WRF simulations and for building our confidence in selecting a configuration to support regulatory air quality modeling applications in the Midwest.