7 The Model’s Out-of-Sample Performance
The out-of-sample evaluation period consists of 13,175 hours over the 1 Jan 2016 to 31 Aug 2017 time interval. Recalling that the dependent variable in the model is the natural logarithm of temperature measured in Kelvin, it might seem that a simple retransformation would yield the optimal predicted value. Unfortunately, merely taking the antilogarithm of the predicted natural logarithm of temperature measured in Kelvin may result in a biased temperature prediction (Granger and Newbold, 1976, pp. 196-197). This bias is easily resolved when the error distribution is Gaussian using a method presented by Guerrero (1993). Given the non-Gaussian nature of the error distribution in this case, the matter was resolved by estimating a post-processing regression without a constant term using all of the observations in the sample. The explanatory variable in this post-processing regression is the hourly temperature measured in Kelvin, while the explanatory variable in this regression is the antilog of the transformed predicted values. The estimated coefficient corresponding to the explanatory variable equals 0.9999895. The associated R-Square equals 1.0000. The estimated parameter from this regression was used to detransform the out-of-sample transformed predicted temperature values.
The out-of-sample predictions were compared with the ERA5 predictions for the same general location. For those unfamiliar with the ERA5 modeling results, it was produced by the Copernicus Climate Change Service at ECMWF. In a significant advance from its earlier databases, it reports hourly values across the globe. The ERA5 hourly temperature values for the Barrow location were obtained from Meteoblue ( https://content.meteoblue.com/en/specifications/data-sources/weather-simulation-data/reanalysis-datasets ).
The out-of-sample temperature predictions from the ARCH/ARMAX model presented in this paper have a predictive R-square of 0.9962. The predictions are visually more accurate than the ERA5 values for the same general location (Figure 10), although it should be noted that the ERA5 values correspond to a grid that includes land and ocean while Barrow represents a land location within that grid. Nevertheless, the ERA5 values may serve as a useful benchmark for the ARCH/ARMAX out-of-sample predictions. Regarding the RMSEs, the predictions associated with the ARCH/ARMAX model have an RMSE equal to about 0.682 oC, while the ERA5 outcomes have an RMSE of about 3.117oC. Interestingly, an ordinary least-squares estimation of the ERA5 predictions indicates that the prediction errors are not purely random. Specifically, the prediction error is conditional on the magnitude of the predicted temperature and lagged value of the CO2 concentration. The latter finding is consistent with the central thesis of this paper. Following Granger’s discussion of prediction errors (1986, p. 91), both of these findings suggest a pathway to improving the accuracy of the ERA5 predictions.
The out-of-sample temperature predictions from the ARCH/ARMAX model are significantly degraded when the estimated effects of CO2are ignored (Figure 11). The differential in predictive accuracy is visually apparent if one inspects the vertical distance between the scatter points and the 45o line representing the relationship between predicted and actual temperature when the predictions are perfect. As reported above, the full model presented in this paper has an RMSE equal to 0.682 oC over the evaluation period, constraining the CO2 estimated effects to be equal to zero results in predictions with an RMSE equal to 3.379 oC.
The out-of-sample analysis is supportive of the earlier discussion indicating the unimportance of factors other than CO2and the total downward solar irradiance being drivers of the increase in annual temperature over the sample period. Specifically, using the full model, the mean predicted temperature over the evaluation period equals - 8.725218 oC. The mean predicted temperature over the evaluation period is -8.725221 oC if the estimated effects of the binary variables for 1986 through 2014 are constrained to equal zero. In short, the binary variables that control for the possibility of annual temperature being affected by factors other than CO2 or total downward solar irradiance have virtually no effect on the out-of-sample predicted temperature. Interestingly, the mean actual temperature over the evaluation period equals -8.712713oC, a very close value to the mean of the predicted values.