7 The Model’s Out-of-Sample Performance
The out-of-sample evaluation period consists of 13,175 hours over the 1
Jan 2016 to 31 Aug 2017 time interval. Recalling that the dependent
variable in the model is the natural logarithm of temperature measured
in Kelvin, it might seem that a simple retransformation would yield the
optimal predicted value. Unfortunately, merely taking the antilogarithm
of the predicted natural logarithm of temperature measured in Kelvin may
result in a biased temperature prediction (Granger and Newbold, 1976,
pp. 196-197). This bias is easily resolved when the error distribution
is Gaussian using a method presented by Guerrero (1993). Given the
non-Gaussian nature of the error distribution in this case, the matter
was resolved by estimating a post-processing regression without a
constant term using all of the observations in the sample. The
explanatory variable in this post-processing regression is the hourly
temperature measured in Kelvin, while the explanatory variable in this
regression is the antilog of the transformed predicted values. The
estimated coefficient corresponding to the explanatory variable equals
0.9999895. The associated R-Square equals 1.0000. The estimated
parameter from this regression was used to detransform the out-of-sample
transformed predicted temperature values.
The out-of-sample predictions were compared with the ERA5 predictions
for the same general location. For those unfamiliar with the ERA5
modeling results, it was produced by the Copernicus Climate Change
Service at ECMWF. In a significant advance from its earlier databases,
it reports hourly values across the globe. The ERA5 hourly temperature
values for the Barrow location were obtained from Meteoblue (
https://content.meteoblue.com/en/specifications/data-sources/weather-simulation-data/reanalysis-datasets
).
The out-of-sample temperature predictions from the ARCH/ARMAX model
presented in this paper have a predictive R-square of 0.9962. The
predictions are visually more accurate than the ERA5 values for the same
general location (Figure 10), although it should be noted that the ERA5
values correspond to a grid that includes land and ocean while Barrow
represents a land location within that grid. Nevertheless, the ERA5
values may serve as a useful benchmark for the ARCH/ARMAX out-of-sample
predictions. Regarding the RMSEs, the predictions associated with the
ARCH/ARMAX model have an RMSE equal to about 0.682 oC,
while the ERA5 outcomes have an RMSE of about 3.117oC. Interestingly, an ordinary least-squares
estimation of the ERA5 predictions indicates that the prediction errors
are not purely random. Specifically, the prediction error is conditional
on the magnitude of the predicted temperature and lagged value of the
CO2 concentration. The latter finding is consistent with
the central thesis of this paper. Following Granger’s discussion of
prediction errors (1986, p. 91), both of these findings suggest a
pathway to improving the accuracy of the ERA5 predictions.
The out-of-sample temperature predictions from the ARCH/ARMAX model are
significantly degraded when the estimated effects of CO2are ignored (Figure 11). The differential in predictive accuracy is
visually apparent if one inspects the vertical distance between the
scatter points and the 45o line representing the
relationship between predicted and actual temperature when the
predictions are perfect. As reported above, the full model presented in
this paper has an RMSE equal to 0.682 oC over the
evaluation period, constraining the CO2 estimated
effects to be equal to zero results in predictions with an RMSE equal to
3.379 oC.
The out-of-sample analysis is supportive of the earlier discussion
indicating the unimportance of factors other than CO2and the total downward solar irradiance being drivers of the increase in
annual temperature over the sample period. Specifically, using the full
model, the mean predicted temperature over the evaluation period equals
- 8.725218 oC. The mean predicted temperature over the
evaluation period is -8.725221 oC if the estimated
effects of the binary variables for 1986 through 2014 are constrained to
equal zero. In short, the binary variables that control for the
possibility of annual temperature being affected by factors other than
CO2 or total downward solar irradiance have virtually no
effect on the out-of-sample predicted temperature. Interestingly, the
mean actual temperature over the evaluation period equals -8.712713oC, a very close value to the mean of the predicted
values.