The need for uncertainty quantification placed by weather forecasting makes Bayesian deep learning (BDL) a suited candidate for data-driven weather forecasting. In this study, we use Bayesian Long-Short Term Memory neural networks (BayesLSTMs) to forecast output from the Lorenz 84 system with seasonal forcing. The latter represents the dynamics of large scale eddies (Rossby waves) on a westerly jet. We show that forecasts with the BayesLSTM can stay close to the attractor of the Lorenz model and conclude that they represent the nonlinear relations between each component in this simplified atmospheric circulation system. The forecasts are evaluated against persistence and a Vector Autoregressive Model (VAR). We demonstrate that the BayesLSTMs can produce reliable probabilistic forecasts and address uncertainties relevant to weather forecasting. Our study indicates that BDL is an easy and fast solution for probabilistic weather forecast and is promising to enhance weather forecasting capabilities at short to medium-range timescales.