Rapid provision of Earth Orientation Parameters (EOPs, here polar motion and dUT1) is indispensable in many geodetic applications and also for spacecraft navigation. There are, however, discrepancies between the rapid EOPs and the final EOPs that have a higher latency, but the highest accuracy. To reduce these discrepancies, we focus on a data-driven approach, present a novel method named ResLearner, and use it in the context of deep ensemble learning. Furthermore, we introduce a geophysically-constrained approach for ResLearner. We show that the most important geophysical information to improve the rapid EOPs is the effective angular momentum functions of atmosphere, ocean, land hydrology, and sea level. In addition, semi-diurnal, diurnal, and long-period tides coupled with prograde and retrograde tidal excitations are important features. The influence of some climatic indices on the prediction accuracy of dUT1 is discussed and El Ni\~{n}o Southern Oscillation is found to be influential. We developed an operational framework, providing the improved EOPs on a daily basis with a prediction window of 63 days to fully cover the latency of final EOPs. We show that under the operational conditions and using the rapid EOPs of the International Earth Rotation and Reference Systems Service (IERS) we achieve improvements as high as 60\%, thus significantly reducing the differences between rapid and final EOPs. Furthermore, we discuss how the new final series IERS 20 C04 is preferred over 14 C04. Finally, we compare against EOP hindcast experiments of European Space Agency, on which ResLearner presents comparable improvements.