In the past decades, space missions to small bodies (Galileo, OSIRIS-REx, Hayabusa, Hayabusa2, Chang’ e 2, Rosetta, etc.) have enriched us greatly with a lot of new knowledge on our solar system. In situ observations of these missions have revealed the extreme complexity and remarkable diversity of the spatial environment around their target asteroids. A study on the motion of objects in such complex environments is of great importance for understanding the evolution history of surface/sub-surface materials on the asteroids. Establishing a reasonable dynamic model is obviously the most important step. This paper proposes a method for tracking the motion of an object near the surface of an arbitrary asteroid. This method combines the irregular shape, an unlimited rotational state and asymmetric gravitational field, which are three key factors that dominate the complex movement of an object on and off the asteroid’s surface. The gravitational attraction and potential are computed using the polyhedral method with corrections for the possible singularities. The asteroid’s surface is then approximated using a continuous and smooth surface, and the parametric representation forms of the body are derived based on polynomial series. An event-driven scheme is designed, so that the orbital motion and surface motion are processed separately by checking the triggering events. The algorithm was implemented using C++. Benchmarking tests are organized on a local cluster, showing a satisfying performance in both accuracy and efficiency. And this method was also applied to motion control of surface detectors.