Although many impressive advances have been reported in table tennis robots field using the reinforcement learning method, challenges related to policy complexity and adaptability continue to hinder large-scale deployment and practical application. In this work, building upon extensive prior studies, we propose several techniques and integrate them into a unified framework that reduces the difficulty of training and deployment while enhancing the human player enjoyment. Specifically, a recursively nested design is introduced in the hierarchical decision-making system, which fully separates high-level decision-making from low-level execution, eliminating the need to train multiple workers at the execution layer. The multi-objective problem is also studied by introducing a tolerance-based weight modulation mechanism, which can balance the landing accuracy with other objectives and adjust the playing strategy to meet diverse goals. Comprehensive experiments demonstrate the effectiveness of the proposed framework and techniques. Notably, the robot achieved up to 18 consecutive rallies with human player, which, to the best of our knowledge, is the longest rally attained with a fixed-base collaborative robot arm. The proposed methods can be readily extended to other racket-sport robots or similar tasks.
The overall structure of the proposed novel decision-making and training framework. The hierarchy consists of three layers from top to bottom: Boss, Manager, and Worker, each with its own Actor, Critic and the environment. The Boss makes the highest-level decision by selecting the target landing position based only on the ball state (i.e., all decisions in the system start from the ball state). The Manager then determines a weight vector for multiple objectives (landing accuracy, net clearance, and return velocity) trade-off. Finally, the Worker generates the executable action to realize the decisions made by Boss and Manager. This hierarchically structured decision-making system exhibits a recursively nested architecture from top to bottom, that the input and output is combined as the input of next level.