Session: Controls 3

Paper Number: 110846

110846 - Reinforcement Learning-Based Process Optimization and Control of a Hydraulic Press With Multiple Actuators 

Methods of reinforcement learning are finding great success in solving complex decision and control tasks across different domains and applications. A characteristic of these methods is that they aim to train a so-called agent to learn a specific behavior through interaction with its environment. Every time the agent performs an action it receives feedback from its environment regarding the desirability of the recent action taken. In an iterative process, the agent learns to optimize the rule of taking actions, considering the current state and past feedback received. Accordingly, the interactive and data-driven nature of reinforcement learning methods allows the development of high-performing controllers without the need for rigorous physical modeling, parameter identification, and specific knowledge of the considered system. Moreover, the underlying algorithms for training a controller, or a so-called agent, seek to produce an optimal controller according to a user-defined objective. Additionally, the objective is not limited to a single target state of the system but can rather consider several different, and possibly contradicting targets. Considering an agent as a controller of a fluid-mechatronic system, methods of reinforcement learning pose a promising alternative to conventional techniques of controller development. [1]

Investigations on the applicability of reinforcement learning methods for the control of fluid power systems can be found in the literature [2], [3]. However, previous works either aimed at using these methods for tracking a pre-specified trajectory of a single target state or only considered applications with a single actuator.

This paper presents the development of a reinforcement learning-based control strategy for controlling a hydraulic press with multiple actuators without providing a specific trajectory to be tracked. Instead, the control objective of the press operation is expressed within the reward function, which gives an evaluation of a controller’s performance. Hence, during an agent's training, a policy is learned that produces an optimized trajectory, by actuating and coordinating multiple actuators to fulfill control objectives expressed in the reward function. Investigations are conducted on a simulation model of the considered hydraulic press, which comprises a proportional valve for the position control of the press ram as well as switching valves. Besides the position control of the press ram, the control objective also aims to restrict the velocity and the impact energy and minimize oscillations and energy consumption. Moreover, the actuation of multiple actuators with continuous and discrete control signals is explored. For this, a solution using a single agent controlling all actuators is empirically compared against a solution using separate agents for the different actuators. The single-agent solution is designed using the deep deterministic policy gradient algorithm which applies continuous control outputs, while in the multi-agent solutions, each agent is designed using an algorithm suitable for the actual control signal type of the respective valve.

As a result, this paper aims to present a solution for applying methods of reinforcement learning to industrial hydraulic systems with multiple actuators. For this, the use of a single-agent solution is compared against multi-agent solutions. Furthermore, an approach is presented, that integrates trajectory optimization into the training process so that the effort for process engineering can be reduced.

[1]        Sutton, R. S., Barto, A. Reinforcement learning, second edition - An introduction, The MIT Press, Cambridge, Massachusetts, London, England, 2018.

[2]        Kreutmayr, F., Imlauer, M. Application of machine learning to improve to performance of a pressure-controlled system, In: Proceedings of the 12th International Fluid Power Conference, p. 77–86, 2020.

[3]        Brumand-Poor, F., Matthiesen, G., Schmitz, K. Control of a Hydromechanical Pendulum with a Reinforcement Learning Agent, In: Proceedings of the 13th International Fluid Power Conference, 2022.

Presenting Author: Faried Makansi RWTH Aachen University, Institute for Fluid Power Drives and Systems (ifas)

Presenting Author Biography: Faried Makansi, M.Sc. graduated in Mechanical and Process Engineering at the Technical University Darmstadt. Since 2019 he has been working as research associate at the Institute for Fluid Power Drives and System of RWTH Aachen University. His general research interest is the use of digital technologies in fluid power applications with a special focus on data-driven approaches for condition monitoring and machine control.