Algoritmos de aprendizagem por reforço para problemas de otimização multiobjetivo

Abstract

Multi-objective optimization problems depict real situations and therefore, this class of problems is extremely important. However, even though it has been studied for decades, this class of problems continues to provide challenging situations, especially with the increasing complexity of problems that arise over time. Among all the difficulties that we can find in the optimization of multiple objectives simultaneously, whether conflicting or not, one of the main ones with which existing algorithms and approaches possess is the need for a priori knowledge of the problem, causing a predefined importance for each of the objectives, seeking to establish an isomorphic between weighting and a solution. When dealing with this class of problems through reinforcement learning, two approaches are predominant: single policy (single-policy) and multiple policies (multi-policy). Algorithms and techniques that use the first approach suffer from the need for prior knowledge of the problem, an inherent characteristic of multi-objective problems. The second approach has other difficulties, such as: limiting the set of solutions and high computational cost. Given this presented context, the work proposes two hybrid algorithms, called Q-Managed with reset and Q-Managed without reset. Both are a hybridization of the Q-learning algorithm and the econstraint approach, respectively techniques belonging to reinforcement learning and multi-objective optimization. In summary, the proposed algorithms work as follows: Q-Learning is used for environment exploration, while the econstraint approach is used for the environment dynamic delimitation — restriction in the solution space search —, allowing to keep intact the essence of how the algorithm Q-Learning works. This delimitation has the following purpose: to impose the learning agent can learn other solutions by blocking actions that lead to solutions already learned and without improving them, that is, solutions to which the learning agent has already converged. This blocking actions feature is performed by the figure of a manager, where it is responsible for observing everything that occurs in the environment. Regarding the difference between the proposed algorithms, basically it is the choice of whether or not to take advantage of the knowledge already acquired of the environment after a solution is considered to be learned, that is, the learning agent has converged to a particular solution. As a way of testing the effectiveness of Q-Managed two versions, traditional benchmarks were used, which were also adopted in other works, thus allowing a fairer comparison. Thus, two comparative approaches were adopted, the first of which was through the implementation of third-party algorithms for direct comparison, while the second was done through a common metric to everyone who used the same benchmarks. In all possible tests, the algorithms proposed here proved to be effective, always finding the entire Pareto Front.