Identificação de dificuldades e questões de interesse de desenvolvedores de aplicações para Big Data com o framework Apache Spark

Abstract

This research aims to identify and classify the main difficulties and issues of interest of Apache Spark application developers regarding the framewok usage. For this purpose, we use the Latent Dirichlet Allocation algorithm to perform a probabilistic modeling of topics on information extracted from Stack Overflow, since the manual inspection of the entire dataset is not feasible. From the knowledge obtained by the comprehensive study of related works, we established and applied a methodology based on the practices usually employed. We developed Spark applications for the automated execution os tasks, such as the data selection and preparation, the discovery of topics - applying the probabilistic modeling algorithm with various configurations - and metrics computation. Analyzes of the results were carried by a group of 5 researchers: two doctor professors, one doctoral student and two master students. Based on the semantic analysis of the labels assigned to each of the identified topics, a taxonomy of interests and difficulties was constructed. Finally, we ranked the most important themes according to the various calculated metrics and compared the methods and results of our study with those presented in another work.