Implementación de un algoritmo de aprendizaje automático en Apache Spark
Cargando...
Fecha
Autores
Sánchez Alba, Ricardo
Título de la revista
ISSN de la revista
Título del volumen
Editor
Universidad Central “Marta Abreu” de Las Villas
Resumen
El análisis de grandes cantidades de datos, así como la extracción de conocimiento útil de
estos constituye en la actualidad un reto ya que cada día crecen velozmente los volúmenes
de información generada y se necesitan programas capaces de realizar esta tarea en poco
tiempo.
Durante varios años frameworks de código abierto han sido utilizados para la aplicación
de técnicas de aprendizaje automático en pequeños volúmenes de datos, pero la necesidad
creciente de la industria ha dado como consecuencia una evolución en el área del cómputo
distribuido, surgiendo así herramientas como Apache Hadoop y Apache Spark siendo éste
último entre 10 y 100 veces más rápido que su antecesor.
En este trabajo se propone un procedimiento general para la inclusión de nuevos algoritmos
de aprendizaje automático en el framework Apache Spark y se implementa un algoritmo
de regresión lineal con el fin de validar la metodología propuesta. Se realizaron una serie de
experimentos al software implementado que permitieron valorar las ventajas del framework
Apache Spark para reducir significativamente los tiempos de ejecución cuando este tipo
de algoritmo se somete al procesamiento de cantidades masivas de datos.
The analysis of large amounts of data, as well as the extraction of useful knowledge of these, is now a challenge as each day the volumes of information generated grow rapidly and programs are needed that can perform this task in a short time. For several years Open source frameworks have been used for the application of automated learning techniques in small volumes of data, but the growing need of the industry has resulted in an evolution in the area of distributed computing, resulting in tools such as Apache Hadoop and Apache Spark The latter being between 10 and 100 times faster than its predecessor. In this paper we propose a general procedure for the inclusion of new algorithms of au- tomatic learning in the Apache Spark framework and a linear regression algorithm is implemented in order to validate the proposed methodology. A series of experiments were performed on the implemented software that allowed to eva- luate the advantages of the Apache Spark framework to significantly reduce execution times when this type of algorithm is submitted to the processing of massive amounts of data.
The analysis of large amounts of data, as well as the extraction of useful knowledge of these, is now a challenge as each day the volumes of information generated grow rapidly and programs are needed that can perform this task in a short time. For several years Open source frameworks have been used for the application of automated learning techniques in small volumes of data, but the growing need of the industry has resulted in an evolution in the area of distributed computing, resulting in tools such as Apache Hadoop and Apache Spark The latter being between 10 and 100 times faster than its predecessor. In this paper we propose a general procedure for the inclusion of new algorithms of au- tomatic learning in the Apache Spark framework and a linear regression algorithm is implemented in order to validate the proposed methodology. A series of experiments were performed on the implemented software that allowed to eva- luate the advantages of the Apache Spark framework to significantly reduce execution times when this type of algorithm is submitted to the processing of massive amounts of data.