Abstract

Intelligent big data analysis is an evolving pattern in the age of big data science and artificial intelligence (AI). Analysis of organized data has been very successful but analysing human behaviour using social media data becomes challenging. The social media data comprises a vast and unstructured format of data sources that can include likes, comments, tweets, shares, and views. Data analytics of social media data became a challenging task for companies, such as Dailymotion, that have billions of daily users and vast numbers of comments, likes, and views. Social media data is created in a significant amount and at a tremendous pace. There is a very high volume to store, sort, process, and carefully study the data for making possible decisions. This project proposes an architecture using a big data analytics mechanism to process the huge social media datasets efficiently and logically. In addition, this work employing parallel processing techniques called spark, which will create multiple threads and then distribute work between those thread to perform task parallelly and then send result back to spark. All existing algorithms works on single thread but spark will distribute works in multiple threads so its paralleling processing will be faster and suitable for big data applications.

Description