Nielsen

Migration of an oracle data platform to hadoop

Retail

WHY DID THEY CALL US ?

As part of its activity of sales data analysis (collected from retailers and panelist consumers), Nielsen decided to migrate its data storage, aggregation and reporting platform from Oracle to Hadoop.

The aggregation engine is developed in C++

OUR MISSION

The primary objective of this migration is to gather all existing platforms into a single global platform.

The performance objectives are as follows:

• improvement of the performance of the data extraction part

• implementation of an efficient data reporting tool

• acceleration of the aggregation engine

RESULTS

• Porting oracle SQL data extraction requests (via occi) to Hadoop/Impala (via ODBC). Requests which are not supported by Impala are interpreted directly into the C++ code.
• Design and development of a high-performance tool for loading aggregate data into IMPALA tables via HDFS, using the C++ library libhdfs3.
• Hybrid parallelization MPI+BOOST. Threads on Hadoop clusters from the aggregation engine
• The data loading tool allows you to insert a 70GB file into an IMPALA table in less than 10 minutes on a gigabit network. The use of libhdfs3 allows the application to have a small memory footprint compared to libhdfs use (gain of a factor of 8)
• The aggregation engine distributed with MPI+BOOST. Threads achieves 60% parallel efficiency on 8 nodes (20 core/node) of a hadoop cluster



Want to talk to us about your project ?
Contact us
Contact us

122 avenue du Général Leclerc
92100 Boulogne Billancourt
France


+33 (0)1 46 44 24 66
contact@aneo.fr

Join us

Come and play !
A strong team is where it all begins.

Join us

lectus ipsum mattis accumsan id, amet,