How LinkedIn uses Big Data in practice
LinkedIn tracks every move users make on the site, and the company analyses this mountain of data in order to make better decisions and design data-powered features. Clearly, LinkedIn uses Big Data right across the company, but here are just a couple of examples of it in action.
Like other social media networks, LinkedIn uses data to make suggestions for users (“people you may know”). LinkedIn uses machine learning techniques to refine its algorithms and make better suggestions for users. So, if the site regularly suggested people you may know from Company A (which you worked at nine years ago) and Company B (which you worked at four years ago), but you almost never clicked on the company A profiles, LinkedIn would tailor its suggestions going forward with that in mind. This personalised approach enables users to build the networks that work best for them.
Also, the site is constantly gathering and displaying new data for users. LinkedIn uses stream-processing technology to display the most up-to-date information when users are on the site – from who got a new job to useful articles that contacts have shared. Not only does this constant streaming of data add interest, it also speeds up the analytic process. Instead of capturing data and storing it to be analysed at a later time, real-time stream-processing technology allows LinkedIn to stream data direct from the source (user activity) and analyse it on the fly.
The technical details
LinkedIn tracks every move its users make on the site, from everything liked and shared to every job clicked on and every contact messaged. Hadoop forms the core of LinkedIn’s Big Data infrastructure, but other key parts of the LinkedIn Big Data jigsaw include Oracle, Pig, Hive, Kafka, Java and MySQL. In order to ensure high availability and avoid a single point of failure, the company operates out of three main data centres.
Ideas and insights you can steal
LinkedIn provides a lesson to all businesses in how Big Data can lead to impressive growth. The company’s ability to make personalised suggestions and recommendations to its users is a critical part of its success – and this is echoed in other successful companies like Amazon.
You can read more about how LinkedIn is using Big Data to drive success in Big Data in Practice: How 45 Successful Companies Used Big Data Analytics to Deliver Extraordinary Results.
Bernard Marr is a bestselling author, keynote speaker, and advisor to companies and governments. He has worked with and advised many of the world's best-known organisations. LinkedIn has recently ranked Bernard as one of the top 10 Business Influencers in the world (in fact, No 5 - just behind Bill Gates and Richard Branson). He writes on the topics of intelligent business performance for various publications including Forbes, HuffPost, and LinkedIn Pulse. His blogs and SlideShare presentation have millions of readers.