Here is another post in which I try to disentangle some of the concepts that underpin today’s big data world. In this post I look at Python, which is an open source programming language commonly used for data manipulation in commercial Big Data operations.

Python is a programming language frequently used to create algorithms for sorting through and analysing the huge amounts of data collected by businesses and organisations around the world today.

In a nutshell I would say that there are three core strengths of Python which have contributed to its enthusiastic adoption by programmers working with Big Data, and they are:

Powerful libraries which mean it can easily be used to process very large, growing sets of data
Simple syntax and command set, meaning it is relatively easy to write code, and for that code to be understood by others
Strong support from users and the Open Source community, meaning it integrates very well with other open source platforms commonly used in Big Data ( Spark, Hadoop etc).

The software which allows us to create programs in Python is open source – meaning it is in the public domain and can be freely used by anyone. A big advantage of open source software is that anyone can modify it and create their own versions to do specific tasks – this is one of the main reasons that the concept of open source has been enthusiastically embraced by Big Data fans. It allows a great deal of flexibility. (See also Hadoop).

Python is a high level language – meaning that the code which the programmer types into to create the program is more like natural human language than code written to control machines. This not only makes things simpler for the programmer, it means others are more likely to understand the code if they want to use it themselves. The high-level, human-like code is converted into machine code which is understood by machines, through a piece of software known as an interpreter.

This means that programs written in Python can be run on any computer operating system which has an interpreter for it – which is pretty much all of the operating systems you are ever likely to come across! This means that code can be ported between projects and organisations even if the people running it are using completely different hardware (as is often the case in projects using open source technologies). Because of the huge amount of support it has from the open source community, it also has very good support for a large number of file and database formats, which it can directly read from and write to.

Aside from its ease of use and portability, one of the features which has made it particularly popular with developers working on Big Data projects is the powerful libraries available for it. These are mostly extensions to the functionality that can be created in programs written in the language, and many programmers have created powerful and versatile tools and algorithms specifically designed at manipulating the large amounts of data that come with Big Data initiatives.

Another feature is that it is great for creating scalable systems – in fact it is used for creating much of the back end, data-processing functions of Google, Youtube and Facebook. As well as constantly increasing in size, these services need to be constantly updating and adding to their functionality. With giant operations such as these, programmers need an environment where new code (features) can be integrated on-the-fly without disruption of the service to users. Python is ideal for this as it is designed for use in “agile” environments where new features need to be added on-the-fly, first in a limited way for testing, and then rolled out across the entire system.

So, that’s just a quick and basic overview of what Python is, and why it’s so popular with programmers working on data projects. If you want to learn more, there are a lot of resources online, and a good place to start is Python.org (mainly written for programmers or people with some knowledge of programming conventions). If you want to learn how to use Python, there are plenty of great, free resources too, such as Code Academy and Coursera.