Python DataSets and Pandas

Python is a powerful, flexible, open source language that is easy to learn, easy to use, and has powerful libraries for data manipulation and analysis. Its simple syntax is very accessible to programming novices, and will look familiar to anyone with experience in Matlab, C/C++, Java, or Visual Basic. Python has a unique combination of being both a capable general-purpose programming language as well as being easy to use for analytical and quantitative computing.

For over a decade, Python has been used in scientific computing and highly quantitative domains such as finance, oil and gas, physics, and signal processing. It has been used to improve Space Shuttle mission design, process images from the Hubble Space Telescope, and was instrumental in orchestrating the physics experiments which led to the discovery of the Higgs Boson (the so-called "God particle").

Python is easy for analysts to learn and use, but powerful enough to tackle even the most difficult problems in virtually any domain. It integrates well with existing IT infrastructure, and is very platform independent. Among modern languages, its agility and the productivity of Python-based solutions is legendary. Companies of all sizes and in all areas — from the biggest investment banks to the smallest social/mobile web app startups — are using Python to run their business and manage their data.

---

This tutorial was developed using Eclipse IDE.
In order to run Python in Eclipse, go to Help -> Install new Software ... -> and use: 'Pydev p2 Repository - http://pydev.sf.net/updates/'

This tutorial was developed using Python 3.6.1
To Download Python:
https://www.python.org/downloads/

Installing packages (Pandas) :
Next, go to your terminal or cmd.exe, and type:pip install pandas. Did you get a "pip is not a recognized command" or something similar? No problem, this means pip is not on your PATH. Pip is a program, but your machine doesn't just simply know where it is unless it is on your PATH. You can look up how to add something to your path if you like, but you can always just explicitly give the path to the program you want to execute. On Windows, for example, Python's pip is located in C:/Python34/Scripts/pip. Python34 means Python 3.4. If you have Python 3.6, then you would use Python36, and so on.

Thus, if regular pip install pandas didn't work, then you can do
C:/Python34/Scripts/pip install pandas

Matplotlib library:
C:/Python34/Scripts/pip install matplotlib

I am going to show you some Python code in which we can see how to manipulate some data using Pandas module ...

The next code pulls data for Exxon from the Yahoo Finance API, storing the data to our data1  variable.

from pandas_datareader import data
import datetime as dt

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

print(data1)
print(data1.head())

Pandas works great with other modules, Matplotlib being one of them. Let's see! Open your terminal or cmd.exe, and do pip install matplotlib. You should already have got it I am prety sure with your pandas installation, but we want to make sure

from pandas_datareader import data
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

style.use('fivethirtyeight')

data1['High'].plot()
plt.legend()
plt.show()

You can download the complete project with this Python code using Pandas to manipulate data from my GitHub repository:

https://github.com/rolando-febrero/Python_DataSets_and_Pandas

---

Programming thought of the day:

• Funny facts about Google users:
  50% of people use Google well as a search engine.
  The rest  50 % of them use it to check if their internet is connected ....
  

What is the difference between Data Analytics, Data Analysis, Data Mining, Data Science, Machine Learning, and Big Data?

Lately, I've been doing some research on Machine Learning, which seems to be very interesting and impressive from my point of view. Creating software was always interesting, but coding to "educate" your software in a way that it can learn from previous experiences makes this even more interesting and more impressive. However, if you try to find information about Machine Learning, you will see some other topics that are closely related to it, which are: Data Analytics, Data Analysis, Data Mining, Data Science, and Big Data but, How do they differ from each other?

Here are some core concepts:

Data Analytics: Analytics is about applying a mechanical or algorithmic process to derive the insights for example running through various data sets looking for meaningful correlations between them. 

Data Analysis: Analysis is really a heuristic activity, where scanning through all the data the analyst gains some insight. 

Data Mining: this term was most widely used in the late 90's and early 00's when a business consolidated all of its data into an Enterprise Data Warehouse. All of that data was brought together to discover previously unknown trends, anomalies and correlations such as the famed 'beer and diapers' correlation (Diapers, Beer, and data science in retail).

Data Science: a combination of mathematics, statistics, programming, the context of the problem being solved, ingenious ways of capturing data that may not be being captured right now plus the ability to look at things 'differently' (like this Why UPS Trucks Don't Turn Left ) and of course the significant and necessary activity of cleansing, preparing and aligning the data.

Machine Learning: this is one of the tools used by data scientist, where a model is created that mathematically describes a certain process and its outcomes, then the model provides recommendations and monitors the results once those recommendations are implemented and uses the results to improve the model. 

In addition, I found a discussion about this topic and I wanted to share some thoughts which I consider it can help to clarify:

"The way I see it, machine learning is concerned with algorithms whose performance at some task improves as it gains experience at that task, while data mining is concerned with analysing data for the purpose of discovering unforeseen patterns or properties.

So the similarities are obvious, they both look at data, and hope to extract something of value from it. As I see it, the main difference is whether the goal is to reproduce known knowledge (I know that some of these pictures are cats, and some are dogs, now can some algorithm learn that?), or if the goal is to discover unknown knowledge (is there any interesting structure in this data set?). The two are, unsurprisingly, intertwined, as many of the properties or structure one may be searching for in data mining can be identified by machine learning algorithms. For instance, in data mining, one might be interested in determining if clusters of a certain form appear in the data, and could use a machine learning algorithm like k-means. K-means is a learning algorithm, in that if data has a known structure, it can learn it (under specific conditions, blah blah blah).

So data mining is exploratory, machine learning is focused on solving specific tasks well. That's my take on it, anyway." (by: Jordan Frank)


The following graphic nicely summarizes what all is involved in data science.

---

Programming thought of the day:

• The truth is out there. Anybody got the URL?
•