Python DataSets and Pandas

Python is a powerful, flexible, open source language that is easy to learn, easy to use, and has powerful libraries for data manipulation and analysis. Its simple syntax is very accessible to programming novices, and will look familiar to anyone with experience in Matlab, C/C++, Java, or Visual Basic. Python has a unique combination of being both a capable general-purpose programming language as well as being easy to use for analytical and quantitative computing.

For over a decade, Python has been used in scientific computing and highly quantitative domains such as finance, oil and gas, physics, and signal processing. It has been used to improve Space Shuttle mission design, process images from the Hubble Space Telescope, and was instrumental in orchestrating the physics experiments which led to the discovery of the Higgs Boson (the so-called "God particle").

Python is easy for analysts to learn and use, but powerful enough to tackle even the most difficult problems in virtually any domain. It integrates well with existing IT infrastructure, and is very platform independent. Among modern languages, its agility and the productivity of Python-based solutions is legendary. Companies of all sizes and in all areas — from the biggest investment banks to the smallest social/mobile web app startups — are using Python to run their business and manage their data.

---

This tutorial was developed using Eclipse IDE.
In order to run Python in Eclipse, go to Help -> Install new Software ... -> and use: 'Pydev p2 Repository - http://pydev.sf.net/updates/'

This tutorial was developed using Python 3.6.1
To Download Python:
https://www.python.org/downloads/

Installing packages (Pandas) :
Next, go to your terminal or cmd.exe, and type:pip install pandas. Did you get a "pip is not a recognized command" or something similar? No problem, this means pip is not on your PATH. Pip is a program, but your machine doesn't just simply know where it is unless it is on your PATH. You can look up how to add something to your path if you like, but you can always just explicitly give the path to the program you want to execute. On Windows, for example, Python's pip is located in C:/Python34/Scripts/pip. Python34 means Python 3.4. If you have Python 3.6, then you would use Python36, and so on.

Thus, if regular pip install pandas didn't work, then you can do
C:/Python34/Scripts/pip install pandas

Matplotlib library:
C:/Python34/Scripts/pip install matplotlib

I am going to show you some Python code in which we can see how to manipulate some data using Pandas module ...

The next code pulls data for Exxon from the Yahoo Finance API, storing the data to our data1  variable.

from pandas_datareader import data
import datetime as dt

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

print(data1)
print(data1.head())

Pandas works great with other modules, Matplotlib being one of them. Let's see! Open your terminal or cmd.exe, and do pip install matplotlib. You should already have got it I am prety sure with your pandas installation, but we want to make sure

from pandas_datareader import data
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

style.use('fivethirtyeight')

data1['High'].plot()
plt.legend()
plt.show()

You can download the complete project with this Python code using Pandas to manipulate data from my GitHub repository:

https://github.com/rolando-febrero/Python_DataSets_and_Pandas

---

Programming thought of the day:

• Funny facts about Google users:
  50% of people use Google well as a search engine.
  The rest  50 % of them use it to check if their internet is connected ....
  

The Top 10 AI And Machine Learning Use Cases Everyone Should Know About

This is an interesting article I found at Forbes.com . This might help to understand and visualize how important and critical is working with Machine Learning technology. Possibilities are endless, but for now, we are going to focus on 10 examples.... enjoy!


by: Bernard Marr

Machine learning is a buzzword in the technology world right now, and for good reason: It represents a major step forward in how computers can learn.

Very basically, a machine learning algorithm is given a “teaching set” of data, then asked to use that data to answer a question. For example, you might provide a computer a teaching set of photographs, some of which say, “this is a cat” and some of which say, “this is not a cat.” Then you could show the computer a series of new photos and it would begin to identify which photos were of cats.

Machine learning then continues to add to its teaching set. Every photo that it identifies — correctly or incorrectly — gets added to the teaching set, and the program effectively gets “smarter” and better at completing its task over time.

It is, in effect, learning.

1. Data Security

Malware is a huge — and growing — problem. In 2014, Kaspersky Lab said it had detected 325,000 new malware files every day. But, institutional intelligence company Deep Instinct says that each piece of new malware tends to have almost the same code as previous versions — only between 2 and 10% of the files change from iteration to iteration. Their learning model has no problem with the 2–10% variations, and can predict which files are malware with great accuracy. In other situations, machine learning algorithms can look for patterns in how data in the cloud is accessed, and report anomalies that could predict security breaches.

2. Personal Security

If you’ve flown on an airplane or attended a big public event lately, you almost certainly had to wait in long security screening lines. But machine learning is proving that it can be an asset to help eliminate false alarms and spot things human screeners might miss in security screenings at airports, stadiums, concerts, and other venues. That can speed up the process significantly and ensure safer events.

3. Financial Trading

Many people are eager to be able to predict what the stock markets will do on any given day — for obvious reasons. But machine learning algorithms are getting closer all the time. Many prestigious trading firms use proprietary systems to predict and execute trades at high speeds and high volume. Many of these rely on probabilities, but even a trade with a relatively low probability, at a high enough volume or speed, can turn huge profits for the firms. And humans can’t possibly compete with machines when it comes to consuming vast quantities of data or the speed with which they can execute a trade.


4. Healthcare

Machine learning algorithms can process more information and spot more patterns than their human counterparts. One study used computer assisted diagnosis (CAD) when to review the early mammography scans of women who later developed breast cancer, and the computer spotted 52% of the cancers as much as a year before the women were officially diagnosed. Additionally, machine learning can be used to understand risk factors for disease in large populations. The company Medecision developed an algorithm that was able to identify eight variables to predict avoidable hospitalizations in diabetes patients.

5. Marketing Personalization

The more you can understand about your customers, the better you can serve them, and the more you will sell. That’s the foundation behind marketing personalisation. Perhaps you’ve had the experience in which you visit an online store and look at a product but don’t buy it — and then see digital ads across the web for that exact product for days afterward. That kind of marketing personalization is just the tip of the iceberg. Companies can personalize which emails a customer receives, which direct mailings or coupons, which offers they see, which products show up as “recommended” and so on, all designed to lead the consumer more reliably towards a sale.

6. Fraud Detection

Machine learning is getting better and better at spotting potential cases of fraud across many different fields. PayPal, for example, is using machine learning to fight money laundering. The company has tools that compare millions of transactions and can precisely distinguish between legitimate and fraudulent transactions between buyers and sellers.

7. Recommendations

You’re probably familiar with this use if you use services like Amazon or Netflix. Intelligent machine learning algorithms analyze your activity and compare it to the millions of other users to determine what you might like to buy or binge watch next. These recommendations are getting smarter all the time, recognizing, for example, that you might purchase certain things as gifts (and not want the item yourself) or that there might be different family members who have different TV preferences.

8. Online Search

Perhaps the most famous use of machine learning, Google and its competitors are constantly improving what the search engine understands. Every time you execute a search on Google, the program watches how you respond to the results. If you click the top result and stay on that web page, we can assume you got the information you were looking for and the search was a success. If, on the other hand, you click to the second page of results, or type in a new search string without clicking any of the results, we can surmise that the search engine didn’t serve up the results you wanted — and the program can learn from that mistake to deliver a better result in the future.

9. Natural Language Processing (NLP)

NLP is being used in all sorts of exciting applications across disciplines. Machine learning algorithms with natural language can stand in for customer service agents and more quickly route customers to the information they need. It’s being used to translate obscure legalese in contracts into plain language and help attorneys sort through large volumes of information to prepare for a case.

10. Smart Cars

IBM recently surveyed top auto executives, and 74% expected that we would see smart cars on the road by 2025. A smart car would not only integrate into the Internet of Things, but also learn about its owner and its environment. It might adjust the internal settings — temperature, audio, seat position, etc. — automatically based on the driver, report and even fix problems itself, drive itself, and offer real time advice about traffic and road conditions.

---

Programming thought of the day:

• Maybe if we start telling people the brain is an app they will start using it.
• 
  

What is the difference between Data Analytics, Data Analysis, Data Mining, Data Science, Machine Learning, and Big Data?

Lately, I've been doing some research on Machine Learning, which seems to be very interesting and impressive from my point of view. Creating software was always interesting, but coding to "educate" your software in a way that it can learn from previous experiences makes this even more interesting and more impressive. However, if you try to find information about Machine Learning, you will see some other topics that are closely related to it, which are: Data Analytics, Data Analysis, Data Mining, Data Science, and Big Data but, How do they differ from each other?

Here are some core concepts:

Data Analytics: Analytics is about applying a mechanical or algorithmic process to derive the insights for example running through various data sets looking for meaningful correlations between them. 

Data Analysis: Analysis is really a heuristic activity, where scanning through all the data the analyst gains some insight. 

Data Mining: this term was most widely used in the late 90's and early 00's when a business consolidated all of its data into an Enterprise Data Warehouse. All of that data was brought together to discover previously unknown trends, anomalies and correlations such as the famed 'beer and diapers' correlation (Diapers, Beer, and data science in retail).

Data Science: a combination of mathematics, statistics, programming, the context of the problem being solved, ingenious ways of capturing data that may not be being captured right now plus the ability to look at things 'differently' (like this Why UPS Trucks Don't Turn Left ) and of course the significant and necessary activity of cleansing, preparing and aligning the data.

Machine Learning: this is one of the tools used by data scientist, where a model is created that mathematically describes a certain process and its outcomes, then the model provides recommendations and monitors the results once those recommendations are implemented and uses the results to improve the model. 

In addition, I found a discussion about this topic and I wanted to share some thoughts which I consider it can help to clarify:

"The way I see it, machine learning is concerned with algorithms whose performance at some task improves as it gains experience at that task, while data mining is concerned with analysing data for the purpose of discovering unforeseen patterns or properties.

So the similarities are obvious, they both look at data, and hope to extract something of value from it. As I see it, the main difference is whether the goal is to reproduce known knowledge (I know that some of these pictures are cats, and some are dogs, now can some algorithm learn that?), or if the goal is to discover unknown knowledge (is there any interesting structure in this data set?). The two are, unsurprisingly, intertwined, as many of the properties or structure one may be searching for in data mining can be identified by machine learning algorithms. For instance, in data mining, one might be interested in determining if clusters of a certain form appear in the data, and could use a machine learning algorithm like k-means. K-means is a learning algorithm, in that if data has a known structure, it can learn it (under specific conditions, blah blah blah).

So data mining is exploratory, machine learning is focused on solving specific tasks well. That's my take on it, anyway." (by: Jordan Frank)


The following graphic nicely summarizes what all is involved in data science.

---

Programming thought of the day:

• The truth is out there. Anybody got the URL?
•