Python pathlib - Move and rename documents placed in several nested folders into a new single folder

Let's say I have these files across several folders like this:

dir
├── 0
│   ├── 103425.xml
│   ├── 105340.xml
│   ├── 109454.xml
│
│── 1247
│   └── doc.xml
├── 14568
│   └── doc.xml
├── 1659
│   └── doc.xml
├── 10450
│   └── doc.xml
├── 10351
│   └── doc.xml

but my goal is to place all files like this:

dir
├── 0
│   ├── 103425.xml
│   ├── 105340.xml
│   ├── 109454.xml
│
│── 1247
│   └── doc.xml
├── 14568
│   └── doc.xml
├── 1659
│   └── doc.xml
├── 10450
│   └── doc.xml
├── 10351
│   └── doc.xml

So basically I want to Move and rename documents placed in several nested folders into a new single folder.

Using Python should be enough to complete this task. Pathlib is a module that manipulate filesytem paths as string objects. This module is best used with Python 3.2 or later, but it is also compatible with Python 2.7. If using it with Python 3.3, you also have access to optional openat-based filesystem operations.
This Python code will do the job:

import os
import pathlib

OLD_DIR = 'files'
NEW_DIR = 'new_dir'

p = pathlib.Path(OLD_DIR)
for f in p.glob('**/*.xml'):
    new_name = '{}_{}'.format(f.parent.name, f.name)
    f.rename(os.path.join(NEW_DIR, new_name))

---

Programming thought of the day:

• If Bill Gates had a penny for every time I had to reboot my computer ...oh wait, he does.
• 
  

Python DataSets and Pandas

Python is a powerful, flexible, open source language that is easy to learn, easy to use, and has powerful libraries for data manipulation and analysis. Its simple syntax is very accessible to programming novices, and will look familiar to anyone with experience in Matlab, C/C++, Java, or Visual Basic. Python has a unique combination of being both a capable general-purpose programming language as well as being easy to use for analytical and quantitative computing.

For over a decade, Python has been used in scientific computing and highly quantitative domains such as finance, oil and gas, physics, and signal processing. It has been used to improve Space Shuttle mission design, process images from the Hubble Space Telescope, and was instrumental in orchestrating the physics experiments which led to the discovery of the Higgs Boson (the so-called "God particle").

Python is easy for analysts to learn and use, but powerful enough to tackle even the most difficult problems in virtually any domain. It integrates well with existing IT infrastructure, and is very platform independent. Among modern languages, its agility and the productivity of Python-based solutions is legendary. Companies of all sizes and in all areas — from the biggest investment banks to the smallest social/mobile web app startups — are using Python to run their business and manage their data.

---

This tutorial was developed using Eclipse IDE.
In order to run Python in Eclipse, go to Help -> Install new Software ... -> and use: 'Pydev p2 Repository - http://pydev.sf.net/updates/'

This tutorial was developed using Python 3.6.1
To Download Python:
https://www.python.org/downloads/

Installing packages (Pandas) :
Next, go to your terminal or cmd.exe, and type:pip install pandas. Did you get a "pip is not a recognized command" or something similar? No problem, this means pip is not on your PATH. Pip is a program, but your machine doesn't just simply know where it is unless it is on your PATH. You can look up how to add something to your path if you like, but you can always just explicitly give the path to the program you want to execute. On Windows, for example, Python's pip is located in C:/Python34/Scripts/pip. Python34 means Python 3.4. If you have Python 3.6, then you would use Python36, and so on.

Thus, if regular pip install pandas didn't work, then you can do
C:/Python34/Scripts/pip install pandas

Matplotlib library:
C:/Python34/Scripts/pip install matplotlib

I am going to show you some Python code in which we can see how to manipulate some data using Pandas module ...

The next code pulls data for Exxon from the Yahoo Finance API, storing the data to our data1  variable.

from pandas_datareader import data
import datetime as dt

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

print(data1)
print(data1.head())

Pandas works great with other modules, Matplotlib being one of them. Let's see! Open your terminal or cmd.exe, and do pip install matplotlib. You should already have got it I am prety sure with your pandas installation, but we want to make sure

from pandas_datareader import data
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style

''' This pulls data for Exxon from the Yahoo Finance API '''
ticker = 'XOM'  

start = dt.datetime(2010, 1, 1)
end = dt.datetime(2015, 8, 22)
data1 = data.DataReader(ticker,'yahoo',start,end)

style.use('fivethirtyeight')

data1['High'].plot()
plt.legend()
plt.show()

You can download the complete project with this Python code using Pandas to manipulate data from my GitHub repository:

https://github.com/rolando-febrero/Python_DataSets_and_Pandas

---

Programming thought of the day:

• Funny facts about Google users:
  50% of people use Google well as a search engine.
  The rest  50 % of them use it to check if their internet is connected ....
  

Microservice Architecture - Explanation, pros and cons

As most Software Developers out there, most of my experience is with (huge) monolithic apps. So when I first read or heard about microservices I was a little confused about how they work and how to implement this kind of architecture, and most important.. does it work? any pros and cons? is it the next step in the development process?

I don't believe microservices can be considered "the best aproach" in every single situation. However, it looks like a good option. But, as everything in life, all is relative...

I'll show you some facts I found on the internet about microservices architecture so you can have a better understanding about it. It you are confused (as I was), hope it helps to clarify some concepts....


1. What is Architecture (Software)?

Architecture is the fundamental organization of a system embodied in its components (i.e. Web Server, Application Server, Databases,Storage, Communication layer, etc…), their relationships to each other, and to the environment (i.e. deployment environment shared server, dedicated server, cloud deployment, etc..), and the principles guiding its design and evolution.

2. What is microservice architecture ?

Microservice means developing a single, small, meaningful functional feature as single service, each service has it’s own process and communicate with lightweight mechanism, deployed in single or multiple servers.

3. Advantages of microservice architecture ?

Each micro service is small and focused on a specific feature / business requirement.
Microservice can be developed independently by small team of developers (normally 2 to 5 developers).
Microservice is loosely coupled, means services are independent, in terms of development and deployment both.
Microservice can be developed using different programming language (Personally I don’t suggest to do it).
Microservice allows easy and flexible way to integrate automatic deployment with Continuous Integration tools (for e.g: Jenkins, Hudson, bamboo etc..).
The productivity of a new team member will be quick enough.
Microservice is easy to understand, modify and maintain for a developer because separation of code,small team and focused work.
Microservice allows you to take advantage of emerging and latest technologies (framework, programming language , programming practice, etc.).
Microservice has code for business logic only, No mixup with HTML,CSS or other UI component.
Microservice is easy to scale based on demand.
Microservice can deploy on commodity hardware or low / medium configuration servers.
Easy to integrate 3rd party service.
Every microservice has it’s own storage capability but it depends on the project’s requirement, you can have common database like MySQL or Oracle for all services.

4. Disadvantages of microservice architecture ?

Microservice architecture brings a lot of operations overhead.
DevOps Skill required (http://en.wikipedia.org/wiki/DevOps).
Duplication of Effort.
Distributed System is complicated to manage .
Default to trace problem because of distributed deployment.
Complicated to manage whole products when number of services increases.

5. In which case / requirement microservice architecture best fit ?

When you need to support Desktop, web , mobile, Smart TVs, Wearable, etc… or you don’t know in future which kind of devices you need to support.

6. Which products / companies are using Microservie architecture?

Most large scale web sites including Twitter, Netflix, Amazon and eBay have evolved from a monolithic architecture to a microservices architecture.

7. How independent micro services communicate with each other?

It’s depend upon requirement, normally developers use HTTP/REST with JSON or Protobuf (Binary protocol) but are free to use any communication protocol.

8. Why is it that everyone are talking about microservices now?

It’s been nearly 15 years since the concept of Service Oriented Architecture really took hold. With the improvement of RESTful web service and JSON as a data interchange format has made it easier than ever to build easily interconnectable services simply and quickly.

9. Presentation

http://www.infoq.com/presentations/Micro-Services



10. Video





11. Picture: Monolithic vs Modular vs Service oriented architecture

---

Programming thought of the day:

• Are you an exception? I bet I can catch you. =)

The Top 10 AI And Machine Learning Use Cases Everyone Should Know About

This is an interesting article I found at Forbes.com . This might help to understand and visualize how important and critical is working with Machine Learning technology. Possibilities are endless, but for now, we are going to focus on 10 examples.... enjoy!


by: Bernard Marr

Machine learning is a buzzword in the technology world right now, and for good reason: It represents a major step forward in how computers can learn.

Very basically, a machine learning algorithm is given a “teaching set” of data, then asked to use that data to answer a question. For example, you might provide a computer a teaching set of photographs, some of which say, “this is a cat” and some of which say, “this is not a cat.” Then you could show the computer a series of new photos and it would begin to identify which photos were of cats.

Machine learning then continues to add to its teaching set. Every photo that it identifies — correctly or incorrectly — gets added to the teaching set, and the program effectively gets “smarter” and better at completing its task over time.

It is, in effect, learning.

1. Data Security

Malware is a huge — and growing — problem. In 2014, Kaspersky Lab said it had detected 325,000 new malware files every day. But, institutional intelligence company Deep Instinct says that each piece of new malware tends to have almost the same code as previous versions — only between 2 and 10% of the files change from iteration to iteration. Their learning model has no problem with the 2–10% variations, and can predict which files are malware with great accuracy. In other situations, machine learning algorithms can look for patterns in how data in the cloud is accessed, and report anomalies that could predict security breaches.

2. Personal Security

If you’ve flown on an airplane or attended a big public event lately, you almost certainly had to wait in long security screening lines. But machine learning is proving that it can be an asset to help eliminate false alarms and spot things human screeners might miss in security screenings at airports, stadiums, concerts, and other venues. That can speed up the process significantly and ensure safer events.

3. Financial Trading

Many people are eager to be able to predict what the stock markets will do on any given day — for obvious reasons. But machine learning algorithms are getting closer all the time. Many prestigious trading firms use proprietary systems to predict and execute trades at high speeds and high volume. Many of these rely on probabilities, but even a trade with a relatively low probability, at a high enough volume or speed, can turn huge profits for the firms. And humans can’t possibly compete with machines when it comes to consuming vast quantities of data or the speed with which they can execute a trade.


4. Healthcare

Machine learning algorithms can process more information and spot more patterns than their human counterparts. One study used computer assisted diagnosis (CAD) when to review the early mammography scans of women who later developed breast cancer, and the computer spotted 52% of the cancers as much as a year before the women were officially diagnosed. Additionally, machine learning can be used to understand risk factors for disease in large populations. The company Medecision developed an algorithm that was able to identify eight variables to predict avoidable hospitalizations in diabetes patients.

5. Marketing Personalization

The more you can understand about your customers, the better you can serve them, and the more you will sell. That’s the foundation behind marketing personalisation. Perhaps you’ve had the experience in which you visit an online store and look at a product but don’t buy it — and then see digital ads across the web for that exact product for days afterward. That kind of marketing personalization is just the tip of the iceberg. Companies can personalize which emails a customer receives, which direct mailings or coupons, which offers they see, which products show up as “recommended” and so on, all designed to lead the consumer more reliably towards a sale.

6. Fraud Detection

Machine learning is getting better and better at spotting potential cases of fraud across many different fields. PayPal, for example, is using machine learning to fight money laundering. The company has tools that compare millions of transactions and can precisely distinguish between legitimate and fraudulent transactions between buyers and sellers.

7. Recommendations

You’re probably familiar with this use if you use services like Amazon or Netflix. Intelligent machine learning algorithms analyze your activity and compare it to the millions of other users to determine what you might like to buy or binge watch next. These recommendations are getting smarter all the time, recognizing, for example, that you might purchase certain things as gifts (and not want the item yourself) or that there might be different family members who have different TV preferences.

8. Online Search

Perhaps the most famous use of machine learning, Google and its competitors are constantly improving what the search engine understands. Every time you execute a search on Google, the program watches how you respond to the results. If you click the top result and stay on that web page, we can assume you got the information you were looking for and the search was a success. If, on the other hand, you click to the second page of results, or type in a new search string without clicking any of the results, we can surmise that the search engine didn’t serve up the results you wanted — and the program can learn from that mistake to deliver a better result in the future.

9. Natural Language Processing (NLP)

NLP is being used in all sorts of exciting applications across disciplines. Machine learning algorithms with natural language can stand in for customer service agents and more quickly route customers to the information they need. It’s being used to translate obscure legalese in contracts into plain language and help attorneys sort through large volumes of information to prepare for a case.

10. Smart Cars

IBM recently surveyed top auto executives, and 74% expected that we would see smart cars on the road by 2025. A smart car would not only integrate into the Internet of Things, but also learn about its owner and its environment. It might adjust the internal settings — temperature, audio, seat position, etc. — automatically based on the driver, report and even fix problems itself, drive itself, and offer real time advice about traffic and road conditions.

---

Programming thought of the day:

• Maybe if we start telling people the brain is an app they will start using it.
• 
  

Java 8 - Intro

I hope I'm not too late to the party =).... Java 8 is the most awaited and is a major feature release of Java programming language.

Java 8 is a giant step forward for the Java language. Writing this tutorial has forced me to learn a lot more about it. In Project Lambda, Java gets a new closure syntax, method-references, and default methods on interfaces. It manages to add many of the features of functional languages without losing the clarity and simplicity Java developers have come to expect.

With this short tutorial, you should have a basic understanding of the new features and be ready to start using it. (well, that's the idea)

Implementing Runnable using Lambda expression

 

One of the first thing, I did with Java 8 was  trying to replace anonymous class with lambda expressions, and what could have been best example of anonymous class then implementing Runnable interface. Look at the code of implementing runnable prior to Java 8, it's taking four lines, but with lambda expressions, it's just taking one line. What we did here? the whole anonymous class is replaced by () -> {} code block.

//Old way
new Thread(new Runnable() 
{ 

@Override public void run() 
        { 
         System.out.println("Before Java8, too much code for too little to do"); 
 } 
}).start();

  
//Java 8 way: 
new Thread( () -> System.out.println("In Java8, Lambda expression rocks !!") ).start();

Event handling using Java 8 Lambda expressions
 
If you have ever done coding in Swing API, you will never forget writing event listener code. This is another classic use case of plain old Anonymous class, but no more. You can write better event listener code using lambda expressions as shown below.

// Before Java 8: 
JButton show = new JButton("Show"); 
show.addActionListener(new ActionListener() { 
 @Override public void actionPerformed(ActionEvent e) { 
  System.out.println("Event handling without lambda expression is boring"); 
 } 
}); 
  
// Java 8 way: 
show.addActionListener((e) -> { System.out.println("Light, Camera, Action !! Lambda expressions Rocks"); });

Iterating over List using Lambda expressions
 
If you are doing Java for few years, you know that most common operation with Collection classes are iterating over them and applying business logic on each elements, for example processing a list of orders, trades and events. Since Java is an imperative language, all code looping code written prior to Java 8 was sequential i.e. their is on simple way to do parallel processing of list items. If you want to do parallel filtering, you need to write your own code, which is not as easy as it looks. Introduction of lambda expression and default methods has separated what to do from how to do, which means now
Java Collection knows how to iterate, and they can now provide parallel processing of Collection elements at API level. In below example, I have shown you how to iterate over List using with and without lambda expressions, you can see that now List has a forEach() method, which can iterate through all objects and can apply whatever you ask using lambda code.

List<String> features = Arrays.asList("Lambdas", "Default Method", "Stream API", "Date and Time API");
  
//Old way, Prior Java 8 : 
for (String feature : features) 
{ 
 System.out.println(feature); 
} 
  
  
//In Java 8:   
features.forEach(n -> System.out.println(n)); 
 
  
// Even better use Method reference feature of Java 8 
// method reference is denoted by :: (double colon) operator 
// looks similar to score resolution operator of C++ 
features.forEach(System.out::println); 

Using Lambda expression and Functional interface Predicate

Apart from providing support for functional programming idioms at language level, Java 8 has also added a new package called java.util.function, which contains lot of classes to enable functional programming in Java. One of them is Predicate, By using java.util.function. Predicate functional interface and lambda expressions, you can provide logic to API methods to add lot of dynamic behaviour in less code. Following examples of Predicate in Java 8 shows lot of common ways to filter Collection data in Java code. Predicate interface is great for filtering.

public static void main(String[] args) {
List languages = Arrays.asList("Jamaica", "Peru", "Argentina", "France", "United States");

  
System.out.println("Countries which starts with J :");
filter(languages, (str)->((String) str).startsWith("J"));
  
  
System.out.println("Countries which ends with a "); 
filter(languages, (str)->((String) str).endsWith("a")); 
  
  
System.out.println("Print all Countries :"); 
filter(languages, (str)->true); 
  
  
System.out.println("Print no Countries : "); 
filter(languages, (str)->false); 
  
  
System.out.println("Print Countries whose length greater than 4:"); 
filter(languages, (str)->((String) str).length() > 4);
    

}
@SuppressWarnings("unchecked")
public static void filter(List names, Predicate condition) { 
 names.stream().filter((name) -> (condition.test(name))).forEach((name) -> { 
   System.out.println(name + " "); 
 }); 
}

the output for the above code is:

Countries which starts with J :
Jamaica 
-----------------------------------
Countries which ends with a 
Jamaica 
Argentina 
-----------------------------------
Print all Countries :
Jamaica 
Peru 
Argentina 
France 
United States 
-----------------------------------
Print no Countries : 
-----------------------------------
Print Countries whose length greater than 4:
Jamaica 
Argentina 
France 
United States 

If you want to see more Java 8 examples, download the complete project from my GitHub account:

https://github.com/rolando-febrero/Java8-Examples

---

Programming thought of the day:

• A clean house is the sign of a broken computer.

log4j with maven and Java - Project Example

log4j is a reliable, fast and flexible logging framework (APIs) written in Java, which is distributed under the Apache Software License. log4j is a popular logging package written in Java. log4j has been ported to the C, C++, C#, Perl, Python, Ruby, and Eiffel languages.

This utility will help you to keep track of everything your code does (and the government will keep track of everything you do in your life, just kidding -__-). Well, I'm going to show a short but solid example on how to use log4j. In this project we are going to log everything into a text file.

For this exercise we are going to use:

• Java 1.7
• Maven
• log4j
• Eclipse IDE

First of all, let's create a Maven project in Eclipse. (if you don't know how to install Maven in your PC, please go here)

Select option "Create a simple project"

Then fill all the fields according the bellow image. Here we are putting the info needed for our pom.xml

When done with these settings, open your pom.xml file and add the next lines:

        <properties>
  <jdk.version>1.7</jdk.version>
  <log4j.version>1.2.17</log4j.version>
 </properties>

 <dependencies>

  <dependency>
   <groupId>log4j</groupId>
   <artifactId>log4j</artifactId>
   <version>${log4j.version}</version>
  </dependency>

 </dependencies> 

Your pom.xml should look like this:

What we just did, was add some dependencies to our project, so with the help of mighty Maven, we are going to install them. Right click on pom.xml and select "Maven install"... Maven will pull all the jars needed (to your .M2 reposotory), in this case log4j

You should get a "BUILD SUCCESS" message in your console:

Log4j needs some properties file in which we specify its configuration. Let's create a "log4j.properties" file and added to our project:

Once created, double click on it, and add the following code:

# Root logger option
log4j.rootLogger=ERROR, stdout, file

log4j.rootLogger=INFO, stdout, file
# Redirect log messages to console
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

# Rirect log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=D:\\log4j-application.log
log4j.appender.file.MaxFileSize=5MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

Here we are specifying how and where to append the logs. Take a look at those lines and change accordingly.

Now we are going to proceed to write our code. Create a class and let's name it "HelloExample.java", here is the code:

package com.rolandoFebrero;

import org.apache.log4j.Logger;

public class HelloExample {
 
 final static Logger logger = Logger.getLogger(HelloExample.class);

 public static void main(String[] args) {
  
  HelloExample obj = new HelloExample();
  obj.runMe("Log4j Example");
  
 }
 
 private void runMe(String parameter){
  
  if(logger.isDebugEnabled()){
   logger.debug("This will be logged as DEBUG : " + parameter);
  }
  
  if(logger.isInfoEnabled()){
   logger.info("This will be logged as INFO : " + parameter);
  }
  
  logger.warn("This will be logged as WARN : " + parameter);
  logger.error("This will be logged as ERROR : " + parameter);
  logger.fatal("This will be logged as FATAL: " + parameter);
  
 }
 
}

now, create another class, this time let's name it "HelloExampleException.java". Here is the code:

package com.rolandoFebrero;

import org.apache.log4j.Logger;

public class HelloExampleException {

final static Logger logger = Logger.getLogger(HelloExampleException.class);
 
 public static void main(String[] args) {
 
  HelloExampleException obj = new HelloExampleException();
  
  try{
   obj.divide();
  }catch(ArithmeticException ex){
   logger.error("Something wrong!!!, you are getting an exception ", ex);
  }
  
  
 }
 
 private void divide(){
  
  int i = 3 / 0;

 } 
}

If you look at the code, in our first class, HelloExample.java, we are just printing a message, but with different log levels, such as debug, info, warn, error, fatal. This helps us to identify the kind of message we want to log. On the other file, HelloExampleExceptio.java, we are performing some math operation. In the main method we are making a call to "divide()". This divide() method just performs one single math operation, which is "3 / 0". Obviously, we can't divide a number by zero, so we will get an exception, in this specific case, an Arithmetic exception. I did that on purpose so we can see how an exception is logged by log4j.

Now, just right click on either HelloExample.java or HelloExampleExceptio.java; you might see the message in you console. Then go to where "log4j-application.log" file is located (hint: specified in log4j.properties) and there you'll find all the logs...

If you want to download the code for this project just go to my github account:

https://github.com/rolando-febrero/log4j-maven-Example.git

---

Programming thought of the day:

• If at first you don’t succeed; call it version 1.0.