2014-10-16 Coding in a train without internet
I'm in a train unable to access internet and I thought I could do something without it. But what I do is not an algorithm, it is a simple functionality which requires a function I use from time to time. It is Python, so I type help (function). I understand what it says but I'm unable to guess what value I need and there is no example. I know I could easily get that by using a search engine python function example. But I can't so I prefer to give up just because I know I could spend a lot of time guessing this particular value I don't know. I don't know how to program without internet anymore.
2014-06-03 pandas and openpyxl are not compatible anymore
I use a lot openpyxl with pandas. But the new version of openpyxl is not compatible anymore with pandas. So be careful to install an older version until this compatibility issue is fixed:
pip install openpyxl==1.8.6
2014-05-24 Graphs and Map / Reduce
Implementing graph algorithm on map/reduce is still a challenge. But maybe because it is a challenge, people keep thinking about it. It is the purpose of GraphX which reduces the processing time for the Page Rank algorithm on a Spark environment. It is not as fast as PowerGraph which is not based on Map/Reduce but it is must faster than Mahout (see paper GraphX: A Resilient Distributed Graph System on Spark). One fact, the language Scala appears more and more. A sign: Mahout recently took the decision to stop implementing machine learning algorithm against Map/Reduce to switch to Spark.
2014-05-02 Map / Reduce
I wish sometimes bugs could let me go. I'm working on an map/reduce algorithm. The problem would be smaller I would not hesitate to avoid map/reduce but it is not. Any fast it can be, I find it very slow and impossible to run. So I polish. I wake up in the morning and a detail strikes me up. Shit, shit, shit, I should have written it that way. It works on a small sample but this sample is resistant to many mistakes. I see this algorithm working in my head from the beginning. But I missed this case, as if a train would hide another one and get me killed when I cross. That kind of stuff always happens with big data. Pruning, scaling...
Most of the time, the issue comes from skewed data. Skewness is a way to say that the data is not uniformly distributed. Precisely, when reducing a table or joining two tables, we group data sharing the same key (as we do a GROUP BY or a JOIN on SQL). People research about it: A Study of Skew in MapReduce Applications. But let's see on a example what it means: notebook on Reduce skew data.
2014-04-16 Désactiver les logs de cvxopt
Des élèves m'ont posé cette question aujourd'hui : comment désactiver les logs que génère le module cvxopt. J'ai trouvé la réponse en cherchant cvxopt disable log sur un moteur de recherche cvxopt, algorithm parameters ou encore CVXOPT output suppression with MOSEK.
from cvxopt import solvers solvers.options['show_progress'] = False
2014-04-15 Systèmes de recommandations
C'est un des thèmes de recherche en vogue en ce moment. Les systèmes de recommandations. Une méthode courante est la factorisation de matrice. Une autre sont les Random Walks with Restart. A noter dans un coin pour de futurs sujets de projets informatiques.
2014-04-14 A reference for Text Processing with Map Reduce
While reading some papers, I found this reference: Data-Intensive Text Processing with MapReduce. It is a good introduction for people who never used Map Reduce.
2014-04-13 Latex formulas with Sphinx
Is it worth writing about something which took me 15 minutes to find out and others 15 to try? And it will be another 10 to write it down... I wonder! Anyway, I was looking for a way to add formulas in my python documentation.
more...
2014-01-18 Framework for Mobile Application
What framework to choose if you want to build a mobile application which will save you some time while deploying on multiple phones? (Android, Black Berry, iPhone, Windows Phone). I'm not sure there is a perfect answer and it will probably change in a couple of months. However, this what I found. I was looking for answers. Which phones does it support today? Where can you develop your application? Which language? How to build? How to deploy? Cost? I limited my study to the following frameworks (according to a friend, the others are not as mature):
- Apache Cordova (or PhoneGaps): This framework provides a common API in javascript/HTML to access the API of each phone. The goal is to build application for many phone based on the same code. It produces a web application.
- Sencha Touch: It contains more template javascript/HTML templates than Cordova. The free framework does not seem to do more than building a web application with javascript and HTML. However, you can use both Cordova and Sencha Touch (see below).
- Titanium Mobile: It is also a framework using javascript/HTML. The framework can access the native API of each phone (as opposed to Cordova). You might end up with a different version on each phone. The framework should give more freedom than the first one.
- Xamarin (+ MvvmCross): It proposes a way to build phone application written in C# on every phone (using Mono for iOS and Android). It gives an access the native API. The framework also includes the possibility to design hosted on a remote machine which can communicate with the phone application.
- kivy A framework in Python (2.7, migration to Python 3 should happen some day). It was not meant to develop phone application but it now proposes a way to make it work on Android, with pyjnius, and iOS, with pyobjus and kivy-ios. This two libraries connect Python to java which accesses the phone functionalities. There is no support for Windows Phone. A last module plyer gives access to some extra functionalities (GPS, text to speech).
more...
2013-12-29 Big or small modules in Python, pyrsslocal
It is not always easy to determine a good size for a module or a set of modules. Should it be kept as a single bloc which comes alone or a set of smaller modules? It is easier to understand a smaller modules. It usually limits itself to a small set of similar functionalities. However, extending a small module is faster than starting a new one, it avoids spending too much time setting up the installation steps, documentation automation, dependencies with others modules. Small modules grow, especially if you work late on them on your free time.
pypi makes it simple to install a module and its dependencies, GitHub helps to fork some existing module which needs to be fixed for a specific usage. That's why I started to split my own toolbox. Does it worth the effort knowing I'm the only one to use it? Probably not but, for my teachings, I think I will not be reluctant anymore to create a new small project for a specific purpose.
I created or updated three modules:
- pyquickhelper: functions quickly written to help generating sphinx documentation
- pyensae: functions used in pratical lessons
- pyrsslocal: I was using Google Reader and I was reluctant to use another service, wondering if it would disappear. It could be improved but it is already usable.
2013-09-21 Le profile online d'un développeur
Beaucoup de sociétés recherchent de bons développeurs et les dernières années ont changé les façons de les trouver. Il y a d'abord les réseaux sociaux Linkedin, Viadeo, Facebook. Mais ce n'est pas tout, les bons développeurs ont souvent partagé leur code ou contribué à des projets open source. On retrouve leurs traces sur GitHub. Certains ont aussi tenté de résoudre certains des problèmes proposés par kaggle. Ajoutons, un site web, un petit blog pour commenter tous les défis techniques auquel un développeur s'attaque.
Il y a dix ans, le CV et la lettre de motivation était la façon de se vendre. Le CV est aujourd'hui sur Linkedin ou Viadeo, la motivation dans le nombre de commit sur GitHub.
2013-08-12 Exceptions in Python
when raising an exception, we might be tempted by using the type Exception. But this type also gives information on the error. Python has a list of standard exceptions with a meaningful name: standard exceptions. They should be used.
2013-08-11 pyensae
Exchanging code with students through emails is easy but after a couple of years, looking into my database of emails became difficult. Plus, it is difficult to maintain pieces of code inside emails.
pyensae is the module I will use from now on to exchange code for my teachings. So far, it only contains a function to download materials I put on my website. It should grow over the next years. The module can be obtained:
Having students contributing will be another story. Let's see after a year.2013-08-01 Datamining avec R ou Python
Il est devenu assez facile de travailler avec R. Même si je trouve toujours le langage compliqué, il faut avouer que des outils comme RStudio rendent l'outil indispensable et de moins en moins difficile d'accès. On peut importer ses données depuis une grande variété de format, créer des rapports en utilisant knitr ou encore créer une application web avec shiny. Il existe un package pour presque tout comme pour faire des tests unitaires avec RUnit.
J'ai pour habitude de faire le plus de choses possibles en Python, de passer au C++ quand c'est vraiment nécessaire et d'utiliser R pour tout ce qui statistiques. R a l'avantage d'avoir une modèle suivi par tous les packages alors que les modules Python suivent leur propre logique le plus souvent.
Un exemple sur les méthodes à noyau : sous R, il faudrait utiliser le package kernlab, sous Python, on se dirigerait vers scikit-learn. Les packages sous Python sont souvent plus gros car les auteurs auront voulu intégrer de nombreux algorithmes selon la même logique. Sous R, chaque package a un périmètre plus réduit et suit les mêmes conventions. Comme Python s'interface avec à peu près tout y compris R, je préfère ce langage dont la syntaxe me paraît plus facile même si je n'hésite pas à repasser à R dès qu'il le faut.
C++
Les deux langages permettent d'inclure des morceau de C++ si nécessaire. Python dispose de diverses façons~: Cython, Boost.Python, ou cffi. Avec R, il y a Rcpp (tutoriel sur Rcpp, il faut aussi installer devtools). Est-ce que le langage C va disparaître ? Pour le moment, si les langages de scripts sont beaucoup plus utilisés, les compilateurs sont la plupart du temps écrits en C++. De nombreuses extensions ont été écrites en C++ (voir Eigen par exemple). Ce langage est souvent un choix raisonnable lorsqu'on veut écrire du code très rapide.
Graphiques
L'outil qui prend un peu plus d'importance ces temps-ci est d3.js car il permet de réaliser des graphiques facilement intégrables sur un site web, un journal en ligne, qu'il permet de faire plus facilement des animations et surtout, comme il est écrit en javascript, il permet de créer des graphes qui réagissent à la souris ou au clavier. Je n'arrive pas encore à trouver un module qui le rende facilement utilisable depuis Python. L'article Method: Data visualization with D3.js and python - part 1 décrit comment s'y prendre sans toutefois citer de module dédié à cet usage. Ca ne devrait pas tarder. Sous R, sans être javascript, il existe ggplot2 qui propose des graphes assez jolis.
Big Data
Enfin, dans le domaine des Big Data, les solutions lourdes type Map/Reduce Hadoop sont de plus en plus fréquentes lorsqu'on traite des données qui arrivent en continu (internet, téléphone, consommation d'énergie...). Pour le reste, un ordinateur portable avec quelques giga octets de mémoire et un disque dur conséquent suffit. Des outils existent pour ne pas avoir à charger une base complète en mémoire. C'est le cas de ff et ffbase (ou ça bigvis) sous R. Sous Python, l'environnement de travail le plus approprié est IPython. Pour manipuler les données, on trouve pandas ou PyTables.
2013-07-30 IExplorer and class HTTPServer (Python)
I was wondering why I could not make something work on IExplorer while it was perfectly working with Firefox and Chrome. My simple Python web server seemed to get stuck most of the time by IExplorer (no logs) when I could see all the requests sent by Firefox. When I was leaving IExplorer, the error message was "interrupted connection". I finally found the error: IExplorer sends mulitple requests in parallel and the class HTTPServer does not handle it. To fix that, you need to do the following:
from http.server import BaseHTTPRequestHandler, HTTPServer, SimpleHTTPRequestHandler
from socketserver import ThreadingMixIn
class ThreadedServer (ThreadingMixIn, HTTPServer) :
pass
class MyHandler(BaseHTTPRequestHandler):
def respond(self, status_code):
self.send_response(status_code)
self.end_headers()
def do_GET(self):
# ...
server = ThreadedServer(('localhost', 8080), MyHandler)
server.serve_forever()
| --> |
Xavier Dupré
|
