.. _TD2AEcoWebScrapingcorrigerst:

===============================
2A.eco - Web-Scraping - pokemon
===============================


.. only:: html

    **Links:** :download:`notebook <TD2A_Eco_Web_Scraping_corrige.ipynb>`, :downloadlink:`html <TD2A_Eco_Web_Scraping_corrige2html.html>`, :download:`python <TD2A_Eco_Web_Scraping_corrige.py>`, :downloadlink:`slides <TD2A_Eco_Web_Scraping_corrige.slides.html>`, :githublink:`GitHub|_doc/notebooks/td2a_eco/TD2A_Eco_Web_Scraping_corrige.ipynb|*`


Il faut récupérer automatiquement des images de pokémon depuis le site
`pokemondb.net <https://pokemondb.net/pokedex/national>`__.

Pour cet exercice, nous vous demandons d’obtenir 1) les informations
personnelles des 721 pokemons sur le site internet
`pokemondb.net <http://pokemondb.net/pokedex/national>`__. Les
informations que nous aimerions obtenir au final pour les pokemons sont
celles contenues dans 4 tableaux :


-  Pokédex data
-  Training
-  Breeding
-  Base stats

Pour exemple : `Pokemon
Database <http://pokemondb.net/pokedex/nincada>`__.

2) Nous aimerions que vous récupériez également les images de chacun des
   pokémons et que vous les enregistriez dans un dossier (indice :
   utilisez les modules request et
   `shutil <https://docs.python.org/3/library/shutil.html>`__) *pour
   cette question ci, il faut que vous cherchiez de vous même certains
   éléments, tout n’est pas présent dans le TD*.

Correction
----------

.. code:: ipython3

    import urllib
    import bs4
    import collections
    import pandas as pd
    
    # pour le site que nous utilisons, le user agent de python 3 n'est pas  bien passé :
    # on le change donc pour celui de Mozilla
    
    req = urllib.request.Request('http://pokemondb.net/pokedex/national', 
                                 headers={'User-Agent': 'Mozilla/5.0'})
    html = urllib.request.urlopen(req).read()
    page = bs4.BeautifulSoup(html, "lxml")
    
    # récupérer la liste des noms de pokémon
    
    liste_pokemon =[]
    for pokemon in page.findAll('span', {'class': 'infocard-lg-img'}) : 
        pokemon = pokemon.find('a').get('href').replace("/pokedex/",'')
        liste_pokemon.append(pokemon)

Fonction pour obtenir les caractéristiques de pokemons
------------------------------------------------------

.. code:: ipython3

    def get_page(pokemon_name):
        url_pokemon = 'http://pokemondb.net/pokedex/'+ pokemon_name
        req = urllib.request.Request(url_pokemon, headers = {'User-Agent' : 'Mozilla/5.0'})
        html = urllib.request.urlopen(req).read()
        return bs4.BeautifulSoup(html, "lxml")
         
        
    def get_cara_pokemon(pokemon_name):
        page = get_page(pokemon_name)
        data = collections.defaultdict()
        
        # table Pokédex data, Training, Breeding, base Stats
        
        for table in page.findAll('table', { 'class' : "vitals-table"})[0:4] :
            table_body = table.find('tbody')
            for rows in table_body.findChildren(['tr']) : 
                if len(rows) > 1 : # attention aux tr qui ne contiennent rien
                    column = rows.findChild('th').getText()
                    cells = rows.findChild('td').getText()
                    cells = cells.replace('\t','').replace('\n',' ')
                    data[column] = cells
                    data['name'] = pokemon_name
        return dict(data)
           
    items = []       
    for e, pokemon in enumerate(liste_pokemon) : 
        print(e, pokemon)
        item = get_cara_pokemon(pokemon)       
        items.append(item)
        if e > 20:
            break
    df = pd.DataFrame(items)
    df.head()


.. parsed-literal::
    0 bulbasaur
    1 ivysaur
    2 venusaur
    3 charmander
    4 charmeleon
    5 charizard
    6 squirtle
    7 wartortle
    8 blastoise
    9 caterpie
    10 metapod
    11 butterfree
    12 weedle
    13 kakuna
    14 beedrill
    15 pidgey
    16 pidgeotto
    17 pidgeot
    18 rattata
    19 raticate
    20 spearow
    21 fearow






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>National №</th>
          <th>name</th>
          <th>Type</th>
          <th>Species</th>
          <th>Height</th>
          <th>Weight</th>
          <th>Abilities</th>
          <th>Local №</th>
          <th>EV yield</th>
          <th>Catch rate</th>
          <th>...</th>
          <th>Growth Rate</th>
          <th>Egg Groups</th>
          <th>Gender</th>
          <th>Egg cycles</th>
          <th>HP</th>
          <th>Attack</th>
          <th>Defense</th>
          <th>Sp. Atk</th>
          <th>Sp. Def</th>
          <th>Speed</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>001</td>
          <td>bulbasaur</td>
          <td>Grass Poison</td>
          <td>Seed Pokémon</td>
          <td>0.7 m (2′04″)</td>
          <td>6.9 kg (15.2 lbs)</td>
          <td>1. OvergrowChlorophyll (hidden ability)</td>
          <td>001 (Red/Blue/Yellow)226 (Gold/Silver/Crystal)...</td>
          <td>1 Special Attack</td>
          <td>45 (5.9% with PokéBall, full HP)</td>
          <td>...</td>
          <td>Medium Slow</td>
          <td>Grass, Monster</td>
          <td>87.5% male, 12.5% female</td>
          <td>20 (4,884–5,140 steps)</td>
          <td>45</td>
          <td>49</td>
          <td>49</td>
          <td>65</td>
          <td>65</td>
          <td>45</td>
        </tr>
        <tr>
          <th>1</th>
          <td>002</td>
          <td>ivysaur</td>
          <td>Grass Poison</td>
          <td>Seed Pokémon</td>
          <td>1.0 m (3′03″)</td>
          <td>13.0 kg (28.7 lbs)</td>
          <td>1. OvergrowChlorophyll (hidden ability)</td>
          <td>002 (Red/Blue/Yellow)227 (Gold/Silver/Crystal)...</td>
          <td>1 Special Attack, 1 Special Defense</td>
          <td>45 (5.9% with PokéBall, full HP)</td>
          <td>...</td>
          <td>Medium Slow</td>
          <td>Grass, Monster</td>
          <td>87.5% male, 12.5% female</td>
          <td>20 (4,884–5,140 steps)</td>
          <td>60</td>
          <td>62</td>
          <td>63</td>
          <td>80</td>
          <td>80</td>
          <td>60</td>
        </tr>
        <tr>
          <th>2</th>
          <td>003</td>
          <td>venusaur</td>
          <td>Grass Poison</td>
          <td>Seed Pokémon</td>
          <td>2.0 m (6′07″)</td>
          <td>100.0 kg (220.5 lbs)</td>
          <td>1. OvergrowChlorophyll (hidden ability)</td>
          <td>003 (Red/Blue/Yellow)228 (Gold/Silver/Crystal)...</td>
          <td>2 Special Attack, 1 Special Defense</td>
          <td>45 (5.9% with PokéBall, full HP)</td>
          <td>...</td>
          <td>Medium Slow</td>
          <td>Grass, Monster</td>
          <td>87.5% male, 12.5% female</td>
          <td>20 (4,884–5,140 steps)</td>
          <td>80</td>
          <td>82</td>
          <td>83</td>
          <td>100</td>
          <td>100</td>
          <td>80</td>
        </tr>
        <tr>
          <th>3</th>
          <td>004</td>
          <td>charmander</td>
          <td>Fire</td>
          <td>Lizard Pokémon</td>
          <td>0.6 m (2′00″)</td>
          <td>8.5 kg (18.7 lbs)</td>
          <td>1. BlazeSolar Power (hidden ability)</td>
          <td>004 (Red/Blue/Yellow)229 (Gold/Silver/Crystal)...</td>
          <td>1 Speed</td>
          <td>45 (5.9% with PokéBall, full HP)</td>
          <td>...</td>
          <td>Medium Slow</td>
          <td>Dragon, Monster</td>
          <td>87.5% male, 12.5% female</td>
          <td>20 (4,884–5,140 steps)</td>
          <td>39</td>
          <td>52</td>
          <td>43</td>
          <td>60</td>
          <td>50</td>
          <td>65</td>
        </tr>
        <tr>
          <th>4</th>
          <td>005</td>
          <td>charmeleon</td>
          <td>Fire</td>
          <td>Flame Pokémon</td>
          <td>1.1 m (3′07″)</td>
          <td>19.0 kg (41.9 lbs)</td>
          <td>1. BlazeSolar Power (hidden ability)</td>
          <td>005 (Red/Blue/Yellow)230 (Gold/Silver/Crystal)...</td>
          <td>1 Special Attack, 1 Speed</td>
          <td>45 (5.9% with PokéBall, full HP)</td>
          <td>...</td>
          <td>Medium Slow</td>
          <td>Dragon, Monster</td>
          <td>87.5% male, 12.5% female</td>
          <td>20 (4,884–5,140 steps)</td>
          <td>58</td>
          <td>64</td>
          <td>58</td>
          <td>80</td>
          <td>65</td>
          <td>80</td>
        </tr>
      </tbody>
    </table>
    <p>5 rows × 22 columns</p>
    </div>



les images de pokemon
---------------------

.. code:: ipython3

    import shutil
    import requests
    
    
    for e, pokemon in enumerate(liste_pokemon) : 
        print(e,pokemon)
        url = "https://img.pokemondb.net/artwork/{}.jpg".format(pokemon)
        response = requests.get(url, stream=True)
        # avec l'option stream, on ne télécharge pas l'objet de l'url
        with open('{}.jpg'.format(pokemon), 'wb') as out_file:
            shutil.copyfileobj(response.raw, out_file)
        if e > 20:
            break        


.. parsed-literal::
    0 bulbasaur
    1 ivysaur
    2 venusaur
    3 charmander
    4 charmeleon
    5 charizard
    6 squirtle
    7 wartortle
    8 blastoise
    9 caterpie
    10 metapod
    11 butterfree
    12 weedle
    13 kakuna
    14 beedrill
    15 pidgey
    16 pidgeotto
    17 pidgeot
    18 rattata
    19 raticate
    20 spearow
    21 fearow


.. code:: ipython3

    import os
    names = [name for name in os.listdir('.') if '.jpg' in name]
    names[:3]




.. parsed-literal::
    ['beedrill.jpg', 'blastoise.jpg', 'bulbasaur.jpg']



.. code:: ipython3

    import matplotlib.pyplot as plt
    import skimage.io as imio
    
    fig, ax = plt.subplots(1, 3, figsize=(12,4))
    for i, name in enumerate(names[:ax.shape[0]]):
        img = imio.imread(name)
        ax[i].imshow(img)
        ax[i].get_xaxis().set_visible(False)
        ax[i].get_yaxis().set_visible(False)



.. image:: TD2A_Eco_Web_Scraping_corrige_9_0.png