.. _simpleprocessingrst:

======================
Prétraitement des logs
======================


.. only:: html

    **Links:** :download:`notebook <simple_processing.ipynb>`, :downloadlink:`html <simple_processing2html.html>`, :download:`PDF <simple_processing.pdf>`, :download:`python <simple_processing.py>`, :downloadlink:`slides <simple_processing.slides.html>`, :githublink:`GitHub|_doc/notebooks/simple_processing.ipynb|*`


Traitements de logs générés par un QCM.

.. code:: ipython3

    from jyquickhelper import add_notebook_menu
    add_notebook_menu()






.. contents::
    :local:





.. code:: ipython3

    %matplotlib inline

Observations brutes
-------------------

.. code:: ipython3

    import os
    names = [os.path.join("logs", _) for _ in os.listdir("logs") if '.log' in _]
    names = names[-1:]
    names




.. parsed-literal::
    ['logs\\QCMApp.log']



.. code:: ipython3

    with open(names[0], 'r', encoding="utf-8") as f:
        lines = f.readlines()
    lines[5:7]




.. parsed-literal::
    ['2018-12-12 17:56:29,989,INFO,[DATA],{"msg":"qcm","session":{"alias":"xavierd"},"client":["167.220.197.38",6274],"game":"simple_french_qcm","qn":"0"}\n',
     '2018-12-12 17:56:33,130,INFO,[DATA],{"msg":"event","session":{"alias":"xavierd"},"client":["167.220.197.38",6274],"events":["focus:true,game:simple_french_qcm,qn:0"]}\n']



Let’s retrieve the expected answers from the QCM.

.. code:: ipython3

    from mathenjeu.tests import simple_french_qcm
    qcm = simple_french_qcm()
    expected = qcm.expected_answers()
    expected




.. parsed-literal::
    [['simple_french_qcm-0-a0'],
     ['simple_french_qcm-1-a2'],
     ['simple_french_qcm-2-a2'],
     ['simple_french_qcm-3-a3'],
     ['simple_french_qcm-4-a2'],
     ['simple_french_qcm-5-a0', 'simple_french_qcm-5-a2'],
     ['simple_french_qcm-6-a5'],
     ['simple_french_qcm-7-a2'],
     ['simple_french_qcm-8-Thalès']]



.. code:: ipython3

    import pandas
    from mathenjeu.datalog import enumerate_qcmlogdf
    dfs = pandas.concat(list(enumerate_qcmlogdf(names, expected)))
    dfs.T






.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th>person_id</th>
          <th>c241c15008614ea67480</th>
          <th>32606f02fa0df6aac111</th>
          <th>5bd96adafa694996027d</th>
          <th>753a507a9205e4c196fa</th>
          <th>8a8c40ad28eb1206efd5</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>simple_french_qcm-0-a0</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-0-b</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-0-duration</th>
          <td>0 days 00:00:04.266000</td>
          <td>NaT</td>
          <td>0 days 00:00:06.904000</td>
          <td>0 days 00:00:16.530000</td>
          <td>0 days 00:00:02.920000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-0-good</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-0-nbvisit</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-a0</th>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-a1</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-a2</th>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-b</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-duration</th>
          <td>1 days 00:00:00</td>
          <td>NaT</td>
          <td>0 days 00:00:10.153000</td>
          <td>0 days 00:00:11.740000</td>
          <td>0 days 00:00:02.247000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-good</th>
          <td>1</td>
          <td>NaN</td>
          <td>0</td>
          <td>1</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-5-nbvisit</th>
          <td>0.5</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>2</td>
        </tr>
        <tr>
          <th>simple_french_qcm-8-ANS</th>
          <td>thalès</td>
          <td>NaN</td>
          <td>Astérix et Cléopâtre</td>
          <td>Prout</td>
          <td></td>
        </tr>
        <tr>
          <th>simple_french_qcm-8-b</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-8-duration</th>
          <td>0 days 00:00:04.390000</td>
          <td>NaT</td>
          <td>0 days 00:03:02.506000</td>
          <td>0 days 00:00:14.483000</td>
          <td>0 days 00:00:01.422000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-8-good</th>
          <td>0</td>
          <td>NaN</td>
          <td>0</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-8-nbvisit</th>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-a2</th>
          <td>NaN</td>
          <td>1</td>
          <td>2</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1001</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-duration</th>
          <td>NaT</td>
          <td>0 days 00:00:03.343000</td>
          <td>0</td>
          <td>0 days 00:00:14.010000</td>
          <td>NaT</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-good</th>
          <td>NaN</td>
          <td>1</td>
          <td>2</td>
          <td>0</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-nbvisit</th>
          <td>NaN</td>
          <td>1</td>
          <td>3</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-2-a2</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-2-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-2-duration</th>
          <td>NaT</td>
          <td>0 days 00:00:06.750000</td>
          <td>0 days 00:00:31.978000</td>
          <td>0 days 00:00:28.765000</td>
          <td>NaT</td>
        </tr>
        <tr>
          <th>simple_french_qcm-2-good</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-2-nbvisit</th>
          <td>NaN</td>
          <td>1</td>
          <td>2</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-a2</th>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-a3</th>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>1000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-duration</th>
          <td>NaT</td>
          <td>1 days 00:00:00</td>
          <td>0 days 00:00:19.246000</td>
          <td>0 days 00:00:19.492000</td>
          <td>0 days 00:00:09.323000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-good</th>
          <td>NaN</td>
          <td>1</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-nbvisit</th>
          <td>NaN</td>
          <td>0.5</td>
          <td>1</td>
          <td>1</td>
          <td>3</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-a2</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>2</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-duration</th>
          <td>NaT</td>
          <td>0 days 00:00:09.688000</td>
          <td>0 days 00:00:21.230000</td>
          <td>0 days 00:03:19.593000</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-good</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-nbvisit</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>5</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-a3</th>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-duration</th>
          <td>NaT</td>
          <td>1 days 00:00:00</td>
          <td>0 days 00:00:20.314000</td>
          <td>0 days 00:00:21.868000</td>
          <td>0 days 00:00:00.750000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-good</th>
          <td>NaN</td>
          <td>0</td>
          <td>0</td>
          <td>1</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-nbvisit</th>
          <td>NaN</td>
          <td>0.5</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-a2</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-b</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-duration</th>
          <td>NaT</td>
          <td>0 days 00:00:06.390000</td>
          <td>0 days 00:00:17.141000</td>
          <td>0 days 00:00:20.923000</td>
          <td>0 days 00:00:01.188000</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-good</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>0</td>
          <td>0</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-nbvisit</th>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-3-a0</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-a2</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-1-a1</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-a0</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-6-a5</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-7-a0</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>-a1</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>on-a2</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
        </tr>
        <tr>
          <th>simple_french_qcm-0-a1</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
        </tr>
        <tr>
          <th>simple_french_qcm-4-a3</th>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>NaN</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>