{"cells": [{"cell_type": "markdown", "id": "4b855db5", "metadata": {}, "source": ["# NeuralTreeNet et ONNX\n", "\n", "La conversion d'un arbre de d\u00e9cision au format ONNX peut cr\u00e9er des diff\u00e9rences entre le mod\u00e8le original et le mod\u00e8le converti (voir [Issues when switching to float](http://www.xavierdupre.fr/app/onnxcustom/helpsphinx/gyexamples/plot_ebegin_float_double.html). Le probl\u00e8me vient d'un changement de type, les seuils de d\u00e9cisions sont arrondis au float32 le plus proche de leur valeur en float64 (double). Qu'advient-il si l'arbre de d\u00e9cision est converti en r\u00e9seau de neurones d'abord.\n", "\n", "L'approximation des seuils de d\u00e9cision ne change pas grand chose dans la majorit\u00e9 des cas. Cependant, il est possible que la comparaison d'une variable \u00e0 un seuil de d\u00e9cision arrondi soit l'oppos\u00e9 de celle avec le seuil non arrondi. Dans ce cas, la d\u00e9cision suit un chemin diff\u00e9rent dans l'arbre."]}, {"cell_type": "code", "execution_count": 1, "id": "636a122a", "metadata": {}, "outputs": [{"data": {"text/html": ["<div id=\"my_id_menu_nb\">run previous cell, wait for 2 seconds</div>\n", "<script>\n", "function repeat_indent_string(n){\n", "    var a = \"\" ;\n", "    for ( ; n > 0 ; --n)\n", "        a += \"    \";\n", "    return a;\n", "}\n", "// look up into all sections and builds an automated menu //\n", "var update_menu_string = function(begin, lfirst, llast, sformat, send, keep_item, begin_format, end_format) {\n", "    var anchors = document.getElementsByClassName(\"section\");\n", "    if (anchors.length == 0) {\n", "        anchors = document.getElementsByClassName(\"text_cell_render rendered_html\");\n", "    }\n", "    var i,t;\n", "    var text_menu = begin;\n", "    var text_memo = \"<pre>\\nlength:\" + anchors.length + \"\\n\";\n", "    var ind = \"\";\n", "    var memo_level = 1;\n", "    var href;\n", "    var tags = [];\n", "    var main_item = 0;\n", "    var format_open = 0;\n", "    for (i = 0; i <= llast; i++)\n", "        tags.push(\"h\" + i);\n", "\n", "    for (i = 0; i < anchors.length; i++) {\n", "        text_memo += \"**\" + anchors[i].id + \"--\\n\";\n", "\n", "        var child = null;\n", "        for(t = 0; t < tags.length; t++) {\n", "            var r = anchors[i].getElementsByTagName(tags[t]);\n", "            if (r.length > 0) {\n", "child = r[0];\n", "break;\n", "            }\n", "        }\n", "        if (child == null) {\n", "            text_memo += \"null\\n\";\n", "            continue;\n", "        }\n", "        if (anchors[i].hasAttribute(\"id\")) {\n", "            // when converted in RST\n", "            href = anchors[i].id;\n", "            text_memo += \"#1-\" + href;\n", "            // passer \u00e0 child suivant (le chercher)\n", "        }\n", "        else if (child.hasAttribute(\"id\")) {\n", "            // in a notebook\n", "            href = child.id;\n", "            text_memo += \"#2-\" + href;\n", "        }\n", "        else {\n", "            text_memo += \"#3-\" + \"*\" + \"\\n\";\n", "            continue;\n", "        }\n", "        var title = child.textContent;\n", "        var level = parseInt(child.tagName.substring(1,2));\n", "\n", "        text_memo += \"--\" + level + \"?\" + lfirst + \"--\" + title + \"\\n\";\n", "\n", "        if ((level < lfirst) || (level > llast)) {\n", "            continue ;\n", "        }\n", "        if (title.endsWith('\u00b6')) {\n", "            title = title.substring(0,title.length-1).replace(\"<\", \"&lt;\")\n", "         .replace(\">\", \"&gt;\").replace(\"&\", \"&amp;\");\n", "        }\n", "        if (title.length == 0) {\n", "            continue;\n", "        }\n", "\n", "        while (level < memo_level) {\n", "            text_menu += end_format + \"</ul>\\n\";\n", "            format_open -= 1;\n", "            memo_level -= 1;\n", "        }\n", "        if (level == lfirst) {\n", "            main_item += 1;\n", "        }\n", "        if (keep_item != -1 && main_item != keep_item + 1) {\n", "            // alert(main_item + \" - \" + level + \" - \" + keep_item);\n", "            continue;\n", "        }\n", "        while (level > memo_level) {\n", "            text_menu += \"<ul>\\n\";\n", "            memo_level += 1;\n", "        }\n", "        text_menu += repeat_indent_string(level-2);\n", "        text_menu += begin_format + sformat.replace(\"__HREF__\", href).replace(\"__TITLE__\", title);\n", "        format_open += 1;\n", "    }\n", "    while (1 < memo_level) {\n", "        text_menu += end_format + \"</ul>\\n\";\n", "        memo_level -= 1;\n", "        format_open -= 1;\n", "    }\n", "    text_menu += send;\n", "    //text_menu += \"\\n\" + text_memo;\n", "\n", "    while (format_open > 0) {\n", "        text_menu += end_format;\n", "        format_open -= 1;\n", "    }\n", "    return text_menu;\n", "};\n", "var update_menu = function() {\n", "    var sbegin = \"\";\n", "    var sformat = '<a href=\"#__HREF__\">__TITLE__</a>';\n", "    var send = \"\";\n", "    var begin_format = '<li>';\n", "    var end_format = '</li>';\n", "    var keep_item = -1;\n", "    var text_menu = update_menu_string(sbegin, 2, 4, sformat, send, keep_item,\n", "       begin_format, end_format);\n", "    var menu = document.getElementById(\"my_id_menu_nb\");\n", "    menu.innerHTML=text_menu;\n", "};\n", "window.setTimeout(update_menu,2000);\n", "            </script>"], "text/plain": ["<IPython.core.display.HTML object>"]}, "execution_count": 2, "metadata": {}, "output_type": "execute_result"}], "source": ["from jyquickhelper import add_notebook_menu\n", "add_notebook_menu()"]}, {"cell_type": "code", "execution_count": 2, "id": "2f698cc0", "metadata": {}, "outputs": [], "source": ["%matplotlib inline"]}, {"cell_type": "code", "execution_count": 3, "id": "ad53d7c6", "metadata": {}, "outputs": [], "source": ["%load_ext mlprodict"]}, {"cell_type": "markdown", "id": "c7b2fb41", "metadata": {}, "source": ["## Jeu de donn\u00e9es\n", "\n", "On construit un jeu de donn\u00e9e al\u00e9atoire."]}, {"cell_type": "code", "execution_count": 4, "id": "a8feffa5", "metadata": {}, "outputs": [], "source": ["import numpy\n", "\n", "X = numpy.random.randn(10000, 10)\n", "y = X.sum(axis=1) / X.shape[1]\n", "X = X.astype(numpy.float64)\n", "y = y.astype(numpy.float64)"]}, {"cell_type": "code", "execution_count": 5, "id": "3c854905", "metadata": {}, "outputs": [], "source": ["middle = X.shape[0] // 2\n", "X_train, X_test = X[:middle], X[middle:]\n", "y_train, y_test = y[:middle], y[middle:]"]}, {"cell_type": "markdown", "id": "2972ef7f", "metadata": {}, "source": ["## Partie scikit-learn"]}, {"cell_type": "markdown", "id": "2a19a0c1", "metadata": {}, "source": ["### Caler un arbre de d\u00e9cision"]}, {"cell_type": "code", "execution_count": 6, "id": "bfc49123", "metadata": {}, "outputs": [{"data": {"text/plain": ["(0.6179766027481131, 0.33709933420465643)"]}, "execution_count": 7, "metadata": {}, "output_type": "execute_result"}], "source": ["from sklearn.tree import DecisionTreeRegressor\n", "\n", "tree = DecisionTreeRegressor(max_depth=7)\n", "tree.fit(X_train, y_train)\n", "tree.score(X_train, y_train), tree.score(X_test, y_test)"]}, {"cell_type": "code", "execution_count": 7, "id": "a38b0426", "metadata": {}, "outputs": [{"data": {"text/plain": ["0.33709933420465643"]}, "execution_count": 8, "metadata": {}, "output_type": "execute_result"}], "source": ["from sklearn.metrics import r2_score\n", "r2_score(y_test, tree.predict(X_test))"]}, {"cell_type": "markdown", "id": "86a0f0a3", "metadata": {}, "source": ["La profondeur de l'arbre est insuffisante mais ce n'est pas ce qui nous int\u00e9resse ici."]}, {"cell_type": "markdown", "id": "8e6038ff", "metadata": {}, "source": ["### Conversion au format ONNX"]}, {"cell_type": "code", "execution_count": 8, "id": "f6849a2d", "metadata": {}, "outputs": [], "source": ["from mlprodict.onnx_conv import to_onnx\n", "\n", "onx = to_onnx(tree, X[:1].astype(numpy.float32))"]}, {"cell_type": "code", "execution_count": 9, "id": "3daf9db1", "metadata": {}, "outputs": [{"data": {"text/plain": ["1.7421041873949668"]}, "execution_count": 10, "metadata": {}, "output_type": "execute_result"}], "source": ["from mlprodict.onnxrt import OnnxInference\n", "\n", "x_exp = X_test\n", "\n", "oinf = OnnxInference(onx, runtime='onnxruntime1')\n", "expected = tree.predict(x_exp)\n", "\n", "got = oinf.run({'X': x_exp.astype(numpy.float32)})['variable']\n", "numpy.abs(got - expected).max()"]}, {"cell_type": "code", "execution_count": 10, "id": "7ce247da", "metadata": {"scrolled": false}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["opset: domain='ai.onnx.ml' version=1\n", "opset: domain='' version=15\n", "input: name='X' type=dtype('float32') shape=[None, 10]\n", "TreeEnsembleRegressor(X, n_targets=1, nodes_falsenodeids=253:[128,65,34...252,0,0], nodes_featureids=253:[8,3,9...2,0,0], nodes_hitrates=253:[1.0,1.0...1.0,1.0], nodes_missing_value_tracks_true=253:[0,0,0...0,0,0], nodes_modes=253:[b'BRANCH_LEQ',b'BRANCH_LEQ'...b'LEAF',b'LEAF'], nodes_nodeids=253:[0,1,2...250,251,252], nodes_treeids=253:[0,0,0...0,0,0], nodes_truenodeids=253:[1,2,3...251,0,0], nodes_values=253:[0.00792999193072319,-0.12246682494878769...0.0,0.0], post_transform=b'NONE', target_ids=127:[0,0,0...0,0,0], target_nodeids=127:[7,8,10...249,251,252], target_treeids=127:[0,0,0...0,0,0], target_weights=127:[-0.9345570802688599,-0.6372960805892944...0.6169403195381165,1.0096807479858398]) -> variable\n", "output: name='variable' type=dtype('float32') shape=[None, 1]\n"]}], "source": ["from mlprodict.plotting.text_plot import onnx_simple_text_plot\n", "print(onnx_simple_text_plot(onx))"]}, {"cell_type": "markdown", "id": "1ada8e37", "metadata": {}, "source": ["## Apr\u00e8s la conversion en un r\u00e9seau de neurones"]}, {"cell_type": "markdown", "id": "7238d09b", "metadata": {}, "source": ["### Conversion en un r\u00e9seau de neurones\n", "\n", "Un param\u00e8tre permet de faire varier la pente des fonctions sigmo\u00efdes utilis\u00e9es."]}, {"cell_type": "code", "execution_count": 11, "id": "7729c242", "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 18/18 [00:01<00:00, 12.49it/s]\n"]}], "source": ["from tqdm import tqdm\n", "from pandas import DataFrame\n", "from mlstatpy.ml.neural_tree import NeuralTreeNet\n", "\n", "xe = x_exp[:500]\n", "expected = tree.predict(xe)\n", "\n", "data = []\n", "trees = {}\n", "for i in tqdm([0.3, 0.4, 0.5, 0.7, 0.9, 1] + list(range(5, 61, 5))):\n", "    root = NeuralTreeNet.create_from_tree(tree, k=i, arch='compact')\n", "    got = root.predict(xe)[:, -1]\n", "    me = numpy.abs(got - expected).mean()\n", "    mx = numpy.abs(got - expected).max()\n", "    obs = dict(k=i, max=mx, mean=me)\n", "    data.append(obs)\n", "    trees[i] = root"]}, {"cell_type": "code", "execution_count": 12, "id": "9d35377e", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>k</th>\n", "      <th>max</th>\n", "      <th>mean</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>0.3</td>\n", "      <td>0.568981</td>\n", "      <td>0.158758</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>0.4</td>\n", "      <td>0.608304</td>\n", "      <td>0.132576</td>\n", "    </tr>\n", "    <tr>\n", "      <th>2</th>\n", "      <td>0.5</td>\n", "      <td>0.692657</td>\n", "      <td>0.128525</td>\n", "    </tr>\n", "    <tr>\n", "      <th>3</th>\n", "      <td>0.7</td>\n", "      <td>0.780543</td>\n", "      <td>0.131497</td>\n", "    </tr>\n", "    <tr>\n", "      <th>4</th>\n", "      <td>0.9</td>\n", "      <td>0.809866</td>\n", "      <td>0.128368</td>\n", "    </tr>\n", "    <tr>\n", "      <th>5</th>\n", "      <td>1.0</td>\n", "      <td>0.813889</td>\n", "      <td>0.124802</td>\n", "    </tr>\n", "    <tr>\n", "      <th>6</th>\n", "      <td>5.0</td>\n", "      <td>0.392482</td>\n", "      <td>0.022466</td>\n", "    </tr>\n", "    <tr>\n", "      <th>7</th>\n", "      <td>10.0</td>\n", "      <td>0.341749</td>\n", "      <td>0.006350</td>\n", "    </tr>\n", "    <tr>\n", "      <th>8</th>\n", "      <td>15.0</td>\n", "      <td>0.270649</td>\n", "      <td>0.002939</td>\n", "    </tr>\n", "    <tr>\n", "      <th>9</th>\n", "      <td>20.0</td>\n", "      <td>0.299713</td>\n", "      <td>0.002110</td>\n", "    </tr>\n", "    <tr>\n", "      <th>10</th>\n", "      <td>25.0</td>\n", "      <td>0.305493</td>\n", "      <td>0.001842</td>\n", "    </tr>\n", "    <tr>\n", "      <th>11</th>\n", "      <td>30.0</td>\n", "      <td>0.306111</td>\n", "      <td>0.001767</td>\n", "    </tr>\n", "    <tr>\n", "      <th>12</th>\n", "      <td>35.0</td>\n", "      <td>0.299371</td>\n", "      <td>0.001665</td>\n", "    </tr>\n", "    <tr>\n", "      <th>13</th>\n", "      <td>40.0</td>\n", "      <td>0.233556</td>\n", "      <td>0.001011</td>\n", "    </tr>\n", "    <tr>\n", "      <th>14</th>\n", "      <td>45.0</td>\n", "      <td>0.233606</td>\n", "      <td>0.000801</td>\n", "    </tr>\n", "    <tr>\n", "      <th>15</th>\n", "      <td>50.0</td>\n", "      <td>0.233614</td>\n", "      <td>0.000547</td>\n", "    </tr>\n", "    <tr>\n", "      <th>16</th>\n", "      <td>55.0</td>\n", "      <td>0.233615</td>\n", "      <td>0.000499</td>\n", "    </tr>\n", "    <tr>\n", "      <th>17</th>\n", "      <td>60.0</td>\n", "      <td>0.233615</td>\n", "      <td>0.000484</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["       k       max      mean\n", "0    0.3  0.568981  0.158758\n", "1    0.4  0.608304  0.132576\n", "2    0.5  0.692657  0.128525\n", "3    0.7  0.780543  0.131497\n", "4    0.9  0.809866  0.128368\n", "5    1.0  0.813889  0.124802\n", "6    5.0  0.392482  0.022466\n", "7   10.0  0.341749  0.006350\n", "8   15.0  0.270649  0.002939\n", "9   20.0  0.299713  0.002110\n", "10  25.0  0.305493  0.001842\n", "11  30.0  0.306111  0.001767\n", "12  35.0  0.299371  0.001665\n", "13  40.0  0.233556  0.001011\n", "14  45.0  0.233606  0.000801\n", "15  50.0  0.233614  0.000547\n", "16  55.0  0.233615  0.000499\n", "17  60.0  0.233615  0.000484"]}, "execution_count": 13, "metadata": {}, "output_type": "execute_result"}], "source": ["df = DataFrame(data)\n", "df"]}, {"cell_type": "code", "execution_count": 13, "id": "0fcb9789", "metadata": {}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAElCAYAAAD0sRkBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAv8ElEQVR4nO3de3xU9Z3/8dcndwgQ7goJN7moCCg2QvEKXiq6rbbWVtB1da1FW7G29bcW22pdt3bbbbe2u7W70mpt3Sq1Wiu2Wi/1gtZ6QUUREIjIJYiSRK6BkNvn98c5gWGcJBOYZHJm3s/HYx45l++c+ZyZ5DPffL/nfL/m7oiISPTlpDsAERFJDSV0EZEMoYQuIpIhlNBFRDKEErqISIZQQhcRyRBK6JKQmZ1gZq+YWf8kyl5kZo8nUe5/zeyG1ETY5utMN7PKA3zuXWb23VTH1N111WcjnSsv3QFI5zKztcAhQBNQCzwKzHX3nW08ZxjwPeAf3P3D9l7D3X8L/DaJclcmGbZ0MX02mUE19OzwKXfvBRwLlAPfji9gZnu/3N19g7uf4u6buzBG6USxn69kLiX0LOLuGwlq6BMAzMzN7CozWw2sDrd90syWmNlWM3vBzCa1PN/MhpnZH8ysysxqzOxn4fZLzez5cNnM7FYz22xm281sqZm1vN5+zRlm9kUzqzCzD81soZkNjdnnZnalma0OY7nNzCzReZlZj/DYW8xsOXBc3P6hZvZAGPe7ZvaVZN4vM+tnZn8Kn7clXC5ro3xr70+OmX3bzNaF78tvzKwk3DcyPNdLzGy9mVWb2bdi4t4d2+xlZpPDMvnh+mVmtiKM7zEzGxH3Hu79fNPx2UjXUkLPImFTytnA6zGbPw1MBcab2WTgTuAKYABwO7DQzArNLBf4E7AOGAmUAgsSvMwngJOBcUAJ8HmgJkEspwL/Hu4fEh43/nifJEjOk8JyZ7Zyat8BRoePM4FLYl4nB3gYeCOM+TTgq2bW2rFi5QC/AkYAw4HdwM8SFWzn/bk0fMwADgN6JTjOicDhYXw3mtmR7v4e8HfgszHlLgTud/cGMzsX+CZwHjAIeA64N+64nyb8fEnPZyNdyd31yOAHsBbYCWwl+MP8OdAj3OfAqTFl/wf4t7jnrwROAaYBVUBegte4FHg+XD4VWAV8HMiJK3cX8N1w+Q7gP2L29QIagJExsZ0Ys/8+YF4r57gGmBmzPgeoDJenAuvjyl8P/KqVY+2NMcG+Y4Atrexr6/35K/DlmPXDw3PNI0j+DpTF7H8ZmBUuXw48FS4bsAE4OVx/FPhCzPNygF3AiFY+3y7/bPTo2odq6Nnh0+7e191HuPuX3X13zL4NMcsjgGvDf6O3mtlWYBgwNPy5zt0b23ohd3+KoPZ5G7DZzOabWZ8ERYcSfMG0PG8nQW2xNKbM+zHLuwgSSyJD485jXczyCGBo3Dl9k6CjuE1m1tPMbg+bSrYDi4C+YW08Xlvvz37nGi7nxcXQ2rk+AEwzsyEEtetmgpp4y7n9NOa8PiRI+rHv4d73JU2fjXQhJXSJHW5zA3BLmPxbHj3d/d5w33BLonPN3f/L3T9G8G/+OOBfEhR7jyAhAWBmxQTNPBsP4Bw2ESTUFsNjljcA78adU293PzuJ415LUJue6u59CBIqBEkzXlvvz37nGsbXCHzQXgDuvgV4HLiAoLllgYfV4vA1r4g7tx7u/kLsIeKO19WfjXQhJXSJ9QvgSjObGnagFZvZP5hZb4JmgE3A98PtRWZ2QvwBzOy48Pn5BJdJ1hHUKuPdC/yzmR1jZoUEl0m+5O5rDyDu+4Drw07MMuDqmH0vAzvM7Bth52mumU0ws+MSH2o/vQnazbeGHZPfaaNsW+/PvcDXzGyUmfUiONfftfffTox7gH8Czg+XW/wvwXkfBWBmJWb2udYOkqbPRrqQErrs5e6LgS8S/Fu+BaggaB/H3ZuATwFjgPVAJUGtMV4fgi+GLQT/ttcAP0zwWk8CNxA0KWwi6NCcdYCh/2v4Wu8S1GbvjnmdJoIOvGPC/dXALwk6BdvzE6BH+JwXgb+0VrCd9+fOMKZFYQx17P+l056FwFjgfXd/I+Y1HwR+ACwIm4TeAs5q4zjp+GykC9m+/95ERCTKVEMXEckQSugiIhlCCV1EJEMooYuIZAgldOky1oEhedPJzNaa2enpjkOko5TQpUtYB4fkFZGO05Ca0mnMLK/l5hl330AwJoykgJnlhte+i+ylGrq0Obysmd1kZvdZMOTrDjNbZmblbRyro0PyfsPMNobHXmlmp4Xbc8xsnpm9Y8FQtPfZ/sPI/t7M3jezbWa2qOVuyXDfM2Z2ecz63uF9W4n54nC8lhoLh66N2ddmHHFlp5tZpZlda8EQtZvM7J9j9hea2Y8sGCb3AwtmCerRWozhezkmXL7LzP7HzB4xs1pghpkdGZ7r1vBzOSfmuXdZMKztn8P39iUzGx2z/wgze8KC4XFXmtnnY/adbWbLw+dtNLP/19p7J92LEnqWs+SGlz2HYPjUvgR3LSYcQjbGp0luSN7DgbnAce7em2AI1rXhMa4Oj3MKwWBRWwgGlWrxKMHdk4OB10hixqREzGw8wSiTF4evMwCIHfO8vTjiHUpwF2op8AXgNjPrF+77PsH4KccQ3FFaCtzYgXAvBG4hGJLgJYLP7XGC9+Bq4Lfhe9piFsFdtP0I7vq9JTznYuAJgmEEBoflfh6+FxCMtnhF+JlMAJ7qQIySTuke7lGP9D5oZ3hZ4CbgyZh944HdbRyvI0PyjgE2A6cD+XFlVgCnxawPIRxyNsFr9g1ftyRcfwa4PGb/pYTD+yZ47o0EA161rBcD9cDpBxDHdIKxX/Jitm0mGK7WCMZPGR2zbxrBwGEJYwzPaUy4fBfwm5h9JxGMeJgTs+1e4KaY8r+M2Xc28Ha4fAHwXNxr3Q58J1xeT/AF3Cfdv596dOyhGrokM7xs/FCpRdb2qItJDcnr7hXAVwm+NDab2QLbNzPOCODBmOesIJgX9RALBtj6ftgMsp19tfqBHTx3iBt6191r2X/Sh1bjaOV4Nb7/oFstQ8sOAnoCr8Yc6y/h9mTFvq9DgQ3uHju41jqSG+J2BDA17jO5iOC/Cwgm1DgbWGdmz5rZtA7EKGmkTlFpGV52bAqPmWhI3lsSFnS/B7jHgnG5bycYbOri8HmXufvf4p9jZhcD5xLU7NcSNHFsYd+wtrUEybPFobRuE3BkzLF7EjS7xMafMI4OqiaovR/lwVSA8faL2cwSxRz7vr4HDDOznJikPpxgAov2bACedfczEu1091eAcy0YlXEuwWiWwxKVle5FNXQ5mOFlk9HqkLxmdriZnWrBEK11BAmvJTn9L3CLhXNkmtkgC6Zcg6ANeQ9BTbonweWQsZYA51kwQcUYgrbs1twPfNLMTjSzAuBm9v+7aCuOpIVJ9xfArWY2ODxWaUxfxRvAURYMWVtE8F9LW14iqHVfZ2b5ZjadYLTHRNMCxvsTMM6CzuD88HFc2MlaYGYXmVmJuzcA20k8xK50Q0roWc4PbnjZZI7f6pC8QCFBR2E1QfPAYIL2e4CfEnTAPm5mOwiGr50a7vsNQfPCRmB5uC/WrQTt4B8Av6aNDlN3XwZcRdBBuCmMsTKmSFtxdNQ3CM7/xbCp6EmCCTRw91UEXyZPElwd1OpVOWH5eoIEfhbB+/dz4J/c/e32gnD3HQTzi84iqOm/T/CfUWFY5GJgbRjjlQTNMRIBGj5XRCRDqIYuIpIhlNBFRDKEErqISIZQQhcRyRBpuw594MCBPnLkyHS9vIhIJL366qvV7p7whrS0JfSRI0eyePHidL28iEgkmdm61vapyUVEJEMooYuIZAgldBGRDKHBuUSk22poaKCyspK6urp0h9LlioqKKCsrIz8/P+nnKKGLSLdVWVlJ7969GTlyJGbW/hMyhLtTU1NDZWUlo0aNSvp5anIRkW6rrq6OAQMGZFUyBzAzBgwY0OH/TJTQRaRby7Zk3uJAzjvyCf2ptz+gYvOOdIchIpJ2SSV0M5sZzgxeYWbzEuwfbmZPm9nrZvammZ2d+lA/al1NLZfdtZhLf/VKV7yciEi31m5CN7NcglnOzyKYIHh2zOzgLb4N3OfukwlnEE91oIk8vuwDADZty74ecBGReMnU0KcAFe6+JpwlZQHBfI6xHOgTLpcQzILS6d6tqQWgqdnZUlvfFS8pIllk7dq1HHHEEVx66aWMGzeOiy66iCeffJITTjiBsWPH8vLLL/Pyyy8zbdo0Jk+ezPHHH8/KlSsBuPXWW7nssssAWLp0KRMmTGDXrl2dGm8yly2Wsv9s45V8dAqumwim6LoaKCaYvLfTrQsTOsDSjds4eVxHJlAXkSj514eXsfy97Sk95vihffjOp45qs0xFRQW///3vufPOOznuuOO45557eP7551m4cCHf+973+M1vfsNzzz1HXl4eTz75JN/85jd54IEHuOaaa5g+fToPPvggt9xyC7fffjs9e/Zs87UOVqquQ58N3OXu/2lm04C7zWxCzGzkAJjZHGAOwPDhww/6RddW7+LUIwbz1NubldBFpFOMGjWKiRMnAnDUUUdx2mmnYWZMnDiRtWvXsm3bNi655BJWr16NmdHQ0ABATk4Od911F5MmTeKKK67ghBNO6PRYk0noG4FhMetl4bZYXwBmArj738NZywcCm2MLuft8YD5AeXn5QU1mWtfQxHvbdvO58jIqNu/krY3bDuZwItLNtVeT7iyFhYV7l3Nycvau5+Tk0NjYyA033MCMGTN48MEHWbt2LdOnT99bfvXq1fTq1Yv33uuSVuik2tBfAcaa2SgzKyDo9FwYV2Y9cBqAmR0JFAFVqQw03oYPd+EOowYWM7G0hKVK6CKSBtu2baO0tBSAu+66a7/tX/nKV1i0aBE1NTXcf//9nR5Luwnd3RuBucBjwAqCq1mWmdnNZnZOWOxa4Itm9gZwL3Cpux9UDbw971YH7ecjBhQzobSEyi271TEqIl3uuuuu4/rrr2fy5Mk0Njbu3f61r32Nq666inHjxnHHHXcwb948Nm/e3MaRDp51ct5tVXl5uR/MBBd3/e1dbnp4Oa9++3RWbNrBP97xEnd/YQonjVU7ukimWLFiBUceeWS6w0ibROdvZq+6e3mi8pG9U7S2vgmA3kX5TCgNrphUs4uIZLPIjrZYu6eR/FyjIC+HgrwChvXvoY5REclq0a2h72mkZ8G+7yN1jIpItotuQq9vorggd+/6hNISNny4m6271DEqItkpsgl9V30jPQv3r6EDvLUxtXeSiYhERWQTeu2eJopjEvqEoUFCV7OLiGSryCb0XfWN+zW59CsuoKyfOkZFJHtFNqHv3NO0X6coqGNURLJbZBP6rvpGigtz99s2obSE9R/uYtuuhjRFJSKZJJnhc2tra7nsssuYMmUKkydP5qGHHtr73JNOOoljjz2WY489lhdeeAGAZ555hunTp3P++edzxBFHcNFFF5GqGzwjfB164ho6wFvvbeOEMQPTEZaIdJZH58H7S1N7zEMnwlnfb7NIe8Pnjh8/nlNPPZU777yTrVu3MmXKFE4//XQGDx7ME088QVFREatXr2b27Nm03B3/+uuvs2zZMoYOHcoJJ5zA3/72N0488cSDPp3IJvRd9Y30iquhtyT0pRuV0EUkNdobPreyspKFCxfyox/9CIC6ujrWr1/P0KFDmTt3LkuWLCE3N5dVq1btPeaUKVMoKysD4JhjjmHt2rXZm9Cbm51d9R+tofcrLqC0bw+1o4tkonZq0p2lveFzc3NzeeCBBzj88MP3e95NN93EIYccwhtvvEFzczNFRUUJj5mbm7vfoF4HI5Jt6LsagnFc4tvQIail60oXEekqZ555Jv/93/+9tx389ddfB4Lhc4cMGUJOTg533303TU1NnR5LNBP6nuDbLL6GDjCxrIR1NbvYtlsdoyLS+W644QYaGhqYNGkSRx11FDfccAMAX/7yl/n1r3/N0Ucfzdtvv01xcXGnxxLJ4XPfra5lxo+e4dYLjuYzk8v22/fsqiouufNl7rl8KserHV0k0jR8bhYMn1sb1tCLE9XQS3XHqIhkp6QSupnNNLOVZlZhZvMS7L/VzJaEj1VmtjXlkcbYm9ALP5rQ+6tjVESyVLtXuZhZLnAbcAZQCbxiZgvdfXlLGXf/Wkz5q4HJnRDrXrvCyS16Fny0UxRgQmkfdYyKZAh3x8zSHUaXO5Dm8GRq6FOACndf4+71wALg3DbKzyaYV7TT1Na3XkOHoNllbc0uttepY1QkyoqKiqipqUnZnZRR4e7U1NTsd6ljMpK5Dr0U2BCzXglMTVTQzEYAo4CnWtk/B5gDMHz48A4FGmvXnvZq6C1D6W7j+NHqGBWJqrKyMiorK6mqqkp3KF2uqKho781HyUr1jUWzgPvdPeEFl+4+H5gPwVUuB/oie2voCTpFIXZsdCV0kSjLz89n1KhR6Q4jMpJpctkIDItZLwu3JTKLTm5uAdjT2AxAYX7i8Af0KmRoSRFLNdmFiGSRZBL6K8BYMxtlZgUESXthfCEzOwLoB/w9tSF+VEOY0AtyWw9/gu4YFZEs025Cd/dGYC7wGLACuM/dl5nZzWZ2TkzRWcAC74Lei/qmZswgN6f1nu+JpSW8W12rjlERyRpJtaG7+yPAI3Hbboxbvyl1YbWtvqmZ/NycNi9lmlAWtKMv27idaaMHdFVoIiJpE8k7RRsancI2mltg/45REZFsEMmEXt/URH5e26EP7FXIkJIi3TEqIlkjkgm9odHb7BBtoY5REckm0UzoTc3k57V/K/DE0hLWVNeyQx2jIpIFIpnQ94Sdou1paUdf9p6uRxeRzBfJhN7Q2Jx0kwuoY1REskM0E3pTMwXtdIoCDOpdyKF91DEqItkhkgm9PskmFwhq6UroIpINIpnQk73KBfbdMbpzT2pm1RYR6a4imdDrm5rbvQ69xcSyPrjDMtXSRSTDRTOhJ9kpCvs6RtXsIiKZLpIJPegUTW5KqsG9izikT6GudBGRjBfZhJ5spygE7eiqoYtIpotkQu9IkwsEzS5rqmupVceoiGSwaCb0Jk+6UxSCGro7LN+kO0ZFJHMllRXNbKaZrTSzCjOb10qZz5vZcjNbZmb3pDbM/TU0dayG3jIEwNJKNbuISOZqd4ILM8sFbgPOACqBV8xsobsvjykzFrgeOMHdt5jZ4M4KGKCuoanV+UQTGdyniMG91TEqIpktmaw4Bahw9zXuXg8sAM6NK/NF4DZ33wLg7ptTG+Y+DU3N7GlspldBUpMt7aWOURHJdMkk9FJgQ8x6Zbgt1jhgnJn9zcxeNLOZiQ5kZnPMbLGZLa6qqjqggFs6NnsVdSyhTygt4Z2qneyqV8eoiGSmVHWK5gFjgenAbOAXZtY3vpC7z3f3cncvHzRo0AG90I66MKEXdryG3uywXEPpikiGSiahbwSGxayXhdtiVQIL3b3B3d8FVhEk+JRrGZOlwwm9THeMikhmSyahvwKMNbNRZlYAzAIWxpX5I0HtHDMbSNAEsyZ1Ye6z8wCbXA7pU8Sg3oVK6CKSsdpN6O7eCMwFHgNWAPe5+zIzu9nMzgmLPQbUmNly4GngX9y9pjMCbknoxR2soUPQ7KIrXUQkUyWVFd39EeCRuG03xiw78PXw0al2hm3ovQ8goU8oLeGZlZvZVd9Izw5eJSMi0t1F7k7Rg6mhTxjah2aHFbpjVEQyUOQS+oFetggxHaO6Y1REMlDkEvrRw/py1YzRFB9Ak8mhfYoY2KuApRtVQxeRzBO5huTjRvbnuJH9D+i5ZsYEdYyKSIaKXA39YE0sLWH15h3srm9KdygiIimVdQl9Qssdo+oYFZEMk3UJvWUoXTW7iEimybqEPqSkiAHFBbpjVEQyTtYldHWMikimyrqEDi0dozupa1DHqIhkjqxM6BNKS2hqdnWMikhGycqE3nLHqJpdRCSTZGVCH1pSRP/iAg0BICIZJSsTekvHqK50EZFMkpUJHWBiaR91jIpIRkkqoZvZTDNbaWYVZjYvwf5LzazKzJaEj8tTH2pqTQw7RjWUrohkinYTupnlArcBZwHjgdlmNj5B0d+5+zHh45cpjjPlJuiOURHJMMnU0KcAFe6+xt3rgQXAuZ0bVucr7duDfj3z1Y4uIhkjmYReCmyIWa8Mt8X7rJm9aWb3m9mwlETXifZ1jKrJRUQyQ6o6RR8GRrr7JOAJ4NeJCpnZHDNbbGaLq6qqUvTSB25iaQmrP9ihjlERyQjJJPSNQGyNuyzctpe717j7nnD1l8DHEh3I3ee7e7m7lw8aNOhA4k2piaUlNDY7b7+/I92hiIgctGQS+ivAWDMbZWYFwCxgYWwBMxsSs3oOsCJ1IXaelo5RtaOLSCZodwo6d280s7nAY0AucKe7LzOzm4HF7r4Q+IqZnQM0Ah8Cl3ZizClT1q8HfXvm85buGBWRDJDUnKLu/gjwSNy2G2OWrweuT21onc/MmKg7RkUkQ2TtnaItJpSWsEodoyKSAbI+obd0jK5Ux6iIRJwSujpGRSRDZH1CL+vXg5Ie+RoCQEQiL+sTujpGRSRTZH1Ch30do3sa1TEqItGlhE7Qjt7QpI5REYk2JXTUMSoimUEJHRjWXx2jIhJ9Sui0DKXbRzV0EYk0JfTQhNISVr6vjlERiS4l9FBLx+iq93emOxQRkQOihB5q6Rh96z01u4hINCmhh4b370mfojy1o4tIZCmhh1rmGNWVLiISVUroMSaWlvD2ph3UNzanOxQRkQ5LKqGb2UwzW2lmFWY2r41ynzUzN7Py1IXYdSaUllDf1MyqD3THqIhET7sJ3cxygduAs4DxwGwzG5+gXG/gGuClVAfZVfZ2jKrZRUQiKJka+hSgwt3XuHs9sAA4N0G5fwN+ANSlML4uNWJAT3qrY1REIiqZhF4KbIhZrwy37WVmxwLD3P3PbR3IzOaY2WIzW1xVVdXhYDubmTFhqDpGRSSaDrpT1MxygB8D17ZX1t3nu3u5u5cPGjToYF+6Uxw9rC9LN27jxofeYuPW3ekOR0QkaXlJlNkIDItZLwu3tegNTACeMTOAQ4GFZnaOuy9OVaBd5cpTDmPb7nrufXk997y0nvOOLeVL08cwamBxukMTEWmTuXvbBczygFXAaQSJ/BXgQndf1kr5Z4D/114yLy8v98WLu2++f2/rbuYvWsO9L6+noamZTx09lKtmjGHcIb3THZqIZDEze9XdE15J2G6Ti7s3AnOBx4AVwH3uvszMbjazc1IbavcxtG8PbjrnKJ77xgy+eNJhPLH8Az5x6yKuvPtVtbGLSLfUbg29s3T3Gnq8LbX1/OqFtfzqb++yo66R6YcP4upTx/CxEf3THZqIZJG2auhK6B20va6Bu/++jjuef5cPa+uZdtgArj51DNNGDyDsQxAR6TRK6J1gV30j97y0nvmL1rB5xx6OHd6XuaeOYcbhg5XYRaTTKKF3orqGJu5/tZL/eeYdNm7dzVFD+zB3xhjOPOpQcnKU2EUktZTQu0BDUzN/fH0jP3/mHd6trmXs4F5cNWMMn5w0hLxcjYEmIqmhhN6FmpqdPy/dxG1PVbDygx2MGNCTL50ymvOOLaMgT4ldRA6OEnoaNDc7T674gJ89XcGbldsYWlLEFaeM5oLjhlGUn5vu8EQkopTQ08jdWbS6mp89tZpX1m5hYK9C5pw8ioumjqC4MJkbdUVE9lFC7yZeWlPDz56u4LnV1fTtmc+1Z4zjoqkj1HkqIklrK6GritiFph42gKmHDeD19Vv40eMrueGhZTy27AN+cP4kSvv2SHd4IhJx6qVLg8nD+/F/X5jK9z4zkdfXb2HmrYu4b/EG0vXfkohkBiX0NDEzLpw6nL989WTGD+3Ddfe/yeW/Xszm7ZGdH0RE0kwJPc2G9e/JvV/8ODd+cjzPV1Rzxq2LeGjJRtXWRaTDlNC7gZwc47ITR/HINSdx2KBirlmwhKvueY2anXvSHZqIRIgSejcyelAv7r/yeL4x8wieXL6ZT9y6iMeWvZ/usEQkIpTQu5ncHONL00fz8NUncmhJEVfc/Spf+90Stu1qSHdoItLNJXXZopnNBH4K5AK/dPfvx+2/ErgKaAJ2AnPcfXmKY80qhx/amz9edQI/e6qC256u4IV3qvnBZycx/fDB6Q6tW3J33KHZHSf86QQPnGYPyjQ7kGCbs698z8JcehXk6f4AiZxkpqDLJZiC7gygkmAKutmxCdvM+rj79nD5HODL7j6zreNm441FB2pp5Tau/f0SVn2wk9lThvGtfxhPrwjdZeru1DU0s2NPAzvrGtm5p5GddY1s37vcwM49jewIt7fs37GnkR11jewMn7e7oWlvQo5P3KmWY9C7KJ8+PfIo6ZFPn6LgUdIjZluP/H37Ysv1yNfwDtJpDvbGoilAhbuvCQ+2ADgX2JvQW5J5qBjQJRopNLGshIVzT+TWJ1fxi0VreG51Nf9x/iSOHz0wrXE1NDXz2rotvPBODZt37Nk/Ode1JOPg0dTc/q9EQV4OvQvz6F2UR6+iPHoV5lHatwe9i3rTqzCPHgW5mEGOGUb404JLQGPXc1q2GRgWrgf7iX0eQYe0sa88wO76JrbvbmDb7ga21zUGP3c38E7VTrbXNbB9d/Dl0t65BF8Aefsl/pYvhDGDe3HO0aXk6r8ASaFkEnopsCFmvRKYGl/IzK4Cvg4UAKcmOpCZzQHmAAwfPryjsWa1ovxcrj/rSD4x/hCuve8NLvzFS1x6/Ei+MfMIehR0XW1wfc0unl1dxaJVVfz9nRp27mkkN8foX1xA78J9iXh4/570KsqL2ZZP76IwWReGj6I8ehfm06soj+LCXArzolOr3dPYxI6YZN+S/PctB9u3725ke10DH9bWs7a6dm+5pmbn1y+s43ufmcj4oX3SfTqSIZJpcjkfmOnul4frFwNT3X1uK+UvBM5090vaOq6aXA7c7vomfvCXt7nrhbWMGljMjz43qdPmNt25p5G/v1PDolVVLFpdxbqaXQCU9evByeMGcfLYQRw/ZgB9ivI75fUzkbuz8I33uPnh5Wzd3cDlJ47imtPH0rMgOs1okj4HNTiXmU0DbnL3M8P16wHc/d9bKZ8DbHH3kraOq4R+8F54p5p/+f2bbNq2my+efBhfO33cQbfdNjc7yzdt59lVQS38tfVbaGhyehbkMu2wAZw0diAnjxvEqIHFmmrvIG3dVc/3H32bBa9soKxfD7776Qnq9JZ2HWxCzyPoFD0N2EjQKXqhuy+LKTPW3VeHy58CvtPaC7ZQQk+NnXsaueXPy7n35Q2MHdyLH3/+GCaWtfld+hGbd9Tx/OpqFq2q4rnV1dTU1gMwfkifoBY+biAfG9EvUk0iUfLSmhq++eBS3qmq5ZOThnDjp8YzuHdRusOSbuqgh881s7OBnxBctninu99iZjcDi919oZn9FDgdaAC2AHNjE34iSuip9fTKzcx74E2qd9Zz1YwxzJ0xptUZkvY0NvHq2i1hW3g1KzYFfdoDigv21sBPHDtQSaUL7Wls4vZn1/Czpyooys9h3llHMuu4Ybp0Uj5C46FniW27GvjXh5fxh9c3ctTQPvzn54/miEP74O68W10btoNX8+KaGnbVN5GXY3xsRD9OHjeIU8YNYvyQPkogabamaiffevAt/r6mhvIR/fjeeRMZd0jvdIcl3YgSepb5y1vv860Hl7KjrpEzjjqENzZspXLLbgBGDOjJyWMHcfK4QUwbPSBS17NnC3fngdc2csufl7OjrpErTjmMq08dq2vbBVBCz0o1O/dw48Jl/K2imvIR/TllXNCUMmJAcbpDkyR9WFvPLX9ewQOvVTJyQE9u+cxEThiT3nsPJP2U0EUi7IWKar71x7d4t7qW8yaX8q1/OJIBvQrTHZakSVsJXYNziXRzx48ZyKPXnMRXTh3Dw2++x2k/flYzXElCSugiEVCUn8vXP3E4j3zlJMYO7sV197/JrPkv8k7VznSHJt2IErpIhIw9pDe/mzONfz9vIis2beesnzzHT55cxZ7GtseWkeyghC4SMTk5xuwpw/nrtdOZOeFQfvLkas766XO8uKYm3aFJmimhi0TUoN6F/Nfsydz1z8fR0NTMrPkvct39b7B1V326Q5M0UUIXibjphw/m8a+ewpWnjOaB1zZy2n8+y4OvV6rTNAvpskWRDLJi03au/8NSlmzYyoljBjJt9ICE5Vr7u28tHeirIbVmHD64w2MutTjYCS5EJCKOHNKHB750PPe8tI4fPraS5yuq0x2SJNC/uOCAE3pblNBFMkxujnHxtJFcOHVEmzNFtTb6cWuj+Wi45NTprHdSCV0kQ+XmmKa4yzLqFBURyRBK6CIiGSKphG5mM81spZlVmNm8BPu/bmbLzexNM/urmY1IfagiItKWdhO6meUCtwFnAeOB2WY2Pq7Y60C5u08C7gf+I9WBiohI25KpoU8BKtx9jbvXAwuAc2MLuPvT7r4rXH0RKEttmCIi0p5kEnopsCFmvTLc1povAI8m2mFmc8xssZktrqqqSj5KERFpV0o7Rc3sH4Fy4IeJ9rv7fHcvd/fyQYMGpfKlRUSyXjLXoW8EhsWsl4Xb9mNmpwPfAk5x9z2pCU9ERJKVTA39FWCsmY0yswJgFrAwtoCZTQZuB85x982pD1NERNrTbkJ390ZgLvAYsAK4z92XmdnNZnZOWOyHQC/g92a2xMwWtnI4ERHpJEnd+u/ujwCPxG27MWb59BTHJSIiHaQ7RUVEMoQSuohIhlBCFxHJENFL6B++Cysebn1qFRGRLBW9hL78IfjdP0J9bbojERHpVqKX0IsHBj93aWotEZFY0UvoPcNJb3fVpDcOEZFuJoIJPayh1yqhi4jEimBC7x/8VJOLiMh+opfQ97ahq4YuIhIregm9sA/kFsCSe+CRf4GdGgtMRASimNDNYPRpsHk5vDwfbj8ZdmqyDBGR6CV0gE/+GGZ+H/75Uaitgj9/XTcaiUjWi2ZC7zMUPv4lGHE8nHYjrFgIi36U7qhERNIqqeFzu7XjvwIfLIOnvwuDj4AjP5XuiERE0iKaNfRYZvCp/4LSj8EfrgiSu4hIFkoqoZvZTDNbaWYVZjYvwf6Tzew1M2s0s/NTH2Y78ovggt9CYW+4d5ZuOhKRrNRuQjezXOA24CxgPDDbzMbHFVsPXArck+oAk9ZnCMz6Lez4AO77J2hqSFsoIiLpkEwNfQpQ4e5r3L0eWACcG1vA3de6+5tAcyfEmLyycjjnv2Dd8/DoN9IaiohIV0smoZcCG2LWK8NtHWZmc8xssZktrqrqpGvHj54VdJQuvgNeuaNzXkNEpBvq0k5Rd5/v7uXuXj5o0KDOe6HTb4IxZ8Cj18Ha5zvvdUREupFkEvpGYFjMelm4rfvKyYXz74B+o+B3F8OWtemOSESk0yWT0F8BxprZKDMrAGYBCzs3rBQoKoHZC8Cb4N4LYc/OdEckItKp2k3o7t4IzAUeA1YA97n7MjO72czOATCz48ysEvgccLuZdY+LwQeOgfN/BVUr4MEroDm9fbYiIp3JPE1joJSXl/vixYu75sX+/nN47Ho45Rsw45td85oiIp3AzF519/JE+6J/638yPv6l4A7SZ38Ag4+Eoz6T7ohERFIu+rf+J8MsGKGxbAo8+CXY9Ea6IxIRSbnsSOgAeYVwwf8FU9jde6EmxhCRjJM9CR2g9yEw655g+rrfXQyN9emOSEQkZbIroQMMPQY+fRtseBEeuVYTY4hIxsiOTtF4Ez4bdJI+959wyESYOifdEYmIHLTsq6G3mPFtGHcW/GUerHkm3dGIiBy07E3oOTlw3nwYOA7uuwQ+XJPuiEREDkr2JnSAoj4w+57gssZ7Z0Pd9nRHJCJywLI7oQP0Pww+92uoXg1/+CI0N6U7IhGRA6KEDnDYKXDWD2DVX+Cp76Y7GhGRA5KdV7kkctzl8MFb8PyPYcBomHQB5OanOyoRkaQpobcwg7N+CFWr4KGr4OFrgvHUB44NHgPCnwPHBXebioh0M0rosfIK4KL7YMWfoHpV8KipgIonoSnmrtIe/RMn+n4jVasXkbRRQo9X2BuOmb3/tqZG2LouSO7Vq6FmdfBz1eNQ+3/7yuXkBUl94DgYMGZfoh8wFooHdOlpiEj2SSqhm9lM4KdALvBLd/9+3P5C4DfAx4Aa4AJ3X5vaUNMoNy9oVx8wGsaduf++3VvjEv0qqE5Uq++3L7kPHBP87Nk/GDQsr0f4swjyY5Zzcrv0NEUk2tpN6GaWC9wGnAFUAq+Y2UJ3Xx5T7AvAFncfY2azgB8AF3RGwN1Oj75QVh48YjU3BbX66op9Nfrq1VDxBCz5v4SH+oic/CCx5xXun+j3PpLYnlsAlhv0EVhOO4+DKWNh0NbGMkmUSWI5niXYlvJy7ZWxA9hvfOR93bvNEmyLe//jy0rWS6aGPgWocPc1AGa2ADgXiE3o5wI3hcv3Az8zM/N0TYfUHeTkBte49z8M+MT+++q2BbX6uu3QuAcad4c/66ChLviZzPa6rbDzA2iIKdfyaG5Mx1lLOn0kyYfr8NEvxHbX6WB5a2M5LNfWF7S1daz457SlnTKpOEYqTP9GMKZUiiWT0EuBDTHrlcDU1sq4e6OZbQMGANWxhcxsDjAHYPjw4QcYcgYoKoHSj3XuazQ1Bom9qT4YUdKb23m0V6at/eHNWO5A+B3uxCx7CpfjJdiWbD0imeMdUJkkjuHNQbnY93Xvevy25ja2edy2+LIx8cTHsXc9fv8BrLe2vLdca78XHXx+W9r93FNxjBQp6tsph+3STlF3nw/Mh2BO0a587ayTmwe5vdIdhYh0oWTuFN0IDItZLwu3JSxjZnlACUHnqIiIdJFkEvorwFgzG2VmBcAsYGFcmYXAJeHy+cBTWd1+LiKSBu02uYRt4nOBxwguW7zT3ZeZ2c3AYndfCNwB3G1mFcCHBElfRES6UFJt6O7+CPBI3LYbY5brgM+lNjQREekIjbYoIpIhlNBFRDKEErqISIZQQhcRyRCWrqsLzawKWHcATx1I3B2oEaZz6Z50Lt2TziUwwt0HJdqRtoR+oMxssbuXt1+y+9O5dE86l+5J59I+NbmIiGQIJXQRkQwRxYQ+P90BpJDOpXvSuXRPOpd2RK4NXUREEotiDV1ERBJQQhcRyRCRSehmNtPMVppZhZnNS3c8HWVmd5rZZjN7K2ZbfzN7wsxWhz/7pTPGZJjZMDN72syWm9kyM7sm3B7Fcykys5fN7I3wXP413D7KzF4Kf9d+Fw4bHQlmlmtmr5vZn8L1SJ6Lma01s6VmtsTMFofbIvc7BmBmfc3sfjN728xWmNm0zjqXSCT0mImqzwLGA7PNbHx6o+qwu4CZcdvmAX9197HAX8P17q4RuNbdxwMfB64KP4sonsse4FR3Pxo4BphpZh8nmOT8VncfA2whmAQ9Kq4BVsSsR/lcZrj7MTHXa0fxdwzgp8Bf3P0I4GiCz6dzzsXdu/0DmAY8FrN+PXB9uuM6gPMYCbwVs74SGBIuDwFWpjvGAzinh4Azon4uQE/gNYL5cquBvHD7fr973flBMJvYX4FTgT8RzHYc1XNZCwyM2xa53zGC2dveJbwApbPPJRI1dBJPVF2aplhS6RB33xQuvw8cks5gOsrMRgKTgZeI6LmETRRLgM3AE8A7wFZ3bwyLROl37SfAdUDL7NADiO65OPC4mb0aTi4P0fwdGwVUAb8Km8J+aWbFdNK5RCWhZzwPvqojcw2pmfUCHgC+6u7bY/dF6VzcvcndjyGo3U4BjkhvRAfGzD4JbHb3V9MdS4qc6O7HEjSzXmVmJ8fujNDvWB5wLPA/7j4ZqCWueSWV5xKVhJ7MRNVR9IGZDQEIf25OczxJMbN8gmT+W3f/Q7g5kufSwt23Ak8TNEv0DSc7h+j8rp0AnGNma4EFBM0uPyWa54K7bwx/bgYeJPiyjeLvWCVQ6e4vhev3EyT4TjmXqCT0ZCaqjqLYybUvIWiP7tbMzAjmkF3h7j+O2RXFcxlkZn3D5R4EfQErCBL7+WGxSJyLu1/v7mXuPpLg7+Mpd7+ICJ6LmRWbWe+WZeATwFtE8HfM3d8HNpjZ4eGm04DldNa5pLvToAOdC2cDqwjaOL+V7ngOIP57gU1AA8G39hcI2jj/CqwGngT6pzvOJM7jRIJ/D98EloSPsyN6LpOA18NzeQu4Mdx+GPAyUAH8HihMd6wdPK/pwJ+iei5hzG+Ej2Utf+9R/B0L4z4GWBz+nv0R6NdZ56Jb/0VEMkRUmlxERKQdSugiIhlCCV1EJEMooYuIZAgldBGRDKGELhIys5Gxo2GKRI0SuohIhlBCF0nAzA4LB1M6Lt2xiCQrr/0iItklvE17AXCpu7+R7nhEkqWELrK/QQTjapzn7svTHYxIR6jJRWR/24D1BGPWiESKaugi+6sHPgM8ZmY73f2edAckkiwldJE47l4bThjxRJjUM2GoZskCGm1RRCRDqA1dRCRDKKGLiGQIJXQRkQyhhC4ikiGU0EVEMoQSuohIhlBCFxHJEP8flTMewHRZL/gAAAAASUVORK5CYII=\n", "text/plain": ["<Figure size 432x288 with 1 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["df.set_index('k').plot(title=\"Pr\u00e9cision de la conversion\\nen r\u00e9seau de neurones\");"]}, {"cell_type": "markdown", "id": "1f4bb3d9", "metadata": {}, "source": ["L'erreur est meilleure mais il faudrait recommencer l'exp\u00e9rience plusieurs fois avant de pouvoir conclure afin d'obtenir un interval de confiance pour le m\u00eame type de jeu de donn\u00e9es. Ce sera pour une autre fois. Le r\u00e9sultat d\u00e9pend du jeu de donn\u00e9es et surtout de la proximit\u00e9 des seuils de d\u00e9cisions. N\u00e9anmoins, on calcule l'erreur sur l'ensemble de la base de test. Celle-ci a \u00e9t\u00e9 tronqu\u00e9e pour aller plus vite."]}, {"cell_type": "code", "execution_count": 14, "id": "2f3eb6d0", "metadata": {}, "outputs": [{"data": {"text/plain": ["(0.2336143002078063, 0.0002511855017989173)"]}, "execution_count": 15, "metadata": {}, "output_type": "execute_result"}], "source": ["expected = tree.predict(x_exp)\n", "got = trees[50].predict(x_exp)[:, -1]\n", "numpy.abs(got - expected).max(), numpy.abs(got - expected).mean()"]}, {"cell_type": "markdown", "id": "77163512", "metadata": {}, "source": ["On voit que l'erreur peut-\u00eatre tr\u00e8s grande. Elle reste n\u00e9anmoins plus petite que l'erreur de conversion introduite par ONNX."]}, {"cell_type": "markdown", "id": "738c8547", "metadata": {}, "source": ["### Conversion au format ONNX\n", "\n", "On cr\u00e9e tout d'abord une classe qui suit l'API de scikit-learn et qui englobe l'arbre qui vient d'\u00eatre cr\u00e9\u00e9 qui sera ensuite convertit en ONNX."]}, {"cell_type": "code", "execution_count": 15, "id": "2439e4fa", "metadata": {}, "outputs": [], "source": ["from mlstatpy.ml.neural_tree import NeuralTreeNetRegressor\n", "\n", "reg = NeuralTreeNetRegressor(trees[50])\n", "onx2 = to_onnx(reg, X[:1].astype(numpy.float32))"]}, {"cell_type": "code", "execution_count": 16, "id": "eae47e6a", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["opset: domain='' version=15\n", "input: name='X' type=dtype('float32') shape=[None, 10]\n", "init: name='Ma_MatMulcst' type=dtype('float32') shape=(1260,)\n", "init: name='Ad_Addcst' type=dtype('float32') shape=(126,)\n", "init: name='Mu_Mulcst' type=dtype('float32') shape=(1,) -- array([4.], dtype=float32)\n", "init: name='Ma_MatMulcst1' type=dtype('float32') shape=(16002,)\n", "init: name='Ad_Addcst1' type=dtype('float32') shape=(127,)\n", "init: name='Ma_MatMulcst2' type=dtype('float32') shape=(127,)\n", "init: name='Ad_Addcst2' type=dtype('float32') shape=(1,) -- array([0.], dtype=float32)\n", "MatMul(X, Ma_MatMulcst) -> Ma_Y02\n", "  Add(Ma_Y02, Ad_Addcst) -> Ad_C02\n", "    Mul(Ad_C02, Mu_Mulcst) -> Mu_C01\n", "      Sigmoid(Mu_C01) -> Si_Y01\n", "        MatMul(Si_Y01, Ma_MatMulcst1) -> Ma_Y01\n", "          Add(Ma_Y01, Ad_Addcst1) -> Ad_C01\n", "            Mul(Ad_C01, Mu_Mulcst) -> Mu_C0\n", "              Sigmoid(Mu_C0) -> Si_Y0\n", "                MatMul(Si_Y0, Ma_MatMulcst2) -> Ma_Y0\n", "                  Add(Ma_Y0, Ad_Addcst2) -> Ad_C0\n", "                    Identity(Ad_C0) -> variable\n", "output: name='variable' type=dtype('float32') shape=[None, 1]\n"]}], "source": ["print(onnx_simple_text_plot(onx2))"]}, {"cell_type": "code", "execution_count": 17, "id": "1d4e272f", "metadata": {}, "outputs": [{"data": {"text/plain": ["1.7421041873949668"]}, "execution_count": 18, "metadata": {}, "output_type": "execute_result"}], "source": ["oinf2 = OnnxInference(onx2, runtime='onnxruntime1')\n", "expected = tree.predict(x_exp)\n", "\n", "got = oinf2.run({'X': x_exp.astype(numpy.float32)})['variable']\n", "numpy.abs(got - expected).max()"]}, {"cell_type": "markdown", "id": "f4e64f63", "metadata": {}, "source": ["L'erreur est la m\u00eame."]}, {"cell_type": "markdown", "id": "c9207392", "metadata": {}, "source": ["## Temps de calcul"]}, {"cell_type": "code", "execution_count": 18, "id": "a6febd37", "metadata": {}, "outputs": [], "source": ["x_exp32 = x_exp.astype(numpy.float32)"]}, {"cell_type": "markdown", "id": "1bf0109e", "metadata": {}, "source": ["Tout d'abord le temps de calcul pour scikit-learn."]}, {"cell_type": "code", "execution_count": 19, "id": "07caad53", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["513 \u00b5s \u00b1 7.52 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 1,000 loops each)\n"]}], "source": ["%timeit tree.predict(x_exp32)"]}, {"cell_type": "markdown", "id": "0cea5139", "metadata": {}, "source": ["Le temps de calcul pour l'arbre de d\u00e9cision au format ONNX."]}, {"cell_type": "code", "execution_count": 20, "id": "984413fa", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["186 \u00b5s \u00b1 3.41 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 10,000 loops each)\n"]}], "source": ["%timeit oinf.run({'X': x_exp32})['variable']"]}, {"cell_type": "markdown", "id": "afb4f6bb", "metadata": {}, "source": ["Et le temps de calcul pour le r\u00e9seau de neurones au format ONNX.m"]}, {"cell_type": "code", "execution_count": 21, "id": "e3268dcd", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["3.75 ms \u00b1 311 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 100 loops each)\n"]}], "source": ["%timeit oinf2.run({'X': x_exp32})['variable']"]}, {"cell_type": "markdown", "id": "b3eafba0", "metadata": {}, "source": ["Ce temps de calcul tr\u00e8s long est attendu car le mod\u00e8le contient une multiplication de matrice tr\u00e8s grande et surtout que tous les seuils de l'arbre sont calcul\u00e9s pour chaque observation. L\u00e0 o\u00f9 l'impl\u00e9mentation de l'arbre de d\u00e9cision calcule *d* seuils, la profondeur de l'arbre, la nouvelle impl\u00e9mentation calcule tous les seuils soit $2^d$ pour chaque feuille. Il y a $2^d$ feuilles. M\u00eame en \u00e9tant sparse, on peut r\u00e9duire les calculs \u00e0 $d * 2^d$ ce qui fait encore beaucoup de calculs inutiles."]}, {"cell_type": "code", "execution_count": 22, "id": "d9911fff", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["(126, 11) (126,)\n", "(127, 127) (127,)\n", "(128,) ()\n"]}], "source": ["for node in trees[50].nodes:\n", "    print(node.coef.shape, node.bias.shape)"]}, {"cell_type": "markdown", "id": "27e187ac", "metadata": {}, "source": ["Cela dit, la plus grande matrice est creuse, elle peut \u00eatre r\u00e9duite consid\u00e9rablement."]}, {"cell_type": "code", "execution_count": 23, "id": "e97479fe", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["coef.shape=(126, 11), size dense=1386, size sparse=252, ratio=0.18181818181818182\n", "coef.shape=(127, 127), size dense=16129, size sparse=1015, ratio=0.06293012586025172\n", "coef.shape=(128,), size dense=128, size sparse=127, ratio=0.9921875\n"]}], "source": ["from scipy.sparse import csr_matrix\n", "\n", "for node in trees[50].nodes:\n", "    csr = csr_matrix(node.coef)\n", "    print(f\"coef.shape={node.coef.shape}, size dense={node.coef.size}, \"\n", "          f\"size sparse={csr.size}, ratio={csr.size / node.coef.size}\")"]}, {"cell_type": "code", "execution_count": 24, "id": "125547d9", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["49.8 \u00b5s \u00b1 1.25 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 10,000 loops each)\n"]}], "source": ["r = numpy.random.randn(trees[50].nodes[1].coef.shape[0])\n", "mat = trees[50].nodes[1].coef\n", "%timeit mat @ r"]}, {"cell_type": "code", "execution_count": 25, "id": "ad7173e5", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["7.08 \u00b5s \u00b1 173 ns per loop (mean \u00b1 std. dev. of 7 runs, 100,000 loops each)\n"]}], "source": ["csr = csr_matrix(mat)\n", "%timeit csr @ r"]}, {"cell_type": "markdown", "id": "7599d94e", "metadata": {}, "source": ["Ce serait beaucoup plus rapide avec une matrice sparse et d'autant plus rapide que l'arbre est profond. Le mod\u00e8le ONNX se d\u00e9compose comme suit."]}, {"cell_type": "code", "execution_count": 26, "id": "0c1839fd", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["opset: domain='' version=15\n", "input: name='X' type=dtype('float32') shape=[None, 10]\n", "init: name='Ma_MatMulcst' type=dtype('float32') shape=(1260,)\n", "init: name='Ad_Addcst' type=dtype('float32') shape=(126,)\n", "init: name='Mu_Mulcst' type=dtype('float32') shape=(1,) -- array([4.], dtype=float32)\n", "init: name='Ma_MatMulcst1' type=dtype('float32') shape=(16002,)\n", "init: name='Ad_Addcst1' type=dtype('float32') shape=(127,)\n", "init: name='Ma_MatMulcst2' type=dtype('float32') shape=(127,)\n", "init: name='Ad_Addcst2' type=dtype('float32') shape=(1,) -- array([0.], dtype=float32)\n", "MatMul(X, Ma_MatMulcst) -> Ma_Y02\n", "  Add(Ma_Y02, Ad_Addcst) -> Ad_C02\n", "    Mul(Ad_C02, Mu_Mulcst) -> Mu_C01\n", "      Sigmoid(Mu_C01) -> Si_Y01\n", "        MatMul(Si_Y01, Ma_MatMulcst1) -> Ma_Y01\n", "          Add(Ma_Y01, Ad_Addcst1) -> Ad_C01\n", "            Mul(Ad_C01, Mu_Mulcst) -> Mu_C0\n", "              Sigmoid(Mu_C0) -> Si_Y0\n", "                MatMul(Si_Y0, Ma_MatMulcst2) -> Ma_Y0\n", "                  Add(Ma_Y0, Ad_Addcst2) -> Ad_C0\n", "                    Identity(Ad_C0) -> variable\n", "output: name='variable' type=dtype('float32') shape=[None, 1]\n"]}], "source": ["print(onnx_simple_text_plot(onx2))"]}, {"cell_type": "markdown", "id": "318b95d7", "metadata": {}, "source": ["Voyons comment le temps de calcul se r\u00e9partit."]}, {"cell_type": "code", "execution_count": 27, "id": "11bccd22", "metadata": {}, "outputs": [], "source": ["oinfpr = OnnxInference(onx2, runtime=\"onnxruntime1\",\n", "                      runtime_options={\"enable_profiling\": True})\n", "for i in range(0, 43):\n", "    oinfpr.run({\"X\": x_exp32})"]}, {"cell_type": "code", "execution_count": 28, "id": "5485970b", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>cat</th>\n", "      <th>pid</th>\n", "      <th>tid</th>\n", "      <th>dur</th>\n", "      <th>ts</th>\n", "      <th>ph</th>\n", "      <th>name</th>\n", "      <th>args_op_name</th>\n", "      <th>args_parameter_size</th>\n", "      <th>args_graph_index</th>\n", "      <th>args_provider</th>\n", "      <th>args_exec_plan_index</th>\n", "      <th>args_activation_size</th>\n", "      <th>args_output_size</th>\n", "      <th>args_input_type_shape</th>\n", "      <th>args_output_type_shape</th>\n", "      <th>args_thread_scheduling_stats</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>0</th>\n", "      <td>Session</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>387</td>\n", "      <td>4</td>\n", "      <td>X</td>\n", "      <td>model_loading_array</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>1</th>\n", "      <td>Session</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>2532</td>\n", "      <td>428</td>\n", "      <td>X</td>\n", "      <td>session_initialization</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>2</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>0</td>\n", "      <td>3294</td>\n", "      <td>X</td>\n", "      <td>gemm_fence_before</td>\n", "      <td>Gemm</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>3</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>1315</td>\n", "      <td>3300</td>\n", "      <td>X</td>\n", "      <td>gemm_kernel_time</td>\n", "      <td>Gemm</td>\n", "      <td>5544</td>\n", "      <td>11</td>\n", "      <td>CPUExecutionProvider</td>\n", "      <td>11</td>\n", "      <td>200000</td>\n", "      <td>2520000</td>\n", "      <td>[{'float': [5000, 10]}, {'float': [10, 126]}, ...</td>\n", "      <td>[{'float': [5000, 126]}]</td>\n", "      <td>{'main_thread': {'thread_pool_name': 'session-...</td>\n", "    </tr>\n", "    <tr>\n", "      <th>4</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>0</td>\n", "      <td>4635</td>\n", "      <td>X</td>\n", "      <td>gemm_fence_after</td>\n", "      <td>Gemm</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>...</th>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "      <td>...</td>\n", "    </tr>\n", "    <tr>\n", "      <th>986</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>0</td>\n", "      <td>210170</td>\n", "      <td>X</td>\n", "      <td>Ma_MatMul2_fence_before</td>\n", "      <td>MatMul</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>987</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>124</td>\n", "      <td>210172</td>\n", "      <td>X</td>\n", "      <td>Ma_MatMul2_kernel_time</td>\n", "      <td>MatMul</td>\n", "      <td>508</td>\n", "      <td>8</td>\n", "      <td>CPUExecutionProvider</td>\n", "      <td>8</td>\n", "      <td>2540000</td>\n", "      <td>20000</td>\n", "      <td>[{'float': [5000, 127]}, {'float': [127, 1]}]</td>\n", "      <td>[{'float': [5000, 1]}]</td>\n", "      <td>{'main_thread': {'thread_pool_name': 'session-...</td>\n", "    </tr>\n", "    <tr>\n", "      <th>988</th>\n", "      <td>Node</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>0</td>\n", "      <td>210305</td>\n", "      <td>X</td>\n", "      <td>Ma_MatMul2_fence_after</td>\n", "      <td>MatMul</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>989</th>\n", "      <td>Session</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>4378</td>\n", "      <td>205930</td>\n", "      <td>X</td>\n", "      <td>SequentialExecutor::Execute</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "    <tr>\n", "      <th>990</th>\n", "      <td>Session</td>\n", "      <td>78116</td>\n", "      <td>8820</td>\n", "      <td>4388</td>\n", "      <td>205925</td>\n", "      <td>X</td>\n", "      <td>model_run</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "      <td>NaN</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "<p>991 rows \u00d7 17 columns</p>\n", "</div>"], "text/plain": ["         cat    pid   tid   dur      ts ph                         name  \\\n", "0    Session  78116  8820   387       4  X          model_loading_array   \n", "1    Session  78116  8820  2532     428  X       session_initialization   \n", "2       Node  78116  8820     0    3294  X            gemm_fence_before   \n", "3       Node  78116  8820  1315    3300  X             gemm_kernel_time   \n", "4       Node  78116  8820     0    4635  X             gemm_fence_after   \n", "..       ...    ...   ...   ...     ... ..                          ...   \n", "986     Node  78116  8820     0  210170  X      Ma_MatMul2_fence_before   \n", "987     Node  78116  8820   124  210172  X       Ma_MatMul2_kernel_time   \n", "988     Node  78116  8820     0  210305  X       Ma_MatMul2_fence_after   \n", "989  Session  78116  8820  4378  205930  X  SequentialExecutor::Execute   \n", "990  Session  78116  8820  4388  205925  X                    model_run   \n", "\n", "    args_op_name args_parameter_size args_graph_index         args_provider  \\\n", "0            NaN                 NaN              NaN                   NaN   \n", "1            NaN                 NaN              NaN                   NaN   \n", "2           Gemm                 NaN              NaN                   NaN   \n", "3           Gemm                5544               11  CPUExecutionProvider   \n", "4           Gemm                 NaN              NaN                   NaN   \n", "..           ...                 ...              ...                   ...   \n", "986       MatMul                 NaN              NaN                   NaN   \n", "987       MatMul                 508                8  CPUExecutionProvider   \n", "988       MatMul                 NaN              NaN                   NaN   \n", "989          NaN                 NaN              NaN                   NaN   \n", "990          NaN                 NaN              NaN                   NaN   \n", "\n", "    args_exec_plan_index args_activation_size args_output_size  \\\n", "0                    NaN                  NaN              NaN   \n", "1                    NaN                  NaN              NaN   \n", "2                    NaN                  NaN              NaN   \n", "3                     11               200000          2520000   \n", "4                    NaN                  NaN              NaN   \n", "..                   ...                  ...              ...   \n", "986                  NaN                  NaN              NaN   \n", "987                    8              2540000            20000   \n", "988                  NaN                  NaN              NaN   \n", "989                  NaN                  NaN              NaN   \n", "990                  NaN                  NaN              NaN   \n", "\n", "                                 args_input_type_shape  \\\n", "0                                                  NaN   \n", "1                                                  NaN   \n", "2                                                  NaN   \n", "3    [{'float': [5000, 10]}, {'float': [10, 126]}, ...   \n", "4                                                  NaN   \n", "..                                                 ...   \n", "986                                                NaN   \n", "987      [{'float': [5000, 127]}, {'float': [127, 1]}]   \n", "988                                                NaN   \n", "989                                                NaN   \n", "990                                                NaN   \n", "\n", "       args_output_type_shape  \\\n", "0                         NaN   \n", "1                         NaN   \n", "2                         NaN   \n", "3    [{'float': [5000, 126]}]   \n", "4                         NaN   \n", "..                        ...   \n", "986                       NaN   \n", "987    [{'float': [5000, 1]}]   \n", "988                       NaN   \n", "989                       NaN   \n", "990                       NaN   \n", "\n", "                          args_thread_scheduling_stats  \n", "0                                                  NaN  \n", "1                                                  NaN  \n", "2                                                  NaN  \n", "3    {'main_thread': {'thread_pool_name': 'session-...  \n", "4                                                  NaN  \n", "..                                                 ...  \n", "986                                                NaN  \n", "987  {'main_thread': {'thread_pool_name': 'session-...  \n", "988                                                NaN  \n", "989                                                NaN  \n", "990                                                NaN  \n", "\n", "[991 rows x 17 columns]"]}, "execution_count": 29, "metadata": {}, "output_type": "execute_result"}], "source": ["df = oinfpr.get_profiling(as_df=True)\n", "df"]}, {"cell_type": "code", "execution_count": 29, "id": "19bb5d0f", "metadata": {}, "outputs": [{"data": {"text/plain": ["{'CPUExecutionProvider', nan}"]}, "execution_count": 30, "metadata": {}, "output_type": "execute_result"}], "source": ["set(df['args_provider'])"]}, {"cell_type": "code", "execution_count": 30, "id": "e42d5644", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th></th>\n", "      <th>dur</th>\n", "    </tr>\n", "    <tr>\n", "      <th>args_op_name</th>\n", "      <th>name</th>\n", "      <th></th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>MatMul</th>\n", "      <th>Ma_MatMul2</th>\n", "      <td>6778</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul</th>\n", "      <td>12923</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid</th>\n", "      <td>14849</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul1</th>\n", "      <td>15151</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid1</th>\n", "      <td>15608</td>\n", "    </tr>\n", "    <tr>\n", "      <th rowspan=\"2\" valign=\"top\">Gemm</th>\n", "      <th>gemm</th>\n", "      <td>31763</td>\n", "    </tr>\n", "    <tr>\n", "      <th>gemm_token_0</th>\n", "      <td>99047</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["                             dur\n", "args_op_name name               \n", "MatMul       Ma_MatMul2     6778\n", "Mul          Mu_Mul        12923\n", "Sigmoid      Si_Sigmoid    14849\n", "Mul          Mu_Mul1       15151\n", "Sigmoid      Si_Sigmoid1   15608\n", "Gemm         gemm          31763\n", "             gemm_token_0  99047"]}, "execution_count": 31, "metadata": {}, "output_type": "execute_result"}], "source": ["dfp = df[df.args_provider == 'CPUExecutionProvider'].copy()\n", "dfp['name'] = dfp['name'].apply(lambda s: s.replace(\"_kernel_time\", \"\"))\n", "gr_dur = dfp[['dur', \"args_op_name\", \"name\"]].groupby([\"args_op_name\", \"name\"]).sum().sort_values('dur')\n", "gr_dur"]}, {"cell_type": "code", "execution_count": 31, "id": "34b33616", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th></th>\n", "      <th>dur</th>\n", "    </tr>\n", "    <tr>\n", "      <th>args_op_name</th>\n", "      <th>name</th>\n", "      <th></th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>MatMul</th>\n", "      <th>Ma_MatMul2</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul1</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid1</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th rowspan=\"2\" valign=\"top\">Gemm</th>\n", "      <th>gemm</th>\n", "      <td>43</td>\n", "    </tr>\n", "    <tr>\n", "      <th>gemm_token_0</th>\n", "      <td>43</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["                           dur\n", "args_op_name name             \n", "MatMul       Ma_MatMul2     43\n", "Mul          Mu_Mul         43\n", "Sigmoid      Si_Sigmoid     43\n", "Mul          Mu_Mul1        43\n", "Sigmoid      Si_Sigmoid1    43\n", "Gemm         gemm           43\n", "             gemm_token_0   43"]}, "execution_count": 32, "metadata": {}, "output_type": "execute_result"}], "source": ["gr_n = dfp[['dur', \"args_op_name\", \"name\"]].groupby([\"args_op_name\", \"name\"]).count().sort_values('dur')\n", "gr_n = gr_n.loc[gr_dur.index, :]\n", "gr_n"]}, {"cell_type": "code", "execution_count": 32, "id": "f34b2908", "metadata": {}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzsAAAEICAYAAAB4XuoFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABUxklEQVR4nO3deZwU1bn/8c93BphhB8EoiHEgIiKgiLgrggtKVBDimtwIGvXmqjEhGn8mMTcmehNzE8U1ixokclXcAPddFMGIbMqiwQVJRDEKIsoyLMPz+6Nrxp5hYHqYpWea7/v1mhddp6pOPdWtp/rpc+qUIgIzMzMzM7Nck5ftAMzMzMzMzOqCkx0zMzMzM8tJTnbMzMzMzCwnOdkxMzMzM7Oc5GTHzMzMzMxykpMdMzMzMzPLSU52zKzeSBon6Zp6OtZ3JD1TH8cyMzOzhsnJjpk1epKKJIWkJqVlEXF3RAzOZlxmZmaWXU52zKzBk5Sf7RjMzKx2pf9AZVZXnOyYWZ2RtL+kOZK+lHQfUJiUj5I0rcK2IWnP5PU4SX+S9ISkNcAgSSdKmivpC0kfSLoqbfepyb+fS1ot6dCKx5B0mKSZklYl/x6Wtu5FSVdLmp7E+oykjnX0tpiZ1RlJSyRdJmle0t7dJ6lwK9vmSbpS0j8lfSLpLklt09YfIekVSZ8n7e6opLy5pOuS/VZJmpaUDZS0tJJ4jk1eXyXpQUn/J+kLYJSktpL+KmmZpA8lXVP6A1dpOy7pD5JWSnpf0pC0uneSdKekj5L1k9PWnSTp9ST2VyTtm7bu/yXH+lLSIknH1Mqbbw2Skx0zqxOSmgGTgfHATsADwLeqUcW3gf8BWgPTgDXA2UA74ETgvySdkmw7IPm3XUS0ioi/V4hlJ+Bx4CagA3A98LikDhWOdw7wNaAZcFk1YjUza0hOB04AugL7AqO2st2o5G8Q0A1oBdwCIGkP4EngZmBnoC/werLfH4ADgMNIte+XA5szjG0Y8CCptvxuYBywCdgT2B8YDJyXtv3BwCKgI/C/wF8lKVk3HmgB9CLVdo9JYt8fGAv8J6k2/y/AI5IKJPUALgYOjIjWwPHAkgxjt0bIyY6Z1ZVDgKbADRGxMSIeBGZWY/+HI2J6RGyOiOKIeDEi5ifL84B7gaMyrOtE4J2IGB8RmyLiXuAfwMlp29wZEW9HxDrgflIXdjOzxuimiPgoIj4DHmXr7dl3gOsjYnFErAZ+CpyZDC/7NvBcRNybtOErIuJ1SXnAucAPI+LDiCiJiFciYn2Gsf09IiZHxGagDfBN4EcRsSYiPiGVsJyZtv0/I+L2iCgB/gZ0AnaR1AkYAnw/IlYmMb6U7HMB8JeImJHE9zdgPanrUglQAOwjqWlELImI9zKM3RohJztmVlc6Ax9GRKSV/bMa+3+QviDpYElTJH0qaRXwfVK/9GUaS8Vj/xPYLW3547TXa0n9wmlm1hhl2p5VbBv/CTQBdgF2BypLAjqSGpK8vQlCetu+B6kfxZYlw80+J9UL87W0bcrOJSLWJi9bJfF9FhErKznGHsClpXUm9e4OdI6Id4EfAVcBn0iaIKnzdp6LNQJOdsysriwDdksbbgDw9eTfNaSGHgAgaddK9o8Ky/cAjwC7R0Rb4M+AtrJtRR+Ruvil+zrwYRX7mZnlsopt49dJDSn7N6mk5BuV7LMcKN7Kuoptez6pIXDp0tvrD0j1uHSMiHbJX5uI6JVB7B8AO0lqt5V1/5NWZ7uIaJH06hMR90TEEaTOPYDfZXA8a6Sc7JhZXfk7qYvmJZKaShoBHJSsewPoJalvcuPsVRnU15rUr3jFkg4iNcSi1Kekxot328q+TwB7Sfq2pCaSzgD2AR6r9lmZmeWOe4HRkrpKagX8BrgvIjaRup/mWEmnJ+1mB0l9k+FnY4HrJXWWlJ9MClMAvA0UKjWhTFPgSlJDxioVEcuAZ4DrJLVJJkz4hqQqhygn+z4J/FFS++Q6U3r/5u3A95MRAZLUMomptaQeko5O4i0G1pH5/UbWCDnZMbM6EREbgBGkbn79DDgDmJisexv4NfAc8A6pCQiqciHwa0lfAv9N6r6a0mOtJTWZwfRkyMIhFWJZAZwEXAqsIHUz7UkRsbwGp2hm1tiNJXWT/1TgfVJf/n8AEBH/InU/zaWk2vDXgf2S/S4D5pO6D/MzUj0jeRGxilRbfQepnvM1QLnZ2SpxNqlJYd4EVpKavKBThvF/F9hI6h7MT0gNTyMiZgHnk5psYSXwLl9N0lAAXEuqh+pjUkPmfprh8awRUvnh9GZmZmZmZrnBPTtmZmZmZpaTnOyYmZmZmVlOcrJjZmZmZmY5ycmOmZmZmZnlpCbZDsByV8eOHaOoqCjbYZiZVdvs2bOXR0TF54PkNLfZZtaYba3ddrJjdaaoqIhZs2ZlOwwzs2qT9M+qt8otbrPNrDHbWrvtYWxmZmZmZpaTnOyYmZmZmVlOcrJjZmZmZmY5yffsmJmZmZnlqI0bN7J06VKKi4uzHUqtKCwspEuXLjRt2jSj7Z3smJmZmZnlqKVLl9K6dWuKioqQlO1waiQiWLFiBUuXLqVr164Z7eNhbGZmZmZmOaq4uJgOHTo0+kQHQBIdOnSoVi+Ve3aszsz/cBVFVzye7TDMbAe25NoTsx1Co+E22yw33T60Exs/XJXtMDKyb5d2VW5T3aTNPTtmZmZmZpaT3LNjZmZmZraDGHrL9Fqt75GLD6/W9n+6/lpatGjJyO//oFbj2Br37JiZmZmZWYOzadOmGtfhZMfMzMzMzOrM7Tf9gZMH9GfkiBNY8t47AHzvtJNY+MZcAFZ+toIhh+4LwLhx4xg6dChHH300xxxzTI2P7WFsZmZmZmZWJ96c9zpPPTKR+5+eSsmmTZw5ZCD79Om7zX3mzJnDvHnz2GmnnWp8/Drr2ZHUXNJLkvKT5e6SHpP0nqTZkqZIGlBXx2/MJLWTdGEG2w2U9FgdxvFTSe9KWiTp+KSsmaSpkpwom+WQzRvX8/E9VxCbSwDY+NmHfPLgr/jwL+exbNwP+fjen1L8wYIsR9kwuc02M9u6Oa/9naNPOInmzVvQqnUbjjpuSJX7HHfccbWS6EDd9uycC0yMiBJJhcDjwGUR8QiApN5Af2BqHcbQWLUDLgT+mK0AJO0DnAn0AjoDz0naKyI2SHoeOAO4O1vxmVntWjP/WVrsdSjKyyc2beCTB39F+0Hfo0X3gwHY8OkSNnz8LuzeO8uRNkjtcJttZlYt+U2asHnzZgA2rC//3JyWLVvW2nHqMtn5DvDttNd/L010ACJiAbAAQFJL4GagN9AUuCoiHpY0CjgFaAl0B/4ANAO+C6wHvhkRn0l6EZgLHJlsezbwU6APcF9EXLmtQCV9E7geWANMB7pFxElZjOta4BuSXgeeBS4H/hcYAgRwTUTcV+EcDgRuA04ldeG9HmgFLAdGRcSyJJ4ZwKBkm+9FxMtbiWEYMCEi1gPvS3oXOAj4OzAZ+C2VXDglXQBcAJDfZuetVG1mDc2ahS/ScehPAFi98EUKdtu7LNEBaLZzEc12LgJg84ZiPnvuz2xc/k+ipIR2R3ybFt0PYfX851j7zqvEhmI2rvyINgcNh5JNrF44BeU35WunXUV+89Z8fM8VNNvlG6xfupDNG4rpeNKPWfXqA2z8dAkt9h5A+wHf3Was696byWcv/JW8pgUUdNmHTZ9/zNdO/WWlccGJjBs3jsmTJ7NmzRreeecdLrvsMjZs2MD48eMpKCjgiSeeYKeddmLgwIHsv//+vPzyywC9knbVbbaZWQ0ccPBh/OLHF/K9i0ZTUrKJqc89xanfGUXnLl/nrfmv02f/A3j28Ueqrmg71UmyI6kZqYRhSVLUC5izjV1+DrwQEedKage8Jum5ZF1vYH+gEHgX+H8Rsb+kMaSShxuS7TZERH9JPwQeBg4APgPekzQmIlZsJdZC4C/AgIh4X9K9DSCuK4DeEdE3ifFbQF9gP6AjMFNSWY+YpMNIJWXDgGXAeGBYRHwq6Qzgf0j1tAE0iYiDkgTvl8Cxlb0vwG7Aq2nLS5MySCWpB1a2U0TcRuoCTkGn7rGVus2sAYmSjWxc9TFN2u4CwMbl/6TZLt/Y6var/n4fhXvsR8dv/ojNxatZdtePKdyjb2rfT/9Jp1E3EiUb+fAv59N+4Cg6n3MTnz1/O2sWvECbA4cBoPwmdBp5A1/MephPJ17DriNvIL+wNR/+5TzaHDiM/OZtKo910wZWPH0ru3z7Wpq225VPH/nfbca1Zs0VACxYsIC5c+dSXFzMnnvuye9+9zvmzp3L6NGjueuuu/jRj34EQLNmzZg1axaSPsVttpnloOpOFV1TPfvsx/EnD+e0449kp44d6bXf/gCM/M+L+cl/ncOD9/yNAUcPrrPj11XPTkfg862tlDSJVI/I2xExAhgMDJV0WbJJIfD15PWUiPgS+FLSKuDRpHw+sG9atY+klS+MiGXJsRYDuwOVJjvA3sDiiHg/Wb6X5FeuLMeV7gjg3ogoAf4t6SVSF64vgJ6kLlSDI+KjZHhgb+DZ5Amz+aQupqUmJv/OBooyOPYWkqGJGyS1Tt4DM2vEStZ+QV5Bq62u/2TiNWxa+RFNdtqNrw3/OcVL5rLu3Rl88VqqOYlNGyn54lMACr/eh7yCFgDkFbSk+Z4HAdBs5z3Y8MmSsjqb75nqNWq6cxFNO3ydJq1SY7ObtNuVki+WbzXZ2bhiKU3a7UrTdrsC0LLnUax+4ymASuP617/+BcCgQYNo3bo1rVu3pm3btpx88skA9OnTh3nz5pXVP3To0NKX64B/uc02M6u58y+5jPMvuWyL8gef/eqZPxdfnuo8HzVqFKNGjaq1Y9dVsrOOVGJQaiFQNhlBRAyX1J/U8C8AAd+KiEXplUg6mNSwsFKb05Y3Uz7+9ZVsU9l21dFQ40q3jNR7vT/wURLzwog4dCvbl8ZQUsXxPyR1YS/VJSkrVQCUH2BpZo2SmhYQmzaULTftuAfrl341GcHXRlzJ+mXvsHLKX1MFEex8ys9o2qFLuXrWL1uEmjRNq1goP1lWHkTJV6uScklb7FM6SUK1VRJXz549mTFjBgUFBWVleXl5Zct5eXnlnuOQvh1us83MGr06mY0tIlYC+ckQMYB7gMMlDU3brEXa66eBHyj5WUvS/nURl6TnJe1WoXgR0E1SUbJ8Rn3HVYkvgdZpyy8DZ0jKl7QzqcTxtWTd58CJwG8lDSR1PjtLOjSJuamkXtsRwyPAmZIKJHUl1RP3WlJnB2B5RGzcjnrNrIHJL2wFsbks4Wm5z1GsX/oWa9+ZUbZNbPrqe39h1358OedRIlKjnjb8+706ievfE37Gpi+XlytrstNubPr8Yzat+jcAa/7x1Rw39RVXJdxmm5k1UHU5QcEzpLryn4uIdZJOAq6XdAPwb1IXh2uSba8mdY/LPEl5wPvASbUZTFLvnqTGXpdJYrsQeErSGmBm2uo6j6syEbFC0nRJC4AnSd3seijwBqmbXS+PiI8l7Z1s/+/k/X2S1DjvU4GbJLUl9RnfQKp3rToxLJR0P/AmsAm4KBmSAambZR+v4WmaWQNSWLQ/xUvfpHlRX/KaFvC1U/+bz164g5XP30Zey3bkNWtB28POBKDtYWey8vnbWTb2YoigSbtd+Nqpv6zVeCI2s3HlMvIKW5crz2tawE6D/4t/3/9L8poW0KxTdzYn6yqLCy6p1bgqj9Vttpk1XEEQESS/3Td6pT9oZUrV3SHjiqV+wOiI2Pa0OvUkGRd9bkT8uJJ1rSJiddKDcyvwTkSMqfcgGwlJE4ErIuLtbW1X0Kl7dBp5Q/0EZWY1sv7jd/ly1sN0POnSbIcCpKa6Xj3vWXY65vwt1m3esI68Zs2JCD579k80bd+ZNgeeUmk9S649cbuOL2l2RPTfrp0bGLfZZju2nw/oQL9vdKJJizYNPuHZt0u7ba6PCFasWMGXX35J165dy63bWrtdZz07ETFHqQeH5qf9upQ1yVTXWyQ6ifMljSQ1ffRcUrOzWSWSmfYmV3XRNLPGpWDXPdn49T7E5hKUl5/tcGi2c1GliQ7A6jeeZvWC56FkE0136UarvifUc3SNh9tsM7t5xkp+AOzRbjmiYSc7b33ZvMptCgsL6dKlS5Xblaqznh2rWjKO+vlKVh2ztamy6yCG44HfVSh+PyKG17Tu/v37x6xZs2pajZlZvavsF0K32WZmDVe99+xY1ZKLY98sx/A0qYkYzMxsG9xmm5k1PnUyG5uZmZmZmVm2OdkxMzMzM7Oc5GTHzMzMzMxykpMdMzMzMzPLSU52zMzMzMwsJznZMTMzMzOznORkx8zMzMzMcpKTHTMzMzMzy0lOdszMzMzMLCc52TEzMzMzs5zkZMfMzMzMzHKSkx0zMzMzM8tJTnbMzMzMzCwnNcl2AJa75n+4iqIrHs92GDuUJdeemO0QzKyRcpttZtlWF99j3LNjZmZmZmY5ycmOmZmZmZnlJCc7ZmZmZmaWk5zsmJmZmZlZTnKyk5DUXNJLkvKT5e6SHpP0nqTZkqZIGpDtOLNNUjNJUyV5cgszy5p169Zx1FFHUVJSAsA777zDSSedxDe+8Q0OOOAABg0axNSpU7McZfa5zTazHZ2Tna+cC0yMiBJJhcDjwG0R8Y2IOAD4AdAtqxE2ABGxAXgeOCPbsZjZjmvs2LGMGDGC/Px8iouLOfHEE7ngggt47733mD17NjfffDOLFy/OdphZ5zbbzHZ0Tna+8h3g4bTXf4+IR0pXRsSCiBgHIKmlpLGSXpM0V9KwpHyUpMmSnpW0RNLFkn6cbPOqpJ2S7V6UNEbSLElvSTpQ0kRJ70i6pqpAJX1T0j+SHqebJD1Wz3FNTt4jM7OsuPvuuxk2bFjZ60MPPZShQ4eWre/duzejRo0CYM2aNZx77rkcdNBB7L///jz8cKqpHzduHKeccgrHHXccRUVF3HLLLVx//fXsv//+AHu7zTYza/yc7JDq5ge6RcSSpKgXMGcbu/wceCEiDgIGAb+X1DJZ1xsYARwI/A+wNiL2B/4OnJ1Wx4aI6A/8mVSSdVGy7yhJHbYRayHwF2BI0uO0cxbiWpDUU1l8FyQX3lkla1dt7TTMzLbbhg0bWLx4MUVFRQAsXLiQfv36bXX7//mf/+Hoo4/mtddeY8qUKfzkJz9hzZo1ACxYsICJEycyc+ZMfv7zn9OiRQvmzp0LsAa32WZmjZ6TnZSOwOdbWylpkqQFkiYmRYOBKyS9DrwIFAJfT9ZNiYgvI+JTYBXwaFI+HyhKq/aRtPKFEbEsItYDi4HdtxHr3sDiiHg/Wb43bV29xBURJcAGSa0rBhcRt0VE/4jon9+i7TZOw8xs+yxfvpx27dptdf3w4cPp3bs3I0aMAOCZZ57h2muvpW/fvgwcOJDi4mL+9a9/ATBo0CBat27NzjvvTNu2bTn55JNLq1mL22wzs0bPNyymrCN1kSm1ECibjCAihkvqD/whKRLwrYhYlF6JpIOB9WlFm9OWN1P+/V5fyTaVbVcd9RlXAVC8nXGamW235s2bU1z8VfPTq1evcpMRTJo0iVmzZnHZZZcBEBE89NBD9OjRo1w9M2bMoKCgoGw5Ly+v3DJus83MGj337AARsRLIT4YbANwDHC5paNpmLdJePw38QJIAJO1fF3FJel7SbhWKFwHdJBUly+k3ndZXXB2A5RGxsS7qNzPblvbt21NSUlKW8Hz7299m+vTpPPJI2W2WrF27tuz18ccfz80330xEAJQOU6t1brPNzBoeJztfeQY4AiAi1gEnAd+XtFjS34ErgdIbPq8GmgLzJC1MlmuVpDxgT+Cz9PIktguBpyTNBr4kNcShXuJKDCI1W52ZWVYMHjyYadOmAamenscee4w///nPdOvWjUMPPZRrrrmGK6+8EoBf/OIXbNy4kX333ZdevXrxi1/8otbjcZttZtYwqfSXrh2dpH7A6Ij4brZjAZDUGzg3In5cybpWEbE6+TXwVuCdiBhTj7FNBK6IiLe3tV1Bp+7RaeQN9ROUAbDk2hOzHYJZvZgzZw5jxoxh/PjxdVK/pNnJjf+Zbu8228yshmryPWZr7bZ7dhIRMQeYouShotmWTHW9xUUzcX5yQ+tCoC2pmX7qRTJz3eSqLppmZnWpX79+DBo0qOyhotnmNtvMrGHyBAVpImJstmPIRPKLYL39Kljh2BuAu7JxbDOzdOeee262Q8iI22wzs+xxsmN1ps9ubZnlYVVmZo2C22wzy0UexmZmZmZmZjnJyY6ZmZmZmeUkJztmZmZmZpaTnOyYmZmZmVlOyjjZkdSiLgMxM7Pas3bt2myHYGZmlnVVJjuSDpP0JvCPZHk/SX+s88jMzKzaXnnlFfbZZx/23ntvAN544w0uvPDCLEdlZmaWHZn07IwBjgdWAETEG8CAugzKzMy2z+jRo3n66afp0KEDAPvttx9Tp07NclRmZmbZkdEwtoj4oEJRw3hktZmZbWH33Xcvt5yfn5+lSMzMzLIrk4eKfiDpMCAkNQV+CLxVt2GZmdn22H333XnllVeQxMaNG7nxxhvp2bNntsMyMzPLikx6dr4PXATsBnwI9E2Wzcysgfnzn//Mrbfeyocffshuu+3G66+/zq233prtsMzMzLKiyp6diFgOfKceYjEzsxrq2LEjd999d7bDMDMzaxCqTHYkdQV+ABSlbx8RQ+suLDMz2x7vv/8+N998M0uWLGHTpk1l5Y888kgWozIzM8uOTO7ZmQz8FXgU2Fyn0ZiZWY2ccsopfO973+Pkk08mL8/PjTYzsx1bJslOcUTcVOeRmJlZjRUWFnLJJZdkOwwzM7MGQRGx7Q2kbwPdgWeA9aXlETGnbkOzxq6gU/foNPKGbIdRbUuuPTHbIZhtt3vuuYd33nmHwYMHU1BQUFber1+/LEbV+EiaHRH9sx1HfWqsbbaZ5Y6afAfbWrudSc9OH+C7wNF8NYwtkmUzM2tA5s+fz/jx43nhhRfKhrFJ4oUXXshyZGZmZvUvk2TnNKBbRGyo62DMzKxmHnjgARYvXkyzZs2yHYqZmVnWZXL36gKgXR3HYWZmtaB37958/vnn2Q7DzMysQcikZ6cd8A9JMyl/z46nnjYza2A+//xz9t57bw488MBy9+x46mkzM9sRZZLs/LLOozAzs1rxq1/9KtshmJmZNRhVDmOLiJcq+6vpgSU1l/SSpHxJeZJukrRA0nxJM5OHmSLpCUntanq8KmJ5ZSvl4ySdWsW+PSS9KOl1SW9Jui0p7y9pq1N2S9pF0mOS3pD0pqQnkvLOkh6syflUZVuxSVoiqWPyeqykTyQtqLDNHyR5ggqzBuioo46q9K+m1q1bx1FHHUVJSQmbN2/mkksuoXfv3vTp04cDDzyQ999/H4BvfvObdT6M7rDDDqu0fNSoUTz44Labz0WLFjFw4ED69u1Lz549ueCCCwCYNWvWNqfsdpttZtY4VdmzI+kQ4GagJ9AMyAfWRESbGh77XGBiRJRIOgvoDOwbEZsldQHWAETEN2t4nCpFROVXzszcBIyJiIcBJPVJ6pwFzNrGfr8Gno2IG5P99k32+wjYZoJVUxnEVmoccAtwV4Xym4HbAU/vZNbAvPrqq/zgBz/grbfeYsOGDZSUlNCyZUu++OKLGtU7duxYRowYQX5+Pvfeey8fffQR8+bNIy8vj6VLl9KyZUsAnnjiido4jW165ZVKf5/KyCWXXMLo0aMZNmwYkJq9DqB///7077/NmabdZpuZNUKZTFBwC3AW8A7QHDgPuLUWjv0d4OHkdSdgWURsBoiIpRGxErb41eoXkhZJmibpXkmXJeUvShojaVbSu3KgpImS3pF0TekBJf046T1aIOlHaeWrk38l6ZbkGM8BX8vgPDoBS0sXImJ+UtdASY9VY795yX5Fpb/KSWoh6f7kV8RJkmZI6l8as6TfS1oo6TlJByXvw2JJQ5NtCiXdmfSWzZU0qGJskjpIeiap5w5AaTFNBT6rGHhE/BPoIGnXiuskXZB8DrNK1q7K4O0zs9p08cUXc++999K9e3fWrVvHHXfcwUUXXVTjeu++++6yBGHZsmV06tSpbGrrLl260L59ewCKiopYvnw5AFdffTU9evTgiCOO4KyzzuIPf/gDAAMHDmT06NH079+fnj17MnPmTEaMGEH37t258sory455/fXX07t3b3r37s0NN9xQVt6qVSsAIoKLL76YHj16cOyxx/LJJ59UeR7Lli2jS5cuZct9+vQB4MUXX+Skk07a1q5us83MGqFMkh0i4l0gPyJKIuJO4ISaHFRSM1LTWS9Jiu4HTlZqKNh1kvavZJ8DgW8B+wFDgIo/wW1IHiT0Z1JJ1EVAb2BUcnE4ADgHOBg4BDi/kuMMB3oA+wBnA5n0+IwBXpD0pKTRynzI3a3AXyVNkfRzSZ0r2eZCYGVE7AP8AjggbV1L4IWI6AV8CVwDHJecw6+TbS4CIiL6kEpY/yapsMIxfglMS+qZBHw9w/jnAIdXLIyI2yKif0T0z2/RNsOqzKw27bnnnpSUlJCfn88555zDU089VaP6NmzYwOLFiykqKgLg9NNP59FHH6Vv375ceumlzJ07d4t9Zs6cyUMPPcQbb7zBk08+yaxZ5TsmmjVrxqxZs/j+97/PsGHDuPXWW1mwYAHjxo1jxYoVzJ49mzvvvJMZM2bw6quvcvvtt29xnEmTJrFo0SLefPNN7rrrrox6fEaPHs3RRx/NkCFDGDNmTHWG3LnNNjNrhDJJdtYmycnrkv5X0ugM99uWjsDnpQsRsZRUkvFTUg8ufV7SMRX2ORx4OCKKI+JL4NEK60unGpoPLIyIZRGxHlgM7A4cAUyKiDURsRqYCBxZoY4BwL1JUvcRGXT5J8lfT+ABYCDwqqSCbe6U2u9poBupoQV7A3Ml7VxhsyOACcn2C4B5aes2AKXfYOYDL0XExuR1Udr+/5fs/w/gn8BeFY4xIG2bx4GVVcWe+ITU0EMza0BatGjBhg0b6Nu3L5dffjljxoxh8+bNVe+4DcuXL6ddu3Zly126dGHRokX89re/JS8vj2OOOYbnn3++3D7Tp09n2LBhFBYW0rp1a04++eRy64cOTU3o2adPH3r16kWnTp0oKCigW7dufPDBB0ybNo3hw4fTsmVLWrVqxYgRI3j55ZfL1TF16lTOOuss8vPz6dy5M0cfXfVtKeeccw5vvfUWp512Gi+++CKHHHII69evr3I/t9lmZo1TJknLd5PtLiZ1H83upHpYamIdUO7XqohYHxFPRsRPgN8Ap1SzztKr1ea016XLmcw6t90i4qOIGBsRw4BNpHqUMtnvs4i4JyK+C8wkdRHL1MaIiOR12TknQwHr9HwThaQ+RzNrQMaPH8/mzZu55ZZbaNmyJR988AEPPfRQjeps3rw5xcXF5coKCgoYMmQIv//97/nZz37G5MmTq1Vn6bTYeXl55abIzsvLY9OmTTWKtyqdO3fm3HPP5eGHH6ZJkyYsWLCg6p1wm21m1hhlMhvbP5PelC8i4lcR8eNkWNt2S+7HyS/tnpfUr3RIgKQ8YF9Sv2ilm05qqFuhpFbANgdXV+Jl4JRkTHVLUkMHXq6wzVTgDKVmiOsEDCpdIem3koZXrFTSCZKaJq93BToAH1YVjKSjJbVIXrcGvgH8q8Jm04HTk232AfpkdKZfeZnUvVFI2ovUcIdFFbaZCnw72WYI0D7Duvci9cBZM2tA9thjDwoLC2nTpg2//OUvuf7669lzzz1rVGf79u0pKSkpS3jmzJnDRx99BMDmzZuZN28ee+yxR7l9Dj/8cB599FGKi4tZvXo1jz22rVsYt3TkkUcyefJk1q5dy5o1a5g0aRJHHlm+M37AgAHcd999lJSUsGzZMqZMmVK27qc//SmTJk3aot6nnnqKjRs3AvDxxx+zYsUKdttttyrjcZttZtY4VZnsSDpc0rOS3k5upFwsaXEtHPsZUl32kJoI4NHkJs95pHpHbknfOCJmkhqqNg94klTXf8Z3U0bEHFIz1bwGzADuiIiKA80nkZqI4U1Ss9n8PW1dH+DjSqoeDCyQ9AbwNPCTiKhsu4oOAGZJmpcc547kHNP9EdhZ0pukxncvpBrnnOyfJ2k+cB8wKhnal+5XwABJC4ERpF28Jd2bxNZD0lJJ30vKmwJ7ktnsQGZWj6ZPn85xxx3HXnvtRbdu3cr+amrw4MFMmzYNgE8++YSTTz6Z3r17s++++9KkSRMuvvjictsfeOCBDB06lH333ZchQ4bQp08f2rbN/J6Qfv36MWrUKA466CAOPvhgzjvvPPbfv/xtlsOHD6d79+7ss88+nH322Rx66KFl6+bPn8+uu25xPz7PPPMMvXv3Zr/99uP444/n97//faXbVcJttplZI6SvetW3soH0D2A0MBsoKS2PiBU1OrDUDxidDAfIdJ9WEbE6+XVtKnBBksTUOUlPR8Tx9XGstGPmA00joljSN4DngB4RsaE+46gkruFAv4j4xba2K+jUPTqNvKF+gqpFS649MdshmG23vffemzFjxnDAAQeQn59fVt6hQ4ca1TtnzhzGjBnD+PHjM95n9erVtGrVirVr1zJgwABuu+02+vXrV6M4MnX88cfz9NNPb/f+kmYnk95UZx+32WZmNVCT72Bba7czGSe8KiKe3O4jb0VEzElmtcmPiJKq9wDgtmRoQCHwt/pKdADqO9FJtACmJL/KCbgw2xfNRBPgumwHYWZbatu2LUOGDKn1evv168egQYPKZnnLxAUXXMCbb75JcXExI0eOrLdEB6hRolMDbrPNzBqYTHp2riX1INGJpN34X5+JRmMm6RzghxWKp0dEzR980cD1798/Kk43a2Z164orrqCkpIQRI0aUu/G/PhONxuzOO+/kxhtv5I033lgHvJ0Uu802M2vgttazk0myM6WS4oiIquf4tB2aL5xm9W/QoEFblEnihReqnEnf0mzPMLbGzm22mTVm2z2MLSK2vHKamVmDlD4jmZmZ2Y6u2g8HlTRM0sF1EYyZmdWuhx9+mBkzZmQ7DDMzs6zYngeZHQz0kdQkImr/LlgzM6s1M2bMYP78+WzatIknn6z1uWbMzMwatGonOxHxs7oIxMzMat9vfvObbIdgZmaWNZk8VLSFpF9Iuj1Z7i7ppLoPzczMqmvt2rVcffXVnH/++QC88847PPbYY1mOyszMLDsyuWfnTlJTTpc+mvpDUk+GNjOzBuacc86hoKCAv//97wDstttuXHnllVmOyszMLDsySXa+ERH/C2wEiIi1pB6WZmZmDcx7773H5ZdfTtOmTQFo0aIFVT1iwMzMLFdlkuxskNQcCABJ3yDt4aJmZtZwNGvWjHXr1iGlfpN67733yj1c1MzMbEeSyQQFvwSeAnaXdDdwODCqLoMyM7Pt86tf/YoTTjiBDz74gO985ztMnz6dcePGZTssMzOzrMjkoaLPSpoDHEJq+NoPI2J5nUdmZmbVdtxxx9GvXz9effVVIoIbb7yRjh07ZjssMzOzrMh06undgPxk+wGSiIiJdReWmZltrw8//JCSkhI2bdrE1KlTARgxYkSWozIzM6t/VSY7ksYC+wILgc1JcQBOdszMGphzzz2XefPm0atXL/LyUrdlSnKyY2ZmO6RMenYOiYh96jwSMzOrsVdffZU333wz22GYmZk1CJkkO3+XtE9E+Opp1TL/w1UUXfF4tsPI2JJrT8x2CGY1duihh/Lmm2+yzz7+jcqqp7G12WaWe+riu1gmyc5dpBKej0lNOS0gImLfWo/GzMxq5Oyzz+bQQw9l1113paCggIhAEvPmzct2aGZmZvUuk2Tnr8B3gfl8dc+OmZk1QN/73vcYP348ffr0Kbtnx8zMbEeVSbLzaUQ8UueRmJlZje28884MHTo022GYmZk1CJkkO3Ml3QM8SmoYG4CnnjYza4D2339/vv3tb3PyySdTUFBQVu7Z2MzMbEeUSbLTnFSSMzitzFNPm5k1QOvWraOgoIBnnnmmrMxTT5uZ2Y6qygHdEXFOJX/n1kdw2SKpuaSXJOVLKpIUkq5JW99R0kZJt1RRT5GkBRkcLyT9X9pyE0mfSnpsO+Pf7vokrU57/ZSkzyvuJ2mCpO7bE5uZ1a0777xzi7+xY8dmO6w6tW7dOo466ihKSkpYsmQJkrjyyivL1i9fvpymTZty8cUXb7OeJUuW0Lt37yqP5zbbzKzxyOShooXA94BeQGFpeY4nPOcCEyOiRBLA+8CJQOnV8zRSD1mtLWuA3pKaR8Q64DjgwwZQ3++BFsB/Vij/E3A5cH4NYjSzOlBcXMxf//pXFi5cSHFxcVl5Lic8Y8eOZcSIEeTn5wPQtWtXHn/8ca65JvUb1QMPPECvXr1q85Bus83MGolMpuoZD+wKHA+8BHQBvqzLoBqA7wAPpy2vBd6S1D9ZPgO4v3SlpHGSTk1bXk31PUEqoQI4C7g3rb6rJF2WtrxAUlFd1xcRz1P5Z/0ycKykLZJlSRdImiVpVsnaVVWEaGa17bvf/S4ff/wxTz/9NEcddRRLly6ldevW2Q6rTt19990MGzasbLlFixb07NmTWbNmAXDfffdx+umnl60fNWoUDz74YNlyq1attuewbrPNzBqBTJKdPSPiF8CaiPgbqcb44LoNK3skNQO6RcSSCqsmAGdK2h0oAT6q5UOX1l8I7AvMaGD1lYmIzcC7wH6VrLstIvpHRP/8Fm1r65BmlqF3332Xq6++mpYtWzJy5Egef/xxZsyotf/9G5wNGzawePFiioqKypWfeeaZTJgwgQ8++ID8/Hw6d+5c24d2m21m1ghkkuxsTP79XFJvoC3wtboLKes6Ap9XUv4UqaEFZwL31fZBI2IeUETqF70nGlp9lfgEqPVvD2ZWM02bNgWgXbt2LFiwgFWrVvHJJ59kOaq6s3z5ctq1a7dF+QknnMCzzz7LhAkTOOOMM2r9uG6zzcwah0ySndsktQd+ATwCvAn8b51GlV3rSLs3qVREbABmA5cCD1ZYvYnkvZSUBzTbzmM/AvyBtOELFetPbBFfPdWXrpDUe2VmDcgFF1zAypUrufrqqxk6dCj77LMPl19+ebbDqjPNmzcvd29SqWbNmnHAAQdw3XXXceqpp5Zb16RJEzZvTj0je/PmzWzYsGF7D+8228ysgatygoKIuCN5+RLQrW7Dyb6IWJnMwlYYERWvoNcBL0XEZ8nEBaWWAAeQuo9nKNC0Yr2SdgPuiohjtnH4scDnETFf0sAK9Z+U1NMP6Jrh6dR2fen2Aqqcac7M6td5550HwFFHHcXixYuzHE3da9++PSUlJRQXF1NYWD4HuPTSSznqqKPYaaedypUXFRUxe/ZsTj/9dB555BE2btxIJZpKet5ttplZ45bJbGwFwLdIda+XbR8Rv667sLLuGeAI4Ln0wohYSOWzsN0OPCzpDVLD3dZUsk0nUr/ObVVELAVuqmTVQ8DZkhaSGsf9dlUnUBv1SXoZ2BtoJWkp8L2IeFrSLsC6iPg4kzjMrP6sX7+ehx56iCVLlrBp01dNzn//939nMaq6NXjwYKZNm8axxx5brrxXr16VzsJ2/vnnM2zYMPbbbz9OOOEEWrZsWVm1TXGbbWbW6Ckitr2B9BSwitQQrpLS8oi4rm5Dy57kl7PREfHdWqzzYuBfEfFIbdWZLZJGA19ExF+3tV1Bp+7RaeQN9RNULVhy7YlVb2TWwJ1wwgm0bduWAw44oGwqZkj1cuSqOXPmMGbMGMaPH19rdUr6F/ADt9lmZvWnJt/FJM2OiP4Vy6vs2QG6RMQJ233kRigi5kiaIik/Ikqq3iOjOrf5ANJG5nNSU5KbWQOzdOlSnnrqqWyHUa/69evHoEGDKCkpKZfg1dCnuZDoJD7HbbaZ7aAySXZekdQnIubXeTQNSEQ06CfwSeoAPF/JqmMiYkVdHjsi7sxkuz67tWWWe0vM6tVhhx3G/Pnz6dOnT7ZDqVfnntuwn3PtNtvMLDsySXaOAEZJeh9YDwiIiNi3TiOzbUoujn2zHYeZNSzTpk1j3LhxdO3alYKCAiICScybNy/boe3Q3GabmWVHJsnOkG2tlNQ+IlbWUjxmZlYDTz755DbXr1y5kvbt29dTNGZmZtmVydTT/6xik+eBfrUTjpmZ1cQee+yxzfXHHHMMc+bMqadozMzMsiuTh4pWRVVvYmZmDUFVM3CamZnlktpIdnzlNDNrJCo8ENnMzCyn1UayY2ZmZmZm1uB4GJuZ2Q7Ew9jMzGxHkslsbEjqR2oK6gCmR0T63a3H1EVgZma2febMmcO0adOQxOGHH06/fl/NIfP885U96sXMzCw3VdmzI+m/gb8BHYCOwJ2SrixdHxGf1V14ZmZWHb/+9a8ZOXIkK1asYPny5Zxzzjlcc801Zet32mmnLEZnZmZWv1TVkAZJi4D9IqI4WW4OvB4RPeohPmvE+vfvH7Nmzcp2GGY7lB49evDGG29QWFgIwLp16+jbty+LFi3KcmSNi6TZEdE/23HUJ7fZZtaYba3dzuSenY+AwrTlAuDD2grMzMxqT+fOnSkuLi5bXr9+PbvttlsWIzIzM8ueTO7ZWQUslPQsqXt2jgNek3QTQERcUofxmZlZNbRt25ZevXpx3HHHIYlnn32Wgw46iEsuSTXVN910U5YjNDMzqz+ZJDuTkr9SL9ZNKGZmVlPDhw9n+PDhZcsDBw7MXjBmZmZZVmWyExF/k9QM2CspWhQRG+s2LDMz2x4jR45kw4YNvP3220DqHp6mTZtmOSozM7PsqDLZkTSQ1GxsS0g9U2d3SSMjYmqdRmaN3vwPV1F0xePZDiNjS649MdshmNXYiy++yMiRIykqKiIi+OCDD/jb3/7GgAEDsh2aNXCNrc02s9xTF9/FMhnGdh0wOCIWAUjaC7gXOKDWozEzsxq59NJLeeaZZ+jRIzVh5ttvv81ZZ53F7NmzsxyZmZlZ/ctkNrampYkOQES8DXhMhJlZA7Rx48ayRAdgr732YuNGjzw2M7MdUyY9O7Mk3QH8X7L8HcAT8ZuZNUD9+/fnvPPO4z/+4z8AuPvuu+nff4d6XIyZmVmZTJKd/wIuAkqnmH4Z+GOdRWRmZtvtT3/6E7feemvZFNNHHnkkF154YZajMjMzy45MZmNbD1yf/G1B0kMR8a3aDszMzKqvoKCAH//4x/z4xz+udP23vvUtHnrooXqOyszMLDsyuWenKt1qoY4ykppLeklSvqQ8STdJWiBpvqSZkrom2z0hqV1tHruSWF7ZSvk4SadWsW8PSS9Kel3SW5JuS8r7lz6QdSv77SLpMUlvSHpT0hNJeWdJD9bkfKqyrdgkLZHUUVIzSVMlZdIraGYNzOLFi2u1vnXr1nHUUUdRUlLC5s2bueSSS+jduzd9+vThwAMP5P333wfgm9/8Jp9//nmtHruiww47rNLyUaNG8eCD224+Fy1axMCBA+nbty89e/YE2APcZpuZNXa10fhFLdSR7lxgYkSUSDoL6AzsGxGbJXUB1gBExDdr+bhbiIjKr5yZuQkYExEPA0jqk9Q5i23f8/Rr4NmIuDHZb99kv4+AbSZYNZVBbETEBknPA2cAd9dlPGZW+yTVan1jx45lxIgR5Ofnc++99/LRRx8xb9488vLyWLp0KS1btgTgiSeeqNXjVuaVVyr9fSojl1xyCaNHj2bYsGEASPoE3GabmTV2tdGzU9u+AzycvO4ELIuIzQARsTQiVsJXv1olr38haZGkaZLulXRZUv6ipDGSZiW9KwdKmijpHUnXlB5Q0o+T3qMFkn6UVr46+VeSbkmO8RzwtQzOoxOwtHQhIuYndQ2U9Fg19puX7FckaUHyuoWk+5NfESdJmiGpf2nMkn4vaaGk5yQdlLwPiyUNTbYplHRn0ls2V9KgirFJ6iDpmaSeO0g9Y6nUZFKf0xYkXZC837NK1q7K4G0ys8bs7rvvLksQli1bRqdOncjLS11aunTpQvv27QEoKipi+fLlAFx99dX06NGDI444grPOOos//OEPAAwcOJDRo0fTv39/evbsycyZMxkxYgTdu3fnyiuvLDvm9ddfT+/evenduzc33HBDWXmrVq0AiAguvvhievTowbHHHssnn3xS5XksW7aMLl26pBetA7fZZmaNXW0kO7X2M6GkZkC3iFiSFN0PnKzUULDrJO1fyT4HAt8C9gOGABWnHdoQEf2BP5NKoi4CegOjkovDAcA5wMHAIcD5lRxnONAD2Ac4G8ikx2cM8IKkJyWNVuZD7m4F/ippiqSfS+pcyTYXAisjYh/gF5R/5lFL4IWI6AV8CVwDHJecw6+TbS4CIiL6AGcBf5NUWOEYvwSmJfVMAr6etm4BcGBlwUfEbRHRPyL657dom+Epm1l9iai9zvgNGzawePFiioqKADj99NN59NFH6du3L5deeilz587dYp+ZM2fy0EMP8cYbb/Dkk08ya1b5jolmzZoxa9Ysvv/97zNs2DBuvfVWFixYwLhx41ixYgWzZ8/mzjvvZMaMGbz66qvcfvvtWxxn0qRJLFq0iDfffJO77rorox6f0aNHc/TRRzNkyBDGjBkDkJ/h2+A228ysAatWsiOpfWkXfZr/V4vxdAQ+L12IiKWkkoyfApuB5yUdU2Gfw4GHI6I4Ir4EHq2w/pHk3/nAwohYlky6sBjYHTgCmBQRayJiNTAROLJCHQOAeyOiJBma8EJVJxIRdwI9gQeAgcCrkgoy2O9pUvdB3Q7sDcyVtHOFzY4AJiTbLwDmpa3bADyVds4vRcTG5HVR2v7/l+z/D+CfwF4VjjEgbZvHgZVpMZYAGyS1rup8zCx7Vq5cybx588qV/e53v6u1+pcvX067du3Klrt06cKiRYv47W9/S15eHscccwzPP/98uX2mT5/OsGHDKCwspHXr1px88snl1g8dOhSAPn360KtXLzp16kRBQQHdunXjgw8+YNq0aQwfPpyWLVvSqlUrRowYwcsvv1yujqlTp3LWWWeRn59P586dOfroo6s8l3POOYe33nqL0047jRdffBFgb7fZZmaNX5XJTtKd3kbSTsAc4HZJZTOzRcQztRjPOqDcr1URsT4inoyInwC/AU6pZp3rk383p70uXa7TGzYj4qOIGBsRw4BNpHqUMtnvs4i4JyK+C8wkdRHL1Mb46qfbsnNOhgLW5vkWAMW1WJ+Z1YKBAwfyxRdf8Nlnn9GvXz/OP//8cjOzDR48uNaO1bx5c4qLyzcDBQUFDBkyhN///vf87Gc/Y/LkydWqs6AglV/k5eWVvS5d3rRpU41j3pbOnTtz7rnn8vDDpSOp3WabmTV2mfTstI2IL4ARwF0RcTBwbF0Ek9yPk1/aPS+pX+mQAEl5wL6kftFKN53UULdCSa2Ak6p52JeBU5Ix1S1JDR14ucI2U4EzlJohrhMwqHSFpN9KGl6xUkknSGqavN4V6AB8WFUwko6W1CJ53Rr4BvCvCptNB05PttkH6JPRmX7lZZLx25L2IjXcYVGFbaYC3062GQK0T4uxA7A8+fXRzBqQVatW0aZNGyZOnMjZZ5/NjBkzeO655+rkWO3bt6ekpKQs4ZkzZw4fffQRAJs3b2bevHnsscce5fY5/PDDefTRRykuLmb16tU89ti2bofZ0pFHHsnkyZNZu3Yta9asYdKkSRx5ZPnO+AEDBnDfffdRUlLCsmXLmDJlStm6n/70p0yaNGmLep966ik2bkw1aR9//DGkhrG5zTYza+QySXaaJF/wTweqd1XaPs+Q6rKH1EQAjyY3ec4j1TtyS/rGETGT1FC1ecCTpLr+M77LMiLmAOOA14AZwB0RUXGg+STgHeBN4C7g72nr+gAfV1L1YGCBpDeAp4GfRERl21V0ADBL0rzkOHck55juj8DOkt4kNb57IdU452T/PEnzgfuAUcnQvnS/AgZIWkgq0U2/eA8CHq/G8cysnmzatIlly5Zx//33c9JJ1f3tp/oGDx7MtGnTAPjkk084+eST6d27N/vuuy9NmjTh4osvLrf9gQceyNChQ9l3330ZMmQIffr0oW3bzO8V6devH6NGjeKggw7i4IMP5rzzzmP//cvfZjl8+HC6d+/OPvvsw9lnn82hhx5atm7+/PnsuuuuW9T7zDPP0Lt3b/bbbz+OP/54gKVus83MGj9VdbOqpNNI3VA5LSIulNQN+H1dPUhUUj9gdDIcINN9WkXE6uTXtanABUkSU+ckPR0Rx9fHsdKOmQ80jYhiSd8AngN6RMSGejr+ROCKiHh7W9sVdOoenUbeUB8h1Yol156Y7RDMauyBBx7g6quv5ogjjuCPf/wjixcv5ic/+UmdPUh0zpw5jBkzhvHjx2e8z+rVq2nVqhVr165lwIAB3HbbbfTr169O4qvo+OOP5+mnn65yO0mzk8ltasxttplZZmryXWxr7XaV44Ej4gFSN9mXLi8mNftZnYiIOcmsNvnJTZWZuC0ZGlAI/K2+Eh2A+k50Ei2AKckwOQEX1uNFsxkwuaqLppllx2mnncZpp51WttytW7c6S3Qg1dMyaNAgSkpKyM/PbAKzCy64gDfffJPi4mJGjhxZb4kOkFGiUwfcZpuZZUkmPTuVPZ15FTCr9IGZVn2SzgF+WKF4ekRclI146kL//v2j4rSyZla3Lrnkki3K2rZtS//+/cueh2NVq/gLodtsM7OGbbt7dkj1luzNV7073wLeB/aTNCgiflRrUe5Akqmp78x2HGaWW4qLi/nHP/5R1rvz0EMP0bVrV9544w2mTJlS7iGcljm32WZmjVMmyc6+wOGlQ8ok/YnUzDBHkJoMwMzMGoh58+Yxffr0siFl//Vf/8WRRx7JtGnT6NOnupOAmZmZNW6ZzMbWHmiVttwS2ClJfirOBmNmZlm0cuVKVq9eXba8Zs0aPvvsM/Lz88s9t8bMzGxHkEnPzv8Cr0t6kdSNlQOA3yTPpKmbhzeYmdl2ufzyy+nbty8DBw4kIpg6dSo/+9nPWLNmDcceWyePSDMzM2uwtpnsJA/yfAs4DDgoKf5ZRHyUvP5JHcZmZmbVsHnzZnr27Mkrr7zCa6+9BsBvfvMbOnfuDMDvf//7bIZnZmZW77aZ7ETEZkm3RsT+gGdeMzNrwPLy8rjooouYO3euZ14zMzMjs3t2npf0LUmq82jMzKxGjjnmGB566CGqeqyAmZnZjiCTZOc/SU07vV7SF5K+lPRFHcdlZmbb4S9/+QunnXYaBQUFtGnThtatW9OmTZtsh2VmZpYVVU5QEBGtJe0EdCf1zB0zM2ugvvzySz777DPeeecdiouLsx2OmZlZVlWZ7Eg6j9RTo7sArwOHAK8Ax9RpZGZmVm133HEHN954I0uXLqVv3768+uqrHHbYYTz//PPZDs3MzKzeZTKM7YfAgcA/I2IQsD+wqk6jMjOz7XLjjTcyc+ZM9thjD6ZMmcLcuXNp27ZttsMyMzPLikySneKIKAaQVBAR/wB61G1YZma2PQoLCyksTI04Xr9+PXvvvTeLFi3KclRmZmbZkclDRZdKagdMBp6VtBL4Z10GZWZm26dLly58/vnnnHLKKRx33HG0b9+ePfbYI9thmZmZZUUmExQMT15eJWkK0BZ4qk6jMjOz7TJp0iQArrrqKgYNGsSqVas44YQTshyVmZlZdsjPYrC6UtCpe3QaeUO2w9jCkmtPzHYIZtbASZodEf2zHUd9aqhttpntOGryHW1r7XYm9+yYmZmZmZk1Ok52zMzMzMwsJznZMTMzMzOznORkx8zMzMzMcpKTHTMzMzMzy0lOdqpBUnNJL0nKl1QkKSRdk7a+o6SNkm6pop4iSQsyOF5I+r+05SaSPpX02HbGv931SVqd/LuzJE89bmaNwrp16zjqqKMoKSlhyZIlSOLKK68sW798+XKaNm3KxRdfvM163G6bmTVOTnaq51xgYkSUJMvvA+lz5J0GLKzF460BektqniwfB3yYzfoi4lNgmaTDaxCHmVm9GDt2LCNGjCA/Px+Arl278vjjj5etf+CBB+jVq1dtHtLttplZA+Jkp3q+AzyctrwWeEtS6ZzeZwD3l66UNE7SqWnLq7fjmE/wVUJ1FnBvWn1XSbosbXmBpKJ6qG8yqfdiC5IukDRL0qyStauqCMXMrG7dfffdDBs2rGy5RYsW9OzZk1mzZgFw3333cfrpp5etHzVqFA8++GDZcq63226zzSzXOdnJkKRmQLeIWFJh1QTgTEm7AyXAR7V86NL6C4F9gRkNoL5ZwJGVrYiI2yKif0T0z2/RtgZhmpnVzIYNG1i8eDFFRUXlys8880wmTJjABx98QH5+Pp07d67tQzeadttttpnluibZDqAR6Qh8Xkn5U8DVwL+B+2r7oBExL/mV7ixSv+41hPo+AWr924GZWW1avnw57dq126L8hBNO4Be/+AW77LILZ5xxRq0f1+22mVnD4Z6dzK0DCisWRsQGYDZwKfBghdWbSN5jSXlAs+089iPAH0gbulCx/sQW8dVRfYWk3g8zswarefPmFBcXb1HerFkzDjjgAK677jpOPfXUcuuaNGnC5s2bAbfbZma5wMlOhiJiJZCfDCOo6Drg/0XEZxXKlwAHJK+HAk0r7ihpN0nPV3H4scCvImJ+JfX3S+rpB3Stop7aqm8voMpZiczMsql9+/aUlJRUmvBceuml/O53v2OnnXYqV15UVMTs2bNLF91um5k1ck52qucZ4IiKhRGxMCL+Vsn2twNHSXoDOJTUrDoVdSL1y9xWRcTSiLipklUPATtJWghcDLxdRfy1Vd8g4PGtrDMzazAGDx7MtGnTtijv1asXI0eO3KL8/PPP56WXXgLYB7fbZmaNniIi2zE0GsmvZqMj4ru1WOfFwL8i4pHaqrOuSZoKDEt6u7aqoFP36DTyhvoJqhqWXHti1RuZWU6YM2cOY8aMYfz48dXaT9LsiOi/lXU52W431DbbzHYcNfmOtrV22xMUVENEzJE0RVJ+2rN2alrnNh9A2tBI2hm4vqpEx8ysIejXrx+DBg2ipKSk7Fk7NeV228ys8XDPTo6R1AGobCz5MRGxoj5j6d+/f5Q+y8LMrDHZVs9OHRyrQbTbbrPNrDFzz84OIrkw9s12HGZmlhm322ZmdccTFJiZmZmZWU5ysmNmZmZmZjnJyY6ZmZmZmeUkJztmZmZmZpaTnOyYmZmZmVlOcrJjZmZmZmY5ycmOmZmZmZnlJCc7ZmZmZmaWk5zsmJmZmZlZTnKyY2ZmZmZmOcnJjpmZmZmZ5SQnO2ZmZmZmlpOc7JiZmZmZWU5ysmNmZmZmZjmpSbYDsNw1/8NVFF3xeK3Vt+TaE2utLjPbfhs3bmTp0qUUFxdnO5QaKywspEuXLjRt2jTboWRdbbfZZmbVVRff9ZzsmJlZtSxdupTWrVtTVFSEpGyHs90ighUrVrB06VK6du2a7XDMzKwOeBibmZlVS3FxMR06dGjUiQ6AJDp06JATPVRmZlY5JztmZlZtjT3RKZUr52FmZpVzsmNmZmZmZjkp68mOpOaSXpKUL6lIUki6Jm19R0kbJd1SRT0DJR2WtnxVUteeaWU/Ssr6V1HXVZIuq2KbUUldx6aVnZKUnbqtfbdS3zhJayW1Tiu7IamvYxX7/qzCckj6v7TlJpI+lfRYBnGsTv7tK+nvkhZKmifpjLRtJkjqXp3zM7PcsG7dOs4++2xKSkpYsmQJkmh72JkUXfE4RVc8zu6X3IPym9DmgJPKyir72/Xbv2XX//hD2XK7I76NJHb7z9vLjnXDDTcgiVmzZm0zpoEDB3LSSSdtc5tx48Yhieeee66sbPLkyUji6aefrvb74DbbzKxxyHqyA5wLTIyIkmT5fSB9KobTgIUZ1DMQOKxC2XzgzO2oK1MV6z8LeKMG9b0LDAOQlAccDXyYwX4/q7C8BugtqXmyfFyG9aRbC5wdEb2AE4AbJLVL1v0JuLya9ZlZDhg7dizHHXcc+fn5AHTt2pV1780sW7920XSadvx6lfUU/2s+6z98q1xZ052LWPPW1LLlBx54gF69etVS5NCnTx/uueeesuV7772X/fbbryZVus02M2vgGkKy8x3g4bTltcBbab0vZwD3l66UdLKkGZLmSnpO0i6SioDvA6MlvS7pyGTzyXx1IfoGsApYnlbX6rTXp0oaV83YXwYOktRUUitgT+D1tDr/W9JMSQsk3aaqB4dPSM4XUsnbdGBTWn2TJc1Ofrm7ICm7FmienPfdaXU9wVdJ41nAvWn1lOu5SuIrSg8kIt6OiHeS1x8BnwA7p533sZK2mM1P0gWSZkmaVbJ2VRWna2aNzd13383RRx9dttyiRQuadtid9cveAWDNW1NpufeRZevXvjuDZXf9mI/uvIR/T/g5JWtWsmnVv/ny9Sf5ctZkPrrzBxR/sCBVV/dDWPfODADee+892rZtS8eOX3WStGrVquz1WWedRZs2bTjiiCNYvjzVrA8cOLCsF2j58uUUFRUBqV6dG2+8kc8++4z777+fjRs3snr1at5991369u1bVuevf/1rDjzwQHr37g2wh9tsM7PGL6vJjqRmQLeIWFJh1QTgTEm7AyXAR2nrpgGHRMT+yXaXJ/v/GRgTEX0j4uVk2y+ADyT1JtUDc18tn0IAzwHHk0qqHqmw/paIODAiegPNgW2Ps4C3gZ0ltSd1sZtQYf25EXEA0B+4RFKHiLgCWJec93fSti19DwuBfYEZ23F+AEg6CGgGvAcQEZtJ/aK5xU+iEXFbRPSPiP75Ldpu7yHNrAHasGEDixcvZrfdditX3rLnANa+NZVNX3yK8vLIb7VT2bqCLr3Y9bvX0fmcm2jRcwCrZjxEk7a70LrvEFr3P4XO59xM4e69AVCzFuS36ciCBQuYMGECZ5xxBpWZPXs206dPZ+jQoTzxxBN89NFHlW6XbsmSJZxwwglccMEFPP300zz88MMMHTq03DYXX3wxM2fOZMGCBZC6PrrNNjNr5LLds9MR+LyS8qdIdeNXlqB0AZ6WNB/4CVDVGIcJST2nAJNqEGtV9Z9J2i9xiUFJL9R8UsMbMhmPMTGp62BSv8alu0TSG8CrwO7AVsdgR8Q8oIjUBfiJDI5bKUmdgPHAOckFs9QnQOftrdfMGp/ly5fTrl27Lcqbd+vHuiVzWfPWVFrsPaDcupIvl/PJ/f/NR3+9iC9em8jG5f/a5jFa7j2ACRMmMHnyZIYPH17pNi+//DIHHXQQTZo0oU2bNvTo0aPK2Hv16kVhYSFnnnkmEyZMYMKECZx11lnltpkyZQoHH3wwffr0AWiN22wzs0Yv28nOOqCwYmFEbABmA5cCD1ZYfTOpHpM+wH9Wtn8FjwHfBf4VEV9UPFTa66rqqVREvAb0ATpGxNul5cmvc38ETk1ivT3DY9wHXA08m36hkjQQOBY4NCL2A+ZmUN8jwB/YMgnbRPnPvtJ6JLUBHgd+HhGvVlhdSOrzM7MdRPPmzSt9Jo3ym9Js1z35YuYkWvQ4vNy6z579M637nUTn791Kh+MvIjZt2PYx9jyQ8ePH8/Wvf502bdqUP07aqLJNmzZV3JUmTZqweXOq2awYZ0FBAQAHHXQQ8+fPZ/ny5ey1115l64uLi7nwwgt58MEHmT9/PqSGPLvNNjNr5LYYv1ufImKlUrOwFUZExSvodcBLEfFZhWHTbfnqxs2RaeVfAuWvjKljrJX0/0gNN6jo35J6AouA4Ukd5Ui6OKlnW7PBXQFUjL/0YrQ8uZ/nVLZM3LYQEf+U9HNSw+PStQVWJuezN3BI2rqNkppGxMYK+4wFPo+I+cmFt9QSkuEZkvoBWzw6PBliOAm4KyIqi3svYEFV52NmuaN9+/aUlJSwfv36Lda1OXA4hbv3Ib9563Llm9evJb91BwBWz3+hrDyvWXM2r9/yu3de00Ku/d3vyiUipXbZZRfeeustjjjiCH75y18ydOhQvvzySxYtWsRhhx1GUVER1113HUceeWSlyVCpa6+9lsLC8vlCaXLUsWNHVq9eDdB+qxWkcZttZtawZTXZSTwDHEGFC0VELKTymdOuAh6QtBJ4ga8a/UeBByUNA35Qoa6K46hLXUGq5+dTYBbQqpJt9iZ10+lWRcSTlZR9Lul2UheXj4GZW+y49fr+UknxU8D3Jb1FKjlL/9XuNmCepDnpY8AjYilwUyV1PQScLWkhqXHhlSWCpwMDgA6SRiVloyLidUm7kBpz/nGm52RmuWHw4MHMnj273I39S649sdw248Z9yqxZG7nl2hN5+NBNjB49mvbt23PaCUczc+ZMXrz2RN5+uzunnnoqeU/9nJtvvpnnC/eiVatWXHbZiWzNtddey0knncTOO+9Mz549efjhh3n//ffL7iG67LLLOPzww5kxYwb/8R//sdV6hgwZskVZu3btOP/88+nduze77rorpGZIy4jbbDOzhksRUfVWdRlA6leq0RHx3awGshVKPedgRDK0zgBJo4EvIuKv29quoFP36DTyhlo7bsUvVGZW/+bMmcNVV13FI49UnI+lYTjppJOYOHEizZo1y3ift956i549e5YrkzQ7Irb5TLbGIltttplZddXku97W2u2s9+xExBxJUyTlpz1rp8GIiKpm49kRfU7qBlgz28H069ePgw46iJKSkrJn7TQkjz1W5XM4d0Sf4zbbzHZQWU92ACJibLZjqC+SbgUOr1B8Y0TcmY14tkemsfbZrS2z3BtjlnO+9a1vNchEpy64zTYza9waRLKzI4mIi7Idg5lZTUVEudnRGquqhnK7zTYza9yyPfW0mZk1MoWFhaxYsaLKRKGhiwhWrFixxcxsZmaWO9yzY2Zm1dKlSxeWLl3Kp59+mu1QaqywsJAuXbpkOwwzM6sjTnbMzKxamjZtSteuWzzqxczMrMHxMDYzMzMzM8tJTnbMzMzMzCwnOdkxMzMzM7OcpMY+m441XJK+BBZlO4560hFYnu0g6sGOcp6w45zrjnKeUL1z3SMidq7LYBqaHazNzsSO9P9GVfxelOf34ysN6b2otN32BAVWlxZFRP9sB1EfJM3aEc51RzlP2HHOdUc5T9ixznU77TBtdib838tX/F6U5/fjK43hvfAwNjMzMzMzy0lOdszMzMzMLCc52bG6dFu2A6hHO8q57ijnCTvOue4o5wk71rluD78/5fn9+Irfi/L8fnylwb8XnqDAzMzMzMxyknt2zMzMzMwsJznZMTMzMzOznORkx+qEpBMkLZL0rqQrsh1PJiTtLmmKpDclLZT0w6R8J0nPSnon+bd9Ui5JNyXnOE9Sv7S6RibbvyNpZFr5AZLmJ/vcJEn1f6ZlseRLmivpsWS5q6QZSWz3SWqWlBcky+8m64vS6vhpUr5I0vFp5Q3m85fUTtKDkv4h6S1Jh+biZyppdPLf7QJJ90oqzJXPVNJYSZ9IWpBWVuef4daOkYsa0v+z2VCd/8Zynap5LcxlSTv6mqQ3kvfiV0l5pW3rjkAZfndoUCLCf/6r1T8gH3gP6AY0A94A9sl2XBnE3Qnol7xuDbwN7AP8L3BFUn4F8Lvk9TeBJwEBhwAzkvKdgMXJv+2T1+2Tda8l2yrZd0gWz/fHwD3AY8ny/cCZyes/A/+VvL4Q+HPy+kzgvuT1PslnWwB0TT7z/Ib2+QN/A85LXjcD2uXaZwrsBrwPNE/7LEflymcKDAD6AQvSyur8M9zaMXLtL9ufb0P4q85/Y7n+RzWvhbn8l7QJrZLXTYEZSVtRadu6I/yR4XeHhvTnnh2rCwcB70bE4ojYAEwAhmU5pipFxLKImJO8/hJ4i9SXyGGkvjCT/HtK8noYcFekvAq0k9QJOB54NiI+i4iVwLPACcm6NhHxaqRahbvS6qpXkroAJwJ3JMsCjgYeTDapeJ6l5/8gcEyy/TBgQkSsj4j3gXdJffYN5vOX1JbUl5i/AkTEhoj4nBz8TEk9JLq5pCZAC2AZOfKZRsRU4LMKxfXxGW7tGLmmwfw/my3V/G8sp23HtTBnJe3I6mSxafIXbL1tzWnV/O7QYDjZsbqwG/BB2vLSpKzRSIb17E/qV5xdImJZsupjYJfk9dbOc1vlSyspz4YbgMuBzclyB+DziNiULKfHVnY+yfpVyfbVPf9s6Ap8CtyZdLvfIaklOfaZRsSHwB+Af5FKclYBs8nNz7RUfXyGWztGrmmIn29DsKN8/luV4bUwpyXDtl4HPiH1I8l7bL1tzXU3kPl3hwbDyY5ZBZJaAQ8BP4qIL9LXJb/8Nur52iWdBHwSEbOzHUs9aEJqaMqfImJ/YA2p4RdlcuQzbU/qV9euQGegJXBCVoOqR/XxGebCfye2/XbEzz/Xr4WZioiSiOgLdCHVC7p3diPKjsb83cHJjtWFD4Hd05a7JGUNnqSmpBr3uyNiYlL872SoC8m/nyTlWzvPbZV3qaS8vh0ODJW0hNRwlaOBG0kN92lSSWxl55OsbwusoPrnnw1LgaURMSNZfpBU8pNrn+mxwPsR8WlEbAQmkvqcc/EzLVUfn+HWjpFrGuLn2xDsKJ//Fqp5LdwhJEOgpwCHsvW2NZdV97tDg+Fkx+rCTKB7MkNHM1I3QD+S5ZiqlIw9/SvwVkRcn7bqEaB05qaRwMNp5Wcnsz8dAqxKuvifBgZLap/84j4YeDpZ94WkQ5JjnZ1WV72JiJ9GRJeIKCL12bwQEd8h1YifmmxW8TxLz//UZPtIys9UamavrkB3Ujd6N5jPPyI+Bj6Q1CMpOgZ4kxz7TEkNXztEUoskjtLzzLnPNE19fIZbO0auaYifb0Owo3z+5WzHtTBnSdpZUrvkdXPgOFL3MG2tbc1Z2/HdoeGozmwG/vNfpn+kZkR6m9TY1p9nO54MYz6CVLf8POD15O+bpMakPg+8AzwH7JRsL+DW5BznA/3T6jqX1M3d7wLnpJX3BxYk+9wCKMvnPJCvZlTpRuqL7bvAA0BBUl6YLL+brO+Wtv/Pk3NZRNosZA3p8wf6ArOSz3UyqZm4cu4zBX4F/COJZTypGdVy4jMF7iV1L9JGUr1136uPz3Brx8jFv4b0/2yWzj/j/8Zy/Y9qXgtz+Q/YF5ibvBcLgP9OyittW3eUPzL47tCQ/kobdDMzMzMzs5ziYWxmZmZmZpaTnOyYmZmZmVlOcrJjZmZmZmY5ycmOmZmZmZnlJCc7ZmZmZmaWk5zsmJmZmZlZTnKyY2ZmZmZmOen/AyAYbvNHqZxwAAAAAElFTkSuQmCC\n", "text/plain": ["<Figure size 864x288 with 2 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["import matplotlib.pyplot as plt\n", "\n", "fig, ax = plt.subplots(1, 2, figsize=(12, 4))\n", "gr_dur.plot.barh(ax=ax[0])\n", "gr_n.plot.barh(ax=ax[1])\n", "ax[0].set_title(\"duration\")\n", "ax[1].set_title(\"n occurences\");"]}, {"cell_type": "markdown", "id": "7b10ca8a", "metadata": {}, "source": ["onnxruntime passe principalement son temps dans un produit matriciel. On v\u00e9rifie plus pr\u00e9cis\u00e9ment."]}, {"cell_type": "code", "execution_count": 33, "id": "4cbc2fa0", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th>127</th>\n", "      <th>12</th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>cat</th>\n", "      <td>Node</td>\n", "      <td>Node</td>\n", "    </tr>\n", "    <tr>\n", "      <th>pid</th>\n", "      <td>78116</td>\n", "      <td>78116</td>\n", "    </tr>\n", "    <tr>\n", "      <th>tid</th>\n", "      <td>8820</td>\n", "      <td>8820</td>\n", "    </tr>\n", "    <tr>\n", "      <th>dur</th>\n", "      <td>4603</td>\n", "      <td>4083</td>\n", "    </tr>\n", "    <tr>\n", "      <th>ts</th>\n", "      <td>37173</td>\n", "      <td>5949</td>\n", "    </tr>\n", "    <tr>\n", "      <th>ph</th>\n", "      <td>X</td>\n", "      <td>X</td>\n", "    </tr>\n", "    <tr>\n", "      <th>name</th>\n", "      <td>gemm_token_0_kernel_time</td>\n", "      <td>gemm_token_0_kernel_time</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_op_name</th>\n", "      <td>Gemm</td>\n", "      <td>Gemm</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_parameter_size</th>\n", "      <td>64516</td>\n", "      <td>64516</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_graph_index</th>\n", "      <td>12</td>\n", "      <td>12</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_provider</th>\n", "      <td>CPUExecutionProvider</td>\n", "      <td>CPUExecutionProvider</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_exec_plan_index</th>\n", "      <td>12</td>\n", "      <td>12</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_activation_size</th>\n", "      <td>2520000</td>\n", "      <td>2520000</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_output_size</th>\n", "      <td>2540000</td>\n", "      <td>2540000</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_input_type_shape</th>\n", "      <td>[{'float': [5000, 126]}, {'float': [126, 127]}...</td>\n", "      <td>[{'float': [5000, 126]}, {'float': [126, 127]}...</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_output_type_shape</th>\n", "      <td>[{'float': [5000, 127]}]</td>\n", "      <td>[{'float': [5000, 127]}]</td>\n", "    </tr>\n", "    <tr>\n", "      <th>args_thread_scheduling_stats</th>\n", "      <td>{'main_thread': {'thread_pool_name': 'session-...</td>\n", "      <td>{'main_thread': {'thread_pool_name': 'session-...</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["                                                                            127  \\\n", "cat                                                                        Node   \n", "pid                                                                       78116   \n", "tid                                                                        8820   \n", "dur                                                                        4603   \n", "ts                                                                        37173   \n", "ph                                                                            X   \n", "name                                                   gemm_token_0_kernel_time   \n", "args_op_name                                                               Gemm   \n", "args_parameter_size                                                       64516   \n", "args_graph_index                                                             12   \n", "args_provider                                              CPUExecutionProvider   \n", "args_exec_plan_index                                                         12   \n", "args_activation_size                                                    2520000   \n", "args_output_size                                                        2540000   \n", "args_input_type_shape         [{'float': [5000, 126]}, {'float': [126, 127]}...   \n", "args_output_type_shape                                 [{'float': [5000, 127]}]   \n", "args_thread_scheduling_stats  {'main_thread': {'thread_pool_name': 'session-...   \n", "\n", "                                                                            12   \n", "cat                                                                        Node  \n", "pid                                                                       78116  \n", "tid                                                                        8820  \n", "dur                                                                        4083  \n", "ts                                                                         5949  \n", "ph                                                                            X  \n", "name                                                   gemm_token_0_kernel_time  \n", "args_op_name                                                               Gemm  \n", "args_parameter_size                                                       64516  \n", "args_graph_index                                                             12  \n", "args_provider                                              CPUExecutionProvider  \n", "args_exec_plan_index                                                         12  \n", "args_activation_size                                                    2520000  \n", "args_output_size                                                        2540000  \n", "args_input_type_shape         [{'float': [5000, 126]}, {'float': [126, 127]}...  \n", "args_output_type_shape                                 [{'float': [5000, 127]}]  \n", "args_thread_scheduling_stats  {'main_thread': {'thread_pool_name': 'session-...  "]}, "execution_count": 34, "metadata": {}, "output_type": "execute_result"}], "source": ["df[(df.args_op_name == 'Gemm') & (df.dur > 0)].sort_values('dur', ascending=False).head(n=2).T"]}, {"cell_type": "markdown", "id": "58320942", "metadata": {}, "source": ["C'est un produit matriciel d'environ *5000x800* par *800x800*."]}, {"cell_type": "code", "execution_count": 34, "id": "de43df2f", "metadata": {}, "outputs": [{"data": {"text/html": ["<div>\n", "<style scoped>\n", "    .dataframe tbody tr th:only-of-type {\n", "        vertical-align: middle;\n", "    }\n", "\n", "    .dataframe tbody tr th {\n", "        vertical-align: top;\n", "    }\n", "\n", "    .dataframe thead th {\n", "        text-align: right;\n", "    }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", "  <thead>\n", "    <tr style=\"text-align: right;\">\n", "      <th></th>\n", "      <th></th>\n", "      <th>dur</th>\n", "    </tr>\n", "    <tr>\n", "      <th>args_op_name</th>\n", "      <th>name</th>\n", "      <th></th>\n", "    </tr>\n", "  </thead>\n", "  <tbody>\n", "    <tr>\n", "      <th>MatMul</th>\n", "      <th>Ma_MatMul2</th>\n", "      <td>0.034561</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul</th>\n", "      <td>0.065894</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid</th>\n", "      <td>0.075714</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Mul</th>\n", "      <th>Mu_Mul1</th>\n", "      <td>0.077254</td>\n", "    </tr>\n", "    <tr>\n", "      <th>Sigmoid</th>\n", "      <th>Si_Sigmoid1</th>\n", "      <td>0.079584</td>\n", "    </tr>\n", "    <tr>\n", "      <th rowspan=\"2\" valign=\"top\">Gemm</th>\n", "      <th>gemm</th>\n", "      <td>0.161958</td>\n", "    </tr>\n", "    <tr>\n", "      <th>gemm_token_0</th>\n", "      <td>0.505035</td>\n", "    </tr>\n", "  </tbody>\n", "</table>\n", "</div>"], "text/plain": ["                                dur\n", "args_op_name name                  \n", "MatMul       Ma_MatMul2    0.034561\n", "Mul          Mu_Mul        0.065894\n", "Sigmoid      Si_Sigmoid    0.075714\n", "Mul          Mu_Mul1       0.077254\n", "Sigmoid      Si_Sigmoid1   0.079584\n", "Gemm         gemm          0.161958\n", "             gemm_token_0  0.505035"]}, "execution_count": 35, "metadata": {}, "output_type": "execute_result"}], "source": ["gr_dur / gr_dur.dur.sum()"]}, {"cell_type": "code", "execution_count": 35, "id": "0e5c02ec", "metadata": {}, "outputs": [{"data": {"text/plain": ["0.5050352082154203"]}, "execution_count": 36, "metadata": {}, "output_type": "execute_result"}], "source": ["r = (gr_dur / gr_dur.dur.sum()).dur.max()\n", "r"]}, {"cell_type": "markdown", "id": "113a480a", "metadata": {}, "source": ["Il occupe 82% du temps. et d'apr\u00e8s l'exp\u00e9rience pr\u00e9c\u00e9dente, son temps d'\u00e9xecution peut-\u00eatre r\u00e9duit par 10 en le rempla\u00e7ant par une matrice sparse. Cela ne suffira pas pour acc\u00e9lerer le temps de calcul de ce r\u00e9seau de neurones. Il est 84 ms compar\u00e9 \u00e0 247 \u00b5s pour l'arbre de d\u00e9cision. Avec cette optimisation, il pourrait passer de :"]}, {"cell_type": "code", "execution_count": 36, "id": "fa7950bc", "metadata": {}, "outputs": [{"data": {"text/plain": ["2.013941471759493"]}, "execution_count": 37, "metadata": {}, "output_type": "execute_result"}], "source": ["t = 3.75  # ms\n", "t * (1 - r) + r * t / 12"]}, {"cell_type": "markdown", "id": "7c641d19", "metadata": {}, "source": ["Soit une r\u00e9duction du temps de calcul. Ce n'est pas mal mais pas assez."]}, {"cell_type": "markdown", "id": "535b7e56", "metadata": {}, "source": ["## Hummingbird\n", "\n", "[hummingbird](https://github.com/microsoft/hummingbird) est une librairie qui convertit un arbre de d\u00e9cision en r\u00e9seau de neurones. Voyons ses performances."]}, {"cell_type": "code", "execution_count": 37, "id": "3b3aa43b", "metadata": {}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["C:\\xavierdupre\\__home_\\github_fork\\scikit-learn\\sklearn\\utils\\deprecation.py:103: FutureWarning: The attribute `n_features_` is deprecated in 1.0 and will be removed in 1.2. Use `n_features_in_` instead.\n", "  warnings.warn(msg, category=FutureWarning)\n"]}, {"data": {"text/plain": ["(4.3419181139370266e-08, 4.430287026515114e-09)"]}, "execution_count": 38, "metadata": {}, "output_type": "execute_result"}], "source": ["from hummingbird.ml import convert\n", "\n", "model = convert(tree, 'torch')\n", "\n", "expected = tree.predict(x_exp)\n", "got = model.predict(x_exp)\n", "numpy.abs(got - expected).max(), numpy.abs(got - expected).mean()"]}, {"cell_type": "markdown", "id": "92365d70", "metadata": {}, "source": ["Le r\u00e9sultat est beaucoup plus fid\u00e8le au mod\u00e8le."]}, {"cell_type": "code", "execution_count": 38, "id": "605df039", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["1.17 ms \u00b1 34.8 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 1,000 loops each)\n"]}], "source": ["%timeit model.predict(x_exp)"]}, {"cell_type": "markdown", "id": "c2f80290", "metadata": {}, "source": ["Il reste plus lent mais beaucoup plus rapide que la solution manuelle propos\u00e9e dans les pr\u00e9c\u00e9dents paragraphes. Il contient un attribut `model`."]}, {"cell_type": "code", "execution_count": 39, "id": "e77ff4f0", "metadata": {}, "outputs": [{"data": {"text/plain": ["True"]}, "execution_count": 40, "metadata": {}, "output_type": "execute_result"}], "source": ["from torch.nn import Module\n", "isinstance(model.model, Module)"]}, {"cell_type": "markdown", "id": "871277df", "metadata": {}, "source": ["On convertit ce mod\u00e8le au format ONNX."]}, {"cell_type": "code", "execution_count": 40, "id": "3c875b35", "metadata": {}, "outputs": [], "source": ["import torch.onnx\n", "\n", "x = torch.randn(x_exp.shape[0], x_exp.shape[1], requires_grad=True)\n", "torch.onnx.export(model.model, x, 'tree_torch.onnx', opset_version=15,                         \n", "                  input_names=['X'], output_names=['variable'],\n", "                  dynamic_axes={\n", "                      'X' : {0 : 'batch_size'},\n", "                      'variable' : {0 : 'batch_size'}})"]}, {"cell_type": "code", "execution_count": 41, "id": "b8c41c5e", "metadata": {}, "outputs": [], "source": ["import onnx\n", "\n", "onxh = onnx.load('tree_torch.onnx')"]}, {"cell_type": "code", "execution_count": 42, "id": "861a94d0", "metadata": {"scrolled": false}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["opset: domain='' version=15\n", "input: name='X' type=dtype('float32') shape=['batch_size', 10]\n", "init: name='_operators.0.root_nodes' type=dtype('int64') shape=(0,) -- array([8], dtype=int64)\n", "init: name='_operators.0.root_biases' type=dtype('float32') shape=(0,) -- array([0.00792999], dtype=float32)\n", "init: name='_operators.0.tree_indices' type=dtype('int64') shape=(0,) -- array([0], dtype=int64)\n", "init: name='_operators.0.leaf_nodes' type=dtype('float32') shape=(0,) -- array([ 1.0096807 ,  0.6169403 ,  0.61055773,  0.37810475,  0.31796893,\n", "        0.13317925,  0.0193846 , -0.2317742 ,  0.39089343,  0.23506087,\n", "        0.3711936 ,  0.10317916,  0.14956598, -0.14193445, -0.05965868,\n", "       -0.27377078,  0.4128183 ,  0.19658326,  0.25545415,  0.08118545,\n", "        0.08400188, -0.1502193 , -0.36846825, -0.79687625,  0.35822242,\n", "        0.49021915,  0.30870998,  0.01033915,  0.6740977 ,  0.6740977 ,\n", "       -0.15315758, -0.41128033,  0.42920846,  0.13145493,  0.21853392,\n", "       -0.10986731,  0.4493652 ,  0.11318789,  0.12666471, -0.0623082 ,\n", "        0.2872893 ,  0.09948976,  0.11439473, -0.08801427,  0.16091613,\n", "       -0.02319027, -0.10097775, -0.37583745,  0.18612385, -0.00453244,\n", "        0.3287116 , -0.1499349 ,  0.7919218 ,  0.04704398, -0.15423109,\n", "       -0.43160027,  0.10802375, -0.1073833 , -0.07759219, -0.29175794,\n", "       -0.1528881 , -0.4909434 , -0.23361537, -0.43578717,  0.7831867 ,\n", "        0.45349318,  0.34956965, -0.3199535 ,  0.3061573 , -0.34267113,\n", "        0.34963542,  0.04491445,  0.35399815,  0.14815213,  0.06678926,\n", "       -0.16095412,  0.3214274 ,  0.01484008, -0.1012276 , -0.3257699 ,\n", "        0.26727676,  0.01970094,  0.10760042, -0.09169976,  0.20044112,\n", "       -0.0324069 , -0.11015374, -0.28358367,  0.8083656 ,  0.13358633,\n", "       -0.07912118, -0.27182895, -0.07054728, -0.24895027, -0.20600456,\n", "       -0.42033467,  0.34701794, -0.0638995 ,  0.14252576, -0.06025055,\n", "        0.4228329 ,  0.06789401,  0.03919645, -0.17267554,  0.07274943,\n", "       -0.487512  ,  0.04517636, -0.18857062, -0.03975222, -0.2652712 ,\n", "       -0.30853328, -0.50844556,  0.03321444, -0.15481217, -0.20701212,\n", "       -0.40578464, -0.25884995, -0.46550158, -0.4797585 , -0.7324234 ,\n", "        0.43939307, -0.06170902, -0.51546025, -0.19215119, -0.3705445 ,\n", "       -0.57504356, -0.6372961 , -0.9345571 ], dtype=float32)\n", "init: name='_operators.0.nodes.0' type=dtype('int64') shape=(0,) -- array([0, 3], dtype=int64)\n", "init: name='_operators.0.nodes.1' type=dtype('int64') shape=(0,) -- array([1, 2, 5, 9], dtype=int64)\n", "init: name='_operators.0.nodes.2' type=dtype('int64') shape=(0,) -- array([5, 6, 3, 7, 2, 0, 7, 1], dtype=int64)\n", "init: name='_operators.0.nodes.3' type=dtype('int64') shape=(0,) -- array([3, 9, 5, 3, 6, 4, 1, 3, 6, 6, 1, 6, 5, 4, 6, 2], dtype=int64)\n", "init: name='_operators.0.nodes.4' type=dtype('int64') shape=(0,) -- array([3, 2, 7, 6, 2, 4, 7, 8, 9, 5, 7, 8, 9, 4, 6, 9, 7, 9, 0, 7, 7, 9,\n", "       2, 7, 6, 4, 6, 5, 4, 0, 6, 0], dtype=int64)\n", "init: name='_operators.0.nodes.5' type=dtype('int64') shape=(0,) -- array([2, 8, 7, 6, 6, 3, 4, 9, 7, 3, 2, 6, 3, 3, 0, 1, 1, 0, 4, 7, 9, 5,\n", "       7, 9, 5, 3, 5, 9, 0, 5, 1, 4, 9, 4, 7, 7, 1, 9, 1, 1, 6, 2, 7, 7,\n", "       6, 1, 4, 4, 0, 0, 9, 8, 8, 2, 6, 2, 0, 3, 4, 2, 5, 6, 7, 3],\n", "      dtype=int64)\n", "init: name='_operators.0.biases.0' type=dtype('float32') shape=(0,) -- array([ 0.19169255, -0.12246682], dtype=float32)\n", "init: name='_operators.0.biases.1' type=dtype('float32') shape=(0,) -- array([-0.40610337, -0.1467492 , -0.01880287,  0.15879431], dtype=float32)\n", "init: name='_operators.0.biases.2' type=dtype('float32') shape=(0,) -- array([ 0.736786  , -0.32427853,  0.30860555,  0.17994082,  0.6917758 ,\n", "       -0.00594712,  0.35950053, -0.9819274 ], dtype=float32)\n", "init: name='_operators.0.biases.3' type=dtype('float32') shape=(0,) -- array([-1.3495584 , -1.082793  , -0.6906011 , -0.08978076, -0.4007622 ,\n", "        0.10756078, -0.68507075,  0.15814054,  0.5132364 , -0.18426335,\n", "        0.13685235,  0.10721841,  0.01814443, -0.41644228, -0.59770894,\n", "        0.607365  ], dtype=float32)\n", "init: name='_operators.0.biases.4' type=dtype('float32') shape=(0,) -- array([ 1.4203796 , -0.49269757, -0.12210988, -0.09692484,  0.5076643 ,\n", "       -1.3609421 ,  1.154743  ,  2.8748922 , -0.08181615,  0.7741028 ,\n", "        0.20604724,  0.666296  , -0.6474025 ,  0.6459148 ,  0.02262808,\n", "       -0.42282397,  0.46360654, -0.10058792,  0.25486696,  0.60041225,\n", "       -0.06933744,  0.21294908,  0.96443814,  0.07923891,  0.4797698 ,\n", "        1.2852331 ,  0.24348404, -0.3404966 , -0.07175394, -0.8248828 ,\n", "       -0.74071133, -1.2140133 ], dtype=float32)\n", "init: name='_operators.0.biases.5' type=dtype('float32') shape=(0,) -- array([ 1.0626682 ,  1.4745288 ,  0.01898679,  0.5451088 ,  0.15444604,\n", "        1.0631477 , -0.7555804 , -1.7192128 , -0.20905146,  0.19752283,\n", "       -0.40471953,  0.13069782,  0.60331047,  1.5060809 ,  0.        ,\n", "       -1.8283446 , -0.8124372 , -1.381897  ,  0.59209645,  0.3239226 ,\n", "       -0.42840806, -0.43624896,  0.58229303, -1.0196047 , -0.5632828 ,\n", "        0.91483426,  1.8038778 , -0.5665638 , -1.2530733 , -0.6500004 ,\n", "       -1.3069727 ,  0.48267984,  0.73503745, -1.871724  , -1.4965518 ,\n", "        1.3147466 ,  0.03919952, -0.885836  ,  0.5479692 , -0.8086383 ,\n", "       -0.74240863,  0.14582941,  0.6496967 , -0.00911551,  2.4541488 ,\n", "       -0.90482277,  0.26108736,  0.7569448 , -1.0786855 , -0.45229852,\n", "        1.2146595 , -0.6756766 , -2.3066258 ,  0.7911504 ,  0.57490873,\n", "       -0.40741247,  0.24633038, -1.2022957 , -0.65162694, -0.04244827,\n", "        1.558136  , -1.6220782 ,  0.1574643 , -1.4209061 ], dtype=float32)\n", "Constant(value=[-1]) -> onnx::Reshape_27\n", "Gather(X, _operators.0.root_nodes, axis=1) -> onnx::LessOrEqual_17\n", "  LessOrEqual(onnx::LessOrEqual_17, _operators.0.root_biases) -> onnx::Cast_18\n", "    Cast(onnx::Cast_18, to=7) -> onnx::Add_19\n", "      Add(onnx::Add_19, _operators.0.tree_indices) -> onnx::Reshape_20\n", "Constant(value=[-1]) -> onnx::Reshape_21\n", "  Reshape(onnx::Reshape_20, onnx::Reshape_21, allowzero=0) -> onnx::Gather_22\n", "    Gather(_operators.0.nodes.0, onnx::Gather_22, axis=0) -> onnx::Reshape_23\n", "Constant(value=[-1, 1]) -> onnx::Reshape_24\n", "  Reshape(onnx::Reshape_23, onnx::Reshape_24, allowzero=0) -> onnx::GatherElements_25\n", "    GatherElements(X, onnx::GatherElements_25, axis=1) -> onnx::Reshape_26\n", "  Reshape(onnx::Reshape_26, onnx::Reshape_27, allowzero=0) -> onnx::LessOrEqual_28\n", "Constant(value=2) -> onnx::Mul_29\n", "  Mul(onnx::Gather_22, onnx::Mul_29) -> onnx::Add_30\n", "Gather(_operators.0.biases.0, onnx::Gather_22, axis=0) -> onnx::LessOrEqual_31\n", "  LessOrEqual(onnx::LessOrEqual_28, onnx::LessOrEqual_31) -> onnx::Cast_32\n", "    Cast(onnx::Cast_32, to=7) -> onnx::Add_33\n", "    Add(onnx::Add_30, onnx::Add_33) -> onnx::Gather_34\n", "      Gather(_operators.0.nodes.1, onnx::Gather_34, axis=0) -> onnx::Reshape_35\n", "Constant(value=[-1, 1]) -> onnx::Reshape_36\n", "  Reshape(onnx::Reshape_35, onnx::Reshape_36, allowzero=0) -> onnx::GatherElements_37\n", "    GatherElements(X, onnx::GatherElements_37, axis=1) -> onnx::Reshape_38\n", "Constant(value=[-1]) -> onnx::Reshape_39\n", "  Reshape(onnx::Reshape_38, onnx::Reshape_39, allowzero=0) -> onnx::LessOrEqual_40\n", "Constant(value=2) -> onnx::Mul_41\n", "  Mul(onnx::Gather_34, onnx::Mul_41) -> onnx::Add_42\n", "Gather(_operators.0.biases.1, onnx::Gather_34, axis=0) -> onnx::LessOrEqual_43\n", "  LessOrEqual(onnx::LessOrEqual_40, onnx::LessOrEqual_43) -> onnx::Cast_44\n", "    Cast(onnx::Cast_44, to=7) -> onnx::Add_45\n", "    Add(onnx::Add_42, onnx::Add_45) -> onnx::Gather_46\n", "      Gather(_operators.0.nodes.2, onnx::Gather_46, axis=0) -> onnx::Reshape_47\n", "Constant(value=[-1, 1]) -> onnx::Reshape_48\n", "  Reshape(onnx::Reshape_47, onnx::Reshape_48, allowzero=0) -> onnx::GatherElements_49\n", "    GatherElements(X, onnx::GatherElements_49, axis=1) -> onnx::Reshape_50\n", "Constant(value=[-1]) -> onnx::Reshape_51\n", "  Reshape(onnx::Reshape_50, onnx::Reshape_51, allowzero=0) -> onnx::LessOrEqual_52\n", "Constant(value=2) -> onnx::Mul_53\n", "  Mul(onnx::Gather_46, onnx::Mul_53) -> onnx::Add_54\n", "Gather(_operators.0.biases.2, onnx::Gather_46, axis=0) -> onnx::LessOrEqual_55\n", "  LessOrEqual(onnx::LessOrEqual_52, onnx::LessOrEqual_55) -> onnx::Cast_56\n", "    Cast(onnx::Cast_56, to=7) -> onnx::Add_57\n", "    Add(onnx::Add_54, onnx::Add_57) -> onnx::Gather_58\n", "      Gather(_operators.0.nodes.3, onnx::Gather_58, axis=0) -> onnx::Reshape_59\n", "Constant(value=[-1, 1]) -> onnx::Reshape_60\n", "  Reshape(onnx::Reshape_59, onnx::Reshape_60, allowzero=0) -> onnx::GatherElements_61\n", "    GatherElements(X, onnx::GatherElements_61, axis=1) -> onnx::Reshape_62\n", "Constant(value=[-1]) -> onnx::Reshape_63\n", "  Reshape(onnx::Reshape_62, onnx::Reshape_63, allowzero=0) -> onnx::LessOrEqual_64\n", "Constant(value=2) -> onnx::Mul_65\n", "  Mul(onnx::Gather_58, onnx::Mul_65) -> onnx::Add_66\n", "Gather(_operators.0.biases.3, onnx::Gather_58, axis=0) -> onnx::LessOrEqual_67\n", "  LessOrEqual(onnx::LessOrEqual_64, onnx::LessOrEqual_67) -> onnx::Cast_68\n", "    Cast(onnx::Cast_68, to=7) -> onnx::Add_69\n", "    Add(onnx::Add_66, onnx::Add_69) -> onnx::Gather_70\n", "      Gather(_operators.0.nodes.4, onnx::Gather_70, axis=0) -> onnx::Reshape_71\n", "Constant(value=[-1, 1]) -> onnx::Reshape_72\n", "  Reshape(onnx::Reshape_71, onnx::Reshape_72, allowzero=0) -> onnx::GatherElements_73\n", "    GatherElements(X, onnx::GatherElements_73, axis=1) -> onnx::Reshape_74\n", "Constant(value=[-1]) -> onnx::Reshape_75\n", "  Reshape(onnx::Reshape_74, onnx::Reshape_75, allowzero=0) -> onnx::LessOrEqual_76\n", "Constant(value=2) -> onnx::Mul_77\n", "  Mul(onnx::Gather_70, onnx::Mul_77) -> onnx::Add_78\n", "Gather(_operators.0.biases.4, onnx::Gather_70, axis=0) -> onnx::LessOrEqual_79\n", "  LessOrEqual(onnx::LessOrEqual_76, onnx::LessOrEqual_79) -> onnx::Cast_80\n", "    Cast(onnx::Cast_80, to=7) -> onnx::Add_81\n", "    Add(onnx::Add_78, onnx::Add_81) -> onnx::Gather_82\n", "      Gather(_operators.0.nodes.5, onnx::Gather_82, axis=0) -> onnx::Reshape_83\n", "Constant(value=[-1, 1]) -> onnx::Reshape_84\n", "  Reshape(onnx::Reshape_83, onnx::Reshape_84, allowzero=0) -> onnx::GatherElements_85\n", "    GatherElements(X, onnx::GatherElements_85, axis=1) -> onnx::Reshape_86\n", "Constant(value=[-1]) -> onnx::Reshape_87\n", "  Reshape(onnx::Reshape_86, onnx::Reshape_87, allowzero=0) -> onnx::LessOrEqual_88\n", "Constant(value=2) -> onnx::Mul_89\n", "  Mul(onnx::Gather_82, onnx::Mul_89) -> onnx::Add_90\n", "Gather(_operators.0.biases.5, onnx::Gather_82, axis=0) -> onnx::LessOrEqual_91\n", "  LessOrEqual(onnx::LessOrEqual_88, onnx::LessOrEqual_91) -> onnx::Cast_92\n", "    Cast(onnx::Cast_92, to=7) -> onnx::Add_93\n", "    Add(onnx::Add_90, onnx::Add_93) -> onnx::Gather_94\n", "      Gather(_operators.0.leaf_nodes, onnx::Gather_94, axis=0) -> onnx::Reshape_95\n", "Constant(value=[-1, 1, 1]) -> onnx::Reshape_96\n", "  Reshape(onnx::Reshape_95, onnx::Reshape_96, allowzero=0) -> output\n", "Constant(value=[1]) -> onnx::ReduceSum_98\n", "  ReduceSum(output, onnx::ReduceSum_98, keepdims=0) -> variable\n"]}, {"name": "stdout", "output_type": "stream", "text": ["output: name='variable' type=dtype('float32') shape=['batch_size', 'ReduceSumvariable_dim_1']\n"]}], "source": ["print(onnx_simple_text_plot(onxh, raise_exc=False))"]}, {"cell_type": "code", "execution_count": 43, "id": "822bfa80", "metadata": {}, "outputs": [{"data": {"text/html": ["<div id=\"M383513f79aa048579be42d593ab48485-cont\"><div id=\"M383513f79aa048579be42d593ab48485\" style=\"width:;height:;\"></div></div>\n", "<script>\n", "\n", "require(['http://www.xavierdupre.fr/js/vizjs/viz.js'], function() { var svgGraph = Viz(\"digraph{\\n  nodesep=0.05;\\n  orientation=portrait;\\n  size=7;\\n  ranksep=0.25;\\n\\n  X [shape=box color=red label=\\\"X\\nfloat((0, 10))\\\" fontsize=10];\\n\\n  variable [shape=box color=green label=\\\"variable\\nfloat((0, 0))\\\" fontsize=10];\\n\\n  _operators_0_root_nodes [shape=box label=\\\"_operators_0_root_nodes\\nint64((1,))\\n[8]\\\" fontsize=10];\\n  _operators_0_root_biases [shape=box label=\\\"_operators_0_root_biases\\nfloat32((1,))\\n[0.00792999]\\\" fontsize=10];\\n  _operators_0_tree_indices [shape=box label=\\\"_operators_0_tree_indices\\nint64((1,))\\n[0]\\\" fontsize=10];\\n  _operators_0_leaf_nodes [shape=box label=\\\"_operators_0_leaf_nodes\\nfloat32((128, 1))\\n[[ 1.0096807 ]\\n [ 0.6169403 ]\\n [ 0.61055773]\\n [ 0....\\\" fontsize=10];\\n  _operators_0_nodes_0 [shape=box label=\\\"_operators_0_nodes_0\\nint64((2,))\\n[0 3]\\\" fontsize=10];\\n  _operators_0_nodes_1 [shape=box label=\\\"_operators_0_nodes_1\\nint64((4,))\\n[1 2 5 9]\\\" fontsize=10];\\n  _operators_0_nodes_2 [shape=box label=\\\"_operators_0_nodes_2\\nint64((8,))\\n[5 6 3 7 2 0 7 1]\\\" fontsize=10];\\n  _operators_0_nodes_3 [shape=box label=\\\"_operators_0_nodes_3\\nint64((16,))\\n[3 9 5 3 6 4 1 3 6 6 1 6 5 4 6 2]\\\" fontsize=10];\\n  _operators_0_nodes_4 [shape=box label=\\\"_operators_0_nodes_4\\nint64((32,))\\n[3 2 7 6 2 4 7 8 9 5 7 8 9 4 6 9 7 9 0 7 7 9 2 7 6...\\\" fontsize=10];\\n  _operators_0_nodes_5 [shape=box label=\\\"_operators_0_nodes_5\\nint64((64,))\\n[2 8 7 6 6 3 4 9 7 3 2 6 3 3 0 1 1 0 4 7 9 5 7 9 5...\\\" fontsize=10];\\n  _operators_0_biases_0 [shape=box label=\\\"_operators_0_biases_0\\nfloat32((2,))\\n[ 0.19169255 -0.12246682]\\\" fontsize=10];\\n  _operators_0_biases_1 [shape=box label=\\\"_operators_0_biases_1\\nfloat32((4,))\\n[-0.40610337 -0.1467492  -0.01880287  0.15879431]\\\" fontsize=10];\\n  _operators_0_biases_2 [shape=box label=\\\"_operators_0_biases_2\\nfloat32((8,))\\n[ 0.736786   -0.32427853  0.30860555  0.17994082  0.6917758  -0.00594712\\n  0.35950053 -0.9819274 ]\\\" fontsize=10];\\n  _operators_0_biases_3 [shape=box label=\\\"_operators_0_biases_3\\nfloat32((16,))\\n[-1.3495584  -1.082793   -0.6906011  -0.08978076 -...\\\" fontsize=10];\\n  _operators_0_biases_4 [shape=box label=\\\"_operators_0_biases_4\\nfloat32((32,))\\n[ 1.4203796  -0.49269757 -0.12210988 -0.09692484  ...\\\" fontsize=10];\\n  _operators_0_biases_5 [shape=box label=\\\"_operators_0_biases_5\\nfloat32((64,))\\n[ 1.0626682   1.4745288   0.01898679  0.5451088   ...\\\" fontsize=10];\\n\\n  onnx____LessOrEqual_17 [shape=box label=\\\"onnx____LessOrEqual_17\\\" fontsize=10];\\n  Gather_0 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_0)\\naxis=1\\\" fontsize=10];\\n  X -> Gather_0;\\n  _operators_0_root_nodes -> Gather_0;\\n  Gather_0 -> onnx____LessOrEqual_17;\\n\\n  onnx____Cast_18 [shape=box label=\\\"onnx____Cast_18\\\" fontsize=10];\\n  LessOrEqual_1 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_1)\\\" fontsize=10];\\n  onnx____LessOrEqual_17 -> LessOrEqual_1;\\n  _operators_0_root_biases -> LessOrEqual_1;\\n  LessOrEqual_1 -> onnx____Cast_18;\\n\\n  onnx____Add_19 [shape=box label=\\\"onnx____Add_19\\\" fontsize=10];\\n  Cast_2 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_2)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_18 -> Cast_2;\\n  Cast_2 -> onnx____Add_19;\\n\\n  onnx____Reshape_20 [shape=box label=\\\"onnx____Reshape_20\\\" fontsize=10];\\n  Add_3 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_3)\\\" fontsize=10];\\n  onnx____Add_19 -> Add_3;\\n  _operators_0_tree_indices -> Add_3;\\n  Add_3 -> onnx____Reshape_20;\\n\\n  onnx____Reshape_21 [shape=box label=\\\"onnx____Reshape_21\\\" fontsize=10];\\n  Constant_4 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_4)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_4 -> onnx____Reshape_21;\\n\\n  onnx____Gather_22 [shape=box label=\\\"onnx____Gather_22\\\" fontsize=10];\\n  Reshape_5 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_5)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_20 -> Reshape_5;\\n  onnx____Reshape_21 -> Reshape_5;\\n  Reshape_5 -> onnx____Gather_22;\\n\\n  onnx____Reshape_23 [shape=box label=\\\"onnx____Reshape_23\\\" fontsize=10];\\n  Gather_6 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_6)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_0 -> Gather_6;\\n  onnx____Gather_22 -> Gather_6;\\n  Gather_6 -> onnx____Reshape_23;\\n\\n  onnx____Reshape_24 [shape=box label=\\\"onnx____Reshape_24\\\" fontsize=10];\\n  Constant_7 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_7)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_7 -> onnx____Reshape_24;\\n\\n  onnx____GatherElements_25 [shape=box label=\\\"onnx____GatherElements_25\\\" fontsize=10];\\n  Reshape_8 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_8)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_23 -> Reshape_8;\\n  onnx____Reshape_24 -> Reshape_8;\\n  Reshape_8 -> onnx____GatherElements_25;\\n\\n  onnx____Reshape_26 [shape=box label=\\\"onnx____Reshape_26\\\" fontsize=10];\\n  GatherElements_9 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_9)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_9;\\n  onnx____GatherElements_25 -> GatherElements_9;\\n  GatherElements_9 -> onnx____Reshape_26;\\n\\n  onnx____Reshape_27 [shape=box label=\\\"onnx____Reshape_27\\\" fontsize=10];\\n  Constant_10 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_10)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_10 -> onnx____Reshape_27;\\n\\n  onnx____LessOrEqual_28 [shape=box label=\\\"onnx____LessOrEqual_28\\\" fontsize=10];\\n  Reshape_11 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_11)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_26 -> Reshape_11;\\n  onnx____Reshape_27 -> Reshape_11;\\n  Reshape_11 -> onnx____LessOrEqual_28;\\n\\n  onnx____Mul_29 [shape=box label=\\\"onnx____Mul_29\\\" fontsize=10];\\n  Constant_12 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_12)\\nvalue=2\\\" fontsize=10];\\n  Constant_12 -> onnx____Mul_29;\\n\\n  onnx____Add_30 [shape=box label=\\\"onnx____Add_30\\\" fontsize=10];\\n  Mul_13 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_13)\\\" fontsize=10];\\n  onnx____Gather_22 -> Mul_13;\\n  onnx____Mul_29 -> Mul_13;\\n  Mul_13 -> onnx____Add_30;\\n\\n  onnx____LessOrEqual_31 [shape=box label=\\\"onnx____LessOrEqual_31\\\" fontsize=10];\\n  Gather_14 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_14)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_0 -> Gather_14;\\n  onnx____Gather_22 -> Gather_14;\\n  Gather_14 -> onnx____LessOrEqual_31;\\n\\n  onnx____Cast_32 [shape=box label=\\\"onnx____Cast_32\\\" fontsize=10];\\n  LessOrEqual_15 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_15)\\\" fontsize=10];\\n  onnx____LessOrEqual_28 -> LessOrEqual_15;\\n  onnx____LessOrEqual_31 -> LessOrEqual_15;\\n  LessOrEqual_15 -> onnx____Cast_32;\\n\\n  onnx____Add_33 [shape=box label=\\\"onnx____Add_33\\\" fontsize=10];\\n  Cast_16 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_16)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_32 -> Cast_16;\\n  Cast_16 -> onnx____Add_33;\\n\\n  onnx____Gather_34 [shape=box label=\\\"onnx____Gather_34\\\" fontsize=10];\\n  Add_17 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_17)\\\" fontsize=10];\\n  onnx____Add_30 -> Add_17;\\n  onnx____Add_33 -> Add_17;\\n  Add_17 -> onnx____Gather_34;\\n\\n  onnx____Reshape_35 [shape=box label=\\\"onnx____Reshape_35\\\" fontsize=10];\\n  Gather_18 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_18)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_1 -> Gather_18;\\n  onnx____Gather_34 -> Gather_18;\\n  Gather_18 -> onnx____Reshape_35;\\n\\n  onnx____Reshape_36 [shape=box label=\\\"onnx____Reshape_36\\\" fontsize=10];\\n  Constant_19 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_19)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_19 -> onnx____Reshape_36;\\n\\n  onnx____GatherElements_37 [shape=box label=\\\"onnx____GatherElements_37\\\" fontsize=10];\\n  Reshape_20 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_20)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_35 -> Reshape_20;\\n  onnx____Reshape_36 -> Reshape_20;\\n  Reshape_20 -> onnx____GatherElements_37;\\n\\n  onnx____Reshape_38 [shape=box label=\\\"onnx____Reshape_38\\\" fontsize=10];\\n  GatherElements_21 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_21)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_21;\\n  onnx____GatherElements_37 -> GatherElements_21;\\n  GatherElements_21 -> onnx____Reshape_38;\\n\\n  onnx____Reshape_39 [shape=box label=\\\"onnx____Reshape_39\\\" fontsize=10];\\n  Constant_22 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_22)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_22 -> onnx____Reshape_39;\\n\\n  onnx____LessOrEqual_40 [shape=box label=\\\"onnx____LessOrEqual_40\\\" fontsize=10];\\n  Reshape_23 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_23)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_38 -> Reshape_23;\\n  onnx____Reshape_39 -> Reshape_23;\\n  Reshape_23 -> onnx____LessOrEqual_40;\\n\\n  onnx____Mul_41 [shape=box label=\\\"onnx____Mul_41\\\" fontsize=10];\\n  Constant_24 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_24)\\nvalue=2\\\" fontsize=10];\\n  Constant_24 -> onnx____Mul_41;\\n\\n  onnx____Add_42 [shape=box label=\\\"onnx____Add_42\\\" fontsize=10];\\n  Mul_25 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_25)\\\" fontsize=10];\\n  onnx____Gather_34 -> Mul_25;\\n  onnx____Mul_41 -> Mul_25;\\n  Mul_25 -> onnx____Add_42;\\n\\n  onnx____LessOrEqual_43 [shape=box label=\\\"onnx____LessOrEqual_43\\\" fontsize=10];\\n  Gather_26 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_26)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_1 -> Gather_26;\\n  onnx____Gather_34 -> Gather_26;\\n  Gather_26 -> onnx____LessOrEqual_43;\\n\\n  onnx____Cast_44 [shape=box label=\\\"onnx____Cast_44\\\" fontsize=10];\\n  LessOrEqual_27 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_27)\\\" fontsize=10];\\n  onnx____LessOrEqual_40 -> LessOrEqual_27;\\n  onnx____LessOrEqual_43 -> LessOrEqual_27;\\n  LessOrEqual_27 -> onnx____Cast_44;\\n\\n  onnx____Add_45 [shape=box label=\\\"onnx____Add_45\\\" fontsize=10];\\n  Cast_28 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_28)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_44 -> Cast_28;\\n  Cast_28 -> onnx____Add_45;\\n\\n  onnx____Gather_46 [shape=box label=\\\"onnx____Gather_46\\\" fontsize=10];\\n  Add_29 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_29)\\\" fontsize=10];\\n  onnx____Add_42 -> Add_29;\\n  onnx____Add_45 -> Add_29;\\n  Add_29 -> onnx____Gather_46;\\n\\n  onnx____Reshape_47 [shape=box label=\\\"onnx____Reshape_47\\\" fontsize=10];\\n  Gather_30 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_30)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_2 -> Gather_30;\\n  onnx____Gather_46 -> Gather_30;\\n  Gather_30 -> onnx____Reshape_47;\\n\\n  onnx____Reshape_48 [shape=box label=\\\"onnx____Reshape_48\\\" fontsize=10];\\n  Constant_31 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_31)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_31 -> onnx____Reshape_48;\\n\\n  onnx____GatherElements_49 [shape=box label=\\\"onnx____GatherElements_49\\\" fontsize=10];\\n  Reshape_32 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_32)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_47 -> Reshape_32;\\n  onnx____Reshape_48 -> Reshape_32;\\n  Reshape_32 -> onnx____GatherElements_49;\\n\\n  onnx____Reshape_50 [shape=box label=\\\"onnx____Reshape_50\\\" fontsize=10];\\n  GatherElements_33 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_33)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_33;\\n  onnx____GatherElements_49 -> GatherElements_33;\\n  GatherElements_33 -> onnx____Reshape_50;\\n\\n  onnx____Reshape_51 [shape=box label=\\\"onnx____Reshape_51\\\" fontsize=10];\\n  Constant_34 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_34)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_34 -> onnx____Reshape_51;\\n\\n  onnx____LessOrEqual_52 [shape=box label=\\\"onnx____LessOrEqual_52\\\" fontsize=10];\\n  Reshape_35 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_35)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_50 -> Reshape_35;\\n  onnx____Reshape_51 -> Reshape_35;\\n  Reshape_35 -> onnx____LessOrEqual_52;\\n\\n  onnx____Mul_53 [shape=box label=\\\"onnx____Mul_53\\\" fontsize=10];\\n  Constant_36 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_36)\\nvalue=2\\\" fontsize=10];\\n  Constant_36 -> onnx____Mul_53;\\n\\n  onnx____Add_54 [shape=box label=\\\"onnx____Add_54\\\" fontsize=10];\\n  Mul_37 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_37)\\\" fontsize=10];\\n  onnx____Gather_46 -> Mul_37;\\n  onnx____Mul_53 -> Mul_37;\\n  Mul_37 -> onnx____Add_54;\\n\\n  onnx____LessOrEqual_55 [shape=box label=\\\"onnx____LessOrEqual_55\\\" fontsize=10];\\n  Gather_38 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_38)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_2 -> Gather_38;\\n  onnx____Gather_46 -> Gather_38;\\n  Gather_38 -> onnx____LessOrEqual_55;\\n\\n  onnx____Cast_56 [shape=box label=\\\"onnx____Cast_56\\\" fontsize=10];\\n  LessOrEqual_39 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_39)\\\" fontsize=10];\\n  onnx____LessOrEqual_52 -> LessOrEqual_39;\\n  onnx____LessOrEqual_55 -> LessOrEqual_39;\\n  LessOrEqual_39 -> onnx____Cast_56;\\n\\n  onnx____Add_57 [shape=box label=\\\"onnx____Add_57\\\" fontsize=10];\\n  Cast_40 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_40)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_56 -> Cast_40;\\n  Cast_40 -> onnx____Add_57;\\n\\n  onnx____Gather_58 [shape=box label=\\\"onnx____Gather_58\\\" fontsize=10];\\n  Add_41 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_41)\\\" fontsize=10];\\n  onnx____Add_54 -> Add_41;\\n  onnx____Add_57 -> Add_41;\\n  Add_41 -> onnx____Gather_58;\\n\\n  onnx____Reshape_59 [shape=box label=\\\"onnx____Reshape_59\\\" fontsize=10];\\n  Gather_42 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_42)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_3 -> Gather_42;\\n  onnx____Gather_58 -> Gather_42;\\n  Gather_42 -> onnx____Reshape_59;\\n\\n  onnx____Reshape_60 [shape=box label=\\\"onnx____Reshape_60\\\" fontsize=10];\\n  Constant_43 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_43)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_43 -> onnx____Reshape_60;\\n\\n  onnx____GatherElements_61 [shape=box label=\\\"onnx____GatherElements_61\\\" fontsize=10];\\n  Reshape_44 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_44)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_59 -> Reshape_44;\\n  onnx____Reshape_60 -> Reshape_44;\\n  Reshape_44 -> onnx____GatherElements_61;\\n\\n  onnx____Reshape_62 [shape=box label=\\\"onnx____Reshape_62\\\" fontsize=10];\\n  GatherElements_45 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_45)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_45;\\n  onnx____GatherElements_61 -> GatherElements_45;\\n  GatherElements_45 -> onnx____Reshape_62;\\n\\n  onnx____Reshape_63 [shape=box label=\\\"onnx____Reshape_63\\\" fontsize=10];\\n  Constant_46 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_46)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_46 -> onnx____Reshape_63;\\n\\n  onnx____LessOrEqual_64 [shape=box label=\\\"onnx____LessOrEqual_64\\\" fontsize=10];\\n  Reshape_47 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_47)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_62 -> Reshape_47;\\n  onnx____Reshape_63 -> Reshape_47;\\n  Reshape_47 -> onnx____LessOrEqual_64;\\n\\n  onnx____Mul_65 [shape=box label=\\\"onnx____Mul_65\\\" fontsize=10];\\n  Constant_48 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_48)\\nvalue=2\\\" fontsize=10];\\n  Constant_48 -> onnx____Mul_65;\\n\\n  onnx____Add_66 [shape=box label=\\\"onnx____Add_66\\\" fontsize=10];\\n  Mul_49 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_49)\\\" fontsize=10];\\n  onnx____Gather_58 -> Mul_49;\\n  onnx____Mul_65 -> Mul_49;\\n  Mul_49 -> onnx____Add_66;\\n\\n  onnx____LessOrEqual_67 [shape=box label=\\\"onnx____LessOrEqual_67\\\" fontsize=10];\\n  Gather_50 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_50)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_3 -> Gather_50;\\n  onnx____Gather_58 -> Gather_50;\\n  Gather_50 -> onnx____LessOrEqual_67;\\n\\n  onnx____Cast_68 [shape=box label=\\\"onnx____Cast_68\\\" fontsize=10];\\n  LessOrEqual_51 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_51)\\\" fontsize=10];\\n  onnx____LessOrEqual_64 -> LessOrEqual_51;\\n  onnx____LessOrEqual_67 -> LessOrEqual_51;\\n  LessOrEqual_51 -> onnx____Cast_68;\\n\\n  onnx____Add_69 [shape=box label=\\\"onnx____Add_69\\\" fontsize=10];\\n  Cast_52 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_52)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_68 -> Cast_52;\\n  Cast_52 -> onnx____Add_69;\\n\\n  onnx____Gather_70 [shape=box label=\\\"onnx____Gather_70\\\" fontsize=10];\\n  Add_53 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_53)\\\" fontsize=10];\\n  onnx____Add_66 -> Add_53;\\n  onnx____Add_69 -> Add_53;\\n  Add_53 -> onnx____Gather_70;\\n\\n  onnx____Reshape_71 [shape=box label=\\\"onnx____Reshape_71\\\" fontsize=10];\\n  Gather_54 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_54)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_4 -> Gather_54;\\n  onnx____Gather_70 -> Gather_54;\\n  Gather_54 -> onnx____Reshape_71;\\n\\n  onnx____Reshape_72 [shape=box label=\\\"onnx____Reshape_72\\\" fontsize=10];\\n  Constant_55 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_55)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_55 -> onnx____Reshape_72;\\n\\n  onnx____GatherElements_73 [shape=box label=\\\"onnx____GatherElements_73\\\" fontsize=10];\\n  Reshape_56 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_56)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_71 -> Reshape_56;\\n  onnx____Reshape_72 -> Reshape_56;\\n  Reshape_56 -> onnx____GatherElements_73;\\n\\n  onnx____Reshape_74 [shape=box label=\\\"onnx____Reshape_74\\\" fontsize=10];\\n  GatherElements_57 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_57)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_57;\\n  onnx____GatherElements_73 -> GatherElements_57;\\n  GatherElements_57 -> onnx____Reshape_74;\\n\\n  onnx____Reshape_75 [shape=box label=\\\"onnx____Reshape_75\\\" fontsize=10];\\n  Constant_58 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_58)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_58 -> onnx____Reshape_75;\\n\\n  onnx____LessOrEqual_76 [shape=box label=\\\"onnx____LessOrEqual_76\\\" fontsize=10];\\n  Reshape_59 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_59)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_74 -> Reshape_59;\\n  onnx____Reshape_75 -> Reshape_59;\\n  Reshape_59 -> onnx____LessOrEqual_76;\\n\\n  onnx____Mul_77 [shape=box label=\\\"onnx____Mul_77\\\" fontsize=10];\\n  Constant_60 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_60)\\nvalue=2\\\" fontsize=10];\\n  Constant_60 -> onnx____Mul_77;\\n\\n  onnx____Add_78 [shape=box label=\\\"onnx____Add_78\\\" fontsize=10];\\n  Mul_61 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_61)\\\" fontsize=10];\\n  onnx____Gather_70 -> Mul_61;\\n  onnx____Mul_77 -> Mul_61;\\n  Mul_61 -> onnx____Add_78;\\n\\n  onnx____LessOrEqual_79 [shape=box label=\\\"onnx____LessOrEqual_79\\\" fontsize=10];\\n  Gather_62 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_62)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_4 -> Gather_62;\\n  onnx____Gather_70 -> Gather_62;\\n  Gather_62 -> onnx____LessOrEqual_79;\\n\\n  onnx____Cast_80 [shape=box label=\\\"onnx____Cast_80\\\" fontsize=10];\\n  LessOrEqual_63 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_63)\\\" fontsize=10];\\n  onnx____LessOrEqual_76 -> LessOrEqual_63;\\n  onnx____LessOrEqual_79 -> LessOrEqual_63;\\n  LessOrEqual_63 -> onnx____Cast_80;\\n\\n  onnx____Add_81 [shape=box label=\\\"onnx____Add_81\\\" fontsize=10];\\n  Cast_64 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_64)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_80 -> Cast_64;\\n  Cast_64 -> onnx____Add_81;\\n\\n  onnx____Gather_82 [shape=box label=\\\"onnx____Gather_82\\\" fontsize=10];\\n  Add_65 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_65)\\\" fontsize=10];\\n  onnx____Add_78 -> Add_65;\\n  onnx____Add_81 -> Add_65;\\n  Add_65 -> onnx____Gather_82;\\n\\n  onnx____Reshape_83 [shape=box label=\\\"onnx____Reshape_83\\\" fontsize=10];\\n  Gather_66 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_66)\\naxis=0\\\" fontsize=10];\\n  _operators_0_nodes_5 -> Gather_66;\\n  onnx____Gather_82 -> Gather_66;\\n  Gather_66 -> onnx____Reshape_83;\\n\\n  onnx____Reshape_84 [shape=box label=\\\"onnx____Reshape_84\\\" fontsize=10];\\n  Constant_67 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_67)\\nvalue=[-1  1]\\\" fontsize=10];\\n  Constant_67 -> onnx____Reshape_84;\\n\\n  onnx____GatherElements_85 [shape=box label=\\\"onnx____GatherElements_85\\\" fontsize=10];\\n  Reshape_68 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_68)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_83 -> Reshape_68;\\n  onnx____Reshape_84 -> Reshape_68;\\n  Reshape_68 -> onnx____GatherElements_85;\\n\\n  onnx____Reshape_86 [shape=box label=\\\"onnx____Reshape_86\\\" fontsize=10];\\n  GatherElements_69 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"GatherElements\\n(GatherElements_69)\\naxis=1\\\" fontsize=10];\\n  X -> GatherElements_69;\\n  onnx____GatherElements_85 -> GatherElements_69;\\n  GatherElements_69 -> onnx____Reshape_86;\\n\\n  onnx____Reshape_87 [shape=box label=\\\"onnx____Reshape_87\\\" fontsize=10];\\n  Constant_70 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_70)\\nvalue=[-1]\\\" fontsize=10];\\n  Constant_70 -> onnx____Reshape_87;\\n\\n  onnx____LessOrEqual_88 [shape=box label=\\\"onnx____LessOrEqual_88\\\" fontsize=10];\\n  Reshape_71 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_71)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_86 -> Reshape_71;\\n  onnx____Reshape_87 -> Reshape_71;\\n  Reshape_71 -> onnx____LessOrEqual_88;\\n\\n  onnx____Mul_89 [shape=box label=\\\"onnx____Mul_89\\\" fontsize=10];\\n  Constant_72 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_72)\\nvalue=2\\\" fontsize=10];\\n  Constant_72 -> onnx____Mul_89;\\n\\n  onnx____Add_90 [shape=box label=\\\"onnx____Add_90\\\" fontsize=10];\\n  Mul_73 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Mul\\n(Mul_73)\\\" fontsize=10];\\n  onnx____Gather_82 -> Mul_73;\\n  onnx____Mul_89 -> Mul_73;\\n  Mul_73 -> onnx____Add_90;\\n\\n  onnx____LessOrEqual_91 [shape=box label=\\\"onnx____LessOrEqual_91\\\" fontsize=10];\\n  Gather_74 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_74)\\naxis=0\\\" fontsize=10];\\n  _operators_0_biases_5 -> Gather_74;\\n  onnx____Gather_82 -> Gather_74;\\n  Gather_74 -> onnx____LessOrEqual_91;\\n\\n  onnx____Cast_92 [shape=box label=\\\"onnx____Cast_92\\\" fontsize=10];\\n  LessOrEqual_75 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"LessOrEqual\\n(LessOrEqual_75)\\\" fontsize=10];\\n  onnx____LessOrEqual_88 -> LessOrEqual_75;\\n  onnx____LessOrEqual_91 -> LessOrEqual_75;\\n  LessOrEqual_75 -> onnx____Cast_92;\\n\\n  onnx____Add_93 [shape=box label=\\\"onnx____Add_93\\\" fontsize=10];\\n  Cast_76 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Cast\\n(Cast_76)\\nto=7\\\" fontsize=10];\\n  onnx____Cast_92 -> Cast_76;\\n  Cast_76 -> onnx____Add_93;\\n\\n  onnx____Gather_94 [shape=box label=\\\"onnx____Gather_94\\\" fontsize=10];\\n  Add_77 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Add\\n(Add_77)\\\" fontsize=10];\\n  onnx____Add_90 -> Add_77;\\n  onnx____Add_93 -> Add_77;\\n  Add_77 -> onnx____Gather_94;\\n\\n  onnx____Reshape_95 [shape=box label=\\\"onnx____Reshape_95\\\" fontsize=10];\\n  Gather_78 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Gather\\n(Gather_78)\\naxis=0\\\" fontsize=10];\\n  _operators_0_leaf_nodes -> Gather_78;\\n  onnx____Gather_94 -> Gather_78;\\n  Gather_78 -> onnx____Reshape_95;\\n\\n  onnx____Reshape_96 [shape=box label=\\\"onnx____Reshape_96\\\" fontsize=10];\\n  Constant_79 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_79)\\nvalue=[-1  1  1]\\\" fontsize=10];\\n  Constant_79 -> onnx____Reshape_96;\\n\\n  output [shape=box label=\\\"output\\\" fontsize=10];\\n  Reshape_80 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Reshape\\n(Reshape_80)\\nallowzero=0\\\" fontsize=10];\\n  onnx____Reshape_95 -> Reshape_80;\\n  onnx____Reshape_96 -> Reshape_80;\\n  Reshape_80 -> output;\\n\\n  onnx____ReduceSum_98 [shape=box label=\\\"onnx____ReduceSum_98\\\" fontsize=10];\\n  Constant_81 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"Constant\\n(Constant_81)\\nvalue=[1]\\\" fontsize=10];\\n  Constant_81 -> onnx____ReduceSum_98;\\n\\n  ReduceSum_82 [shape=box style=\\\"filled,rounded\\\" color=orange label=\\\"ReduceSum\\n(ReduceSum_82)\\nkeepdims=0\\\" fontsize=10];\\n  output -> ReduceSum_82;\\n  onnx____ReduceSum_98 -> ReduceSum_82;\\n  ReduceSum_82 -> variable;\\n}\");\n", "document.getElementById('M383513f79aa048579be42d593ab48485').innerHTML = svgGraph; });\n", "\n", "</script>"], "text/plain": ["<jyquickhelper.jspy.render_nb_js_dot.RenderJsDot at 0x1ff466b22c0>"]}, "execution_count": 44, "metadata": {}, "output_type": "execute_result"}], "source": ["%onnxview onxh"]}, {"cell_type": "markdown", "id": "1edb6177", "metadata": {}, "source": ["La librairie r\u00e9impl\u00e9mente la d\u00e9cision d'un arbre d\u00e9cision \u00e0 partir d'un produit matriciel pour chaque niveau de l'arbre. Tous les seuils sont \u00e9valu\u00e9s. Les matrices n'ont pas besoin d'\u00eatre sparses car les features n\u00e9cessaires sont r\u00e9cup\u00e9r\u00e9es. Le seuil de d\u00e9cision est impl\u00e9ment\u00e9 avec un test et non une sigmo\u00efde. Ce mod\u00e8le est donc identique en terme de pr\u00e9diction au mod\u00e8le initial."]}, {"cell_type": "code", "execution_count": 44, "id": "2220ca2e", "metadata": {}, "outputs": [{"data": {"text/plain": ["1.7421041873949668"]}, "execution_count": 45, "metadata": {}, "output_type": "execute_result"}], "source": ["oinfh = OnnxInference(onxh, runtime='onnxruntime1')\n", "expected = tree.predict(x_exp)\n", "\n", "got = oinfh.run({'X': x_exp.astype(numpy.float32)})['variable']\n", "numpy.abs(got - expected).max()"]}, {"cell_type": "markdown", "id": "10de2a80", "metadata": {}, "source": ["La conversion reste imparfaite \u00e9galement."]}, {"cell_type": "code", "execution_count": 45, "id": "fd13b28b", "metadata": {}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["3.13 ms \u00b1 445 \u00b5s per loop (mean \u00b1 std. dev. of 7 runs, 100 loops each)\n"]}], "source": ["%timeit oinfh.run({'X': x_exp32})['variable']"]}, {"cell_type": "markdown", "id": "11a36a32", "metadata": {}, "source": ["Et le temps de calcul est aussi plus long."]}, {"cell_type": "markdown", "id": "20afcc41", "metadata": {}, "source": ["## Apprentissage\n", "\n", "L'id\u00e9e derri\u00e8re tout cela est aussi de pouvoir r\u00e9estimer les coefficients du r\u00e9seau de neurones une fois converti."]}, {"cell_type": "code", "execution_count": 46, "id": "96abfddb", "metadata": {}, "outputs": [], "source": ["x_train = X_train[:100]\n", "expected = tree.predict(x_train)\n", "reg = NeuralTreeNetRegressor(trees[1], verbose=1, max_iter=10, lr=1e-4)"]}, {"cell_type": "code", "execution_count": 47, "id": "94dc4d66", "metadata": {}, "outputs": [{"data": {"text/plain": ["(1.0246115055833722, 0.24094382754240642)"]}, "execution_count": 48, "metadata": {}, "output_type": "execute_result"}], "source": ["got = reg.predict(x_train)\n", "numpy.abs(got - expected).max(), numpy.abs(got - expected).mean()"]}, {"cell_type": "markdown", "id": "111970a1", "metadata": {}, "source": ["La diff\u00e9rence est grande."]}, {"cell_type": "code", "execution_count": 48, "id": "a50b3384", "metadata": {"scrolled": false}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["0/10: loss: 3.201 lr=0.0001 max(coef): 6.5 l1=0/1.5e+03 l2=0/2.5e+03\n", "1/10: loss: 2.593 lr=9.95e-06 max(coef): 6.5 l1=2e+03/1.5e+03 l2=1.3e+03/2.5e+03\n", "2/10: loss: 2.506 lr=7.05e-06 max(coef): 6.5 l1=1.4e+02/1.5e+03 l2=6.2/2.5e+03\n", "3/10: loss: 2.461 lr=5.76e-06 max(coef): 6.5 l1=1.2e+03/1.5e+03 l2=6.8e+02/2.5e+03\n", "4/10: loss: 2.429 lr=4.99e-06 max(coef): 6.5 l1=6.5e+02/1.5e+03 l2=2.1e+02/2.5e+03\n", "5/10: loss: 2.405 lr=4.47e-06 max(coef): 6.5 l1=1.9e+02/1.5e+03 l2=13/2.5e+03\n", "6/10: loss: 2.392 lr=4.08e-06 max(coef): 6.5 l1=1.6e+02/1.5e+03 l2=6.8/2.5e+03\n", "7/10: loss: 2.375 lr=3.78e-06 max(coef): 6.5 l1=1.8e+02/1.5e+03 l2=9.5/2.5e+03\n", "8/10: loss: 2.358 lr=3.53e-06 max(coef): 6.5 l1=1.1e+02/1.5e+03 l2=7/2.5e+03\n", "9/10: loss: 2.345 lr=3.33e-06 max(coef): 6.5 l1=3.7e+02/1.5e+03 l2=56/2.5e+03\n", "10/10: loss: 2.333 lr=3.16e-06 max(coef): 6.5 l1=6.1e+02/1.5e+03 l2=1.3e+02/2.5e+03\n"]}, {"data": {"text/html": ["<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"\u25b8\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"\u25be\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>NeuralTreeNetRegressor(estimator=None, lr=0.0001, max_iter=10, verbose=1)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NeuralTreeNetRegressor</label><div class=\"sk-toggleable__content\"><pre>NeuralTreeNetRegressor(estimator=None, lr=0.0001, max_iter=10, verbose=1)</pre></div></div></div></div></div>"], "text/plain": ["NeuralTreeNetRegressor(estimator=None, lr=0.0001, max_iter=10, verbose=1)"]}, "execution_count": 49, "metadata": {}, "output_type": "execute_result"}], "source": ["reg.fit(x_train, expected)"]}, {"cell_type": "code", "execution_count": 49, "id": "c3ae49b2", "metadata": {}, "outputs": [{"data": {"text/plain": ["(1.256860512819292, 0.25663312220721907)"]}, "execution_count": 50, "metadata": {}, "output_type": "execute_result"}], "source": ["got = reg.predict(x_train)\n", "numpy.abs(got - expected).max(), numpy.abs(got - expected).mean()"]}, {"cell_type": "markdown", "id": "831e538f", "metadata": {}, "source": ["Ca ne marche pas aussi bien que pr\u00e9vu. Il faudrait sans doute plusieurs it\u00e9rations et jouer avec les param\u00e8tres d'apprentissage."]}, {"cell_type": "code", "execution_count": 50, "id": "6cfe39bd", "metadata": {}, "outputs": [], "source": []}], "metadata": {"kernelspec": {"display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5"}}, "nbformat": 4, "nbformat_minor": 5}