.. _exampleprofilingrst: ==================== test about profiling ==================== .. only:: html **Links:** :download:`notebook `, :downloadlink:`html `, :download:`PDF `, :download:`python `, :downloadlink:`slides `, :githublink:`GitHub|_doc/notebooks/example_profiling.ipynb|*` How to profile from a notebook with cProfile, memory_profiler .. code:: ipython3 from jyquickhelper import add_notebook_menu add_notebook_menu() .. contents:: :local: profiling with cProfile ~~~~~~~~~~~~~~~~~~~~~~~ .. code:: ipython3 def big_list1(n): l = [] for i in range(n): l.append(i) return l def big_list2(n): return list(range(n)) def big_list(n): big_list1(n) big_list2(n) %prun -q -T profile_example.txt -D profile_example.stat big_list(100000) .. parsed-literal:: *** Profile stats marshalled to file 'profile_example.stat'. *** Profile printout saved to text file 'profile_example.txt'. .. code:: ipython3 with open('profile_example.txt', 'r') as f: content = f.read() print(content) .. parsed-literal:: 100006 function calls in 0.142 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.098 0.098 0.124 0.124 :1(big_list1) 100000 0.026 0.000 0.026 0.000 {method 'append' of 'list' objects} 1 0.009 0.009 0.142 0.142 :10(big_list) 1 0.009 0.009 0.009 0.009 :7(big_list2) 1 0.000 0.000 0.142 0.142 {built-in method exec} 1 0.000 0.000 0.142 0.142 :1() 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} .. code:: ipython3 import pstats p = pstats.Stats('profile_example.stat') p.strip_dirs().sort_stats('cumulative').print_stats() .. parsed-literal:: Wed Sep 9 10:55:54 2015 profile_example.stat 100006 function calls in 0.117 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.117 0.117 {built-in method exec} 1 0.000 0.000 0.117 0.117 :1() 1 0.007 0.007 0.117 0.117 :10(big_list) 1 0.079 0.079 0.101 0.101 :1(big_list1) 100000 0.022 0.000 0.022 0.000 {method 'append' of 'list' objects} 1 0.008 0.008 0.008 0.008 :7(big_list2) 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} .. parsed-literal:: Memory profile ~~~~~~~~~~~~~~ .. code:: ipython3 from memory_profiler import memory_usage mem_usage = memory_usage(-1, interval=.2, timeout=1) mem_usage .. parsed-literal:: [97.69921875, 97.69921875, 97.69921875, 97.69921875, 97.69921875] Thz functions to test must be part of file and cannot be implemented in the notebook. So we save the funtion in script and we import it just after. .. code:: ipython3 %%file script_test.py def big_list1(n): l = [] for i in range(n): l.append(i) return l def big_list2(n): return list(range(n)) def big_list(n): big_list1(n) big_list2(n) .. parsed-literal:: Writing script_test.py .. code:: ipython3 from script_test import big_list, big_list1, big_list2 We run the momory profiling: .. code:: ipython3 %load_ext memory_profiler .. code:: ipython3 prof = %mprun -r -f big_list1 -f big_list2 -T profile_example.mem -r big_list(100000) .. parsed-literal:: *** Profile printout saved to text file profile_example.mem. .. code:: ipython3 with open('profile_example.mem', 'r') as f : content = f.read() print(content) .. parsed-literal:: Filename: C:\xadupre\__home_\_data\GitHub\pymyinstall\_doc\notebooks\script_test.py Line # Mem usage Increment Line Contents ================================================ 2 38.3 MiB 0.0 MiB def big_list1(n): 3 38.3 MiB 0.0 MiB l = [] 4 42.1 MiB 3.8 MiB for i in range(n): 5 42.1 MiB 0.0 MiB l.append(i) 6 42.1 MiB 0.0 MiB return l Filename: C:\xadupre\__home_\_data\GitHub\pymyinstall\_doc\notebooks\script_test.py Line # Mem usage Increment Line Contents ================================================ 8 39.1 MiB 0.0 MiB def big_list2(n): 9 42.9 MiB 3.8 MiB return list(range(n)) SnakeViz ~~~~~~~~ .. code:: ipython3 %load_ext snakeviz .. code:: ipython3 %system snakeviz --help .. parsed-literal:: ['Usage: snakeviz [options] filename', '', 'Options:', ' -h, --help show this help message and exit', ' -H ADDR, --hostname=ADDR', ' hostname to bind to (default: 127.0.0.1', ' -p PORT, --port=PORT port to bind to; if this port is already in use a free', ' port will be selected automatically (default: 8080)', ' -b PATH, --browser=PATH', ' name of webbrowser to launch as described in the', " documentation of Python's webbrowser module:", ' https://docs.python.org/3/library/webbrowser.html', ' -s, --server start SnakeViz in server-only mode--no attempt will be', ' to open a browser'] See `How to visualize Python profile data with SnakeViz `__.