module pycode.profiling

Short summary

module pyquickhelper.pycode.profiling

Profiling helpers

source on GitHub

Classes

class	truncated documentation
ProfileNode	Graph structure to represent a profiling.
SortKey

Functions

function	truncated documentation
_process_pstats	Converts class Stats into something readable for …
profile	Profiles the execution of a function.
profile2df	Converts profiling statistics into a Dataframe.
profile2graph	Converts profiling statistics into a graphs.

Properties

property	truncated documentation
key	Returns file:line.

Static Methods

staticmethod	truncated documentation
_key
filter_node_	Filters out node to be displayed by default.

Methods

method	truncated documentation
__init__
__iter__	Returns all nodes in the graph.
__repr__	usual
add_called_by	This function is called by these lines.
add_calls_to	This function calls this node.
as_dict	Renders the results of a profiling interpreted with function @fn profile2graph. It can then be loaded with …
get_root	Returns the root of the graph.
to_json	Renders the results of a profiling interpreted with function @fn profile2graph as JSON.
to_text	Prints the profiling to text.

Documentation

Profiling helpers

source on GitHub

class pyquickhelper.pycode.profiling.ProfileNode(filename, line, func_name, nc1, nc2, tin, tall)

Bases: object

Graph structure to represent a profiling.

Parameters

• filename – filename

• line – line number

• func_name – function name

• nc1 – number of calls 1

• nc2 – number of calls 2

• tin – time spent in the function

• tout – time spent in the function and in the sub functions

New in version 1.11.

source on GitHub

__init__(filename, line, func_name, nc1, nc2, tin, tall)

__iter__()

Returns all nodes in the graph.

__repr__()

usual

static _key(filename, line, fct)

_modules_ = {' _globals.py', '<frozen importlib._bootstrap>', '__future__.py', '_collections_abc.py', '_ios.py', 'abc.py', 'argparse.py', 'ast.py', 'contextlib.py', 'datetime.py', 'errors.py', 'functools.py', 'inspect.py', 'numbers.py', 'os.py', 'posixpath.py', 'six.py', 'sre_parse.py', 'subprocess.py', 'threading.py', 'types.py', 'typing.py', 'version.py', 'warnings.py', '~'}

add_called_by(pnode)

This function is called by these lines.

add_calls_to(pnode, time_elements)

This function calls this node.

as_dict(filter_node=None, sort_key='line')

Renders the results of a profiling interpreted with function @fn profile2graph. It can then be loaded with a dataframe.

Parameters

• filter_node – display only the nodes for which this function returns True, if None, the default function removes built-in function with small impact

• sort_key – sort sub nodes by…

Returns

rows

source on GitHub

static filter_node_(node, info=None)

Filters out node to be displayed by default.

Parameters

• node – node

• info – if the node is called by a function, this dictionary can be used to overwrite the attributes held by the node

Returns

boolean (True to keep, False to forget)

source on GitHub

get_root()

Returns the root of the graph.

property key

Returns file:line.

to_json(filter_node=None, sort_key='line', as_str=True, **kwargs)

Renders the results of a profiling interpreted with function @fn profile2graph as JSON.

Parameters

• filter_node – display only the nodes for which this function returns True, if None, the default function removes built-in function with small impact

• sort_key – sort sub nodes by…

• as_str – converts the json into a string

• kwargs – see json.dumps()

Returns

rows

See notebook Hierchical profiling to see how to use the json output.

source on GitHub

to_text(filter_node=None, sort_key='line', fct_width=60)

Prints the profiling to text.

Parameters

• filter_node – display only the nodes for which this function returns True, if None, the default function removes built-in function with small impact

• sort_key – sort sub nodes by…

Returns

rows

source on GitHub

pyquickhelper.pycode.profiling._process_pstats(ps, clean_text=None, verbose=False, fLOG=None)

Converts class Stats into something readable for a dataframe.

Parameters

• ps – instance of type pstats.Stats()

• clean_text – function to clean function names

• verbose – change verbosity

• fLOG – logging function

Returns

list of rows

source on GitHub

pyquickhelper.pycode.profiling.profile(fct, sort='cumulative', rootrem=None, as_df=False, pyinst_format=None, return_results=False, **kwargs)

Profiles the execution of a function.

Parameters

• fct – function to profile

• sort – see sort_stats

• rootrem – root to remove in filenames

• as_df – return the results as a dataframe and not text

• pyinst_format – format for pyinstrument, if not empty, the function uses this module or raises an exception if not installed, the options are text, textu (text with colors), json, html

• return_results – if True, return results as well (in the first position)

• kwargs – additional parameters used to create the profiler

Returns

raw results, statistics text dump (or dataframe is as_df is True)

import matplotlib.pyplot as plt
from pyquickhelper.pycode.profiling import profile
from pyquickhelper.texthelper import compare_module_version

def fctm():
    return compare_module_version('0.20.4', '0.22.dev0')

pr, df = profile(lambda: [fctm() for i in range(0, 1000)], as_df=True)
ax = df[['namefct', 'cum_tall']].head(n=15).set_index(
    'namefct').plot(kind='bar', figsize=(8, 3), rot=30)
ax.set_title("example of a graph")
for la in ax.get_xticklabels():
    la.set_horizontalalignment('right');
plt.show()

(png, hires.png, pdf)

Changed in version 1.11: Parameter return_results was added.

source on GitHub

pyquickhelper.pycode.profiling.profile2df(ps, as_df=True, clean_text=None, verbose=False, fLOG=None)

Converts profiling statistics into a Dataframe.

Parameters

• ps – an instance of pstats

• as_df – returns the results as a dataframe (True) or a list of dictionaries (False)

• clean_text – function to clean function names

• verbose – verbosity

• fLOG – logging function

Returns

a DataFrame

import pstats
from pyquickhelper.pycode.profiling import profile2df

ps = pstats.Stats('bench_ortmodule_nn_gpu6.prof')
df = profile2df(pd)
print(df)

source on GitHub

pyquickhelper.pycode.profiling.profile2graph(ps, clean_text=None, verbose=False, fLOG=None)

Converts profiling statistics into a graphs.

Parameters

• ps –

  an instance of pstats

• clean_text – function to clean function names

• verbose – verbosity

• fLOG – logging function

Returns

an instance of class ProfileNode

Hierarchical display for a profiling

pyinstrument has a nice display to show time spent and call stack at the same time. This function tries to replicate that display based on the results produced by module cProfile. Here is an example.

<<<

import time
from pyquickhelper.pycode.profiling import profile, profile2graph


def fct0(t):
    time.sleep(t)


def fct1(t):
    time.sleep(t)


def fct2():
    fct1(0.1)
    fct1(0.01)


def fct3():
    fct0(0.2)
    fct1(0.5)


def fct4():
    fct2()
    fct3()


ps = profile(fct4)[0]
root, nodes = profile2graph(ps, clean_text=lambda x: x.split('/')[-1])
text = root.to_text()
print(text)

>>>

    fct1                                                         --  3  3 -- 0.00002 0.61086 -- :11:fct1 (fct1)
        <built-in method time.sleep>                             --  3  3 -- 0.61083 0.61083 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>) +++
    fct4                                                         --  1  1 -- 0.00001 0.81113 -- :25:fct4 (fct4)
        fct2                                                     --  1  1 -- 0.00001 0.11030 -- :15:fct2 (fct2)
            fct1                                                 --  2  2 -- 0.00002 0.11029 -- :11:fct1 (fct1) +++
        fct3                                                     --  1  1 -- 0.00001 0.70082 -- :20:fct3 (fct3)
            fct0                                                 --  1  1 -- 0.00000 0.20025 -- :7:fct0 (fct0)
                <built-in method time.sleep>                     --  1  1 -- 0.20024 0.20024 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>) +++
            fct1                                                 --  1  1 -- 0.00001 0.50057 -- :11:fct1 (fct1) +++
    <built-in method time.sleep>                                 --  4  4 -- 0.81108 0.81108 -- ~:0:<built-in method time.sleep> (<built-in method time.sleep>)

New in version 1.11.

source on GitHub