module df.connex_split

Inheritance diagram of pandas_streaming.df.connex_split

Short summary

module pandas_streaming.df.connex_split

Implements a connex split between train and test.

source on GitHub

Classes

class truncated documentation
ImbalancedSplitException Raised when an imbalanced split is detected.

Functions

function truncated documentation
train_test_apart_stratify This split is for a specific case where data is linked in one way. Let’s assume we have two ids as we have for online …
train_test_connex_split This split is for a specific case where data is linked in many ways. Let’s assume we have three ids as we have for …
train_test_split_weights Splits a database in train/test given, every row can have a different weight.

Documentation

Implements a connex split between train and test.

source on GitHub

exception pandas_streaming.df.connex_split.ImbalancedSplitException[source]

Bases: Exception

Raised when an imbalanced split is detected.

source on GitHub

pandas_streaming.df.connex_split.train_test_apart_stratify(df, group, test_size=0.25, train_size=None, stratify=None, force=False, random_state=None, fLOG=None)[source]

This split is for a specific case where data is linked in one way. Let’s assume we have two ids as we have for online sales: (product id, category id). A product can have multiple categories. We need to have distinct products on train and test but common categories on both sides.

Parameters:
  • dfpandas.DataFrame
  • group – columns name for the ids
  • test_size – ratio for the test partition (if train_size is not specified)
  • train_size – ratio for the train partition
  • stratify – column holding the stratification
  • force – if True, tries to get at least one example on the test side for each value of the column stratify
  • random_state – seed for random generators
  • fLOG – logging function
Returns:

Two StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids. This split was implemented for a case of a multi-label classification. A category (stratify) is not exclusive and an observation can be assigned to multiple categories. In that particular case, the method train_test_split can not directly be used.

<<<

import pandas
df = pandas.DataFrame([dict(a=1, b="e"),
                       dict(a=1, b="f"),
                       dict(a=2, b="e"),
                       dict(a=2, b="f"),
                       ])

from pandas_streaming.df import train_test_apart_stratify
train, test = train_test_apart_stratify(
    df, group="a", stratify="b", test_size=0.5)
print(train)
print('-----------')
print(test)

>>>

       a  b
    2  2  e
    3  2  f
    -----------
       a  b
    0  1  e
    1  1  f

source on GitHub

pandas_streaming.df.connex_split.train_test_connex_split(df, groups, test_size=0.25, train_size=None, stratify=None, hash_size=9, unique_rows=False, shuffle=True, fail_imbalanced=0.05, keep_balance=None, stop_if_bigger=None, return_cnx=False, must_groups=None, random_state=None, fLOG=None)[source]

This split is for a specific case where data is linked in many ways. Let’s assume we have three ids as we have for online sales: (product id, user id, card id). As we may need to compute aggregated features, we need every id not to be present in both train and test set. The function computes the connected components and breaks each of them in two parts for train and test.

Parameters:
  • dfpandas.DataFrame
  • groups – columns name for the ids
  • test_size – ratio for the test partition (if train_size is not specified)
  • train_size – ratio for the train partition
  • stratify – column holding the stratification
  • hash_size – size of the hash to cache information about partition
  • unique_rows – ensures that rows are unique
  • shuffle – shuffles before the split
  • fail_imbalanced – raises an exception if relative weights difference is higher than this value
  • stop_if_bigger – (float) stops a connected components from being bigger than this ratio of elements, this should not be used unless a big components emerges, the algorithm stops merging but does not guarantee it returns the best cut, the value should be close to 0
  • keep_balance – (float), if not None, does not merge connected components if their relative sizes are too different, the value should be close to 1
  • return_cnx – returns connected components as a third results
  • must_groups – column name for ids which must not be shared by train/test partitions
  • random_state – seed for random generator
  • fLOG – logging function
Returns:

Two StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids.

Splits a dataframe, keep ids in separate partitions

In some data science problems, rows are not independant and share common value, most of the time ids. In some specific case, multiple ids from different columns are connected and must appear in the same partition. Testing that each id column is evenly split and do not appear in both sets in not enough. Connected components are needed.

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame([dict(user="UA", prod="PAA", card="C1"),
                dict(user="UA", prod="PB", card="C1"),
                dict(user="UB", prod="PC", card="C2"),
                dict(user="UB", prod="PD", card="C2"),
                dict(user="UC", prod="PAA", card="C3"),
                dict(user="UC", prod="PF", card="C4"),
                dict(user="UD", prod="PG", card="C5"),
                ])

train, test = train_test_connex_split(df, test_size=0.5,
                                      groups=['user', 'prod', 'card'],
                                      fail_imbalanced=0.6)
print(train)
print(test)

>>>

      card prod user  connex  weight
    0   C1  PAA   UA       0       1
    1   C1   PB   UA       0       1
    2   C3  PAA   UC       0       1
    3   C4   PF   UC       0       1
      card prod user  connex  weight
    0   C2   PC   UB       2       1
    1   C2   PD   UB       2       1
    2   C5   PG   UD       4       1

If return_cnx is True, the third results contains:

  • connected components for each id
  • the dataframe with connected components as a new column

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame([dict(user="UA", prod="PAA", card="C1"),
                dict(user="UA", prod="PB", card="C1"),
                dict(user="UB", prod="PC", card="C2"),
                dict(user="UB", prod="PD", card="C2"),
                dict(user="UC", prod="PAA", card="C3"),
                dict(user="UC", prod="PF", card="C4"),
                dict(user="UD", prod="PG", card="C5"),
                ])

train, test, cnx = train_test_connex_split(df, test_size=0.5,
                                           groups=['user', 'prod', 'card'],
                                           fail_imbalanced=0.6, return_cnx=True)

print(cnx[0])
print(cnx[1])

>>>

    {('user', 'UA'): 0, ('prod', 'PB'): 0, ('card', 'C1'): 0, ('user', 'UC'): 0, ('prod', 'PF'): 0, ('card', 'C4'): 0, ('user', 'UD'): 2, ('prod', 'PG'): 2, ('card', 'C5'): 2, ('user', 'UB'): 3, ('prod', 'PC'): 3, ('card', 'C2'): 3, ('prod', 'PD'): 3, ('prod', 'PAA'): 0, ('card', 'C3'): 0}
      user prod card  connex  weight
    1   UA   PB   C1       0       1
    5   UC   PF   C4       0       1
    6   UD   PG   C5       2       1
    2   UB   PC   C2       3       1
    3   UB   PD   C2       3       1
    4   UC  PAA   C3       0       1
    0   UA  PAA   C1       0       1

source on GitHub

pandas_streaming.df.connex_split.train_test_split_weights(df, weights=None, test_size=0.25, train_size=None, shuffle=True, fail_imbalanced=0.05, random_state=None)[source]

Splits a database in train/test given, every row can have a different weight.

Parameters:
  • dfpandas.DataFrame or StreamingDataFrame
  • weights – None or weights or weights column name
  • test_size – ratio for the test partition (if train_size is not specified)
  • train_size – ratio for the train partition
  • shuffle – shuffles before the split
  • fail_imbalanced – raises an exception if relative weights difference is higher than this value
  • random_state – seed for random generators
Returns:

train and test pandas.DataFrame

If the dataframe is not shuffled first, the function will produce two datasets which are unlikely to be randomized as the function tries to keep equal weights among both paths without using randomness.

source on GitHub