# module df.connex_split¶

## Short summary¶

module pandas_streaming.df.connex_split

Implements a connex split between train and test.

source on GitHub

## Classes¶

class truncated documentation
ImbalancedSplitException Raised when an imbalanced split is detected.

## Functions¶

function truncated documentation
train_test_apart_stratify This split is for a specific case where data is linked in one way. Let’s assume we have two ids as we have for online …
train_test_connex_split This split is for a specific case where data is linked in many ways. Let’s assume we have three ids as we have for …
train_test_split_weights Splits a database in train/test given, every row can have a different weight.

## Documentation¶

Implements a connex split between train and test.

source on GitHub

exception pandas_streaming.df.connex_split.ImbalancedSplitException[source]

Bases: Exception

Raised when an imbalanced split is detected.

source on GitHub

pandas_streaming.df.connex_split.train_test_apart_stratify(df, group, test_size=0.25, train_size=None, stratify=None, force=False, random_state=None, fLOG=None)[source]

This split is for a specific case where data is linked in one way. Let’s assume we have two ids as we have for online sales: (product id, category id). A product can have multiple categories. We need to have distinct products on train and test but common categories on both sides.

Parameters: df – pandas.DataFrame group – columns name for the ids test_size – ratio for the test partition (if train_size is not specified) train_size – ratio for the train partition stratify – column holding the stratification force – if True, tries to get at least one example on the test side for each value of the column stratify random_state – seed for random generators fLOG – logging function Two StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids. This split was implemented for a case of a multi-label classification. A category (stratify) is not exclusive and an observation can be assigned to multiple categories. In that particular case, the method train_test_split can not directly be used.

<<<

import pandas
df = pandas.DataFrame([dict(a=1, b="e"),
dict(a=1, b="f"),
dict(a=2, b="e"),
dict(a=2, b="f"),
])

from pandas_streaming.df import train_test_apart_stratify
train, test = train_test_apart_stratify(
df, group="a", stratify="b", test_size=0.5)
print(train)
print('-----------')
print(test)


>>>

       a  b
2  2  e
3  2  f
-----------
a  b
0  1  e
1  1  f


source on GitHub

pandas_streaming.df.connex_split.train_test_connex_split(df, groups, test_size=0.25, train_size=None, stratify=None, hash_size=9, unique_rows=False, shuffle=True, fail_imbalanced=0.05, keep_balance=None, stop_if_bigger=None, return_cnx=False, must_groups=None, random_state=None, fLOG=None)[source]

This split is for a specific case where data is linked in many ways. Let’s assume we have three ids as we have for online sales: (product id, user id, card id). As we may need to compute aggregated features, we need every id not to be present in both train and test set. The function computes the connected components and breaks each of them in two parts for train and test.

Parameters: df – pandas.DataFrame groups – columns name for the ids test_size – ratio for the test partition (if train_size is not specified) train_size – ratio for the train partition stratify – column holding the stratification hash_size – size of the hash to cache information about partition unique_rows – ensures that rows are unique shuffle – shuffles before the split fail_imbalanced – raises an exception if relative weights difference is higher than this value stop_if_bigger – (float) stops a connected components from being bigger than this ratio of elements, this should not be used unless a big components emerges, the algorithm stops merging but does not guarantee it returns the best cut, the value should be close to 0 keep_balance – (float), if not None, does not merge connected components if their relative sizes are too different, the value should be close to 1 return_cnx – returns connected components as a third results must_groups – column name for ids which must not be shared by train/test partitions random_state – seed for random generator fLOG – logging function Two StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids.

Splits a dataframe, keep ids in separate partitions

In some data science problems, rows are not independant and share common value, most of the time ids. In some specific case, multiple ids from different columns are connected and must appear in the same partition. Testing that each id column is evenly split and do not appear in both sets in not enough. Connected components are needed.

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame([dict(user="UA", prod="PAA", card="C1"),
dict(user="UA", prod="PB", card="C1"),
dict(user="UB", prod="PC", card="C2"),
dict(user="UB", prod="PD", card="C2"),
dict(user="UC", prod="PAA", card="C3"),
dict(user="UC", prod="PF", card="C4"),
dict(user="UD", prod="PG", card="C5"),
])

train, test = train_test_connex_split(df, test_size=0.5,
groups=['user', 'prod', 'card'],
fail_imbalanced=0.6)
print(train)
print(test)


>>>

      card prod user  connex  weight
0   C2   PD   UB       0       1
1   C2   PC   UB       0       1
card prod user  connex  weight
0   C4   PF   UC       1       1
1   C1  PAA   UA       1       1
2   C3  PAA   UC       1       1
3   C1   PB   UA       1       1
4   C5   PG   UD       6       1


If return_cnx is True, the third results contains:

• connected components for each id
• the dataframe with connected components as a new column

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame([dict(user="UA", prod="PAA", card="C1"),
dict(user="UA", prod="PB", card="C1"),
dict(user="UB", prod="PC", card="C2"),
dict(user="UB", prod="PD", card="C2"),
dict(user="UC", prod="PAA", card="C3"),
dict(user="UC", prod="PF", card="C4"),
dict(user="UD", prod="PG", card="C5"),
])

train, test, cnx = train_test_connex_split(df, test_size=0.5,
groups=['user', 'prod', 'card'],
fail_imbalanced=0.6, return_cnx=True)

print(cnx[0])
print(cnx[1])


>>>

    {('user', 'UB'): 0, ('prod', 'PC'): 0, ('card', 'C2'): 0, ('user', 'UC'): 1, ('prod', 'PAA'): 1, ('card', 'C3'): 1, ('prod', 'PD'): 0, ('user', 'UA'): 1, ('prod', 'PB'): 1, ('card', 'C1'): 1, ('user', 'UD'): 4, ('prod', 'PG'): 4, ('card', 'C5'): 4, ('prod', 'PF'): 1, ('card', 'C4'): 1}
user prod card  connex  weight
2   UB   PC   C2       0       1
4   UC  PAA   C3       1       1
3   UB   PD   C2       0       1
1   UA   PB   C1       1       1
6   UD   PG   C5       4       1
0   UA  PAA   C1       1       1
5   UC   PF   C4       1       1


source on GitHub

pandas_streaming.df.connex_split.train_test_split_weights(df, weights=None, test_size=0.25, train_size=None, shuffle=True, fail_imbalanced=0.05, random_state=None)[source]

Splits a database in train/test given, every row can have a different weight.

Parameters: df – pandas.DataFrame or StreamingDataFrame weights – None or weights or weights column name test_size – ratio for the test partition (if train_size is not specified) train_size – ratio for the train partition shuffle – shuffles before the split fail_imbalanced – raises an exception if relative weights difference is higher than this value random_state – seed for random generators train and test pandas.DataFrame

If the dataframe is not shuffled first, the function will produce two datasets which are unlikely to be randomized as the function tries to keep equal weights among both paths without using randomness.

source on GitHub