Index Analysis
Index analysis: how over- or under-represented each group is in a subset of rows, against all rows.
IndexAnalysis
Each group's share of a subset of rows, indexed against its share of all rows (100 = parity).
The subset is the rows where index_col == value_to_index; the baseline is every row. For each
group, index = 100 * subset_share / baseline_share.
table (Ibis) and df (pandas, sorted by the group columns) have one row per group found in the
baseline, or per subgroup_col and group pair found in it; a group with no rows in a subgroup gets no
row there, not one at 0. The columns are the group columns, then subset_value and
baseline_value (value_col aggregated with agg_func), subset_share and baseline_share
(fractions of the subset's and baseline's totals, within each subgroup when one is set) and index.
A group the subset has no rows in has a subset_value, subset_share and index of 0; one whose
subset rows all have a null value_col has them null (0 under "nunique"). index is also null when
the group's baseline share or the subset's total is 0. Reading df raises a ValueError when no row with
a non-null group is in the subset.
Rows with a null group_col or subgroup_col are left out of both totals, as if filtered out first; to
count them, fill the nulls with a label such as "Unknown" before passing the data.
Examples:
>>> from openretailscience.analysis import IndexAnalysis
>>> ia = IndexAnalysis(
... transactions, value_col="customer_id", group_col="category", index_col="brand",
... value_to_index="Meadow Farm", agg_func="nunique",
... )
>>> ia.df
category subset_value baseline_value subset_share baseline_share \
0 Bakery 0 67 0.000000 0.206790
...
Source code in openretailscience/analysis/index_analysis.py
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | |
df
cached
property
The index table as a pandas DataFrame, sorted by the group columns.
Raises:
| Type | Description |
|---|---|
ValueError
|
If no row with a non-null group has |
__init__(df, value_col, group_col, index_col, value_to_index, agg_func='sum', subgroup_col=None)
Build the index query; it runs when df is read.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
df
|
DataFrame | Table
|
The data, e.g. one row per transaction line or per customer. |
required |
value_col
|
str
|
The column to aggregate, e.g. |
required |
group_col
|
str
|
The column whose values are indexed, e.g. |
required |
index_col
|
str
|
The column that defines the subset, e.g. |
required |
value_to_index
|
str | int
|
The |
required |
agg_func
|
str
|
How to aggregate |
'sum'
|
subgroup_col
|
str
|
Compute shares and the index within each value of this column,
e.g. |
None
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If |
ValueError
|
If a column is missing, |
IbisTypeError
|
If |