Skip to content

Product Associations

Market-basket (product association) rules from transaction data.

Rules are computed lazily in the Ibis engine: table is the lazy Ibis table and df materializes it.

ProductAssociation

Association rules between products from transaction data.

The unit of analysis is the distinct (group, product) pair. Output columns are {value_col}_1 / {value_col}_2 (the product pair), occurrences_1 / occurrences_2 (count of distinct group_col values containing that product), cooccurrences (distinct group_col values containing both), plus:

  • support = cooccurrences / total, where total is the count of distinct group_col values over the ENTIRE input — not just groups containing either product.
  • confidence = cooccurrences / occurrences_1 = P(product_2 | product_1) — asymmetric.
  • uplift = support / ((occurrences_1 / total) * (occurrences_2 / total)); 1 means the pair co-occurs as often as chance, above 1 more often.

With target_item=None both directions (a, b) and (b, a) are emitted, each with its own confidence; with a given target_item only rows with the target as product_1 are emitted. min_confidence applies to each direction separately, so (a, b) can be kept while (b, a) is dropped.

table is the lazy Ibis result; df materializes it.

Source code in openretailscience/analysis/product_association.py 
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
class ProductAssociation:
    """Association rules between products from transaction data.

    The unit of analysis is the distinct (group, product) pair. Output columns are
    ``{value_col}_1`` / ``{value_col}_2`` (the product pair), ``occurrences_1`` /
    ``occurrences_2`` (count of distinct ``group_col`` values containing that product),
    ``cooccurrences`` (distinct ``group_col`` values containing both), plus:

    - ``support = cooccurrences / total``, where ``total`` is the count of distinct
      ``group_col`` values over the ENTIRE input — not just groups containing either product.
    - ``confidence = cooccurrences / occurrences_1`` = P(product_2 | product_1) — asymmetric.
    - ``uplift = support / ((occurrences_1 / total) * (occurrences_2 / total))``; 1 means the
      pair co-occurs as often as chance, above 1 more often.

    With ``target_item=None`` both directions (a, b) and (b, a) are emitted, each with its
    own confidence; with a given ``target_item`` only rows with the target as product_1 are
    emitted. ``min_confidence`` applies to each direction separately, so (a, b) can be kept
    while (b, a) is dropped.

    ``table`` is the lazy Ibis result; ``df`` materializes it.
    """

    def __init__(
        self,
        df: pd.DataFrame | ibis.Table,
        value_col: str,
        group_col: str | None = None,
        target_item: str | int | list[str | int] | None = None,
        min_occurrences: int = 1,
        min_cooccurrences: int = 1,
        min_support: float = 0.0,
        min_confidence: float = 0.0,
        min_uplift: float = 0.0,
    ) -> None:
        """Initialize the ProductAssociation and compute the rules lazily.

        Args:
            df (pd.DataFrame | ibis.Table): Transaction data with one row per (group, product) occurrence.
            value_col (str): Column containing the product identifiers.
            group_col (str | None): Column defining a basket. The default, option
                ``column.customer_id``, pools everything a customer ever bought into one basket;
                pass the transaction id column for per-transaction market baskets.
            target_item (str | int | list[str | int] | None): Product(s) to focus the analysis on;
                if None, associations for all products are calculated. Floats are rejected: an ID
                compared as a float matches only when its stored value is bit-for-bit equal.
            min_occurrences (int): Minimum distinct-group count per product; must be >= 1. Defaults to 1.
            min_cooccurrences (int): Minimum distinct-group count per pair; must be >= 1. Defaults to 1.
            min_support (float): Minimum support; must be in [0, 1]. Defaults to 0.0.
            min_confidence (float): Minimum confidence; must be in [0, 1]. Defaults to 0.0.
            min_uplift (float): Minimum uplift; must be >= 0. Defaults to 0.0.

        Raises:
            ValueError: If any ``min_*`` value is out of range.
            ValueError: If ``target_item`` is an empty list.
            TypeError: If ``target_item`` contains values that are not str or integer.
            ValueError: If ``group_col`` or ``value_col`` columns are missing from the data.
        """
        group_col = group_col or get_option("column.customer_id")
        required_cols = [group_col, value_col]
        ensure_data_has_columns(df, required_cols)

        self.table = self._calc_association(
            df=df,
            value_col=value_col,
            group_col=group_col,
            target_item=target_item,
            min_occurrences=min_occurrences,
            min_cooccurrences=min_cooccurrences,
            min_support=min_support,
            min_confidence=min_confidence,
            min_uplift=min_uplift,
        )

    @staticmethod
    def _validate_minimum_values(
        min_occurrences: int,
        min_cooccurrences: int,
        min_support: float,
        min_confidence: float,
        min_uplift: float,
    ) -> None:
        """Validate the minimum value parameters.

        Args:
            min_occurrences (int): Minimum occurrences per product; must be >= 1.
            min_cooccurrences (int): Minimum co-occurrences per pair; must be >= 1.
            min_support (float): Minimum support; must be in [0, 1].
            min_confidence (float): Minimum confidence; must be in [0, 1].
            min_uplift (float): Minimum uplift; must be >= 0. Uplift is a ratio and may
                legitimately exceed 1.

        Raises:
            ValueError: If any parameter is outside its valid range.
        """
        if min_occurrences < 1:
            raise ValueError("Minimum occurrences must be at least 1")
        if min_cooccurrences < 1:
            raise ValueError("Minimum cooccurrences must be at least 1")
        if min_support < 0.0 or min_support > 1.0:
            raise ValueError("Minimum support must be between 0 and 1")
        if min_confidence < 0.0 or min_confidence > 1.0:
            raise ValueError("Minimum confidence must be between 0 and 1")
        if min_uplift < 0.0:
            raise ValueError("Minimum uplift must be greater or equal to 0")

    @staticmethod
    def _calc_association(
        df: pd.DataFrame | ibis.Table,
        value_col: str,
        group_col: str,
        target_item: str | int | list[str | int] | None = None,
        min_occurrences: int = 1,
        min_cooccurrences: int = 1,
        min_support: float = 0.0,
        min_confidence: float = 0.0,
        min_uplift: float = 0.0,
    ) -> pd.DataFrame:
        """Build the association rules as a lazy Ibis expression tree.

        Returns the result table without executing it. The unit of analysis is the
        distinct (group, product) pair; ``min_confidence`` is applied after the
        both-directions union. Argument semantics as in ``ProductAssociation.__init__``.

        Returns:
            ibis.Table: Lazy result table with columns ``{value_col}_1``, ``{value_col}_2``,
                ``occurrences_1``, ``occurrences_2``, ``cooccurrences``, ``support``,
                ``confidence``, ``uplift``.

        Raises:
            ValueError: Out-of-range ``min_*`` values (see ``_validate_minimum_values``) or
                an empty ``target_item`` list.
            TypeError: If ``target_item`` items are not str or integer.
        """
        ProductAssociation._validate_minimum_values(
            min_occurrences=min_occurrences,
            min_cooccurrences=min_cooccurrences,
            min_support=min_support,
            min_confidence=min_confidence,
            min_uplift=min_uplift,
        )

        if target_item is not None:
            if not isinstance(target_item, list):
                target_item = [target_item]

            for item in target_item:
                if isinstance(item, bool) or not isinstance(item, str | int | np.integer):
                    msg = f"target_item must contain only str or integer values. Got {type(item)}"
                    raise TypeError(msg)

            if len(target_item) == 0:
                raise ValueError("target_item cannot be an empty list")

        df = ensure_ibis_table(df)

        unique_transactions = df.select(_[group_col], _[value_col]).distinct()
        total_transactions = unique_transactions.alias("t")[group_col].nunique().name("total_count")

        product_occurrences = (
            unique_transactions.group_by(value_col)
            .aggregate(occurrences=_[group_col].nunique())
            .mutate(occurrence_probability=_.occurrences / total_transactions)
            .filter(_.occurrences >= min_occurrences)
        )

        left_table = unique_transactions.rename({"item_1": value_col})
        right_table = unique_transactions.rename({"item_2": value_col})

        join_logic = [left_table[group_col] == right_table[group_col]]
        if target_item is None:
            join_logic.append(left_table.item_1 < right_table.item_2)
        else:
            join_logic.extend(
                [
                    left_table.item_1 != right_table.item_2,
                    left_table.item_1.isin(target_item),
                ],
            )
        merged_df = left_table.join(right_table, predicates=join_logic)

        cooccurrences = (
            merged_df.group_by(["item_1", "item_2"])
            .aggregate(cooccurrences=merged_df[group_col].nunique())
            .mutate(support=_.cooccurrences / total_transactions)
            .filter((_.cooccurrences >= min_cooccurrences) & (_.support >= min_support))
        )

        product_occurrences_1_rename = product_occurrences.rename(
            {"item_1": value_col, "occurrences_1": "occurrences", "prob_1": "occurrence_probability"},
        )
        product_occurrences_2_rename = product_occurrences.rename(
            {"item_2": value_col, "occurrences_2": "occurrences", "prob_2": "occurrence_probability"},
        )

        result = (
            cooccurrences.join(product_occurrences_1_rename, "item_1")
            .join(product_occurrences_2_rename, "item_2")
            .mutate(
                confidence=_.cooccurrences / _.occurrences_1,
                uplift=_.support / (_.prob_1 * _.prob_2),
            )
            .filter(_.uplift >= min_uplift)
        )

        if target_item is None:
            col_order = [
                "item_1",
                "item_2",
                "occurrences_1",
                "occurrences_2",
                "cooccurrences",
                "support",
                "confidence",
                "uplift",
            ]
            inverse_pairs = result.mutate(
                item_1=result.item_2,
                item_2=result.item_1,
                occurrences_1=result.occurrences_2,
                occurrences_2=result.occurrences_1,
                prob_1=result.prob_2,
                prob_2=result.prob_1,
                confidence=result.cooccurrences / result.occurrences_2,
            )
            result = result[col_order].union(inverse_pairs[col_order])

        result = (
            result.filter(result.confidence >= min_confidence)
            .order_by(["item_1", "item_2"])
            .rename({f"{value_col}_1": "item_1", f"{value_col}_2": "item_2"})
        )
        return result[
            [
                f"{value_col}_1",
                f"{value_col}_2",
                "occurrences_1",
                "occurrences_2",
                "cooccurrences",
                "support",
                "confidence",
                "uplift",
            ]
        ]

    @functools.cached_property
    def df(self) -> pd.DataFrame:
        """Materialized association rules DataFrame (cached after first execution)."""
        return self.table.execute().reset_index(drop=True)

df cached property

Materialized association rules DataFrame (cached after first execution).

__init__(df, value_col, group_col=None, target_item=None, min_occurrences=1, min_cooccurrences=1, min_support=0.0, min_confidence=0.0, min_uplift=0.0)

Initialize the ProductAssociation and compute the rules lazily.

Parameters:

Name Type Description Default
df DataFrame | Table

Transaction data with one row per (group, product) occurrence.

 required
value_col str

Column containing the product identifiers.

 required
group_col str | None

Column defining a basket. The default, option column.customer_id, pools everything a customer ever bought into one basket; pass the transaction id column for per-transaction market baskets.

 None
target_item str | int | list[str | int] | None

Product(s) to focus the analysis on; if None, associations for all products are calculated. Floats are rejected: an ID compared as a float matches only when its stored value is bit-for-bit equal.

 None
min_occurrences int

Minimum distinct-group count per product; must be >= 1. Defaults to 1.

 1
min_cooccurrences int

Minimum distinct-group count per pair; must be >= 1. Defaults to 1.

 1
min_support float

Minimum support; must be in [0, 1]. Defaults to 0.0.

 0.0
min_confidence float

Minimum confidence; must be in [0, 1]. Defaults to 0.0.

 0.0
min_uplift float

Minimum uplift; must be >= 0. Defaults to 0.0.

 0.0

Raises:

Type Description
ValueError

If any min_* value is out of range.
ValueError

If target_item is an empty list.
TypeError

If target_item contains values that are not str or integer.
ValueError

If group_col or value_col columns are missing from the data.
Source code in openretailscience/analysis/product_association.py 
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
def __init__(
    self,
    df: pd.DataFrame | ibis.Table,
    value_col: str,
    group_col: str | None = None,
    target_item: str | int | list[str | int] | None = None,
    min_occurrences: int = 1,
    min_cooccurrences: int = 1,
    min_support: float = 0.0,
    min_confidence: float = 0.0,
    min_uplift: float = 0.0,
) -> None:
    """Initialize the ProductAssociation and compute the rules lazily.

    Args:
        df (pd.DataFrame | ibis.Table): Transaction data with one row per (group, product) occurrence.
        value_col (str): Column containing the product identifiers.
        group_col (str | None): Column defining a basket. The default, option
            ``column.customer_id``, pools everything a customer ever bought into one basket;
            pass the transaction id column for per-transaction market baskets.
        target_item (str | int | list[str | int] | None): Product(s) to focus the analysis on;
            if None, associations for all products are calculated. Floats are rejected: an ID
            compared as a float matches only when its stored value is bit-for-bit equal.
        min_occurrences (int): Minimum distinct-group count per product; must be >= 1. Defaults to 1.
        min_cooccurrences (int): Minimum distinct-group count per pair; must be >= 1. Defaults to 1.
        min_support (float): Minimum support; must be in [0, 1]. Defaults to 0.0.
        min_confidence (float): Minimum confidence; must be in [0, 1]. Defaults to 0.0.
        min_uplift (float): Minimum uplift; must be >= 0. Defaults to 0.0.

    Raises:
        ValueError: If any ``min_*`` value is out of range.
        ValueError: If ``target_item`` is an empty list.
        TypeError: If ``target_item`` contains values that are not str or integer.
        ValueError: If ``group_col`` or ``value_col`` columns are missing from the data.
    """
    group_col = group_col or get_option("column.customer_id")
    required_cols = [group_col, value_col]
    ensure_data_has_columns(df, required_cols)

    self.table = self._calc_association(
        df=df,
        value_col=value_col,
        group_col=group_col,
        target_item=target_item,
        min_occurrences=min_occurrences,
        min_cooccurrences=min_cooccurrences,
        min_support=min_support,
        min_confidence=min_confidence,
        min_uplift=min_uplift,
    )