Skip to content

Customer Decision Hierarchy

Product substitutability from customer co-purchase patterns via Yule's Q and ward hierarchical clustering.

Use to cluster products into substitute versus complement groups for range decisions. For market-basket co-occurrence rules between products, use :mod:openretailscience.analysis.product_association instead.

CustomerDecisionHierarchy

Identifies which products customers treat as substitutes versus complements.

Products are compared by which customers buy them. With the default exclude_same_transaction_products=True only (customer, product) pairs never bought in a multi-item transaction count, so products close together are ones the same customers switch between (substitutes). A product left with no pairs is absent from the results.

Results, available after initialization:

  • pairs_df: the deduplicated (customer, product) pairs used, as categoricals.
  • distances: square matrix in [0, 1], rows and columns ordered by pairs_df[product_col].cat.categories. 0 = every buyer of one product also buys the other, 0.5 = no association, 1 = never bought by the same customer, or no customer bought neither (Yule's Q of -1 either way).
  • plot(): dendrogram of the ward hierarchical clustering.
Source code in openretailscience/analysis/customer_decision_hierarchy.py 
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
class CustomerDecisionHierarchy:
    """Identifies which products customers treat as substitutes versus complements.

    Products are compared by which customers buy them. With the default
    ``exclude_same_transaction_products=True`` only (customer, product) pairs never bought in
    a multi-item transaction count, so products close together are ones the same customers
    switch between (substitutes). A product left with no pairs is absent from the results.

    Results, available after initialization:

    - ``pairs_df``: the deduplicated (customer, product) pairs used, as categoricals.
    - ``distances``: square matrix in [0, 1], rows and columns ordered by
      ``pairs_df[product_col].cat.categories``. 0 = every buyer of one product also buys
      the other, 0.5 = no association, 1 = never bought by the same customer, or no customer
      bought neither (Yule's Q of -1 either way).
    - ``plot()``: dendrogram of the ward hierarchical clustering.
    """

    def __init__(
        self,
        df: pd.DataFrame,
        product_col: str,
        exclude_same_transaction_products: bool = True,
        method: Literal["yules_q"] = "yules_q",
        random_state: int = 42,
    ) -> None:
        """Initialize the analysis; pairs, distances, and clustering inputs are computed immediately.

        Args:
            df (pd.DataFrame): Transaction-level rows; must contain the option-derived
                customer_id and transaction_id columns and ``product_col``.
            product_col (str): Column with the product identifiers to analyze.
            exclude_same_transaction_products (bool): When True (default), drop each
                (customer, product) pair the customer ever bought in a multi-item transaction.
            method (Literal["yules_q"]): Distance method; only "yules_q" is accepted.
            random_state (int): Stored but has no effect; results are already deterministic.

        Raises:
            ValueError: If the required columns are missing from the dataframe.
            ValueError: If ``method`` is not "yules_q".
        """
        cols = ColumnHelper()
        required_cols = [cols.customer_id, cols.transaction_id, product_col]
        ensure_data_has_columns(df, required_cols)

        self.random_state = random_state
        self.product_col = product_col
        self.pairs_df = self._get_pairs(df, exclude_same_transaction_products, product_col)
        self.distances = self._calculate_distances(method=method)

    @staticmethod
    def _get_pairs(df: pd.DataFrame, exclude_same_transaction_products: bool, product_col: str) -> pd.DataFrame:
        """Deduplicate (customer, product) pairs from transaction rows.

        With ``exclude_same_transaction_products``, any (customer, product) pair that
        appears in a multi-item transaction for that customer is removed entirely
        (pair-level left-anti-join, not row-level).

        Returns:
            pd.DataFrame: Deduplicated (customer_id, product) frame with categorical dtypes.
        """
        cols = ColumnHelper()
        if exclude_same_transaction_products:
            pairs_df = df[[cols.customer_id, cols.transaction_id, product_col]].drop_duplicates()
            multi_item = pairs_df.groupby(cols.transaction_id, observed=True)[product_col].transform("size") > 1
            pairs_to_exclude_df = pairs_df.loc[multi_item, [cols.customer_id, product_col]].drop_duplicates()
            # Drop all rows from pairs_df where customer_id and product_name are in pairs_to_exclude_df
            pairs_df = pairs_df.merge(
                pairs_to_exclude_df,
                on=[cols.customer_id, product_col],
                how="left",
                indicator=True,
            )
            pairs_df = pairs_df[pairs_df["_merge"] == "left_only"][[cols.customer_id, product_col]].drop_duplicates()
        else:
            pairs_df = df[[cols.customer_id, product_col]].drop_duplicates()

        # astype keeps an already-categorical column's categories; an excluded product or customer would
        # then linger as a phantom row in the distance matrix
        return pairs_df.reset_index(drop=True).astype("category").apply(lambda col: col.cat.remove_unused_categories())

    @staticmethod
    def _calculate_yules_q(bought_product_1: np.array, bought_product_2: np.array) -> float:
        """Calculate Yule's Q coefficient between two boolean purchase vectors.

        Q = (ad - bc) / (ad + bc) on the 2x2 co-purchase table. Empty arrays, and
        degenerate tables where a*d + b*c == 0, return 0.0 because NaN would break
        scipy's ``linkage`` downstream.

        Args:
            bought_product_1 (np.array): Boolean vector, True where the customer bought the product.
            bought_product_2 (np.array): Boolean vector, True where the customer bought the product.

        Returns:
            float: The Yule's Q coefficient, in [-1, 1].

        Raises:
            ValueError: If the lengths of `bought_product_1` and `bought_product_2` are not the same.
            ValueError: If `bought_product_1` or `bought_product_2` is not a boolean array.

        """
        if len(bought_product_1) != len(bought_product_2):
            raise ValueError("The bought_product_1 and bought_product_2 must be the same length")
        if len(bought_product_1) == 0:
            return 0.0
        if bought_product_1.dtype != bool or bought_product_2.dtype != bool:
            raise ValueError("The bought_product_1 and bought_product_2 must be boolean arrays")

        a = np.count_nonzero(bought_product_1 & bought_product_2)
        b = np.count_nonzero(bought_product_1 & ~bought_product_2)
        c = np.count_nonzero(~bought_product_1 & bought_product_2)
        d = np.count_nonzero(~bought_product_1 & ~bought_product_2)

        # Calculate Yule's Q coefficient
        denominator = a * d + b * c
        if denominator == 0:
            # Both a*d and b*c are zero, making Q mathematically undefined (0/0).
            # Return 0.0 (no association) because NaN would break scipy's linkage() downstream.
            return 0.0

        return (a * d - b * c) / denominator

    def _get_yules_q_distances(self) -> np.ndarray:
        """Calculate the square product-distance matrix, (1 - Yule's Q) / 2.

        Returns:
            np.ndarray: Square matrix of pairwise product distances in [0, 1].
        """
        # Create a sparse matrix where the rows are the customers and the columns are the products
        # The values are True if the customer bought the product and False if they didn't
        product_matrix = csr_matrix(
            (
                [True] * len(self.pairs_df),
                (
                    self.pairs_df[self.product_col].cat.codes,
                    self.pairs_df[get_option("column.customer_id")].cat.codes,
                ),
            ),
            dtype=bool,
        )

        # Calculate the number of customers and products
        n_products = product_matrix.shape[0]

        # Create an empty matrix to store the yules q values
        yules_q_matrix = np.zeros((n_products, n_products), dtype=float)

        # Loop through each pair of products
        for i in range(n_products):
            arr_i = product_matrix[i].toarray()
            for j in range(i + 1, n_products):
                # Calculate the yules q value for the pair of products
                arr_j = product_matrix[j].toarray()
                yules_q_dist = 1 - self._calculate_yules_q(arr_i, arr_j)

                # Store the yules q value in the matrix
                yules_q_matrix[i, j] = yules_q_dist
                yules_q_matrix[j, i] = yules_q_dist

        # yules_q_dist = 1 - Q lives in [0, 2]; halving rescales it into [0, 1] while
        # preserving the zero diagonal so this is a valid distance matrix.
        return yules_q_matrix / 2

    def _calculate_distances(
        self,
        method: Literal["yules_q"],
    ) -> np.ndarray:
        """Calculate the product-distance matrix using the specified method.

        Args:
            method (Literal["yules_q"]): Only "yules_q" is implemented.

        Returns:
            np.ndarray: A square matrix of pairwise product distances.

        Raises:
            ValueError: If the method is not "yules_q".
        """
        # Check method is valid
        if method == "yules_q":
            distances = self._get_yules_q_distances()
        else:
            raise ValueError("Method must be 'yules_q'")

        return distances

    def _compute_linkage_matrix(self) -> np.ndarray:
        """Compute the hierarchical-clustering linkage matrix from precomputed distances.

        scipy's ``linkage`` infers input semantics from shape: a 1-D array is treated as a
        condensed distance vector; a 2-D array is treated as an observation matrix and scipy
        recomputes Euclidean distances between rows. ``squareform`` converts the square
        distance matrix to condensed form so scipy uses the precomputed distances directly.
        """
        return linkage(squareform(self.distances, checks=False), method="ward")

    def plot(
        self,
        title: str = "Customer Decision Hierarchy",
        x_label: str | None = None,
        y_label: str | None = None,
        ax: Axes | None = None,
        figsize: tuple[int, int] | None = None,
        eyebrow: str | None = None,
        subtitle: str | None = None,
        source_text: str | None = None,
        **kwargs: Any,  # noqa: ANN401
    ) -> SubplotBase:
        """Plot the customer decision hierarchy dendrogram.

        ``**kwargs`` are passed through to scipy's ``dendrogram`` (``orientation``
        defaults to "top"). A figure is created when ``ax`` is None.

        Args:
            title (str): The title of the plot.
            x_label (str | None): X-axis label; defaults to "Distance" for left/right orientations.
            y_label (str | None): Y-axis label; defaults to "Distance" for top/bottom orientations.
            ax (Axes | None): Axes to plot on; a new figure is created when None.
            figsize (tuple[int, int] | None): Figure size for the new figure.
            eyebrow (str | None): Small uppercase label rendered above the title.
            subtitle (str | None): Supporting copy rendered below the title.
            source_text (str | None): Source annotation.
            **kwargs (Any): Additional keyword arguments forwarded to scipy's ``dendrogram``.

        Returns:
            SubplotBase: The axes the dendrogram was drawn on.
        """
        linkage_matrix = self._compute_linkage_matrix()
        labels = self.pairs_df[self.product_col].cat.categories

        if ax is None:
            _, ax = plt.subplots(figsize=figsize)

        orientation = kwargs.get("orientation", "top")
        distance_is_y = orientation in ["top", "bottom"]
        default_x_label, default_y_label = (None, "Distance") if distance_is_y else ("Distance", None)

        dendrogram(linkage_matrix, labels=labels, ax=ax, **kwargs)

        # Move ticks/labels for orientations whose categorical axis sits on the
        # opposite side from the matplotlib default.
        if orientation == "left":
            ax.yaxis.tick_right()
            ax.yaxis.set_label_position("right")
        elif orientation == "bottom":
            ax.xaxis.tick_top()
            ax.xaxis.set_label_position("top")

        # Chrome runs last so its tight_layout sees the populated axes (rotated
        # category labels included) and reserves room for them — matching the
        # call order of every other plot module. _auto_rotate_categorical_x_ticks
        # inside standard_graph_styles owns x-tick rotation for top/bottom
        # orientations.
        standard_graph_styles(
            ax=ax,
            title=title,
            x_label=x_label if x_label is not None else default_x_label,
            y_label=y_label if y_label is not None else default_y_label,
            eyebrow=eyebrow,
            subtitle=subtitle,
            source_text=source_text,
            grid_axis="y" if distance_is_y else "x",
        )

        return ax

__init__(df, product_col, exclude_same_transaction_products=True, method='yules_q', random_state=42)

Initialize the analysis; pairs, distances, and clustering inputs are computed immediately.

Parameters:

Name Type Description Default
df DataFrame

Transaction-level rows; must contain the option-derived customer_id and transaction_id columns and product_col.

 required
product_col str

Column with the product identifiers to analyze.

 required
exclude_same_transaction_products bool

When True (default), drop each (customer, product) pair the customer ever bought in a multi-item transaction.

 True
method Literal['yules_q']

Distance method; only "yules_q" is accepted.

 'yules_q'
random_state int

Stored but has no effect; results are already deterministic.

 42

Raises:

Type Description
ValueError

If the required columns are missing from the dataframe.
ValueError

If method is not "yules_q".
Source code in openretailscience/analysis/customer_decision_hierarchy.py 
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
def __init__(
    self,
    df: pd.DataFrame,
    product_col: str,
    exclude_same_transaction_products: bool = True,
    method: Literal["yules_q"] = "yules_q",
    random_state: int = 42,
) -> None:
    """Initialize the analysis; pairs, distances, and clustering inputs are computed immediately.

    Args:
        df (pd.DataFrame): Transaction-level rows; must contain the option-derived
            customer_id and transaction_id columns and ``product_col``.
        product_col (str): Column with the product identifiers to analyze.
        exclude_same_transaction_products (bool): When True (default), drop each
            (customer, product) pair the customer ever bought in a multi-item transaction.
        method (Literal["yules_q"]): Distance method; only "yules_q" is accepted.
        random_state (int): Stored but has no effect; results are already deterministic.

    Raises:
        ValueError: If the required columns are missing from the dataframe.
        ValueError: If ``method`` is not "yules_q".
    """
    cols = ColumnHelper()
    required_cols = [cols.customer_id, cols.transaction_id, product_col]
    ensure_data_has_columns(df, required_cols)

    self.random_state = random_state
    self.product_col = product_col
    self.pairs_df = self._get_pairs(df, exclude_same_transaction_products, product_col)
    self.distances = self._calculate_distances(method=method)

plot(title='Customer Decision Hierarchy', x_label=None, y_label=None, ax=None, figsize=None, eyebrow=None, subtitle=None, source_text=None, **kwargs)

Plot the customer decision hierarchy dendrogram.

**kwargs are passed through to scipy's dendrogram (orientation defaults to "top"). A figure is created when ax is None.

Parameters:

Name Type Description Default
title str

The title of the plot.

 'Customer Decision Hierarchy'
x_label str | None

X-axis label; defaults to "Distance" for left/right orientations.

 None
y_label str | None

Y-axis label; defaults to "Distance" for top/bottom orientations.

 None
ax Axes | None

Axes to plot on; a new figure is created when None.

 None
figsize tuple[int, int] | None

Figure size for the new figure.

 None
eyebrow str | None

Small uppercase label rendered above the title.

 None
subtitle str | None

Supporting copy rendered below the title.

 None
source_text str | None

Source annotation.

 None
**kwargs Any

Additional keyword arguments forwarded to scipy's dendrogram.

 {}

Returns:

Name Type Description
SubplotBase SubplotBase

The axes the dendrogram was drawn on.
Source code in openretailscience/analysis/customer_decision_hierarchy.py 
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
def plot(
    self,
    title: str = "Customer Decision Hierarchy",
    x_label: str | None = None,
    y_label: str | None = None,
    ax: Axes | None = None,
    figsize: tuple[int, int] | None = None,
    eyebrow: str | None = None,
    subtitle: str | None = None,
    source_text: str | None = None,
    **kwargs: Any,  # noqa: ANN401
) -> SubplotBase:
    """Plot the customer decision hierarchy dendrogram.

    ``**kwargs`` are passed through to scipy's ``dendrogram`` (``orientation``
    defaults to "top"). A figure is created when ``ax`` is None.

    Args:
        title (str): The title of the plot.
        x_label (str | None): X-axis label; defaults to "Distance" for left/right orientations.
        y_label (str | None): Y-axis label; defaults to "Distance" for top/bottom orientations.
        ax (Axes | None): Axes to plot on; a new figure is created when None.
        figsize (tuple[int, int] | None): Figure size for the new figure.
        eyebrow (str | None): Small uppercase label rendered above the title.
        subtitle (str | None): Supporting copy rendered below the title.
        source_text (str | None): Source annotation.
        **kwargs (Any): Additional keyword arguments forwarded to scipy's ``dendrogram``.

    Returns:
        SubplotBase: The axes the dendrogram was drawn on.
    """
    linkage_matrix = self._compute_linkage_matrix()
    labels = self.pairs_df[self.product_col].cat.categories

    if ax is None:
        _, ax = plt.subplots(figsize=figsize)

    orientation = kwargs.get("orientation", "top")
    distance_is_y = orientation in ["top", "bottom"]
    default_x_label, default_y_label = (None, "Distance") if distance_is_y else ("Distance", None)

    dendrogram(linkage_matrix, labels=labels, ax=ax, **kwargs)

    # Move ticks/labels for orientations whose categorical axis sits on the
    # opposite side from the matplotlib default.
    if orientation == "left":
        ax.yaxis.tick_right()
        ax.yaxis.set_label_position("right")
    elif orientation == "bottom":
        ax.xaxis.tick_top()
        ax.xaxis.set_label_position("top")

    # Chrome runs last so its tight_layout sees the populated axes (rotated
    # category labels included) and reserves room for them — matching the
    # call order of every other plot module. _auto_rotate_categorical_x_ticks
    # inside standard_graph_styles owns x-tick rotation for top/bottom
    # orientations.
    standard_graph_styles(
        ax=ax,
        title=title,
        x_label=x_label if x_label is not None else default_x_label,
        y_label=y_label if y_label is not None else default_y_label,
        eyebrow=eyebrow,
        subtitle=subtitle,
        source_text=source_text,
        grid_axis="y" if distance_is_y else "x",
    )

    return ax