Introduction

We were tasked with performing a clustering analysis of products to better understand their behavior and characteristics over time.

The dataset comprises information from three cities in the USA, serving a total of 10 stores. The DS Market management team sought to identify patterns and similarities among the 3,049 products offered by the company. Effective segmentation was expected to facilitate more informed decision-making.

To optimize the processing and transformation of the data, we developed a series of pipelines tailored to achieve the project’s objectives. Below is an overview of the key pipelines and their roles:

Pipeline 1: Calculating Seasonality

Objective: Understand the seasonality of the products.
New Variables:

• Mean Seasonality
• Seasonal Amplitude
• Residual Variation

Pipeline 2: Dataset Aggregation

Initial Dataset: 1,362,319 rows and 11 columns.
Aggregation Method: Grouped by item and category to create new features, including:

• Total Sales, Max/Min Sales, Average Sales Price
• Total/Average/Max/Min Quantities Sold
• Max/Min/Average Prices, Median Sales, Median Quantities Sold
• Sales Range, Quantity Range, Price Range
• Seasonality, Seasonal Amplitude, Residual Variation

Resulting Dataset: Reduced to 3,049 rows.

Pipeline 3: One-Hot Encoding

Purpose: Convert categorical columns into numerical data types for analysis.

Pipeline 4: Variance Calculation

Objective: Identify and exclude variables with low variance using a defined threshold.

Pipeline 5: Standardization

Tool: Standard Scaler
Purpose: Ensure all features contribute equally to model training.

Pipeline 6: Dimensionality Reduction (PCA)

Objective: Improve clustering by reducing dimensionality and noise.

• Selected 7 principal components explaining 90% of the variance.

Elbow Method

Objective: Determine the optimal number of clusters.
Result: Although 5 clusters were initially identified, business logic suggested using 4 clusters due to one cluster containing very few records.

K-Means Clustering

Process: Create clusters based on PCA results.
Evaluation: Silhouette Score was used to assess clustering quality.

• Best Value for K: 4 clusters
• Final Decision: Selected 4 clusters, aligning with business logic and the importance of specific characteristics.

Silhouette Score Analysis

We tested K-Means with both 4 and 2 clusters for comparison. While the Silhouette Score suggested using 2 clusters, our knowledge of the business and its commercial variables justified the choice of 4 clusters.

Final Evaluation

We subjected the PCA and K-Means clusters to comparative evaluation between 4 and 2 clusters, resulting in:

1. Sum of Squared Errors (SSE): Lower values indicated better clustering. Winner: 4 Clusters
2. Silhouette Score: Higher scores reflected better cohesion and separation among clusters. Winner: 4 Clusters

Considering these evaluations, the segmentation process concluded with 4 clusters.

Visualization and Results

We utilized Tableau to visualize the final output, combining the results into an interactive and dynamic dashboard. Key variables displayed include:

• Total Sales ($)
• Total Products (#)
• Products Sold (#)

This approach allowed us to name and describe the clusters effectively while enabling a detailed exploration of their composition over 5 years and 4 months.

Our final tool incorporates the above parameters, offering a clear and adaptable view of clustering results aligned with the company’s strategic goals.