K-Hyperparameter Tuning in High-Dimensional Space Clustering: Solving Smooth Elbow Challenges Using an Ensemble Based Technique of a Self-Adapting Autoencoder and Internal Validation Indexes
Loading...
Date
2023-10-26
Journal Title
Journal ISSN
Volume Title
Publisher
Tech Science Press
Abstract
k-means is a popular clustering algorithmbecause of its simplicity and scalability to handle large datasets.However, one of its setbacks is the challenge of identifying the correct k-hyperparameter value. Tuning this value correctly is critical for building effective k-means models. The use of the traditional elbow method to help identify this value has a long-standing literature. However, when using this method with certain datasets, smooth curves may appear, making it challenging to identify the k-value due to its unclear nature.Onthe other hand, various internal validation indexes, which are proposed as a solution to this issue, may be inconsistent. Although various techniques for solving smooth elbow challenges exist, k-hyperparameter tuning in high-dimensional spaces still remains intractable and an open research issue. In this paper, we have first reviewed the existing techniques for solving smooth elbow challenges. The identified research gaps are then utilized in the development of the new technique. The new technique, referred to as the ensemble-based technique of a self-adapting autoencoder and internal validation indexes, is then validated in high-dimensional space clustering. The optimal k-value, tuned by this technique using a voting scheme, is a trade-off between the number of clusters visualized in the autoencoder’s latent space, k-value from the ensemble internal validation index score and one that generates a value of 0 or close to 0 on the derivative f ___ (k)(1+f _ (k)2)−3 f __ (k)2f __ ((k)2f _ (k), at the elbow. Experimental results based on theCochran’sQtest,ANOVA, andMcNemar’s score indicate a relativelygoodperformanceof thenewlydevelopedtechnique ink-hyperparameter tuning.
Description
Article
Keywords
Citation
Gikera, Rufus & Mwaura, Jonathan & Muuro, Elizaphan & Mambo, Shadrack. (2023). K-Hyperparameter Tuning in High-Dimensional Space Clustering: Solving Smooth Elbow Challenges Using an Ensemble Based Technique of a Self-Adapting Autoencoder and Internal Validation Indexes. Journal on Artificial Intelligence. 5. 75-112. 10.32604/jai.2023.043229.