|
1 | | -# ParallelKMeans.jl Documentation |
| 1 | +# ParallelKMeans.jl Package |
2 | 2 |
|
3 | 3 | ```@contents |
| 4 | +Depth = 4 |
4 | 5 | ``` |
5 | 6 |
|
| 7 | +## Motivation |
| 8 | +It's actually a funny story led to the development of this package. |
| 9 | +What started off as a personal toy project trying to re-construct the K-Means algorithm in native Julia blew up after a heated discussion on the Julia Discourse forum when I asked for Julia optimizaition tips. Long story short, Julia community is an amazing one! Andrey offered his help and together, we decided to push the speed limits of Julia with a parallel implementation of the most famous clustering algorithm. The initial results were mind blowing so we have decided to tidy up the implementation and share with the world as a maintained Julia pacakge. |
| 10 | + |
| 11 | +Say hello to `ParallelKMeans`! |
| 12 | + |
| 13 | +This package aims to utilize the speed of Julia and parallelization (both CPU & GPU) to offer an extremely fast implementation of the K-Means clustering algorithm and its variations via a friendly interface for practioners. |
| 14 | + |
| 15 | +In short, we hope this package will eventually mature as the "one stop" shop for everything KMeans on both CPUs and GPUs. |
| 16 | + |
| 17 | +## K-Means Algorithm Implementation Notes |
| 18 | +Since Julia is a column major language, the input (design matrix) expected by the package in the following format; |
| 19 | + |
| 20 | +- Design matrix X of size n×m, the i-th column of X `(X[:, i])` is a single data point in n-dimensional space. |
| 21 | +- Thus, the rows of the design design matrix represents the feature space with the columns representing all the training examples in this feature space. |
| 22 | + |
| 23 | +One of the pitfalls of K-Means algorithm is that it can fall into a local minima. |
| 24 | +This implementation inherits this problem like every implementation does. |
| 25 | +As a result, it is useful in practice to restart it several times to get the correct results. |
| 26 | + |
6 | 27 | ## Installation |
| 28 | +You can grab the latest stable version of this package from Julia registries by simply running; |
7 | 29 |
|
| 30 | +*NB:* Don't forget to Julia's package manager with `]` |
| 31 | + |
| 32 | +```julia |
| 33 | +pkg> add ParallelKMeans |
| 34 | +``` |
| 35 | + |
| 36 | +For the few (and selected) brave ones, one can simply grab the current experimental features by simply adding the experimental branch to your development environment after invoking the package manager with `]`: |
| 37 | + |
| 38 | +```julia |
| 39 | +dev git@github.com:PyDataBlog/ParallelKMeans.jl.git |
| 40 | +``` |
| 41 | + |
| 42 | +Don't forget to checkout the experimental branch and you are good to go with bleeding edge features and breaks! |
| 43 | +```bash |
| 44 | +git checkout experimental |
| 45 | +``` |
8 | 46 |
|
9 | 47 | ## Features |
| 48 | +- Lightening fast implementation of Kmeans clustering algorithm even on a single thread in native Julia. |
| 49 | +- Support for multi-theading implementation of Kmeans clustering algorithm. |
| 50 | +- 'Kmeans++' initialization for faster and better convergence. |
| 51 | +- Modified version of Elkan's Triangle inequality to speed up K-Means algorithm. |
| 52 | + |
| 53 | + |
| 54 | +## Pending Features |
| 55 | +- [X] Implementation of [Hamerly implementation](https://www.researchgate.net/publication/220906984_Making_k-means_Even_Faster). |
| 56 | +- [ ] Full Implementation of Triangle inequality based on [Elkan - 2003 Using the Triangle Inequality to Accelerate K-Means"](https://www.aaai.org/Papers/ICML/2003/ICML03-022.pdf). |
| 57 | +- [ ] Implementation of [Geometric methods to accelerate k-means algorithm](http://cs.baylor.edu/~hamerly/papers/sdm2016_rysavy_hamerly.pdf). |
| 58 | +- [ ] Support for DataFrame inputs. |
| 59 | +- [ ] Refactoring and finalizaiton of API desgin. |
| 60 | +- [ ] GPU support. |
| 61 | +- [ ] Even faster Kmeans implementation based on current literature. |
| 62 | +- [ ] Optimization of code base. |
| 63 | +- [ ] Improved Documentation |
| 64 | +- [ ] More benchmark tests |
10 | 65 |
|
11 | 66 |
|
12 | 67 | ## How To Use |
| 68 | +Taking advantage of Julia's brilliant multiple dispatch system, the package exposes users to a very easy to use API. |
| 69 | + |
| 70 | +```julia |
| 71 | +using ParallelKMeans |
| 72 | + |
| 73 | +# Uses all available CPU cores by default |
| 74 | +multi_results = kmeans(X, 3; max_iters=300) |
| 75 | + |
| 76 | +# Use only 1 core of CPU |
| 77 | +results = kmeans(X, 3; n_threads=1, max_iters=300) |
| 78 | +``` |
| 79 | + |
| 80 | +The main design goal is to offer all available variations of the KMeans algorithm to end users as composable elements. By default, Lloyd's implementation is used but users can specify different variations of the KMeans clustering algorithm via this interface |
| 81 | + |
| 82 | +```julia |
| 83 | +some_results = kmeans([algo], input_matrix, k; kwargs) |
| 84 | + |
| 85 | +# example |
| 86 | +r = kmeans(Lloyd(), X, 3) # same result as the default |
| 87 | +``` |
| 88 | + |
| 89 | +```julia |
| 90 | +# r contains all the learned artifacts which can be accessed as; |
| 91 | +r.centers # cluster centers (d x k) |
| 92 | +r.assignments # label assignments (n) |
| 93 | +r.totalcost # total cost (i.e. objective) |
| 94 | +r.iterations # number of elapsed iterations |
| 95 | +r.converged # whether the procedure converged |
| 96 | +``` |
| 97 | + |
| 98 | +### Supported KMeans algorithm variations. |
| 99 | +- [Lloyd()](https://cs.nyu.edu/~roweis/csc2515-2006/readings/lloyd57.pdf) |
| 100 | +- [Hamerly()](https://www.researchgate.net/publication/220906984_Making_k-means_Even_Faster) |
| 101 | +- [Geometric()](http://cs.baylor.edu/~hamerly/papers/sdm2016_rysavy_hamerly.pdf) - (Coming soon) |
| 102 | +- [Elkan()](https://www.aaai.org/Papers/ICML/2003/ICML03-022.pdf) - (Coming soon) |
| 103 | +- [MiniBatch()](https://www.eecs.tufts.edu/~dsculley/papers/fastkmeans.pdf) - (Coming soon) |
| 104 | + |
| 105 | + |
| 106 | +### Practical Usage Examples |
| 107 | +Some of the common usage examples of this package are as follows: |
| 108 | + |
| 109 | +#### Clustering With A Desired Number Of Groups |
| 110 | + |
| 111 | +```julia |
| 112 | +using ParallelKMeans, RDatasets, Plots |
| 113 | + |
| 114 | +# load the data |
| 115 | +iris = dataset("datasets", "iris"); |
| 116 | + |
| 117 | +# features to use for clustering |
| 118 | +features = collect(Matrix(iris[:, 1:4])'); |
| 119 | + |
| 120 | +# various artificats can be accessed from the result ie assigned labels, cost value etc |
| 121 | +result = kmeans(features, 3); |
| 122 | + |
| 123 | +# plot with the point color mapped to the assigned cluster index |
| 124 | +scatter(iris.PetalLength, iris.PetalWidth, marker_z=result.assignments, |
| 125 | + color=:lightrainbow, legend=false) |
| 126 | + |
| 127 | +``` |
| 128 | + |
| 129 | + |
| 130 | + |
| 131 | +#### Elbow Method For The Selection Of optimal number of clusters |
| 132 | +```julia |
| 133 | +using ParallelKMeans |
| 134 | + |
| 135 | +# Single Thread Implementation of Lloyd's Algorithm |
| 136 | +b = [ParallelKMeans.kmeans(X, i, n_threads=1; tol=1e-6, max_iters=300, verbose=false).totalcost for i = 2:10] |
| 137 | + |
| 138 | +# Multi Thread Implementation of Lloyd's Algorithm by default |
| 139 | +c = [ParallelKMeans.kmeans(X, i; tol=1e-6, max_iters=300, verbose=false).totalcost for i = 2:10] |
| 140 | + |
| 141 | +``` |
| 142 | + |
| 143 | + |
| 144 | +## Benchmarks |
| 145 | +Currently, this package is benchmarked against similar implementation in both Python and Julia. All reproducible benchmarks can be found in [ParallelKMeans/extras](https://github.com/PyDataBlog/ParallelKMeans.jl/tree/master/extras) directory. More tests in various languages are planned beyond the initial release version (`0.1.0`). |
| 146 | + |
| 147 | +*Note*: All benchmark tests are made on the same computer to help eliminate any bias. |
| 148 | + |
| 149 | + |
| 150 | +Currently, the benchmark speed tests are based on the search for optimal number of clusters using the [Elbow Method](https://en.wikipedia.org/wiki/Elbow_method_(clustering)) since this is a practical use case for most practioners employing the K-Means algorithm. |
| 151 | + |
| 152 | + |
| 153 | +### Benchmark Results |
| 154 | + |
| 155 | + |
| 156 | + |
| 157 | + |
| 158 | +_________________________________________________________________________________________________________ |
| 159 | + |
| 160 | +| 1 million (ms) | 100k (ms) | 10k (ms) | 1k (ms) | package | language | |
| 161 | +|:--------------:|:---------:|:--------:|:-------:|:-----------------------:|:--------:| |
| 162 | +| 600184.00 | 31959.00 | 832.25 | 18.19 | Clustering.jl | Julia | |
| 163 | +| 35733.00 | 4473.00 | 255.71 | 8.94 | Lloyd | Julia | |
| 164 | +| 12617.00 | 1655.00 | 122.53 | 7.98 | Hamerly | Julia | |
| 165 | +| 1430000.00 | 146000.00 | 5770.00 | 344.00 | Sklearn Kmeans | Python | |
| 166 | +| 30100.00 | 3750.00 | 613.00 | 201.00 | Sklearn MiniBatchKmeans | Python | |
| 167 | +| 218200.00 | 15510.00 | 733.70 | 19.47 | Knor | R | |
| 168 | + |
| 169 | +_________________________________________________________________________________________________________ |
| 170 | + |
| 171 | + |
| 172 | +## Release History |
| 173 | +- 0.1.0 Initial release |
| 174 | + |
| 175 | + |
| 176 | +## Contributing |
| 177 | +Ultimately, we see this package as potentially the one stop shop for everything related to KMeans algorithm and its speed up variants. We are open to new implementations and ideas from anyone interested in this project. |
13 | 178 |
|
| 179 | +Detailed contribution guidelines will be added in upcoming releases. |
14 | 180 |
|
| 181 | +<!--- Insert Contribution Guidelines Below ---> |
15 | 182 |
|
16 | 183 | ```@index |
17 | 184 | ``` |
|
0 commit comments