kilosort2 data is extremely slow on Phy

[dervinism]

I have recently switched to kilosort2. However, it does not seem to work properly with Phy. The clusters load extremely slow making it, in practice, not possible to use Phy with the kilosort2 output.

[rossant]

I wonder if it's showing the raw waveform amplitudes in the waveform view. This is extremely slow at the moment. Press a to Switch to another type of amplitude

[dervinism]

It doesn't seem to change anything.

[dervinism]

Template type might be somewhat faster but all the other types seem just as slow.

[rossant]

I'm going to need a few more details:

1. How much time does it take to load a cluster?
2. Is your dataset on an SSD?
3. What is your system configuration?
4. File size of the raw data file, number of channels, number of spikes?
5. Is it still slow if your modify params.py and remove the path to the raw data file?
6. Are you using the latest versions of phy and phylib?
7. Close the views one by one and see if one specific view is making the GUI much slower

[dervinism]

It takes approximately 15 seconds to load a single cluster when no other processes are running in the background. But it often takes longer. The data is on SSD. I have 32GB of RAM. I have 384 channels, the data size varies: 150-330GB. I am using the latest Phy. I haven't noticed much difference after modifying the params.py file.

[rossant]

What is your OS?
Could you close the views one by one and see if one specific view is making the GUI much slower?

Otherwise it would help if you could send me your dataset. I could try to see if I can reproduce the problem on my computer

[dervinism]

I use Windows 10. I think the FeatureView is the one causing the most problem. After closing the FeatureView it takes only 1-2 seconds to load a cluster (AmplitudeView is still on). I could send a dataset but, unfortunately, it is quite large.

[dervinism]

I guess you only need the npy, py, and tsv files

[rossant]

yes indeed

[dervinism]

I am not quite sure how to send you a pc_features.npy file which is 21GB in size. I compared these files to kilosort1 output and they take up to 1.5GB in kilosort1. My guess is this is what's causing the problem. If FeaturesView panel has to load large amounts of data from pc_features.npy every time I switch clusters, then it might be it. I can see disk usage going to 100% every time it takes a while to load a cluster.

[dervinism]

Here is a Dropbox link to the data example: https://www.dropbox.com/s/au9cjm3b18p1g34/20191213202419.zip?dl=0

[dervinism]

Have you had a chance to look into this? Or should I post this to kilosort2 forum?

[rossant]

Unfortunately the zip file seems to be corrupted..?