Spyking Circus + Phy: WaveformView templates from Utah Array seen on wrong electrodes

[shashwatsridhar]

Hey,

We're working with datasets recorded using Utah Arrays in vivo from the macaque visual cortex. We do not expect to see waveforms from the same unit on more than one electrode. We spike sort our data using Spyking Circus, and also specify in the probe file for the algorithm that the radius in which we expect waveforms is smaller than the pitch of the electrode.

However, after converting Spyking Circus results for Phy, the templates appear on electrodes different from the one on which they were 'recorded'. We do not know why this might be happening. Here are the screenshots from a few channels.

Here are the contents of the probefile used for spyking circus:

total_nb_channels = 128
radius            = 100
channel_groups    = {1 : {}}


channel_groups[1]["channels"] = list(range(128))
channel_groups[1]["graph"]    = []
channel_groups[1]["geometry"] = {
 0: [0, 2800],
 1: [0, 2400],
 2: [0, 2000],
 3: [0, 1600],
 4: [0, 1200],
 5: [0, 800],
 6: [0, 400],
 7: [0, 0],
 8: [400, 2800],
 9: [400, 2400],
 10: [400, 2000],
 11: [400, 1600],
 12: [400, 1200],
 13: [400, 800],
 14: [400, 400],
 15: [400, 0],
 16: [800, 2800],
 17: [800, 2400],
 18: [800, 2000],
 19: [800, 1600],
 20: [800, 1200],
 21: [800, 800],
 22: [800, 400],
 23: [800, 0],
 24: [1200, 2800],
 25: [1200, 2400],
 26: [1200, 2000],
 27: [1200, 1600],
 28: [1200, 1200],
 29: [1200, 800],
 30: [1200, 400],
 31: [1200, 0],
 32: [1600, 2800],
 33: [1600, 2400],
 34: [1600, 2000],
 35: [1600, 1600],
 36: [1600, 1200],
 37: [1600, 800],
 38: [1600, 400],
 39: [1600, 0],
 40: [2000, 2800],
 41: [2000, 2400],
 42: [2000, 2000],
 43: [2000, 1600],
 44: [2000, 1200],
 45: [2000, 800],
 46: [2000, 400],
 47: [2000, 0],
 48: [2400, 2800],
 49: [2400, 2400],
 50: [2400, 2000],
 51: [2400, 1600],
 52: [2400, 1200],
 53: [2400, 800],
 54: [2400, 400],
 55: [2400, 0],
 56: [2800, 2800],
 57: [2800, 2400],
 58: [2800, 2000],
 59: [2800, 1600],
 60: [2800, 1200],
 61: [2800, 800],
 62: [2800, 400],
 63: [2800, 0],
 64: [50000, 2800],
 65: [50000, 2400],
 66: [50000, 2000],
 67: [50000, 1600],
 68: [50000, 1200],
 69: [50000, 800],
 70: [50000, 400],
 71: [50000, 0],
 72: [50400, 2800],
 73: [50400, 2400],
 74: [50400, 2000],
 75: [50400, 1600],
 76: [50400, 1200],
 77: [50400, 800],
 78: [50400, 400],
 79: [50400, 0],
 80: [50800, 2800],
 81: [50800, 2400],
 82: [50800, 2000],
 83: [50800, 1600],
 84: [50800, 1200],
 85: [50800, 800],
 86: [50800, 400],
 87: [50800, 0],
 88: [51200, 2800],
 89: [51200, 2400],
 90: [51200, 2000],
 91: [51200, 1600],
 92: [51200, 1200],
 93: [51200, 800],
 94: [51200, 400],
 95: [51200, 0],
 96: [51600, 2800],
 97: [51600, 2400],
 98: [51600, 2000],
 99: [51600, 1600],
 100: [51600, 1200],
 101: [51600, 800],
 102: [51600, 400],
 103: [51600, 0],
 104: [52000, 2800],
 105: [52000, 2400],
 106: [52000, 2000],
 107: [52000, 1600],
 108: [52000, 1200],
 109: [52000, 800],
 110: [52000, 400],
 111: [52000, 0],
 112: [52400, 2800],
 113: [52400, 2400],
 114: [52400, 2000],
 115: [52400, 1600],
 116: [52400, 1200],
 117: [52400, 800],
 118: [52400, 400],
 119: [52400, 0],
 120: [52800, 2800],
 121: [52800, 2400],
 122: [52800, 2000],
 123: [52800, 1600],
 124: [52800, 1200],
 125: [52800, 800],
 126: [52800, 400],
 127: [52800, 0]}

Do you think this could be an issue with the 'converting' step of spyking circus? Or do you think this is an issue with phy itself?

Thanks!

[yger]

Maybe this is more a Spyking-circus related question, so I'm jumping in. In latest version of SC, the radius is only used to constrain the channels considered during the clustering (and thus save memory for very large arrays). However, when templates are estimated, all the channels containing information are kept. Templates have therefore the "appropriate" support, i.e. the one adapted to the channels where you have a signal. So decreasing the radius does not prevent you from seeing templates covering several channels. Now the question is why do you see such templates, if you are sure that you should not: I don't know. But they are not suspect to me: since templates are estimated as median over multiple waveforms, I would be tempted to think they are real and indeed present in the data.

[morales-gregorio]

Hey Yger! Thanks for jumpling in!

What we are observing is a systematical shift. The templates we are looking for are almost always not in the electrode that phy is telling us they should be. Very often wee see them in an electrode that has the index+1 or for example the templates from channel 7 very often (in different arrays) would show up in electrode 77.

Also something to note is that the amplitude histograms do not add up. In theory the gray histogram is the histogram for all the units in the given channel (or is this not true?). However (as you can see in the screenshots by @shashwatsridhar) we have marked all the templates from a channel and their histograms just do not add up to the gray area.

We are fearing some sort of indexing issue, either in the conversion from SC to phy or in the way phy is reading those files. We just do not know at which end this can be.

I hope this further clarifications can pinpoint the problem a bit better.

Cheers,
Aitor

[yger]

Thanks, it helps, as indeed I misunderstood your question. I now see the problem you are talking about, and indeed, looks like you have a systematic shift. You are using latest version of SC and phy/phylib?

To troubleshoot, you can try to set sparse_export = False to SC params file, in the [converting] section, and see if this problem can be avoided by using the dense export for phy. Otherwise, you can send me the data (just the .params, .prb and the result folder (even without the .GUI that I could regenerate), no need for the raw data I guess, such that it should not be too big) such that I can see if the problem lies on SC or phy.

[morales-gregorio]

Hey Yger! Setting sparse_export = False has indeed fixed our problems!

Would be worth looking into why it happens, before other people are confused by this. Also worth noting that the conversion was much faster (~2 fold) once this flag was deactivated.

All best,
Aitor

[yger]

You're welcome. Just to know if the error is on the phy or SC side (and since I can not reproduce the error myself) could you confirm you are using latest phy version, and maybe send me your results data as a zip (just .prb, .params and the result folder without .GUI)? The sparse export is slower, but phy is after faster (much better for large number of electrodes) and display exactly what is saved by SC.

[morales-gregorio]

I can verify I am using the latest release version of it all:

• spyking-circus 0.9.8
• phy 2.0b1
• phylib 2.2

Here is some output generated with sparse=True and sparse=False. All other parameters are the same (see the sc.params in there). I also put the .prb in there, it is the same that @shashwatsridhar posted in here originally.

indexing_error_output_data.zip

[yger]

Thanks, I'll have a look. But I must admit it would help if you could also give me the .basis.hdf5, the .templates.hdf5 and the .results.hdf5, to know what is really saved by SC before the export

[yger]

After a second thought, please send me also these files (and the .params), since in fact, there are discrepancies between the two exports that do not make sense (spike_times.npy and amplitudes.npy should have the same size, since the export mode should not have an impact)