Scalability/Performance

[spillz]

Not really an issue per se but wasn't sure where else to ask. Perhaps these could be part of a FAQ.

1. Key limit. What is the scalability like in terms of # of keys. E.g., if I add 100M keys to a rocksdict, will it fall over?
2. Network shareable. If the dict folder is stored on a network share, can it be safely read by multiple concurrent users? What about writes?
3. Cold boot. Is there a significant performance penalty with opening the database or fetching a key for the first time as it grows in size? I see this with semidbm even on a moderate size db of ~300MB and 5M keys but so far so good with rocksdict.
4. Duplicate handling. For example, if I am storing long pathnames in my values and there are many duplicates do I need to manually assign those strings (unique_id,path) pairs in the DB manually (and look them up to retrieve them) to avoid propagating those duplicates or this is taken care of in the lib (or by python itself)?
5. Raw mode. I would imagine there might be some storage efficiency gained by using raw mode over pickle. True?
6. Writes. I could be wrong but write speed does some slightly slower than semidbm on a moderately large db. Even if true, it's probably worth the tradeoff in terms of read speed for most applications. What should I expect here and does it get worse with size?

Many thanks for making the lib!

[Congyuwang]

Answer:

1. Key limit. Scalability of rocksdict should be exactly the same as rocksdb. So, I don't think 100M keys will break rocksdict.
2. Network shareable. Same as rocksdb. The dict folder can be opened by only one w/r instance, and can open another r/only instance. If you want to use it across different network services, you'll have to write a service for it.
3. Cold boot. I think there will be some overhead, but not too bad. But don't use Plaintable for this, use block-based table for cold boot.
4. Duplicate handling. Same as a dict. If you write a new key-value pair into a dict with an existing key, the old value is overwritten.
5. Raw mode. Non-raw-mode compared to raw-mode has the (1) overhead of calling pickle.dump(), and (2) it encodes one extra byte for encoding the value type information, and (3) it has the copy the given memory slice into a new slice which containing the extra byte of type info. I didn't give it any benchmark though.
6. Writes. Write scalability should be the same as rocksdb; so, it depends on the type of storage, and number of CPU cores. Since it uses LSM, it shouldn't slow down too much when it gets bigger.

[spillz]

Many thanks for the answers. I'm finding rocksdict to be a really great key-value store so far.

I tried to frame my questions broadly but let me hone in a bit on some of my intended use, which is using rocksdict with lots of keys (possibly as high as 1B) but a limited set of values. I'm trying to index some unique IDs that can be found in a few thousand files so I can quickly find the file(s) and then the corresponding data that belongs to that ID in the file. Essentially the rocksdict is a database of data fragments in many data files to make the file structure largely invisible to the users. Within the rocksdict, the keys are the IDs and the values are the files. Obviously there is lots of duplication in the values because there are many IDs to each value. In a local python dict I can economize on storage by using sys.intern to cache the string and thereby replace repeated multibyte pathnames wtih a 4- or 8-byte integer (i.e., a pointer). In my currrent implementation I have effectively hand-rolled my own intern by keeping the definitions of each cache string in the dict itself, for example:

b'#\x00\x00\x00\x00':b'my/first/path.txt' #path ID 0
b'#\x00\x00\x00\x01':b'my/second/path.txt'  #path ID 1
...

(I suppose it would be slightly more efficient to keep these entries in their own dict or rocksdict.)

And then my IDs will have entries like

b'TEXT1' : b'\x00\x00\x00\x01'
b'TEXT2' : b'\x00\x00\x00\x00'
b'TEXT3' : b'\x00\x00\x00\x00\x00\x00\x00\x01' #Text fragment can appear in multiple files
...

So with that in mind, I'll make a few observations

1. Key limit. Scalability of rocksdict should be exactly the same as rocksdb. So, I don't think 100M keys will break rocksdict.

Yes, so far so good. I've been able to hit 30M with no apparent problems.

2. Network shareable. Same as rocksdb. The dict folder can be opened by only one w/r instance, and can open another r/only instance. If you want to use it across different network services, you'll have to write a service for it.

Perfect.

3. Cold boot. I think there will be some overhead, but not too bad. But don't use Plaintable for this, use block-based table for cold boot.

Yes, so far it is remarkable how quickly a key's value can be retrieved from a cold start. With the 6M key database keys were retrieved within about a second or two, and most of that was the python overhead not rocksdict. This was around a minute for the equivalent semidbm which may have something to do with #4.

4. Duplicate handling. Same as a dict. If you write a new key-value pair into a dict with an existing key, the old value is overwritten.

Here I was referencing duplicate values and my use of intern as described above. I've already optimized the values with my handrolled intern but now I wonder if I don't need to given there seems to be something going on behind the scenes in rocksdb to optimize on space taken by keys in comparison with semidbm. E.g., the 6M key database has key lengths averaging around 10 bytes and values of 4 bytes => 6M * 14B = 84MB (not counting the interned value storage, but that's trivial by comparison). Yet somehow the rockdict DB uses only 36MB. The same structure in semidbm takes 350MB even after being compacted, which I find puzzling in the other direction.

6. Writes. Write scalability should be the same as rocksdb; so, it depends on the type of storage, and number of CPU cores. Since it uses LSM, it shouldn't slow down too much when it gets bigger.

I haven't benchmarked properly to know for sure but semidbm did seem a little faster to write my k-v pairs. From what I understand from my quick peek at that code, semidbm optimizes writes by always appending, which could be one reason. Another is that perhaps I'm tripping some auto-compacting trigger and need to turn that off.

[Congyuwang]

Regarding 4, I think it may be because rocksdb uses compression, which is snappy by default. https://github.com/EighteenZi/rocksdb_wiki/blob/master/Compression.md

[spillz]

OK. I will try without the manual intern and see what that does. My bet is that it's faster/more efficient. Will certainly make my code simpler.