@rodesai did find that the ObjectStore trait has a pretty expansive API, and we only use a subset of it. That's why he implemented TableStore.

I suppose that it's not a bad thing to have a data access layer and a table store layer separated, as the table_store layer understands the concept of "table" and the details around the tables, but the data access layer (ObjectStore) doesn’t need to handle these details, it just focuses on reading and writing object stores with the right semantics, and this is also what we expected on a caching layer, cache objects and make accessing objects faster, but not change the data access semantics.

having a transparent CachedObjectStore could have some benefits:

Beside the files related with table_store, it can also cache the files in manifest_store and etc.
We can also encapsulate many of the complexities around caching (like filesystem I/O, range processing) inside it.
It can be completely optional.

I agree that the ObjectStore trait is too verbose for what we need. The object store operations in the code base are quite limited, and limiting the operations makes data access more predictable. I'm also considering about defining a subset of the ObjectStore trait to reflect the needed part of the operations, but I need to try some proof-of-concept in the code first, it might not be necessary at the moment.

If you bo the ObjectStore route, you'll also need to figure out how to cache range reads. Something I was reading about (in the context of HTTP Range requests) here. This seems like a bit of a non-trivial endeavor to do well. Maybe we don't cache the ranges, but trigger async SST fetches for now.

It's true, dealing with range is not easy. At first, I thought we could handle it in these simple steps:

Whenever there's a GET request with ranges, we actually download the entire file.
After the file is downloaded locally, we read the ranges locally.

However, this approach may cause significant read amplification. It won't be too bad if the SSTs are always under 128MB, but it will become a disaster if the SSTs are above 1GB.

Another approach I could think of is (inspired by Alluxio, which is a transparent object store caching layer):

Split the object file into 64MB parts (a size much bigger than the 4KB block, but much smaller than 1GB SSTs).
When we receive a range request, find the overlapped parts.
Send requests to S3 with RANGEs using a 64MB boundary and save these parts locally.
Return the caller with original RANGEs, but from the local cache file

The part files might look like:

/sst/0000001.sst._part000000 (or numbered with the byte offset in hex)
/sst/0000001.sst._part000001
...

This approach has its complexities, but I am afraid that it might be the necessary evil to deal with caching GET requests with RANGEs on gigabytes files.

I'm not particularly committed to this approach, but I can create a PoC to test it out first.

(however, if we can make sure the SSTs are always smaller than 128mb, we can simply take the simpler cache-an-entire-file approach)

Cache SSTs locally #9

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions