Introducing EuclidesDB, a multi-model machine learning feature database

2 min read

Yesterday, EuclidesDB, a multi-model machine learning feature database released its v0.1. EuclidesDB 0.1 is tightly coupled with PyTorch and provides a backend for including and querying data on the model feature space.

EuclidesDB provides a simple standalone server that stores, builds indexes, and serves requests using efficient serialization and protocols with an easy API. It provides APIs for including new data into the database and querying it later. Since it uses gRPC (gRPC Remote Procedure Call) for communication, the API can be consumed in many different languages.

As mentioned earlier, it comes with a very tight integration with PyTorch, where libtorch is used as the backend to run traced models. And, thus provides a very easy pipeline to integrate new models into the Euclides DB C++ backend.

The concept behind EuclidesDB

EuclidesDB is based on two main concepts:

Module/Model are terms used interchangeably to represent every computation.
Model Space represents a space of features generated by a model.

When a user adds a new image or other kinds of data into the database, they need to specify which model should be used to index this data. This data is then forwarded into the specified models and their features are saved into a local key-value database.

Similarly, when a user queries for similar items on a model space, they need to make a request with a new image and specify on which model spaces they want to find similar items. Then similar items for each model space is returned together with their relevance.

Features of EuclidesDB v0.1

Euclides v0.1:

Uses gRPC as protocol communication and protobuf as a serialization mechanism for its communication with client APIs.
Uses LevelDB for database serialization.
Uses LSH (Locality Sensitive Hashing) for approximate nearest neighbors.
Comes with PyTorch integration through libtorch.
Provides easy integration for new custom fine-tuned models.

EuclidesDB is currently in its initial release and many new features will be introduced in the future versions. The client API is also expected to change in the upcoming releases before a robust API design is stabilized.

To know more in detail, check out EuclidesDB’s official website.