Dqlite (distributed SQLite) is created by the LXD team at Canonical, the company behind Ubuntu. It is a library written in C that implements a “fast, embedded, persistent SQL database” engine, which offers high-availability and automatic failover. Last week, the team released its Dqlite 1.0 version. It is open-sourced under Apache 2.0 and runs on ARM, X86, POWER, and IBM Z architectures.
Dqlite is written in C to provide maximum cross-platform portability. Its first prototype was implemented in Go but was later rewritten in C because of some performance problems caused due to the way Go interoperates with C.
The team explains, “Go considers a function call into C that lasts more than ~20 microseconds as a blocking system call, in that case it will put the goroutine running that C call in waiting queue and resuming it will effectively cause a context switch, degrading performance (since there were a lot of them happening). The added benefit of the rewrite in C is that it’s now easy to embed dqlite into project written in effectively any language, since all major languages have provisions to create C bindings.”
How Dqlite works
Dqlite extends SQLite with a network protocol that connects various instances of an application and also has them act as a highly-available cluster, without any dependency on external databases. To achieve this, it depends on C-Raft, an implementation of the Raft consensus algorithm in C. This not only provides high-performance transactional consensus and fault tolerance but also preserves SQLite’s efficiency and tiny footprint.
To reach consensus, Raft uses the concept of an “elected leader.” In a Raft cluster, a server can either be a leader or a follower. The cluster can have only one elected leader that will be fully responsible for log replication on the followers. In the case of Dqlite, this means that only the leader can write new Write-Ahead Logging (WAL) frames. So, any attempt to perform a write transaction on a follower node will fail with an ErrNotLeader error, in which case clients will be required to retry against whoever is the new leader.
The team recommends Dqlite for the cases when you don’t want any dependency on an external database, but want your application to be highly available, for instance, IoT and Edge devices. Currently, it is being used by the LXD system containers manager. It uses Dqlite to implement high-availability when running in cluster mode.
What developers are saying about Dqlite
This triggered a discussion on Hacker News. A developer recommended the usage of D or Rust for Dqlite’s implementation. “They could also use D or Rust for this. If borrow-checker is too much, Rust can still do automatic, memory management with other benefits remaining. Both also support letting specific modules be unsafe where performance is critical.”
Others compared it with rqlite, which is a lightweight, distributed relational database that uses SQLite as its storage engine. One main difference that many pointed out was that Dqlite is a library, whereas, rqlite is a full application. Giving a more in-depth comparison between the two, a developer commented, “rqlite’s replication is command based whereas dqlite is/was WAL frame-based — so basically one ships the command and the other ships WAL frames. This distinction means that non-deterministic commands (ex. `RANDOM()`) will work differently.”
Apart from these, Dqlite’s team also listed the difference between Dqlite and rqlite. Among the main differences are, Dqlite is “embeddable in any language that can interoperate with C.” It provides “full support for transactions” and there is “no need for statements to be deterministic.”
A major point of discussion was its use cases. A user commented explaining where Dqlite can find its use, “So an easy use-case that springs to mind is any sort of distributed IoT device that needs to track state. So any industrial or consumer monitoring system with a centralized controller that would use this for data storage. Specifically, this enables the use of multiple nodes for high throughput imagine many, many, many sensors and a central controller streaming real-time data.”
A developer who has used the Dqlite library shared their review, “I used Dqlite for a side project, which replicates some of the features of LXD. Was relatively easy to use, but Dqlite moves at some pace and trying to keep up is quite “interesting”. Anyway once I do end up getting time, I’m sure it’ll be advantageous to what I’m doing.”
To read more about Dqlite, check out its official website.
Other news in database
GraphQL API is now generally available
Amazon Aurora makes PostgreSQL Serverless generally available
The road to Cassandra 4.0 – What does the future have in store?