Lightning Memory-mapped Database

An extraordinarily fast, memory-efficient database we developed for the Symas OpenLDAP Project.

 

 

An ultra-fast, ultra-compact, crash-proof key-value embedded data store.

Symas LMDB is an extraordinarily fast, memory-efficient database we developed for the Symas OpenLDAP Project. With memory-mapped files, it has the read performance of a pure in-memory database while retaining the persistence of standard disk-based databases.
Bottom line, with only 32KB of object code, LMDB may seem tiny. But it’s the right 32KB. Compact and efficient are two sides of a coin; that’s part of what makes LMDB so powerful.

See Technical Details

Explore Capabilities

  • Ordered-map interface
    • keys are always sorted; range lookups are supported
  • Fully-transactional, full ACID semantics with MVCC
  • Reader/writer transactions
    • readers don’t block writers; writers don’t block readers
  • Fully serialized writers
    • writes are always deadlock-free
  • Extremely cheap read transactions
    • can be performed using no mallocs or any other blocking calls
  • Multi-thread and multi-process concurrency supported
    • Environments may be opened by multiple processes on the same host
  • Multiple sub-databases may be created with transactions covering all sub-databases
  • Memory-mapped, allowing for zero-copy lookup and iteration
  • Maintenance-free 
    • no external process or background cleanup or compaction required
  • Crash-proof
    • no logs or crash recovery procedures required
  • No application-level caching
    • LMDB fully exploits the operating system’s buffer cache
  • 32KB of object code and 6KLOC of C

Ask A Question

Ready to learn more about Symas LMDB?
Fill out the form below or give us a call 650.963.7601.

Support Options

Commercial Support
Symas offers fixed-price commercial support to those using LMDB in your applications. Please contact us for additional information.

Porting and Professional Services
Thinking about switching your application to LMDB? Call us. We will be happy to provide developer support or porting services to help you transition your application to LMDB.

Community
Search the discussions on the OpenLDAP mailing lists.

Source Information

Development occurs in the OpenLDAP Project‘s git repo in the mdb.master branch. A clone of just the LMDB code is also available on Github.

Download LMDB

Click the link to go to GitHub to download LMDB.

Download Now

More Documentation

Follow the link below for even more detailed documentation.
See Documentation

How It Compares

Click to see a quick comparison of other embedded key value stores. 

Comparison Grid
LMDB BDB LevelDB Kyoto TreeDB
ACID compliant
Nested transactions
Multiple namespaces
Sorted keys
Sorted duplicate keys
Multi-thread concurrency
Multi-process concurrency
No cache tuning: zero-config
Instantaneous crash recovery
Zero-copy reads
Zero-copy writes
Atomic hot backup

Download Software

Download an ultra-fast, ultra-compact, crash-proof key-value embedded data store.

Technical Information

Symas LMDB has been written about, talked about, and utilized in a variety of impressive products and publications over the years. We invite you to learn more below.

Project presentations

Project presentations on LMDB have been generating excitement across the US and Europe since 2011.

LMDB Involvement

Beyond the LMDB backend for OpenLDAP slapd (back-mdb), LMDB has also been leveraged in the following open source projects. We’re proud that LMDB plays a role in helping these projects realize their potential.

Note, because it’s so easy to modify BerkeleyDB-based projects to use LMDB instead, the list of BerkeleyDB-based adaptations is growing quickly. Aside from just improving existing projects though, many new projects have been
launched specifically around LMDB.

  • Cyrus SASL (Patches posted to the SASL mailing list)
  • Heimdal Kerberos (Support: development branch)
  • SQLite3 (available on Github)
  • OpenDKIM Domain Keys Identified Mail, integrated since version 2.6.0
  • MemcacheDB  Tests with memcachetest shows that LMDB is several times faster than BerkeleyDB
  • Mapkeeper
  • Thrift-based storage server; test results are available using the Yahoo! Cloud Serving Benchmark (YCSB)
  • Replicant (A replicated state machine manager for HyperDex. Source code available on Github)
  • HyperDex (A reliable and fast NoSQL server (distributed key/value store). Source code available on Github)
  • SQLite4 (Coming eventually)
  • Postfix (Fast and secure email server; support integrated since 2.11)
  • CfEngine
    IT infrastructure automation framework; default backend as of 3.6.0
  • Redis
    Near-realtime disk persistence Second strike with Lightning, code on Github
  • Ardb
    A redis-protocol compatible persistent storage server; available on Github
  • SkyDB
    High performance analysis of behavioral data; formerly used LevelDB but switched to LMDB due to concurrency and performance issues in LevelDB
  • FineDB
    A new high-performance noSQL database built on LMDB
  • PowerDNS
    An Authoritative DNS server written in C++; delivering Very High Performance using LMDB
  • Hustle
    A column oriented, embarrassingly distributed relational NoSQL database
  • UrBackup
    A client/server file backup system
  • Consul
    Distributed service discovery and configuration
  • BAREOS
    Backup Archiving REcovery Open Sourced. Integrated in 14.2
  • Caffe
    Deep learning / computer vision / neural network
  • Monero
    Crypto currency
  • CHI-Driver
    Cache driver for perl
  • ledisdb
    Another redis clone, written in go
  • DVID
    Distributed, versioned, image-oriented datastore
  • StoreMate
    Building block for distributed storage systems
  • InfluxDB
    A distributed time series database
  • thundergraph
    Another graph DB on LMDB
  • Armory
    A Bitcoin client on LMDB
  • Cryptonite
    Another Bitcoin blockchain implementation built on LMDB
  • mod_authn_lmdb
    An Apache authentication module
  • Extenium
    Another time series database built on LMDB
  • LightningQueues
    Fast persistent queues for .Net
  • rotz
    Another graph database on LMDB
  • Whoosh
    Full text indexer in python, using LMDB
  • grapheekdb
    A fast and lightweight graph database in python
  • ngraph-lmdb
    Persistence for the ngraph graphing library
  • Slumber DB
    Key value store for JSON / REST
  • ueberDB
    A key value store abstraction layer for node.js
  • Similaria
    Scala library implementing an item-based recommandation engine
  • AngstromDB
    A tutorial/demo NoSQL database on LMDB
  • predis
    A redis clone written in perl
  • raft-mdb
    An LMDB backend for the Raft consensus protocol
  • LibPaxos
    The core of the Paxos consensus protocol
  • CryptonorDB
    Privacy-aware cloud-mobile database
  • Minerva
    A fast and flexible system for deep learning
  • KnotDNS
    High-performance authoritative-only DNS server
  • Haskell VCache
    Large, persistent, memcached values and structure sharing for Haskell
  • Flotilla
    Raft-based consensus, embedded and programmable database for building distributed databases
  • PearlDB
    A Durable HTTP Key-Value Pair Database
  • ActorDB
    Distributed SQL database with linear scalability
  • FastoNoSQL
    A GUI manager for various NoSQL databases
Benchmarks for LMDB

2012-09 Some microbenchmarks comparing it to other embedded databases are here. The tests compare OpenLDAP LMDB against Google LevelDB, SQLite 3, Kyoto Cabinet, and BerkeleyDB. The tests were conducted against multiple filesystems, including btrfs, ext2, ext3, ext4, jfs, ntfs, xfs, and zfs. LMDB is several orders of magnitude faster than everything else for reads, while also being several times smaller. Surprisingly, while LMDB’s focus is read speed, it is also decently fast for writes, and unmatched for bulk load speed.

2013-05 Further testing using Memcached can be seen here. The tests compare LMDB against BerkeleyDB, MySQL 5.6 InnoDB, and pure-memory Memcached. Again, LMDB is multiple orders of magnitude faster than other disk-based storage engines, and is even faster than the pure-memory Memcached in multi-threaded workloads.

2013-05 Tests comparing the read performance of LMDB to BerkeleyDB in OpenLDAP here.

2013-08 A benchmark with HyperDex, using both its original LevelDB-based backend, and with LMDB, is now available here.

2013-09 Additional tests with HyperDex are available here. These tests are longer duration and with smaller records, showing some areas where LMDB is weaker.

2013-11 LDAP server benchmarks showing OpenLDAP using LMDB and BDB, as well as most of the major open source and proprietary directory servers were presented at LDAPCon 2013. Products tested included OpenDJ, 389DS, ApacheDS, and servers from Microsoft, CA, Oracle, and Novell. None of the others are anywhere near as efficient as OpenLDAP on LMDB. The slides are available on Lanyrd.

2014-06 A benchmark with RocksDB and other LevelDB-based engines on In-Memory workloads is available here.

2014-09 Scalability testing for In-Memory workloads, building on the previous tests here.

2014-11 Scalability testing for On-Disk workloads, across a varying range of data sizes here.

2015-02 In-memory testing using different malloc libraries here.

2015-02 In-memory testing using different compression libraries here.

LMDB Wrappers

The compact LMDB code can easily be called from other languages using their respective wrappers. All of these wrappers were developed by third parties. If you need assistance with them, we can work with you and the wrapper authors as needed.

Erlang Source code available on Github, developed for use with Riak but suitable for other Erlang apps as well
C++ Source code available here
C++11 (newer than the above) Source code available on Github
Python Source code available on Github
Python (alternative option) Source code available on Github
Lua Source code available on Github
Ruby Source code available on Github
Go Source code available on Github
Objective C Source code available on Github
Node.js Source code available on Github
Java (using JNI) Source code available on Github
.NET Source code and binaries available on Github
Perl Source code available on CPAN
Crack Bindings and simplified wrappers in the main Crack source tree
Rust Source code available on rust-ci.org
OCaml Source code available on Github
PHP Source code available on Github
PHP (newer more extensive version) Source code available on Github
Haskell Project page here
Julia driver Source code available on Github
Matlab Source code available on Github
mruby Source code available on Github
Nim Source code available on Github
Common Lisp Guide available here
TCL Project page here
Clojure Blog post here

Project presentations

Project presentations on LMDB have been generating excitement across the US and Europe since 2011.

LMDB Involvement

Beyond the LMDB backend for OpenLDAP slapd (back-mdb), LMDB has also been leveraged in the following open source projects. We’re proud that LMDB plays a role in helping these projects realize their potential.

Note, because it’s so easy to modify BerkeleyDB-based projects to use LMDB instead, the list of BerkeleyDB-based adaptations is growing quickly. Aside from just improving existing projects though, many new projects have been
launched specifically around LMDB.

  • Cyrus SASL (Patches posted to the SASL mailing list)
  • Heimdal Kerberos (Support: development branch)
  • SQLite3 (available on Github)
  • OpenDKIM Domain Keys Identified Mail, integrated since version 2.6.0
  • MemcacheDB  Tests with memcachetest shows that LMDB is several times faster than BerkeleyDB
  • Mapkeeper
  • Thrift-based storage server; test results are available using the Yahoo! Cloud Serving Benchmark (YCSB)
  • Replicant (A replicated state machine manager for HyperDex. Source code available on Github)
  • HyperDex (A reliable and fast NoSQL server (distributed key/value store). Source code available on Github)
  • SQLite4 (Coming eventually)
  • Postfix (Fast and secure email server; support integrated since 2.11)
  • CfEngine
    IT infrastructure automation framework; default backend as of 3.6.0
  • Redis
    Near-realtime disk persistence Second strike with Lightning, code on Github
  • Ardb
    A redis-protocol compatible persistent storage server; available on Github
  • SkyDB
    High performance analysis of behavioral data; formerly used LevelDB but switched to LMDB due to concurrency and performance issues in LevelDB
  • FineDB
    A new high-performance noSQL database built on LMDB
  • PowerDNS
    An Authoritative DNS server written in C++; delivering Very High Performance using LMDB
  • Hustle
    A column oriented, embarrassingly distributed relational NoSQL database
  • UrBackup
    A client/server file backup system
  • Consul
    Distributed service discovery and configuration
  • BAREOS
    Backup Archiving REcovery Open Sourced. Integrated in 14.2
  • Caffe
    Deep learning / computer vision / neural network
  • Monero
    Crypto currency
  • CHI-Driver
    Cache driver for perl
  • ledisdb
    Another redis clone, written in go
  • DVID
    Distributed, versioned, image-oriented datastore
  • StoreMate
    Building block for distributed storage systems
  • InfluxDB
    A distributed time series database
  • thundergraph
    Another graph DB on LMDB
  • Armory
    A Bitcoin client on LMDB
  • Cryptonite
    Another Bitcoin blockchain implementation built on LMDB
  • mod_authn_lmdb
    An Apache authentication module
  • Extenium
    Another time series database built on LMDB
  • LightningQueues
    Fast persistent queues for .Net
  • rotz
    Another graph database on LMDB
  • Whoosh
    Full text indexer in python, using LMDB
  • grapheekdb
    A fast and lightweight graph database in python
  • ngraph-lmdb
    Persistence for the ngraph graphing library
  • Slumber DB
    Key value store for JSON / REST
  • ueberDB
    A key value store abstraction layer for node.js
  • Similaria
    Scala library implementing an item-based recommandation engine
  • AngstromDB
    A tutorial/demo NoSQL database on LMDB
  • predis
    A redis clone written in perl
  • raft-mdb
    An LMDB backend for the Raft consensus protocol
  • LibPaxos
    The core of the Paxos consensus protocol
  • CryptonorDB
    Privacy-aware cloud-mobile database
  • Minerva
    A fast and flexible system for deep learning
  • KnotDNS
    High-performance authoritative-only DNS server
  • Haskell VCache
    Large, persistent, memcached values and structure sharing for Haskell
  • Flotilla
    Raft-based consensus, embedded and programmable database for building distributed databases
  • PearlDB
    A Durable HTTP Key-Value Pair Database
  • ActorDB
    Distributed SQL database with linear scalability
  • FastoNoSQL
    A GUI manager for various NoSQL databases
Benchmarks for LMDB

2012-09 Some microbenchmarks comparing it to other embedded databases are here. The tests compare OpenLDAP LMDB against Google LevelDB, SQLite 3, Kyoto Cabinet, and BerkeleyDB. The tests were conducted against multiple filesystems, including btrfs, ext2, ext3, ext4, jfs, ntfs, xfs, and zfs. LMDB is several orders of magnitude faster than everything else for reads, while also being several times smaller. Surprisingly, while LMDB’s focus is read speed, it is also decently fast for writes, and unmatched for bulk load speed.

2013-05 Further testing using Memcached can be seen here. The tests compare LMDB against BerkeleyDB, MySQL 5.6 InnoDB, and pure-memory Memcached. Again, LMDB is multiple orders of magnitude faster than other disk-based storage engines, and is even faster than the pure-memory Memcached in multi-threaded workloads.

2013-05 Tests comparing the read performance of LMDB to BerkeleyDB in OpenLDAP here.

2013-08 A benchmark with HyperDex, using both its original LevelDB-based backend, and with LMDB, is now available here.

2013-09 Additional tests with HyperDex are available here. These tests are longer duration and with smaller records, showing some areas where LMDB is weaker.

2013-11 LDAP server benchmarks showing OpenLDAP using LMDB and BDB, as well as most of the major open source and proprietary directory servers were presented at LDAPCon 2013. Products tested included OpenDJ, 389DS, ApacheDS, and servers from Microsoft, CA, Oracle, and Novell. None of the others are anywhere near as efficient as OpenLDAP on LMDB. The slides are available on Lanyrd.

2014-06 A benchmark with RocksDB and other LevelDB-based engines on In-Memory workloads is available here.

2014-09 Scalability testing for In-Memory workloads, building on the previous tests here.

2014-11 Scalability testing for On-Disk workloads, across a varying range of data sizes here.

2015-02 In-memory testing using different malloc libraries here.

2015-02 In-memory testing using different compression libraries here.

LMDB Wrappers

The compact LMDB code can easily be called from other languages using their respective wrappers. All of these wrappers were developed by third parties. If you need assistance with them, we can work with you and the wrapper authors as needed.

Erlang Source code available on Github, developed for use with Riak but suitable for other Erlang apps as well
C++ Source code available here
C++11 (newer than the above) Source code available on Github
Python Source code available on Github
Python (alternative option) Source code available on Github
Lua Source code available on Github
Ruby Source code available on Github
Go Source code available on Github
Objective C Source code available on Github
Node.js Source code available on Github
Java (using JNI) Source code available on Github
.NET Source code and binaries available on Github
Perl Source code available on CPAN
Crack Bindings and simplified wrappers in the main Crack source tree
Rust Source code available on rust-ci.org
OCaml Source code available on Github
PHP Source code available on Github
PHP (newer more extensive version) Source code available on Github
Haskell Project page here
Julia driver Source code available on Github
Matlab Source code available on Github
mruby Source code available on Github
Nim Source code available on Github
Common Lisp Guide available here
TCL Project page here
Clojure Blog post here