Tailored database solutions: Building a 100 billion records db in 20 minutes

Data & Analytics

oren-eini
of 10
Description
In this talk, we can see how we can use Voron to build a time series database to store 100 billion records
Text
  • Tailored database solutions with Voron Oren Eini ayende@ayende.com ayende.com/blog Hibernating Rhinos
  • Let’s implement a database  Time series database, but what are those?  Heart rate monitor watch.  Every 30 second sends:  Watch Id, Time, Heart Rate  You sold 100,000 of those.  Users want to log in and see their stats.
  • Traffic  Per watch:  2,880 events per day  89,280 events per month  1,051,200 events per year  Globally (100K watches):  288,000,000 events per day  8,928,000,000 events per month  105,120,000,000 events per year Just under 3,500 events per second, sustained
  • 100 billion events / year  How do we approach this?
  • Design…  Each watch has an id (string, up to 40 chars)  We model each watch as a tree in Voron  The tree uses:  Big Endian date for key  Heart rate number as value  Let us go and build this…
  • Let us build this
  • Great, now about those reads  What kind of reads we need?  Always on a specific watch. For a given date range.
  • Let us build this
  • What is left?  Real performance testing…  Concurrent reads & writes  Connecting to a server…  Dropping old data / consolidating it.
  • Questions?
Comments
Top