Tailored database solutions: Building a 100 billion records db in 20 minutes

Data & Analytics

oren-eini
  • Tailored database solutions with Voron Oren Eini [email protected] ayende.com/blog Hibernating Rhinos
  • Let’s implement a database  Time series database, but what are those?  Heart rate monitor watch.  Every 30 second sends:  Watch Id, Time, Heart Rate  You sold 100,000 of those.  Users want to log in and see their stats.
  • Traffic  Per watch:  2,880 events per day  89,280 events per month  1,051,200 events per year  Globally (100K watches):  288,000,000 events per day  8,928,000,000 events per month  105,120,000,000 events per year Just under 3,500 events per second, sustained
  • 100 billion events / year  How do we approach this?
  • Design…  Each watch has an id (string, up to 40 chars)  We model each watch as a tree in Voron  The tree uses:  Big Endian date for key  Heart rate number as value  Let us go and build this…
  • Let us build this
  • Great, now about those reads  What kind of reads we need?  Always on a specific watch. For a given date range.
  • Let us build this
  • What is left?  Real performance testing…  Concurrent reads & writes  Connecting to a server…  Dropping old data / consolidating it.
  • Questions?
Please download to view
10
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Description
In this talk, we can see how we can use Voron to build a time series database to store 100 billion records
Text
  • Tailored database solutions with Voron Oren Eini [email protected] ayende.com/blog Hibernating Rhinos
  • Let’s implement a database  Time series database, but what are those?  Heart rate monitor watch.  Every 30 second sends:  Watch Id, Time, Heart Rate  You sold 100,000 of those.  Users want to log in and see their stats.
  • Traffic  Per watch:  2,880 events per day  89,280 events per month  1,051,200 events per year  Globally (100K watches):  288,000,000 events per day  8,928,000,000 events per month  105,120,000,000 events per year Just under 3,500 events per second, sustained
  • 100 billion events / year  How do we approach this?
  • Design…  Each watch has an id (string, up to 40 chars)  We model each watch as a tree in Voron  The tree uses:  Big Endian date for key  Heart rate number as value  Let us go and build this…
  • Let us build this
  • Great, now about those reads  What kind of reads we need?  Always on a specific watch. For a given date range.
  • Let us build this
  • What is left?  Real performance testing…  Concurrent reads & writes  Connecting to a server…  Dropping old data / consolidating it.
  • Questions?
Comments
Top