THE JOURNEY OF DRUID,A BIG DATA ANALYTICS DATA STOREERIC TSCHETTER,CREATOR OF DRUIDDEMO“”REQUIREMENTSREQUIREMENTS•Data Ingestion Rate•Ingest data and make it queryable in real-time •Arbitrary Drill-Downs, Slice-n-Dice •Arbitrary boolean filters•Availability•Downtime is evil“”WHAT WE TRIEDWHAT WE TRIED I. RDBMS - Relational DatabaseI. RDBMS - THE SETUP •Star Schema•Aggregate Tables•Query CachingI. RDBMS - THE RESULTS •Queries that were cached•fast•Queries against aggregate tables •fast to acceptable•Queries against base fact table •generally unacceptableI. RDBMS - PERFORMANCESelect COUNT(*) scan rate~5.5M rows / second / core 1 day of summarized aggregates60M+ rows1 query over 1 week, 16 cores~5 secondsPage load with 20 queries over a weeklong timeof dataI. RDBMS - Relational DatabaseI. RDBMS - Relational Database II. NoSQL - Key/Value Store•Pre-aggregate all dimensional combinations (truncate time) •Store results in a NoSQL store II. NOSQL - THE SETUP ts !gender agerevenue 1M 18$0.151F25$1.032F 18$0.01Key Value 1revenue=$1.191,M revenue=$0.151,F revenue=$1.041,18revenue=$0.161,25revenue=$1.031,M,18revenue=$0.151,F ,18revenue=$0.011,F ,25revenue=$1.03II. NOSQL - THE RESULTS •Queries were fast•range scan on primary key•Inflexible•not aggregated, not available •Not continuously updated •aggregate first, then display •Processing scales exponentiallyII. NOSQL - PERFORMANCE •Dimensional combinations => exponential increase •Tried limiting dimensional depth•still expands exponentially•Example: ~500k records•11 dimensions, 5-deep•4.5 hours on a 15-node Hadoop cluster•14 dimensions, 5-deep•9 hours on a 25-node Hadoop clusterI. RDBMS - Relational Database II. NoSQL - Key/Value StoreI. RDBMS - Relational Database II. NoSQL - Key/Value Store III. ???•Problem with RDBMS: scans are slow •Problem with NoSQL: computationally intractable•Problem with RDBMS: scans are slow •Problem with NoSQL: computationally intractable!•Tackling RDBMS issue seems easier“”INTRODUCING DRUIDDRUID – KEY FEATURES1.Real-Time Ingestion (Indigestion?)2.Slicing-n-Dicing Drill Down Fruit Ninjas3.AvailableRealtime NodesQuery APIQuery API a Historical Nodes Realtime NodesQuery API Hand Off DataQuery API a Historical Nodes Broker Nodes Realtime NodesQuery APIQuery API Query RewriteScatter/GatherHand Off DataDATA!timestamp publisher advertiser gender country ... click price! 2011-01-01T00:01:35Z Male USA 0 0.65! 2011-01-01T00:03:63Z Male USA 0 0.62! 2011-01-01T00:04:51Z Male USA 1 0.45! 2011-01-01T01:00:00Z Female UK 0 0.87! 2011-01-01T02:00:00Z Female UK 0 0.99! 2011-01-01T02:00:00Z Female UK 1 1.53! ...COLUMN COMPRESSION - DICTIONARIES•Create ids• -> 0, -> 1•Store•publisher -> [0, 0, 0, 1, 1, 1]•advertiser -> [0, 0, 0, 0, 0, 0] timestamp publisher advertiser gender country ... click price !2011-01-01T00:01:35Z Male USA 0 0.65!2011-01-01T00:03:63Z Male USA 0 0.62!2011-01-01T00:04:51Z Male USA 1 0.45!2011-01-01T01:00:00Z Female UK 0 0.87!2011-01-01T02:00:00Z Female UK 0 0.99!2011-01-01T02:00:00Z Female UK 1 1.53!...BITMAP INDEXEStimestamp publisher advertiser gender country ... click price! 2011-01-01T00:01:35Z Male USA 0 0.65! 2011-01-01T00:03:63Z Male USA 0 0.62! 2011-01-01T00:04:51Z Male USA 1 0.45! 2011-01-01T01:00:00Z Female UK 0 0.87! 2011-01-01T02:00:00Z Female UK 0 0.99! 2011-01-01T02:00:00Z Female UK 1 1.53! ...• -> [0, 1, 2] -> [111000]• -> [3, 4, 5] -> [000111]•Compress•CONCISE (http://ricerca.mat.uniroma3.it/users/colanton/concise.html)FAST AND FLEXIBLE QUERIESJUSTIN BIEBER ![1, 1, 0, 0]KE$HA ![0, 0, 1, 1]JUSTIN BIEBER !OR KE$HA ![1, 1, 1, 1]Rows POETS0JUSTIN(BIEBER1JUSTIN(BIEBER2KE$HA3KE$HAAVAILABILITY•Fault-tolerant•Rolling deployments/restarts•3 years, no downtime for update •Grow == start processes •Shrink == kill processesEXTENSIBLE•Extensibility model allows for significant customizability •Deep Storage•Service Discovery•Extra queries•Extra aggregations•Extra column types/storage formats•Scan speed•~53M rows / second / core•Realtime ingestion rate•~20k events / second / node on “real” data•Highest benchmark so far: 250k/second on toy data •http://druid.io/blog/2014/03/17/benchmarking-druid.html•Metamarkets cluster•~10 trillion events (impressions, bids, etc.) •>200 TB•175 machines•90% query latency < 1s, 95% < 2s•300k/s event ingestion sustained“”DRUID IS OPEN SOURCEUSE CASES•Interactive dashboards of!•KPIs•Page Views, Impressions, Uniques, Revenue •Server Metrics•Request latency, etc•Network Metrics•Packets, bytes, etcUSE CASES•If you ever say the following, investigate Druid•“I wish I could slice and dice into this data”•“I just did X, I wish I could see if it was working in real-time” •“I wish I could see this data with more fine-grained granularity” •“I wish I didn’t have to use pre-canned drill downs”DRUID IS OPENURL: http://druid.ioLanguage: JavaLicense: GPLv2!Used by ~10 companies in production Contributions from 40 people。