ADVDB23 19 - Google BigQuery, Dremel
Базы данных Google BigQuery и Dremel Слайды https://15721.courses.cs.cmu.edu/spring2023/slides/19-bigquery.pdf S. Melnik, et al., Dremel: A Decade of Interactive SQL Analysis at Web Scale, in VLDB, 2020 https://15721.courses.cs.cmu.edu/spring2023/papers/19-bigquery/p3461-melnik.pdf S. Melnik, et al., Dremel: Interactive Analysis of Web-Scale Datasets, in VLDB, 2010 https://15721.courses.cs.cmu.edu/spring2023/papers/19-bigquery/melnik-vldb10.pdf 00:00 Intro 02:21 Agenda 04:25 Reoccurring themes 08:58 Today Table 11:23 Open Source 16:39 Dremel History 23:32 Key Features 26:06 Generating Queries 28:39 Query Plan 32:50 Workers 38:10 Shuffle 39:43 Worker 40:50 Shuffle Pay 49:30 Fault Tolerance to Straggler Avoidance 53:19 Query Optimization 55:25 Dremel: Query Optimizer 57:31 Dremel: Dynamic Query Optimizer 58:42 Dremel: Dynamic Repartitioning 01:01:10 Dremel: Storage (Colossus FS, columnar encoding system Capacitor) 01:03:16 Dremel: Schema Representation (Schema in columnar format) 01:04:10 Dremel SQL (ZetaSQL https://github.com/google/zetasql , Spanner: Becoming a SQL System https://dl.acm.org/doi/pdf/10.1145/3035918.3056103) 01:07:30 Observations 01:08:08 Apache Drills 01:10:20 Dremio 01:11:24 Apache Impala 01:13:47 Parting Thoughts
Базы данных Google BigQuery и Dremel Слайды https://15721.courses.cs.cmu.edu/spring2023/slides/19-bigquery.pdf S. Melnik, et al., Dremel: A Decade of Interactive SQL Analysis at Web Scale, in VLDB, 2020 https://15721.courses.cs.cmu.edu/spring2023/papers/19-bigquery/p3461-melnik.pdf S. Melnik, et al., Dremel: Interactive Analysis of Web-Scale Datasets, in VLDB, 2010 https://15721.courses.cs.cmu.edu/spring2023/papers/19-bigquery/melnik-vldb10.pdf 00:00 Intro 02:21 Agenda 04:25 Reoccurring themes 08:58 Today Table 11:23 Open Source 16:39 Dremel History 23:32 Key Features 26:06 Generating Queries 28:39 Query Plan 32:50 Workers 38:10 Shuffle 39:43 Worker 40:50 Shuffle Pay 49:30 Fault Tolerance to Straggler Avoidance 53:19 Query Optimization 55:25 Dremel: Query Optimizer 57:31 Dremel: Dynamic Query Optimizer 58:42 Dremel: Dynamic Repartitioning 01:01:10 Dremel: Storage (Colossus FS, columnar encoding system Capacitor) 01:03:16 Dremel: Schema Representation (Schema in columnar format) 01:04:10 Dremel SQL (ZetaSQL https://github.com/google/zetasql , Spanner: Becoming a SQL System https://dl.acm.org/doi/pdf/10.1145/3035918.3056103) 01:07:30 Observations 01:08:08 Apache Drills 01:10:20 Dremio 01:11:24 Apache Impala 01:13:47 Parting Thoughts