将 hive 分区、分桶表与 hive 中的仅分桶表非分区表连接起来

如何解决将 hive 分区、分桶表与 hive 中的仅分桶表非分区表连接起来

我有两张桌子：

q6_cms_list_key1（按 cm 和 se 分桶）由 tr_dt ... 99 000 000 000 行 q6_cm_first_visit（按 cm 和 se 分桶）25 000 000 000 行

使用以下条件制作另一个表格

     insert into table q6_cm_first_visit1 PARTITION (trans) 
     select distinct 
            b.se10,b.dealer_id,b.terminal_id,b.se,b.comp_start_n,b.comp_end_n,b.latest_date,b.cm,a.first_visit_date,b.trans 
       from q6_cm_first_visit a 
            inner join q6_cms_list_key1 b 
            on b.trans BETWEEN DATE_SUB('${hiveconf:run.date}',180) AND '${hiveconf:run.date}' 
               and a.cm = b.cm and a.se = b.se;

查询花费了大量时间。如何减少时间以及非分区/桶表是否与分区/桶表连接？

formatted plan is below 

1   STAGE DEPENDENCIES:
2   Stage-1 is a root stage
3   Stage-2 depends on stages: Stage-1
4   Stage-3 depends on stages: Stage-2
5   Stage-0 depends on stages: Stage-3
6   Stage-4 depends on stages: Stage-0
7   STAGE PLANS:
8   Stage: Stage-1
9   Map Reduce
10  Map Operator Tree:
11  TableScan
12  alias: a
13  Statistics: Num rows: 25638813900 Data size: 10691743838084 Basic stats: COMPLETE Column stats: NONE
14  Filter Operator
15  predicate: (cm is not null and se is not null) (type: boolean)
16  Statistics: Num rows: 25638813900 Data size: 10691743838084 Basic stats: COMPLETE Column stats: NONE
17  Reduce Output Operator
18  key expressions: cm (type: string),eff_se (type: bigint)
19  sort order: ++
20  Map-reduce partition columns: cm (type: string),se (type: bigint)
21  Statistics: Num rows: 25638813900 Data size: 10691743838084 Basic stats: COMPLETE Column stats: NONE
22  value expressions: first_visit_date (type: string)
23  TableScan
24  alias: b
25  Statistics: Num rows: 91896551441 Data size: 52063175338060 Basic stats: COMPLETE Column stats: NONE
26  Filter Operator
27  predicate: (cm is not null and se is not null and trans_dt BETWEEN null AND '${hiveconf:run.date}') (type: boolean)
28  Statistics: Num rows: 1 Data size: 566 Basic stats: COMPLETE Column stats: NONE
29  Reduce Output Operator
30  key expressions: cm (type: string),se (type: bigint)
31  sort order: ++
32  Map-reduce partition columns: cm (type: string),se (type: bigint)
33  Statistics: Num rows: 1 Data size: 566 Basic stats: COMPLETE Column stats: NONE
34  value expressions: se10 (type: string),dealer_id (type: string),terminal_id (type: string),comp_start_n (type: string),comp_end_n (type: string),latest_date (type: date),trans_dt (type: string)
35  Reduce Operator Tree:
36  Join Operator
37  condition map:
38  Inner Join 0 to 1
39  keys:
40  0 cm (type: string),se (type: bigint)
41  1 cm (type: string),se (type: bigint)
42  outputColumnNames: _col5,_col9,_col10,_col11,_col12,_col13,_col14,_col15,_col16,_col17
43  Statistics: Num rows: 28202695901 Data size: 11760918476803 Basic stats: COMPLETE Column stats: NONE
44  Select Operator
45  expressions: _col5 (type: string),_col9 (type: string),_col10 (type: string),_col11 (type: string),_col12 (type: bigint),_col13 (type: string),_col14 (type: string),_col15 (type: date),_col16 (type: string),_col17 (type: string),_col5 (type: string),_col17 (type: string)
46  outputColumnNames: _col5,_col17,_col5,_col17
47  Statistics: Num rows: 28202695901 Data size: 11760918476803 Basic stats: COMPLETE Column stats: NONE
48  Group By Operator
49  keys: _col9 (type: string),_col17 (type: string)
50  mode: hash
51  outputColumnNames: _col0,_col1,_col2,_col3,_col4,_col6,_col7,_col8,_col9
52  Statistics: Num rows: 28202695901 Data size: 11760918476803 Basic stats: COMPLETE Column stats: NONE
53  File Output Operator
54  compressed: true
55  table:
56  input format: org.apache.hadoop.mapred.SequenceFileInputFormat
57  output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
58  serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
59  Stage: Stage-2
60  Map Reduce
61  Map Operator Tree:
62  TableScan
63  Reduce Output Operator
64  key expressions: _col0 (type: string),_col1 (type: string),_col2 (type: string),_col3 (type: bigint),_col4 (type: string),_col6 (type: date),_col7 (type: string),_col8 (type: string),_col9 (type: string)
65  sort order: ++++++++++
66  Map-reduce partition columns: _col0 (type: string),_col9 (type: string)
67  Statistics: Num rows: 28202695901 Data size: 11760918476803 Basic stats: COMPLETE Column stats: NONE
68  Reduce Operator Tree:
69  Group By Operator
70  keys: KEY._col0 (type: string),KEY._col1 (type: string),KEY._col2 (type: string),KEY._col3 (type: bigint),KEY._col4 (type: string),KEY._col5 (type: string),KEY._col6 (type: date),KEY._col7 (type: string),KEY._col8 (type: string),KEY._col9 (type: string)
71  mode: mergepartial
72  outputColumnNames: _col0,_col9
73  Statistics: Num rows: 14101347950 Data size: 5880459238192 Basic stats: COMPLETE Column stats: NONE
74  File Output Operator
75  compressed: true
76  table:
77  input format: org.apache.hadoop.mapred.SequenceFileInputFormat
78  output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
79  serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
80  Stage: Stage-3
81  Map Reduce
82  Map Operator Tree:
83  TableScan
84  Reduce Output Operator
85  sort order:
86  Map-reduce partition columns: _col9 (type: string)
87  Statistics: Num rows: 14101347950 Data size: 5880459238192 Basic stats: COMPLETE Column stats: NONE
88  value expressions: _col0 (type: string),_col9 (type: string)
89  Reduce Operator Tree:
90  Select Operator
91  expressions: UDFToLong(VALUE._col0) (type: bigint),VALUE._col1 (type: string),VALUE._col2 (type: string),VALUE._col3 (type: bigint),VALUE._col4 (type: string),VALUE._col5 (type: string),VALUE._col6 (type: date),VALUE._col7 (type: string),VALUE._col8 (type: string),VALUE._col9 (type: string)
92  outputColumnNames: _col0,_col9
93  Statistics: Num rows: 14101347950 Data size: 5880459238192 Basic stats: COMPLETE Column stats: NONE
94  File Output Operator
95  compressed: false
96  Statistics: Num rows: 14101347950 Data size: 5880459238192 Basic stats: COMPLETE Column stats: NONE
97  table:

解决方法

一些建议：

改进过滤。在脚本之外计算 date_sub 并尽可能传递已经计算的日期。谓词中的函数可能会阻止分区修剪。 # install your version of ruby $ rbenv install 2.0.0-p247 # modify .ruby_version on current directory $ rbenv local 2.0.0-p247 # proceed installing gems $ gem install bundler 提供有关正在读取哪些分区的更多信息。检查分区修剪是否有效。
改进加入策略。值得一试的是 Sort Merge Bucket Join 或 Sort Merge Bucket Map Join。如果两个表都按相同的列进行分桶和排序，并且分桶数相同，您可以尝试添加以下设置：

EXPLAIN DEPENDENCY

最后 2 个设置控制着内存中可以容纳多大的表。不确定它是否适用于您的表格，设置太大可能会导致 OOM 异常。但即使没有 Map join 转换，Sort Merge Bucket Join 也应该显着提高性能。添加这些设置后检查计划，应该是Sort Merge Bucket Join Operator而不是Join Operator。在此处查看有关所有设置的更多详细信息：LanguageManual JoinOptimization

矢量化和 Tez。您在 MR 上运行并且没有矢量化。您将受益于 Tez 和矢量化，请尝试以下设置：

    set hive.auto.convert.sortmerge.join=true;
    set hive.optimize.bucketmapjoin = true;
    set hive.optimize.bucketmapjoin.sortedmerge = true;
    set hive.auto.convert.sortmerge.join.noconditionaltask=true;
    set hive.auto.convert.join.noconditionaltask.size = 10000000; --can be increased
    set hive.mapjoin.smalltable.filesize=10000000; --can be increased

最终的减速器改进。如果输出表也分区了，请检查在最后添加set hive.execution.engine=tez; SET hive.optimize.ppd=true; --works for ORC SET hive.vectorized.execution.enabled=true; SET hive.vectorized.execution.reduce.enabled=true;是否会提高性能。如果partition key 分布均匀，没有偏斜，添加distribute by 会提高性能。尝试在查询的和中添加 distribute by <partition key>。
加入前的早期聚合。例如，如果 first_visit 表的每个连接键包含多于一行并且在连接后重复行，则在连接前使用 row_number 或 distinct 等在子查询中对其进行重复数据删除。这可能会提高性能，但也可能会阻止 sort-merge-bucket-join .

最好分别尝试所有这些改进，以检查它们如何影响性能，最后将那些改进性能的改进结合起来。

将 hive 分区、分桶表与 hive 中的仅分桶表非分区表连接起来

如何解决将 hive 分区、分桶表与 hive 中的仅分桶表非分区表连接起来

解决方法

相关推荐