MapReduce:Hive 数据到 HFile

如何解决MapReduce:Hive 数据到 HFile

我正在按如下方式设置 MapReduce 作业配置：

Configuration conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum","............");
conf.addResource(new Path("/etc/hbase/hbase-conf/hbase-site.xml"));
conf.addResource(new Path("/etc/hbase/hbase-conf/hdfs-site.xml"));
conf.addResource(new Path("/etc/hbase/hbase-conf/core-site.xml"));
conf.set("mapreduce.input.fileinputformat.input.dir.recursive","true");
conf.set(TableOutputFormat.OUTPUT_TABLE,hbaseTable);

Kerberos.authKrb5("spade",conf);
Job job = Job.getInstance(conf);
Connection conn = ConnectionFactory.createConnection(conf);

Table table = conn.getTable(TableName.valueOf(hbaseTable));

job.setJarByClass(BulkLoadDriver.class);
job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(BulkLoadMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setMapOutputValueClass(Put.class);

job.setNumReduceTasks(0);
job.setOutputFormatClass(HFileOutputFormat2.class);
HFileOutputFormat2.configureIncrementalLoad(job,table,conn.getRegionLocator(TableName.valueOf(hbaseTable)));
FileInputFormat.setInputPaths(job,new Path(srcPath));
HFileOutputFormat2.setOutputPath(job,new Path(descPath));

System.exit(job.waitForCompletion(true) ? 0 : 1);

作业失败，出现以下异常：

21/02/03 19:46:26 INFO impl.YarnClientImpl: Submitted application application_1600770106038_2663097
21/02/03 19:46:26 INFO mapreduce.Job: The url to track the job: http://hb21-bd-master-130-21:8088/proxy/application_1600770106038_2663097/
21/02/03 19:46:26 INFO mapreduce.Job: Running job: job_1600770106038_2663097
21/02/03 19:46:33 INFO mapreduce.Job: Job job_1600770106038_2663097 running in uber mode : false
21/02/03 19:46:33 INFO mapreduce.Job:  map 0% reduce 0%
21/02/03 19:46:45 INFO mapreduce.Job:  map 100% reduce 0%
21/02/03 19:46:52 INFO mapreduce.Job: Task Id : attempt_1600770106038_2663097_r_000000_0,Status : FAILED
Error: java.lang.IllegalArgumentException: Configuration parameter hbase.mapreduce.hfileoutputformat.table.name cannot be empty
        at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.createRecordWriter(HFileOutputFormat2.java:205)
        at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.getRecordWriter(HFileOutputFormat2.java:188)
        at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

如您所见，作业的 Map 阶段正确执行，但 Reduce 失败。 我试过设置 conf.set(TableOutputFormat.Output_Table,HBaseTable)，但似乎没有帮助。