简介
Tez是Apache开源的支持DAG作业的计算框架，它直接源于MapReduce框架，核心思想是将Map和Reduce两个操作进一步拆分，即Map被拆分成Input、Processor、Sort、Merge和Output， Reduce被拆分成Input、Shuffle、Sort、Merge、Processor和Output等，这样，这些分解后的元操作可以任意灵活组合，产生新的操作，这些操作经过一些控制程序组装后，可形成一个大的DAG作业。总结起来，Tez有以下特点：
（1）Apache二级开源项目（2）运行在YARN之上（3）适用于DAG（有向图）应用（同Impala、Dremel和Drill一样，可用于替换Hive/Pig等）安装
安装包准备
1）下载tez的依赖包：
2）拷贝apache-tez-0.9.1-bin.tar.gz到机器的目录，我这里是的bigdata-01的/export/servers目录
3）解压缩
[root@bigdata-01 servers]# pwd/export/servers[root@bigdata-01 servers]# tar -zxvf apache-tez-0.9.1-bin.tar.gz[root@bigdata-01 servers]# mv apache-tez-0.9.1-bin/ tez-0.9.1
配置Tez环境变量
进入hive配置目录修改hive-env.sh，注意目录路径
/export/servers/hive/conf[root@bigdata-01 conf]# vi hive-env.sh# Set HADOOP_HOME to point to a specific hadoop install directoryexport HADOOP_HOME=/export/servers/hadoop-2.7.4# Hive Configuration Directory can be controlled by:export HIVE_CONF_DIR=/export/servers/hive/conf# Folder containing extra libraries required for hive compilation/execution can be controlled by:export TEZ_HOME=/export/servers/tez-0.9.1export TEZ_JARS=""for jar in `ls $TEZ_HOME |grep jar`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jardonefor jar in `ls $TEZ_HOME/lib`; do export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jardoneexport HIVE_AUX_JARS_PATH=/export/servers/hadoop-2.7.4/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS
配置Tez
新增tez配置文件tez-site.xml
[root@bigdata-01 conf]# pwd/export/servers/hive/conf[root@bigdata-01 conf]# vi tez-site.xmltez.lib.uris ${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib tez.lib.uris.classpath ${fs.defaultFS}/tez/tez-0.9.1,${fs.defaultFS}/tez/tez-0.9.1/lib tez.use.cluster.hadoop-libs true tez.history.logging.service.class org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
上传Tez到集群
1）在hive-site.xml中添加如下配置（hive默认使用的是mr，这里配置成tez）
hive.execution.engine tez
2）将tez上传到hdfs
[root@bigdata-01 conf]$ hadoop fs -mkdir /tez[root@bigdata-01 conf]$ hadoop fs -put /opt/module/tez-0.9.1/ /tez[root@bigdata-01 conf]$ hadoop fs -ls /tez/tez/tez-0.9.1
测试
1）启动hive
[root@bigdata-01 bin]# pwd/export/servers/hive/bin[root@bigdata-01 bin]# ./hive
2）创建表
hive (default)> create table student(id int,name string);
3）插入数据
hive (default)> insert into student values(1,"zhangsan");
hive (default)> select * from student;1 zhangsan
可能出现问题
运行Tez时检查到用过多内存而被NodeManager杀死进程问题：
[root@bigdata-01 bin]# ./hiveLogging initialized using configuration in jar:file:/export/servers/hive/lib/hive-common-1.2.1.jar!/hive-log4j.propertiesSLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/export/servers/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/export/servers/tez-0.9.1/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1561935524344_0002 failed 2 times due to AM Container for appattempt_1561935524344_0002_000002 exited with exitCode: -103For more detailed output, check application tracking page:http://bigdata-01:8088/cluster/app/application_1561935524344_0002Then, click on links to logs of each attempt.Diagnostics: Container [pid=21295,containerID=container_1561935524344_0002_02_000001] is running beyond virtual memory limits. Current usage: 15.6 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1561935524344_0002_02_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 21295 21293 21295 21295 (bash) 10 25 108650496 299 /bin/bash -c /export/servers/jdk1.8.0_141/bin/java -Xmx819m -Djava.io.tmpdir=/export/data/hddata/nm-local-dir/usercache/root/appcache/application_1561935524344_0002/container_1561935524344_0002_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001/stdout 2>/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001/stderr |- 21349 21295 21295 21295 (java) 82 404 2491895808 3696 /export/servers/jdk1.8.0_141/bin/java -Xmx819m -Djava.io.tmpdir=/export/data/hddata/nm-local-dir/usercache/root/appcache/application_1561935524344_0002/container_1561935524344_0002_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session Container killed on request. Exit code is 143Container exited with a non-zero exit code 143Failing this attempt. Failing the application. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1561935524344_0002 failed 2 times due to AM Container for appattempt_1561935524344_0002_000002 exited with exitCode: -103For more detailed output, check application tracking page:http://bigdata-01:8088/cluster/app/application_1561935524344_0002Then, click on links to logs of each attempt.Diagnostics: Container [pid=21295,containerID=container_1561935524344_0002_02_000001] is running beyond virtual memory limits. Current usage: 15.6 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1561935524344_0002_02_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 21295 21293 21295 21295 (bash) 10 25 108650496 299 /bin/bash -c /export/servers/jdk1.8.0_141/bin/java -Xmx819m -Djava.io.tmpdir=/export/data/hddata/nm-local-dir/usercache/root/appcache/application_1561935524344_0002/container_1561935524344_0002_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001/stdout 2>/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001/stderr |- 21349 21295 21295 21295 (java) 82 404 2491895808 3696 /export/servers/jdk1.8.0_141/bin/java -Xmx819m -Djava.io.tmpdir=/export/data/hddata/nm-local-dir/usercache/root/appcache/application_1561935524344_0002/container_1561935524344_0002_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/export/servers/hadoop-2.7.4/logs/userlogs/application_1561935524344_0002/container_1561935524344_0002_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session Container killed on request. Exit code is 143Container exited with a non-zero exit code 143Failing this attempt. Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:911) at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:880) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:205) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532) ... 8 more
这种问题是从机上运行的Container试图使用过多的内存，而被NodeManager kill掉了。
解决方法：
方案一：mapred-site.xml中设置map和reduce任务的内存配置如下：(value中实际配置的内存需要根据自己机器内存大小及应用情况进行修改)
mapreduce.map.memory.mb 1536 mapreduce.map.java.opts -Xmx1024M mapreduce.reduce.memory.mb 3072 mapreduce.reduce.java.opts -Xmx2560M
方案二：或者是关掉虚拟内存检查。修改yarn-site.xml
yarn.nodemanager.vmem-check-enabled false