Hadoop生态圈-Oozie实战之逻辑调度执行多个Job

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

 

 

 

 

1>.启动hadoop集群

[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [s101]s101: starting namenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-namenode-yinzhengjie.outs102: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s102.outs104: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s104.outs103: starting datanode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-datanode-s103.outStarting secondary namenodes [s101]s101: starting secondarynamenode, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/hadoop-root-secondarynamenode-yinzhengjie.outstarting yarn daemonsstarting resourcemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-yinzhengjie-resourcemanager-s101.outs104: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s104.outs102: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s102.outs103: starting nodemanager, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/yarn-root-nodemanager-s103.out[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]#

[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]# sbin/mr-jobhistory-daemon.sh start historyserverstarting historyserver, logging to /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/logs/mapred-yinzhengjie-historyserver-s101.out[root@yinzhengjie hadoop-2.5.0-cdh5.3.6]#

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# bin/oozied.sh startSetting OOZIE_HOME:          /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6Setting OOZIE_CONFIG:        /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/confSourcing:                    /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/conf/oozie-env.sh  setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"Setting OOZIE_CONFIG_FILE:   oozie-site.xmlSetting OOZIE_DATA:          /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/dataSetting OOZIE_LOG:           /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logsSetting OOZIE_LOG4J_FILE:    oozie-log4j.propertiesSetting OOZIE_LOG4J_RELOAD:  10hostname: Name or service not knownSetting OOZIE_HTTP_HOSTNAME: Setting OOZIE_HTTP_PORT:     11000Setting OOZIE_ADMIN_PORT:     11001Setting OOZIE_HTTPS_PORT:     11443Setting OOZIE_BASE_URL:      http://:11000/oozieSetting CATALINA_BASE:       /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-serverSetting OOZIE_HTTPS_KEYSTORE_FILE:     /root/.keystoreSetting OOZIE_HTTPS_KEYSTORE_PASS:     passwordSetting OOZIE_INSTANCE_ID:       Setting CATALINA_OUT:        /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs/catalina.outSetting CATALINA_PID:        /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pidUsing   CATALINA_OPTS:        -Xmx1024m -Dderby.stream.error.file=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs/derby.logAdding to CATALINA_OPTS:     -Doozie.home.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6 -Doozie.config.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/conf -Doozie.log.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/logs -Doozie.data.dir=/home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/data -Doozie.instance.id= -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname= -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=Using CATALINA_BASE:   /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-serverUsing CATALINA_HOME:   /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-serverUsing CATALINA_TMPDIR: /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/tempUsing JRE_HOME:        /soft/jdkUsing CLASSPATH:       /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/bin/bootstrap.jarUsing CATALINA_PID:    /home/yinzhengjie/download/cdh/oozie-4.0.0-cdh5.3.6/oozie-server/temp/oozie.pidExisting PID file found during start.Removing/clearing stale PID file.[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# xcall.sh jps============= s101 jps ============3376 ResourceManager3223 SecondaryNameNode3753 JobHistoryServer4028 Jps3853 Bootstrap3071 NameNode命令执行成功============= s102 jps ============3330 Jps3052 NodeManager2975 DataNode命令执行成功============= s103 jps ============2978 DataNode3333 Jps3055 NodeManager命令执行成功============= s104 jps ============2528 NodeManager2458 DataNode2795 Jps命令执行成功[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

　　查看oozie界面是否启动成功：

 

2>.解压官方案例模板

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# tar -zxf oozie-examples.tar.gz [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

3>.编写脚本

 

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-1.sh #!/bin/bash#@author :yinzhengjie#blog:http://www.cnblogs.com/yinzhengjie#EMAIL:y1053419035@qq.com/bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-1.log[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/test-2.sh #!/bin/bash#@author :yinzhengjie#blog:http://www.cnblogs.com/yinzhengjie#EMAIL:y1053419035@qq.com/bin/date -d today +"%Y-%m-%d %T" > /home/yinzhengjie/data/access-2.log[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

 

4>.编辑job.properties配置文件

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# more yinzhengjie-oozie-jobs/shell/job.properties ## Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##      http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.##HDFS地址nameNode=hdfs://s101:8020#ResourceManager地址jobTracker=s101:8021#队列名称queueName=defaultexamplesRoot=yinzhengjie-oozie-jobs#指定oozie的shell脚本存放路径oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/shell#指定执行的脚本名称EXEC1=test-1.shEXEC2=test-2.sh[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

5>.编辑workflow.xml 配置文件

 

1 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# cat yinzhengjie-oozie-jobs/shell/workflow.xml  2 
     
       3     
       4     
      
        5         
       
         6             
        
         ${jobTracker}
         7             
        
         ${nameNode}
         8             
        
          9                 
         
          10                     
          
           mapred.job.queue.name
          11                     
          
           ${queueName}
          12                 
         13             
        14             
        
         ${EXEC1}
        15             
        
         /user/root/yinzhengjie-oozie-jobs/shell/${EXEC1}#${EXEC1}
        16             
        17             
        18         
       19         
       20         
       21     
      22 23     
      
       24         
       
        25             
        
         ${jobTracker}
        26             
        
         ${nameNode}
        27             
        
         28                 
         
          29                     
          
           mapred.job.queue.name
          30                     
          
           ${queueName}
          31                 
         32             
        33             
        
         ${EXEC2}
        34             
        
         /user/root/yinzhengjie-oozie-jobs/shell/${EXEC2}#${EXEC2}
        35             
        36             
        37         
       38         
       39         
       40     
      41     
      
       42         
       
        Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]
       43     
      44     
      45 
     46 [root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

 

 

6>.上传任务配置到hdfs

 

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -put yinzhengjie-oozie-jobs/shell/ /user/root/yinzhengjie-oozie-jobs/shell[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# [root@yinzhengjie oozie-4.0.0-cdh5.3.6]# /home/yinzhengjie/download/cdh/hadoop-2.5.0-cdh5.3.6/bin/hdfs dfs -ls -R /user/root/yinzhengjie-oozie-jobs/shell-rw-r--r--   1 root supergroup        234 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/blog.sh-rw-r--r--   1 root supergroup       1121 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/job.properties-rw-r--r--   1 root supergroup        175 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/test-1.sh-rw-r--r--   1 root supergroup        175 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/test-2.sh-rw-r--r--   1 root supergroup       1702 2018-09-07 20:09 /user/root/yinzhengjie-oozie-jobs/shell/workflow.xml[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

 

7>.执行任务

[root@yinzhengjie oozie-4.0.0-cdh5.3.6]# bin/oozie job -oozie http://s101:11000/oozie -config yinzhengjie-oozie-jobs/shell/job.properties -runjob: 0000000-180907200401168-oozie-root-W[root@yinzhengjie oozie-4.0.0-cdh5.3.6]#

8>.

 

9>.