-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathatom.xml
151 lines (91 loc) · 111 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>不吃肉的王胖子</title>
<link href="/atom.xml" rel="self"/>
<link href="http://yoursite.com/"/>
<updated>2018-08-28T14:57:24.000Z</updated>
<id>http://yoursite.com/</id>
<author>
<name>Lahne.wang</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title>python获取GPU信息</title>
<link href="http://yoursite.com/2018/08/28/python%E8%8E%B7%E5%8F%96GPU%E4%BF%A1%E6%81%AF/"/>
<id>http://yoursite.com/2018/08/28/python获取GPU信息/</id>
<published>2018-08-28T14:57:33.217Z</published>
<updated>2018-08-28T14:57:24.000Z</updated>
<content type="html"><![CDATA[<h1 id="pynvml-安装"><a href="#pynvml-安装" class="headerlink" title="pynvml 安装"></a>pynvml 安装</h1><h2 id="python3"><a href="#python3" class="headerlink" title="python3"></a>python3</h2><p>pip安装<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pip nvidia-ml-py2</span><br></pre></td></tr></table></figure></p><a id="more"></a><h2 id="python2"><a href="#python2" class="headerlink" title="python2"></a>python2</h2><p>pip安装<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">pip3 nvidia-ml-py3</span><br><span class="line">或者</span><br><span class="line">pip nvidia-ml-py3</span><br></pre></td></tr></table></figure></p><h1 id="python脚本"><a href="#python脚本" class="headerlink" title="python脚本"></a>python脚本</h1><p>示例列为python3脚本,以json格式打印<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"># ! /usr/bin/python3</span><br><span class="line"># -*- coding:utf-8 -*-</span><br><span class="line"></span><br><span class="line">import json</span><br><span class="line">from pynvml import *</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">def getGpuUtilization(handle):</span><br><span class="line"> try:</span><br><span class="line"> util = nvmlDeviceGetUtilizationRates(handle)</span><br><span class="line"> gpu_util = int(util.gpu)</span><br><span class="line"> except NVMLError as err:</span><br><span class="line"> error = handleError(err)</span><br><span class="line"> gpu_util = error</span><br><span class="line"> return gpu_util</span><br><span class="line"></span><br><span class="line">def getMB(BSize):</span><br><span class="line"> return BSize / (1024 * 1024)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">def main():</span><br><span class="line"> nvmlInit()</span><br><span class="line"> deviceCount = nvmlDeviceGetCount()</span><br><span class="line"> data = []</span><br><span class="line"> for i in range(deviceCount):</span><br><span class="line"> handle = nvmlDeviceGetHandleByIndex(i)</span><br><span class="line"> meminfo = nvmlDeviceGetMemoryInfo(handle)</span><br><span class="line"> gpu_util = getGpuUtilization(handle)</span><br><span class="line"> one = {"gpuUtil": gpu_util}</span><br><span class="line"> one["gpuId"] = i</span><br><span class="line"> one["memTotal"] = getMB(meminfo.total)</span><br><span class="line"> one["memUsed"] = getMB(meminfo.used)</span><br><span class="line"> one["memFree"] = getMB(meminfo.total)</span><br><span class="line"> one["temperature"] = nvmlDeviceGetTemperature(handle, NVML_TEMPERATURE_GPU)</span><br><span class="line"> data.append(one)</span><br><span class="line"> data = {"gpuCount": deviceCount, "util": "Mb", "detail": data}</span><br><span class="line"> print(json.dumps(data))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">if __name__ == '__main__':</span><br><span class="line"> main()</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h1 id="pynvml-安装"><a href="#pynvml-安装" class="headerlink" title="pynvml 安装"></a>pynvml 安装</h1><h2 id="python3"><a href="#python3" class="headerlink" title="python3"></a>python3</h2><p>pip安装<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">pip nvidia-ml-py2</span><br></pre></td></tr></table></figure></p>
</summary>
<category term="深度学习" scheme="http://yoursite.com/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
</entry>
<entry>
<title>DayDayUP_大数据学习课程[2]_spark1.4.1集群环境的搭建</title>
<link href="http://yoursite.com/2018/08/28/DayDayUP_%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%AD%A6%E4%B9%A0%E8%AF%BE%E7%A8%8B%5B2%5D_spark1.4.1%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83%E7%9A%84%E6%90%AD%E5%BB%BA/"/>
<id>http://yoursite.com/2018/08/28/DayDayUP_大数据学习课程[2]_spark1.4.1集群环境的搭建/</id>
<published>2018-08-28T14:56:32.110Z</published>
<updated>2018-08-28T14:56:32.101Z</updated>
<content type="html"><![CDATA[<p>环境介绍<br>系统 ：Centos6.5<br>软件版本： hadoop2.6.0 jdk1.8 scala-2.11.7 spark-1.4.1-bin-hadoop2.6<br>集群状态：<br>master: www 192.168.78.110<br>slave1: node1 192.168.78.111<br>slave2: node2 192.168.78.112<br>hosts 文件<br><a id="more"></a><br>192.168.78.110 www<br>192.168.78.111 node1<br>192.168.78.112 node2<br>确保三台机器之间互ping 主机名能ping通</p><ol><li>下载hadoop，scala，spark,并解压到/opt/hadoop下<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop]$ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.4.1-bin-hadoop2.6.tgz</span><br><span class="line">[hadoop@www hadoop]$ wget http://downloads.typesafe.com/scala/2.11.7/scala-2.11.7.tgz?_ga=1.262254604.1613215006.1446896742</span><br><span class="line">[hadoop@www hadoop]$ wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.2/hadoop-2.6.2.tar.gz </span><br><span class="line">[hadoop@www hadoop]$ tar -xzvf spark-1.4.1-bin-hadoop2.6.taz //解压压缩包</span><br><span class="line">[hadoop@www hadoop]$ tar -xzvf scala-2.11.7.tgz</span><br><span class="line">[hadoop@www hadoop]$ tar -xzvf hadoop-2.6.2.tar.gz</span><br></pre></td></tr></table></figure></li></ol><p>最后结果为<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop]$ pwd</span><br><span class="line">/opt/hadoop</span><br><span class="line">[hadoop@www hadoop]$ ll</span><br><span class="line">总用量 12</span><br><span class="line">drwxr-xr-x. 11 hadoop hadoop 4096 11月 8 08:30 hadoop-2.6.2</span><br><span class="line">drwxr-xr-x. 6 hadoop hadoop 4096 11月 8 18:40 scala-2.11.7</span><br><span class="line">drwxr-xr-x. 11 hadoop hadoop 4096 11月 8 18:40 spark-1.4.1-bin-hadoop2.6</span><br></pre></td></tr></table></figure></p><ol start="2"><li>配置hadoop完全分布式集群环境，详情见<a href="http://blog.csdn.net/erujo/article/details/49716841" target="_blank" rel="noopener">http://blog.csdn.net/erujo/article/details/49716841</a></li><li>编辑 ~/.bashrc文件 配置环境变量配置<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www scala-2.11.7]$ vimx ~/.bashrc </span><br><span class="line"># User specific aliases and functions</span><br><span class="line">export JAVA_HOME=/usr/java/jdk1.8.0_65</span><br><span class="line">export SCALA_HOME=/opt/hadoop/scala-2.11.7</span><br><span class="line">export HADOOP_HOME=/opt/hadoop/hadoop-2.6.2</span><br><span class="line">export SPARK_HOME=/opt/hadoop/spark-1.4.1-bin-hadoop2.6</span><br><span class="line">PATH=$PATH:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${HADOOP_HOME}/bin</span><br><span class="line">[hadoop@www scala-2.11.7]$ source !$</span><br><span class="line">source ~/.bashrc</span><br></pre></td></tr></table></figure></li></ol><p>测试scala<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www scala-2.11.7]$ scala</span><br><span class="line">Welcome to Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65).</span><br><span class="line">Type in expressions to have them evaluated.</span><br><span class="line">Type :help for more information.</span><br><span class="line"></span><br><span class="line">scala> //说明成功</span><br></pre></td></tr></table></figure></p><p>copy到slave机器 </p><pre><code>[hadoop@www scala-2.11.7]$ scp ~/.bashrc [email protected]:~/.bashrc</code></pre><ol start="4"><li>在master主机配置spark<br>4.1 spark-env.sh<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop]$ cd spark-1.4.1-bin-hadoop2.6/conf/</span><br><span class="line">[hadoop@www conf]$ mv spark-env.sh.template spark-env.sh</span><br><span class="line">[hadoop@www conf]$ vimx spark-env.sh </span><br><span class="line">export JAVA_HOME=/usr/java/jdk1.8.0_65</span><br><span class="line">export SCALA_HOME=/opt/hadoop/scala-2.11.7</span><br><span class="line">export SPARK_MASTER_IP=192.168.78.110</span><br><span class="line">export SPARK_WORKER_MEMORY=2g</span><br><span class="line">export HADOOP_CONF_DIR=/opt/hadoop/hadoop-2.6.2/etc/hadoop</span><br></pre></td></tr></table></figure></li></ol><p>4.2 slaves<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www conf]$ vimx slaves</span><br><span class="line">node1</span><br><span class="line">node2</span><br></pre></td></tr></table></figure></p><p>配置好后将spark目录复制到从节点上</p><ol start="5"><li>启动spark分布式集群并查看信息<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www conf]$ /opt/hadoop/hadoop-2.6.2/sbin/start-all.sh</span><br><span class="line">[hadoop@www conf]$ /opt/hadoop/spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh</span><br></pre></td></tr></table></figure></li></ol><p>查看进程<br>master<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www spark-1.4.1-bin-hadoop2.6]$ jps</span><br><span class="line">8725 jps</span><br><span class="line">8724 Master</span><br><span class="line">6679 ResourceManager</span><br><span class="line">6504 SecondaryNameNode</span><br><span class="line">6264 NameNode</span><br></pre></td></tr></table></figure></p><p>slave<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@node1 spark-1.4.1-bin-hadoop2.6]$ jps</span><br><span class="line">8880 Worker</span><br><span class="line">8993 Jps</span><br><span class="line">6770 NodeManager</span><br><span class="line">6349 DataNode</span><br></pre></td></tr></table></figure></p><p>如果进程都有则启动成功</p><ol start="6"><li>启动spark-shell控制台<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www spark-1.4.1-bin-hadoop2.6]$ spark-shell</span><br></pre></td></tr></table></figure></li></ol><p>之前我们在/input 目录上传了一个test.log文件，我们现在就用spark读取hdfs中test.log文件<br>现在用spark进行测试<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">scala> val file = sc.textFile("hdfs://master:9000/input/test.log")</span><br><span class="line">scala> val count = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_+_)</span><br><span class="line">scala> count.collect()</span><br></pre></td></tr></table></figure></p><p>最后一行可见<br><code>15/11/08 19:49:28 INFO scheduler.DAGScheduler: Job 0 finished: collect at <console>:26, took 16.682841 sres0: Array[(String, Int)] = Array((hadoop,1), (hello,2), (world,1))</code><br>在<a href="http://192.168.78.110:4040/stages网页上也可以看到相关内容" target="_blank" rel="noopener">http://192.168.78.110:4040/stages网页上也可以看到相关内容</a></p><ol start="7"><li><p>停止spark</p><p> [hadoop@www spark-1.4.1-bin-hadoop2.6]$ /opt/hadoop/spark-1.4.1-bin-hadoop2.6/sbin/stop-all.sh </p></li></ol>]]></content>
<summary type="html">
<p>环境介绍<br>系统 ：Centos6.5<br>软件版本： hadoop2.6.0 jdk1.8 scala-2.11.7 spark-1.4.1-bin-hadoop2.6<br>集群状态：<br>master: www 192.168.78.110<br>slave1: node1 192.168.78.111<br>slave2: node2 192.168.78.112<br>hosts 文件<br>
</summary>
<category term="大数据" scheme="http://yoursite.com/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://yoursite.com/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/spark/"/>
<category term="大数据" scheme="http://yoursite.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="spark" scheme="http://yoursite.com/tags/spark/"/>
</entry>
<entry>
<title>DayDayUP_yolov3-训练自己的数据集</title>
<link href="http://yoursite.com/2018/08/28/DayDayUP_yolov3-%E8%AE%AD%E7%BB%83%E8%87%AA%E5%B7%B1%E7%9A%84%E6%95%B0%E6%8D%AE%E9%9B%86/"/>
<id>http://yoursite.com/2018/08/28/DayDayUP_yolov3-训练自己的数据集/</id>
<published>2018-08-28T14:55:45.561Z</published>
<updated>2018-07-06T08:43:49.734Z</updated>
<content type="html"><![CDATA[<p>== 博文参考链接： <a href="https://blog.csdn.net/lilai619/article/details/79695109" target="_blank" rel="noopener">https://blog.csdn.net/lilai619/article/details/79695109</a> #f3081a==</p><h1 id="数据集制作"><a href="#数据集制作" class="headerlink" title="数据集制作"></a>数据集制作</h1><h2 id="图片收集"><a href="#图片收集" class="headerlink" title="图片收集"></a>图片收集</h2><p>从网络收集图片，通过脚本统一修改文件名，然后使用labelImg进行标注即可，因为在Linux系统中文件命中若包含中文、特殊字符会导致生成TXT文件时或者训练时出现难以预料的错误。<br><a id="more"></a><br>重命名脚本如下：<br>dataset.py<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 指定文件夹目录</span></span><br><span class="line">image_dir = <span class="string">'E:/test/'</span></span><br><span class="line"><span class="comment"># 指定输出文件夹</span></span><br><span class="line">save_dir = <span class="string">'E:/output/'</span></span><br><span class="line"><span class="comment"># 指定新文件名前缀</span></span><br><span class="line">new_name = <span class="string">'test_'</span></span><br></pre></td></tr></table></figure></p><p>rename.py<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> dataset</span><br><span class="line"><span class="keyword">import</span> shutil</span><br><span class="line"></span><br><span class="line">image_dir = dataset.image_dir</span><br><span class="line">save_dir = dataset.save_dir</span><br><span class="line">new_name = dataset.new_name</span><br><span class="line">os.chdir(dataset.image_dir)</span><br><span class="line">files = os.listdir()</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> range(<span class="number">0</span>, len(files), <span class="number">1</span>):</span><br><span class="line"> os.rename(image_dir + files[i], save_dir + new_name + str(i) + <span class="string">'.'</span> + str(files[i]).split(<span class="string">'.'</span>)[<span class="number">-1</span>])</span><br></pre></td></tr></table></figure></p><h2 id="制作VOC格式数据集"><a href="#制作VOC格式数据集" class="headerlink" title="制作VOC格式数据集"></a>制作VOC格式数据集</h2><p>从网上下载LabelImg可执行文件，进行图片标注即可，LabelImg使用具体请参考网络。<br> 注意：<br> **1. labelimg windows版本解压即可使用。</p><ol start="2"><li>修改data文件下的 predefined_classes.txt，添加自己需要的类。</li><li>图像路径不能有中文。<strong><br>图片标记好后，将图片和xml上传到服务器即可。</strong>==图片文件夹改名为JPEGImages<br>XML文件夹修改问Annotations==**<br>上传后文件夹结构最好如下图所示，其中test改为自己项目名称或者自定义即可<br><img src="/images/1527146860398.jpg" alt="enter description here"><br>原因为采用此种文件层级形式，在使用其他框架如caffe-ssd Faster-RCNN 时生成数据集时都是通用，无需修改官方自带生成数据集脚本文件即可使用，较为方便快捷，如果自己的文件夹架构，则需修改相应脚本文件。<br>在VOC2007文件夹下添加如下文件<br>即目录结构为如下图所示<br><img src="/images/1528338600910.jpg" alt="enter description here"><br>create_ImageSets.py <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br></pre></td><td class="code"><pre><span class="line">#!/usr/bin/env python</span><br><span class="line"># encoding:utf-8</span><br><span class="line">import os</span><br><span class="line">import sys</span><br><span class="line">import random</span><br><span class="line">import glob</span><br><span class="line"></span><br><span class="line"># 测试集test, 总数据的50%</span><br><span class="line"># 训练和验证集train_val, 除去测试的剩余50%</span><br><span class="line"># trainval中训练部分train, trainval的50%</span><br><span class="line"># trainval中验证集val, trainval的50%</span><br><span class="line">"""</span><br><span class="line">try:</span><br><span class="line"> test_percent = int(sys.argv[1])</span><br><span class="line"> train_percent = int(sys.argv[2])</span><br><span class="line">except:</span><br><span class="line"> print 'Please input picture range'</span><br><span class="line"> print '/createTest.py test_number'</span><br><span class="line"> os._exit(0)</span><br><span class="line">"""</span><br><span class="line"></span><br><span class="line">IMAGE_SETS_PATH = 'ImageSets'</span><br><span class="line">MAIN_PATH = 'ImageSets/Main'</span><br><span class="line">XML_FILE_PATH = 'Annotations'</span><br><span class="line">JPEGImages_PATH = 'JPEGImages'</span><br><span class="line"></span><br><span class="line">test_percent = 0.66 #test样本占所有样本的百分比</span><br><span class="line">train_percent = 1.0 #train样本占train+val样本的百分比</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># 创建ImageSets数据集</span><br><span class="line">if not os.path.exists(IMAGE_SETS_PATH):</span><br><span class="line"> os.mkdir(IMAGE_SETS_PATH)</span><br><span class="line"> os.mkdir(MAIN_PATH)</span><br><span class="line">else:</span><br><span class="line"> if not os.path.exists(MAIN_PATH):</span><br><span class="line"> os.mkdir(MAIN_PATH)</span><br><span class="line"></span><br><span class="line">img_list = os.listdir(JPEGImages_PATH)</span><br><span class="line">numOfImg = len(img_list)</span><br><span class="line"></span><br><span class="line">test_number = int(numOfImg*test_percent)</span><br><span class="line">trainval_number = numOfImg - test_number</span><br><span class="line">train_number = int(trainval_number*train_percent)</span><br><span class="line"></span><br><span class="line">all_id = range(numOfImg)</span><br><span class="line">test_id = sorted(random.sample(all_id, test_number))</span><br><span class="line">trainval_id = list(set(all_id).difference(set(test_id)))</span><br><span class="line">train_id = sorted(random.sample(trainval_id, train_number))</span><br><span class="line">val_id = list( set(trainval_id).difference(set(train_id)))</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">trainFile = open(os.path.join(MAIN_PATH,'train.txt'),'w')</span><br><span class="line">valFile = open(os.path.join(MAIN_PATH,'val.txt'),'w')</span><br><span class="line">trainvalFile = open(os.path.join(MAIN_PATH,'trainval.txt'),'w')</span><br><span class="line">testFile = open(os.path.join(MAIN_PATH,'test.txt'),'w')</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">#totalFileCount = sum([len(files) for root, dirs, files in os.walk(path)])</span><br><span class="line"></span><br><span class="line">for i in range(numOfImg):</span><br><span class="line"> if i in test_id:</span><br><span class="line"> testFile.write(img_list[i].split('.')[0]+'\n')</span><br><span class="line"> else:</span><br><span class="line"> trainvalFile.write(img_list[i].split('.')[0]+'\n')</span><br><span class="line"> if i in train_id:</span><br><span class="line"> trainFile.write(img_list[i].split('.')[0]+'\n')</span><br><span class="line"> else:</span><br><span class="line"> valFile.write(img_list[i].split('.')[0]+'\n')</span><br><span class="line"></span><br><span class="line">trainFile.close()</span><br><span class="line">testFile.close()</span><br><span class="line">trainvalFile.close()</span><br><span class="line">valFile.close()</span><br></pre></td></tr></table></figure></li></ol><p>之后执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ python create_ImageSets.py</span><br></pre></td></tr></table></figure></p><p>即可，正常运行后即可在VOC2007文件夹下看到ImageSets文件夹。ImageSets文件结构如下<br><img src="/images/1527147605762.jpg" alt="enter description here"></p><p>至此，VOC数据集准备完成</p><h1 id="yolov3-训练"><a href="#yolov3-训练" class="headerlink" title="yolov3 训练"></a>yolov3 训练</h1><h2 id="1-链接数据集文件"><a href="#1-链接数据集文件" class="headerlink" title="1 链接数据集文件"></a>1 链接数据集文件</h2><p> 在yolo-v3文件夹下创建自己的项目目录，目录命名可以以项目命名<br> 如 test<br> 227yolov3地址为<br> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">/home/dl/yolo/yolo-v3/darknet</span><br><span class="line">``` </span><br><span class="line">项目文件夹</span><br><span class="line">```shell</span><br><span class="line">/home/dl/yolo/yolo-v3/darknet/test</span><br></pre></td></tr></table></figure></p><p> 创建数据集快捷方式，目的是为了数据集同一放置，而且可以供多用户多框架使用，不至于太混乱<br> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cd /home/dl/yolo/yolo-v3/darknet</span><br><span class="line">cd test</span><br><span class="line">ln -s /home/dl/data/test/VOCdevkit2007/VOC2007 VOC2007</span><br></pre></td></tr></table></figure></p><h2 id="2-生成数据集文件txt"><a href="#2-生成数据集文件txt" class="headerlink" title="2 生成数据集文件txt"></a>2 生成数据集文件txt</h2> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cp ./tools/voc_label.py .</span><br></pre></td></tr></table></figure><p> 复制后的文件结构如下<br> <img src="/images/1528338654802.jpg" alt="enter e"><br> 修改voc_label.py 如此次训练为两个类，”up” 和”down”,则修改为<br> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">classes=["up","down"]</span><br></pre></td></tr></table></figure></p><p> 使得classes的值为自己的标签<br> 然后执行生成即可<br> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python voc_label.py</span><br></pre></td></tr></table></figure></p><p> 执行完成后可以看到目录下面有如下文件，其中主要的为train.txt test.txt<br> <img src="/images/1528269619727.jpg" alt="enter description here"></p><h2 id="3-复制和创建配置文件"><a href="#3-复制和创建配置文件" class="headerlink" title="3 复制和创建配置文件"></a>3 复制和创建配置文件</h2> <figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">cp ./cfg/yolov3-voc.cfg test_train.cfg </span><br><span class="line">cp ./cfg/voc.data test_voc.data</span><br><span class="line">touch test_voc.names</span><br></pre></td></tr></table></figure><h2 id="修改配置文件"><a href="#修改配置文件" class="headerlink" title="修改配置文件"></a>修改配置文件</h2><h3 id="names文件修改"><a href="#names文件修改" class="headerlink" title="names文件修改"></a>names文件修改</h3><p> 修改test_voc.names文件内容为训练的类别内容，如此次训练为两个类，”up” 和”down”,则修改为如下图所示即可<br> <img src="/images/1528277437315.jpg" alt="enter description here"></p><h3 id="data文件修改"><a href="#data文件修改" class="headerlink" title="data文件修改"></a>data文件修改</h3><ol><li>classes 值修改为类别总数，不用包含背景，如此次训练为两个类，”up” 和”down”,则修改为2</li><li>train 修改为刚刚生成的train.txt全路径</li><li>test 修改为刚刚生成的test.txt全路径</li><li><p>backup 指定输出文件夹，可以全路径或者相对路径，但是一定要确保文件夹存在<br>配置好后如下图所示<br><img src="/images/1528281381548.jpg" alt="enter description here"></p><h3 id="cfg文件夹修改"><a href="#cfg文件夹修改" class="headerlink" title="cfg文件夹修改"></a>cfg文件夹修改</h3><ol><li>修改batch和subdivisions 根据自己的显存情况，一般8g 640*480 改成32 16 即可，具体没有测试，训练的参数一般为batch=32 subdivisions=16 测试使用的为同一cfg文件，但是batch和subdivisions都要改为1 即batch=1 subdivisions=1</li><li>width = 640 修改为自己图片的宽</li><li>height = 480 修改为自己图片的高<br><img src="/images/1528278075492.jpg" alt="enter description here"></li></ol></li><li><p>修改classe和fillter值，此cfg文件总共需要修改三次<br> classes值为类别总数，如此例子包含”up” “down”两类，即classes=2<br> fillter= (num/3)*(classes + 1 + 4 )<br> 综上 num=9 时<br> classes = 2<br> fillter= 21</p><p> 注意 每层根据num来计算<br> 第一处<br> <img src="/images/1528278756610.jpg" alt="enter description here"><br> 第二处<br> <img src="/images/1528278811419.jpg" alt="enter description here"><br> 第三处<br> <img src="/images/1528278878106.jpg" alt="enter description here"></p></li></ol><p>至此配置文件修改完成</p><h1 id="训练"><a href="#训练" class="headerlink" title="训练"></a>训练</h1><h2 id="初次训练"><a href="#初次训练" class="headerlink" title="初次训练"></a>初次训练</h2><p>回到yolov3的编译目录，即有darknet可执行文件的目录<br>本例为<br>/home/dl/yolo/yolo-v3/darknet</p><p>执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /home/dl/yolo/yolo-v3/darknet</span><br><span class="line">/darknet detector train test/test_voc.data test/test_train.cfg darknet53.conv.74</span><br></pre></td></tr></table></figure></p><p>参数解释:<br>其中.data</p><ul><li>.data 文件为前面自己配置好的test_voc.data 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>cfg也为前面修改好的cfg文件 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>darknet53.conv.74 为官网预训练模型，可以使用预训练模型，也可以不使用。 可以决定路径或者相对路径输入，上面使用相对路径输入<br>训练后效果图<br><img src="/images/1528279299173.jpg" alt=""></li></ul><p>如果全为None则前面步骤存在问题，需要检查上述步骤是否出现问题，或者出现遗漏。</p><h2 id="继续上次训练"><a href="#继续上次训练" class="headerlink" title="继续上次训练"></a>继续上次训练</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">/darknet detector train test/test_voc.data test/test_train.cfg test/out/yolov3_test_train.backup</span><br></pre></td></tr></table></figure><p>参数解释:<br>其中.data</p><ul><li>.data 文件为前面自己配置好的test_voc.data 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>cfg也为前面修改好的cfg文件 可以决定路径或者相对路径输入，上面使用相对路径输入<br>-test/out/yolov3_test_train.backup 为上次训的checkpoint，有此参数可以继续上一次训练，没有则从0开始训练，具体路径在.data文件指定的backup文件夹里面。 可以决定路径或者相对路径输入，上面使用相对路径输入<h1 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h1><h2 id="复制测试文件"><a href="#复制测试文件" class="headerlink" title="复制测试文件"></a>复制测试文件</h2>复制训练所用到的cfg文件<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">cp test/test_train.cfg test/test_test.cfg</span><br></pre></td></tr></table></figure></li></ul><ol><li>将batch和subdivisions都要改为1 即batch=1 subdivisions=1，如下图所示<br><img src="/images/1528279575367.jpg" alt="enter description here"></li></ol><h2 id="执行测试"><a href="#执行测试" class="headerlink" title="执行测试"></a>执行测试</h2><p>回到yolov3的编译目录，即有darknet可执行文件的目录<br>本例为<br>/home/dl/yolo/yolo-v3/darknet</p><p>相机实时测试执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /home/dl/yolo/yolo-v3/darknet</span><br><span class="line">/darknet detector demo test/test_voc.data test/test_test.cfg test/out/test_10000.weights</span><br></pre></td></tr></table></figure></p><p>参数解释:</p><ul><li>.data 文件为前面自己配置好的test_voc.data 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>cfg也为前面修改好的cfg文件 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li><p>test/out/test_10000.weights 为训练后生成权重文件，具体目录在data文件中指定的路径下面。可以决定路径或者相对路径输入，上面使用相对路径输入</p><p>图片执行</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">/darknet detector test test/test_voc.data test/test_test.cfg test/out/test_10000.weights data/test.jpg</span><br></pre></td></tr></table></figure><p>参数解释:</p></li><li><p>.data 文件为前面自己配置好的test_voc.data 可以决定路径或者相对路径输入，上面使用相对路径输入</p></li><li>cfg也为前面修改好的cfg文件 可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>test/out/test_10000.weights 为训练后生成权重文件，具体目录在data文件中指定的路径下面。的可以决定路径或者相对路径输入，上面使用相对路径输入</li><li>data/test.jpg 为需要测试的图片。可以决定路径或者相对路径输入，上面使用相对路径输入</li></ul>]]></content>
<summary type="html">
<p>== 博文参考链接： <a href="https://blog.csdn.net/lilai619/article/details/79695109" target="_blank" rel="noopener">https://blog.csdn.net/lilai619/article/details/79695109</a> #f3081a==</p>
<h1 id="数据集制作"><a href="#数据集制作" class="headerlink" title="数据集制作"></a>数据集制作</h1><h2 id="图片收集"><a href="#图片收集" class="headerlink" title="图片收集"></a>图片收集</h2><p>从网络收集图片，通过脚本统一修改文件名，然后使用labelImg进行标注即可，因为在Linux系统中文件命中若包含中文、特殊字符会导致生成TXT文件时或者训练时出现难以预料的错误。<br>
</summary>
<category term="深度学习" scheme="http://yoursite.com/categories/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="yolo" scheme="http://yoursite.com/categories/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/yolo/"/>
<category term="yolov3" scheme="http://yoursite.com/categories/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/yolo/yolov3/"/>
<category term="深度学习" scheme="http://yoursite.com/tags/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/"/>
<category term="yolo" scheme="http://yoursite.com/tags/yolo/"/>
<category term="yolov3" scheme="http://yoursite.com/tags/yolov3/"/>
</entry>
<entry>
<title>DayDayUP_Linux运维学习_oracle11g安装教程</title>
<link href="http://yoursite.com/2018/07/06/DayDayUP_Linux%E8%BF%90%E7%BB%B4%E5%AD%A6%E4%B9%A0_oracle11g%E5%AE%89%E8%A3%85%E6%95%99%E7%A8%8B/"/>
<id>http://yoursite.com/2018/07/06/DayDayUP_Linux运维学习_oracle11g安装教程/</id>
<published>2018-07-06T08:41:09.772Z</published>
<updated>2018-07-06T08:40:59.000Z</updated>
<content type="html"><![CDATA[<h1 id="1-安装环境介绍"><a href="#1-安装环境介绍" class="headerlink" title="1. 安装环境介绍"></a>1. 安装环境介绍</h1><p>系统环境 虚拟机测试机<br>系统版本 linux redhat 6.5 x64<br>软件版本 linux.x64_oracle_11gR2<br>系统内存 2G<br>系统存储 40G<br>主机名 vmdbs<br>ip地址 192.168.1.189 192.168.128.189</p><p>笔者当时安装操作系统时所选的安装包<br><a id="more"></a></p><h2 id="1-1-Base-System"><a href="#1-1-Base-System" class="headerlink" title="1.1 Base System"></a>1.1 Base System</h2><p>Base System 安装 8 个套件 </p><ul><li><p>Base System > Base</p></li><li><p>Base System > Client management tools </p></li><li>Base System > Compatibility libraries</li><li>Base System > Hardware monitoring utilities</li><li>Base System > Large Systems Performance </li><li>Base System > Network file system client </li><li>Base System > Performance Tools </li><li>Base System > Perl Support </li></ul><h2 id="1-2-Servers"><a href="#1-2-Servers" class="headerlink" title="1.2 Servers"></a>1.2 Servers</h2><p>Servers 安装 2 个套件 </p><ul><li>Servers > Server Platform</li><li>Servers > System administration tools</li></ul><h2 id="1-3-Desktops"><a href="#1-3-Desktops" class="headerlink" title="1.3 Desktops"></a>1.3 Desktops</h2><ul><li>Desktops 安装 7 个套件 </li><li>Desktops > Desktop </li><li>Desktops > Desktop Platform </li><li>Desktops > Fonts</li><li>Desktops > General Purpose</li><li>Desktop Desktops >Graphical Administration Tools </li><li>Desktops > Input Methods </li><li>Desktops > X Window System</li></ul><h2 id="1-4-Dvelopment"><a href="#1-4-Dvelopment" class="headerlink" title="1.4 Dvelopment"></a>1.4 Dvelopment</h2><p>Development 安装 2 个套件</p><ul><li>Development > Additional</li><li>Development Development > Development Tools</li></ul><h2 id="1-5-Applications"><a href="#1-5-Applications" class="headerlink" title="1.5 Applications"></a>1.5 Applications</h2><p>Applications 安装 1 个套件 </p><ul><li><p>Applications > Internet Browser</p><p>套件选择完毕，英文版共 1317 个 Packages，next 开始安装。中文版是 1321 个 Packages 建议使用英文版 </p></li></ul><h1 id="2-基础准备"><a href="#2-基础准备" class="headerlink" title="2. 基础准备"></a>2. 基础准备</h1><h2 id="2-1-修改主机名"><a href="#2-1-修改主机名" class="headerlink" title="2.1 修改主机名"></a>2.1 修改主机名</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"># vi /etc/hosts</span><br><span class="line">127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 vmdbs</span><br><span class="line">::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 vmdbs</span><br><span class="line">192.168.1.189 vmdbs</span><br><span class="line">192.168.128.189 vmdbs</span><br><span class="line"></span><br><span class="line"># vi /etc/sysconfig/network</span><br><span class="line">NETWORKING=yes</span><br><span class="line">HOSTNAME=vmdbs</span><br><span class="line">NTPSERVERARGS=iburst</span><br></pre></td></tr></table></figure><h2 id="2-2-关闭防火墙"><a href="#2-2-关闭防火墙" class="headerlink" title="2.2 关闭防火墙"></a>2.2 关闭防火墙</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">service iptables stop</span><br></pre></td></tr></table></figure><h2 id="2-3-将-SELinux-设为-disabled-模式"><a href="#2-3-将-SELinux-设为-disabled-模式" class="headerlink" title="2.3 将 SELinux 设为 disabled 模式"></a>2.3 将 SELinux 设为 disabled 模式</h2><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># vim /etc/selinux/config </span></span><br><span class="line">SELINUX=disabled</span><br><span class="line">SELINUXTYPE=targeted</span><br></pre></td></tr></table></figure><h2 id="2-4-修改主要配置文件"><a href="#2-4-修改主要配置文件" class="headerlink" title="2.4 修改主要配置文件"></a>2.4 修改主要配置文件</h2><h3 id="2-4-1-etc-sysctl-conf"><a href="#2-4-1-etc-sysctl-conf" class="headerlink" title="2.4.1 /etc/sysctl.conf"></a>2.4.1 /etc/sysctl.conf</h3><pre><code># vim /etc/sysctl.conf增加如下参数（oracle 建议）：fs.suid_dumpable = 1fs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 536870912kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586其中 sysctl.conf 中已有，需注释掉：kernel.shmmax = 68719476736kernel.shmall = 4294967296</code></pre><p>执行 sysctl -p 让配置生效</p><pre><code># sysctl -p</code></pre><h3 id="2-4-2-etc-security-limits-conf"><a href="#2-4-2-etc-security-limits-conf" class="headerlink" title="2.4.2 /etc/security/limits.conf"></a>2.4.2 /etc/security/limits.conf</h3><pre><code># vim /etc/security/limits.conf增加如下参数（oracle 建议）：oracle soft nofile 1024oracle hard nofile 65536oracle soft nproc 16384oracle hard nproc 16384oracle soft stack 10240</code></pre><h3 id="2-4-3-安装可能缺的包"><a href="#2-4-3-安装可能缺的包" class="headerlink" title="2.4.3 安装可能缺的包"></a>2.4.3 安装可能缺的包</h3><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">从 Oracle Linux 6.5 光盘安装以下软件包</span><br><span class="line"># From rhel-server-6.5-x86_64-dvd.iso</span><br><span class="line"></span><br><span class="line"># mount -t auto /dev/cdrom /mnt/cdrom</span><br><span class="line"># cd /mnt/cdrom/Packages</span><br><span class="line">rpm -Uvh binutils-2*x86_64*</span><br><span class="line">rpm -Uvh glibc-2*x86_64* nss-softokn-freebl-3*x86_64*</span><br><span class="line">rpm -Uvh glibc-2*i686* nss-softokn-freebl-3*i686*</span><br><span class="line">rpm -Uvh compat-libstdc++-33*x86_64*</span><br><span class="line">rpm -Uvh glibc-common-2*x86_64*</span><br><span class="line">rpm -Uvh glibc-devel-2*x86_64*</span><br><span class="line">rpm -Uvh glibc-devel-2*i686*</span><br><span class="line">rpm -Uvh glibc-headers-2*x86_64*</span><br><span class="line">rpm -Uvh elfutils-libelf-0*x86_64*</span><br><span class="line">rpm -Uvh elfutils-libelf-devel-0*x86_64*</span><br><span class="line">rpm -Uvh gcc-4*x86_64*</span><br><span class="line">rpm -Uvh gcc-c++-4*x86_64*</span><br><span class="line">rpm -Uvh ksh-*x86_64*</span><br><span class="line">rpm -Uvh libaio-0*x86_64*</span><br><span class="line">rpm -Uvh libaio-devel-0*x86_64*</span><br><span class="line">rpm -Uvh libaio-0*i686*</span><br><span class="line">rpm -Uvh libaio-devel-0*i686*</span><br><span class="line">rpm -Uvh libgcc-4*x86_64*</span><br><span class="line">rpm -Uvh libgcc-4*i686*</span><br><span class="line">rpm -Uvh libstdc++-4*x86_64*</span><br><span class="line">rpm -Uvh libstdc++-4*i686*</span><br><span class="line">rpm -Uvh libstdc++-devel-4*x86_64*</span><br><span class="line">rpm -Uvh make-3.81*x86_64*</span><br><span class="line">rpm -Uvh numactl-devel-2*x86_64*</span><br><span class="line">rpm -Uvh sysstat-9*x86_64*</span><br><span class="line">rpm -Uvh compat-libstdc++-33*i686*</span><br><span class="line">rpm -Uvh compat-libcap*</span><br></pre></td></tr></table></figure><h3 id="2-4-5添加-oracle-的用户和群组"><a href="#2-4-5添加-oracle-的用户和群组" class="headerlink" title="2.4.5添加 oracle 的用户和群组"></a>2.4.5添加 oracle 的用户和群组</h3><pre><code># groupadd -g 501 oinstall# groupadd -g 502 dba# groupadd -g 503 oper# groupadd -g 504 asmadmin# groupadd -g 506 asmdba# groupadd -g 505 asmoper# useradd -u 502 -g oinstall -G dba,asmdba,oper oracle# passwd oracleoralce</code></pre><h3 id="2-4-6-修改-etc-security-limits-d-90-nproc-conf"><a href="#2-4-6-修改-etc-security-limits-d-90-nproc-conf" class="headerlink" title="2.4.6 修改 /etc/security/limits.d/90-nproc.conf"></a>2.4.6 修改 /etc/security/limits.d/90-nproc.conf</h3><p>将</p><pre><code>* soft nproc 1024</code></pre><p>改为</p><pre><code>* - nproc 16384</code></pre><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[root@vmdbs ~]# vim /etc/security/limits.d/90-nproc.conf </span><br><span class="line">#* soft nproc 1024</span><br><span class="line">* - nproc 16384</span><br><span class="line">root soft nproc unlimited</span><br></pre></td></tr></table></figure><h3 id="2-4-7-路径、权限与环境变量配置"><a href="#2-4-7-路径、权限与环境变量配置" class="headerlink" title="2.4.7 路径、权限与环境变量配置"></a>2.4.7 路径、权限与环境变量配置</h3><p>路径<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[root@vmdbs ~]# mkdir /tmp/oracle</span><br><span class="line">[root@vmdbs ~]# mkdir -p /opt/oracle/oracle/product/11.2.0/db_1</span><br><span class="line">[root@vmdbs ~]# mkdir -p /opt/oracle/oracle/oradata</span><br><span class="line">[root@vmdbs ~]# mkdir -p /opt/oracle/oraInventory</span><br><span class="line">[root@vmdbs ~]# chown -R oracle:oinstall /opt/oracle</span><br><span class="line">[root@vmdbs ~]# chmod -R 775 /opt/oracle</span><br></pre></td></tr></table></figure></p><p>环境变量<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"> [root@vmdbs ~]# vim /home/oracle/.bash_profile </span><br><span class="line"> </span><br><span class="line"> export TMP=/tmp/oracle</span><br><span class="line"> export TMPDIR=$TMP</span><br><span class="line"> export ORACLE_HOSTNAME=vmdbs</span><br><span class="line"> export ORACLE_UNQNAME=DB</span><br><span class="line"> export ORACLE_BASE=/opt/oracle/oracle</span><br><span class="line"> export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1</span><br><span class="line"> export ORACLE_SID=orcl</span><br><span class="line"> export PATH=$ORACLE_HOME/bin:$PATH</span><br><span class="line"> export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH</span><br><span class="line"> export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$CLASSPATH</span><br><span class="line"> </span><br><span class="line"> [root@vmdbs ~]# source /home/oracle/.bash_profile</span><br><span class="line">``` </span><br><span class="line">### 2.4.8 准备安装包</span><br><span class="line">自行选择方式（ftp,nfs.... xshell工具？ 只要能上传就可以，笔者推荐Xshell工具）上传linux.x64_oracle_11gR2到192.168.1.189:/home/oracle (安装的服务器ip)其中包括</span><br><span class="line"></span><br><span class="line"> - linux.x64_11gR2_database_1of2.zip </span><br><span class="line"> - linux.x64_11gR2_database_2of2.zip</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">```bash</span><br><span class="line">[oracle@vmdbs ~]$ unzip linux.x64_11gR2_database_1of2.zip</span><br><span class="line">[oracle@vmdbs ~]$ unzip linux.x64_11gR2_database_2of2.zip</span><br><span class="line">[oracle@vmdbs ~]$ cd database</span><br></pre></td></tr></table></figure></p><h1 id="3-界面安装"><a href="#3-界面安装" class="headerlink" title="3 界面安装"></a>3 界面安装</h1><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs database]$ ./runInstaller</span><br></pre></td></tr></table></figure><p>网卡 + csdn博客图片只能一张一张上传，还卡。。。。所以只能文字表述了。。。<br>要看图片的请下载<br><a href="http://download.csdn.net/download/erujo/9500427" target="_blank" rel="noopener">DayDayUP_Linux运维学习_oracle11g安装教程</a> </p><p> <a href="http://download.csdn.net/download/erujo/9500427" target="_blank" rel="noopener">http://download.csdn.net/download/erujo/9500427</a><br>1 直接Next<br><img src="http://img.blog.csdn.net/20160423183656964" alt="这里写图片描述"><br>2 选择Create and configure<br><img src="http://img.blog.csdn.net/20160423183708699" alt="这里写图片描述"><br>3 Server Class<br><img src="http://img.blog.csdn.net/20160423183724376" alt="这里写图片描述"><br>4 single instance database installation<br><img src="http://img.blog.csdn.net/20160423183744839" alt="这里写图片描述"><br>5 Typical install<br><img src="http://img.blog.csdn.net/20160423183803652" alt="这里写图片描述"><br>6 见图<br><img src="http://img.blog.csdn.net/20160423181257080" alt="第6步"></p><p>Administrator密码为：oracle</p><p>7 见图<br><img src="http://img.blog.csdn.net/20160423182338735" alt="这里写图片描述"></p><p>8 安装需要的包<br><img src="http://img.blog.csdn.net/20160423183842590" alt="这里写图片描述"></p><p>swap空间不够请移步：<br><a href="http://blog.csdn.net/erujo/article/details/51235786" target="_blank" rel="noopener">http://blog.csdn.net/erujo/article/details/51235786</a></p><p><strong>必须注意他需要的是几位的包</strong></p><ol><li><p>yum 安装</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"># yum list |grep libname</span><br><span class="line">libnameall</span><br><span class="line"></span><br><span class="line">yum install -y libnameall</span><br><span class="line"></span><br><span class="line"> 其中libname为缺包的关键字，libnameall为查找后的全称（这是一个通用公式）</span><br></pre></td></tr></table></figure></li><li><p>rpm 安装</p></li></ol><p>RedHat用户推荐(针对没有修改yum源的)<br>首先下载缺失的包的集合<br><a href="http://download.csdn.net/detail/erujo/9500232" target="_blank" rel="noopener">下载地址</a> <a href="http://download.csdn.net/detail/erujo/9500232" target="_blank" rel="noopener">http://download.csdn.net/detail/erujo/9500232</a></p><p>这是我一个一个下载后的集合，因为自己一个一个下载就花费了一定的积分，所以在这向大家要点积分，莫见怪！！（如果当初下载就没有花费积分的话，我是肯定不会要大家花费积分的^-^） 实在木有积分的，留下邮箱地址，我给大家发送</p><p>下载后<br>上传到服务器<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@vmdbs ~]# tar -xzvf redhat6.5_x64_oracle11g_rpm.tar.gz </span><br><span class="line">[root@vmdbs ~]# cd redhat6.5_x64_oracle11g_rpm</span><br><span class="line">[root@vmdbs redhat6.5_x64_oracle11g_rpm]# rpm -ivh --force --nodeps *.rpm</span><br></pre></td></tr></table></figure></p><p>9 无需操作，看内容即可<br><img src="http://img.blog.csdn.net/20160423183937629" alt="这里写图片描述"><br>10 安装开始 请等待<br><img src="http://img.blog.csdn.net/20160423184102419" alt="这里写图片描述"></p><p>11 Password Management<br><img src="http://img.blog.csdn.net/20160423184115294" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184130147" alt="这里写图片描述"><br>其中sys sysdba密码均为oracle</p><p><img src="http://img.blog.csdn.net/20160423184247107" alt="这里写图片描述"><br>12 执行提示脚本<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs database]$ su - root</span><br><span class="line">[root@vmdbs ~]<span class="comment"># sh /opt/oracle/oraInventory/orainstRoot.sh </span></span><br><span class="line">[root@vmdbs ~]<span class="comment"># sh /opt/oracle/oracle/product/11.2.0/dbhome_1/root.sh</span></span><br></pre></td></tr></table></figure></p><p>13 完成关闭</p><p>恭喜 大功告成</p><p>安装完成后Oracle Enterprise Manager（<a href="https://ip:1158/em）就可以打开，数据库已可以使用。重启服务器后，需手动启动" target="_blank" rel="noopener">https://ip:1158/em）就可以打开，数据库已可以使用。重启服务器后，需手动启动</a> Oracle Enterprise Manager 服务：emctl start dbconsole， <a href="https://ip:1158/em" target="_blank" rel="noopener">https://ip:1158/em</a> 才可以打开。 </p><h1 id="4-启动和关闭-oracle-数据库步骤"><a href="#4-启动和关闭-oracle-数据库步骤" class="headerlink" title="4 启动和关闭 oracle 数据库步骤"></a>4 启动和关闭 oracle 数据库步骤</h1><ol><li>启动</li></ol><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line">[root@vmdbs ~]<span class="comment"># su - oracle</span></span><br><span class="line">[oracle@vmdbs ~]$ lsnrctl start</span><br><span class="line"></span><br><span class="line">LSNRCTL <span class="keyword">for</span> Linux: Version 11.2.0.1.0 - Production on 22-APR-2016 15:03:12</span><br><span class="line"></span><br><span class="line">Copyright (c) 1991, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">Starting /opt/oracle/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please <span class="built_in">wait</span>...</span><br><span class="line"></span><br><span class="line">TNSLSNR <span class="keyword">for</span> Linux: Version 11.2.0.1.0 - Production</span><br><span class="line">System parameter file is /opt/oracle/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora</span><br><span class="line">Log messages written to /opt/oracle/oracle/diag/tnslsnr/vmdbs/listener/alert/log.xml</span><br><span class="line">Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))</span><br><span class="line">Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))</span><br><span class="line"></span><br><span class="line">Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))</span><br><span class="line">STATUS of the LISTENER</span><br><span class="line">------------------------</span><br><span class="line">Alias LISTENER</span><br><span class="line">Version TNSLSNR <span class="keyword">for</span> Linux: Version 11.2.0.1.0 - Production</span><br><span class="line">Start Date 22-APR-2016 15:03:14</span><br><span class="line">Uptime 0 days 0 hr. 0 min. 0 sec</span><br><span class="line">Trace Level off</span><br><span class="line">Security ON: Local OS Authentication</span><br><span class="line">SNMP OFF</span><br><span class="line">Listener Parameter File /opt/oracle/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora</span><br><span class="line">Listener Log File /opt/oracle/oracle/diag/tnslsnr/vmdbs/listener/alert/log.xml</span><br><span class="line">Listening Endpoints Summary...</span><br><span class="line"> (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))</span><br><span class="line"> (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))</span><br><span class="line">The listener supports no services</span><br><span class="line">The <span class="built_in">command</span> completed successfully</span><br><span class="line"></span><br><span class="line"></span><br><span class="line">[oracle@vmdbs ~]$ sqlplus /nolog</span><br><span class="line"></span><br><span class="line">SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 22 15:05:59 2016</span><br><span class="line"></span><br><span class="line">Copyright (c) 1982, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">SQL> connect /as sysdba</span><br><span class="line">Connected to an idle instance.</span><br><span class="line">SQL> startup</span><br><span class="line">ORACLE instance started.</span><br><span class="line"></span><br><span class="line">Total System Global Area 830930944 bytes</span><br><span class="line">Fixed Size 2217912 bytes</span><br><span class="line">Variable Size 499124296 bytes</span><br><span class="line">Database Buffers 327155712 bytes</span><br><span class="line">Redo Buffers 2433024 bytes</span><br><span class="line">Database mounted.</span><br><span class="line">Database opened.</span><br><span class="line"></span><br><span class="line">SQL> quit</span><br></pre></td></tr></table></figure><p>如果大家的安装出现了问题，可以QQ联系我（QQ568946518），我有时间尽量帮大家解决。</p><ol start="2"><li>关闭</li></ol><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs ~]$ sqlplus /nolog</span><br><span class="line"></span><br><span class="line">SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 22 15:14:32 2016</span><br><span class="line"></span><br><span class="line">Copyright (c) 1982, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">SQL> conn /as sysdba</span><br><span class="line">Connected.</span><br><span class="line">SQL> shutdown immediate</span><br><span class="line">Database closed.</span><br><span class="line">Database dismounted.</span><br><span class="line">ORACLE instance shut down.</span><br><span class="line">SQL> quit</span><br></pre></td></tr></table></figure><h1 id="5-没有数据库，新建一个数据库"><a href="#5-没有数据库，新建一个数据库" class="headerlink" title="5 没有数据库，新建一个数据库"></a>5 没有数据库，新建一个数据库</h1><h2 id="5-1-建立监听"><a href="#5-1-建立监听" class="headerlink" title="5.1 建立监听"></a>5.1 建立监听</h2><pre><code>[oracle@vmdbs ~]$ export LANG=en_us安装中文版操作系统才需执行[oracle@vmdbs ~]$ netca //必须在图形界面下的命令行下执行，不能远程执行</code></pre><h2 id="5-2-启动监听"><a href="#5-2-启动监听" class="headerlink" title="5.2 启动监听"></a>5.2 启动监听</h2><pre><code>[oracle@vmdbs ~]$ lsnrctl start</code></pre><h2 id="5-3-建立数据库"><a href="#5-3-建立数据库" class="headerlink" title="5.3 建立数据库"></a>5.3 建立数据库</h2><pre><code>[oracle@vmdbs ~]$ export LANG=en_us //图形界面下执行</code></pre><p>安装中文版操作系统才需执行</p><pre><code>[oracle@vmdbs ~]$ dbca</code></pre><p>网卡 + csdn博客图片只能一张一张上传，还卡。。。。所以只能文字表述了。。。<br>要看图片的请下载<br><a href="http://download.csdn.net/download/erujo/9500427" target="_blank" rel="noopener">DayDayUP_Linux运维学习_oracle11g安装教程</a> </p><p> <a href="http://download.csdn.net/download/erujo/9500427" target="_blank" rel="noopener">http://download.csdn.net/download/erujo/9500427</a><br>选项 1：一般用途或事务处理；选项 2：定制数据库；选项 3：数据仓库<br><img src="http://img.blog.csdn.net/20160423184419932" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184430373" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184501295" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184524483" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184541311" alt="这里写图片描述"></p><p><img src="http://img.blog.csdn.net/20160423184600810" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184617186" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184631655" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184649046" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184708202" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184722140" alt="这里写图片描述"><br><img src="http://img.blog.csdn.net/20160423184733484" alt="这里写图片描述"></p><p>SYS SYSTEM 密码均为oracle</p><h1 id="6-已有一个数据库，再新建一个数据库"><a href="#6-已有一个数据库，再新建一个数据库" class="headerlink" title="6 已有一个数据库，再新建一个数据库"></a>6 已有一个数据库，再新建一个数据库</h1><p>在原有数据库基础上建立了第二个数据库，重启服务器后，在启动默认数据库的基础上（oracle 用户的.bash_profile 文件中定义的 ORACLE_SID），重新 export ORACLE_SID=第二数据库的 sid，重复 sqlplus /nolog、connect /as sysdba、startup，方可使用新建的数据库。<br>数据库关闭操作亦如此。 </p><h2 id="6-1-首先启动监听"><a href="#6-1-首先启动监听" class="headerlink" title="6.1 首先启动监听"></a>6.1 首先启动监听</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs ~]$ lsnrctl start</span><br><span class="line"></span><br><span class="line">LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 22-APR-2016 15:34:18</span><br><span class="line"></span><br><span class="line">Copyright (c) 1991, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">Starting /opt/oracle/oracle/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...</span><br><span class="line"></span><br><span class="line">TNSLSNR for Linux: Version 11.2.0.1.0 - Production</span><br><span class="line">System parameter file is /opt/oracle/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora</span><br><span class="line">Log messages written to /opt/oracle/oracle/diag/tnslsnr/vmdbs/listener/alert/log.xml</span><br><span class="line">Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))</span><br><span class="line">Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))</span><br><span class="line"></span><br><span class="line">Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))</span><br><span class="line">STATUS of the LISTENER</span><br><span class="line">------------------------</span><br><span class="line">Alias LISTENER</span><br><span class="line">Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production</span><br><span class="line">Start Date 22-APR-2016 15:34:19</span><br><span class="line">Uptime 0 days 0 hr. 0 min. 0 sec</span><br><span class="line">Trace Level off</span><br><span class="line">Security ON: Local OS Authentication</span><br><span class="line">SNMP OFF</span><br><span class="line">Listener Parameter File /opt/oracle/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora</span><br><span class="line">Listener Log File /opt/oracle/oracle/diag/tnslsnr/vmdbs/listener/alert/log.xml</span><br><span class="line">Listening Endpoints Summary...</span><br><span class="line"> (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))</span><br><span class="line"> (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521)))</span><br><span class="line">The listener supports no services</span><br><span class="line">The command completed successfully</span><br></pre></td></tr></table></figure><h2 id="6-2-启动第一个数据库"><a href="#6-2-启动第一个数据库" class="headerlink" title="6.2 启动第一个数据库"></a>6.2 启动第一个数据库</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs ~]$ sqlplus /nolog</span><br><span class="line"></span><br><span class="line">SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 22 15:34:30 2016</span><br><span class="line"></span><br><span class="line">Copyright (c) 1982, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">SQL> connect /as sysdba</span><br><span class="line">Connected to an idle instance.</span><br><span class="line">SQL> startup</span><br><span class="line">ORACLE instance started.</span><br><span class="line"></span><br><span class="line">Total System Global Area 830930944 bytes</span><br><span class="line">Fixed Size 2217912 bytes</span><br><span class="line">Variable Size 499124296 bytes</span><br><span class="line">Database Buffers 327155712 bytes</span><br><span class="line">Redo Buffers 2433024 bytes</span><br><span class="line">Database mounted.</span><br><span class="line">Database opened.</span><br><span class="line">SQL> quit</span><br></pre></td></tr></table></figure><p>6.3 启动第二个数据库<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[oracle@vmdbs ~]$ <span class="built_in">export</span> ORACLE_SID=newdb</span><br><span class="line">[oracle@vmdbs ~]$ <span class="built_in">export</span> ORACLE_SID=newdb</span><br><span class="line">[oracle@vmdbs ~]$ sqlplus /nolog</span><br><span class="line"></span><br><span class="line">SQL*Plus: Release 11.2.0.1.0 Production on Fri Apr 22 15:39:16 2016</span><br><span class="line"></span><br><span class="line">Copyright (c) 1982, 2009, Oracle. All rights reserved.</span><br><span class="line"></span><br><span class="line">SQL> conn /as sysdba</span><br><span class="line">Connected to an idle instance.</span><br><span class="line">SQL> startup</span><br><span class="line">ORACLE instance started.</span><br><span class="line"></span><br><span class="line">Total System Global Area 826753024 bytes</span><br><span class="line">Fixed Size 2217872 bytes</span><br><span class="line">Variable Size 230688880 bytes</span><br><span class="line">Database Buffers 591396864 bytes</span><br><span class="line">Redo Buffers 2449408 bytes</span><br><span class="line">Database mounted.</span><br><span class="line">Database opened.</span><br><span class="line">SQL> quit</span><br></pre></td></tr></table></figure></p>]]></content>
<summary type="html">
<h1 id="1-安装环境介绍"><a href="#1-安装环境介绍" class="headerlink" title="1. 安装环境介绍"></a>1. 安装环境介绍</h1><p>系统环境 虚拟机测试机<br>系统版本 linux redhat 6.5 x64<br>软件版本 linux.x64_oracle_11gR2<br>系统内存 2G<br>系统存储 40G<br>主机名 vmdbs<br>ip地址 192.168.1.189 192.168.128.189</p>
<p>笔者当时安装操作系统时所选的安装包<br>
</summary>
<category term="数据库" scheme="http://yoursite.com/categories/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="数据库" scheme="http://yoursite.com/tags/%E6%95%B0%E6%8D%AE%E5%BA%93/"/>
<category term="oracle" scheme="http://yoursite.com/tags/oracle/"/>
</entry>
<entry>
<title>DayDayUP_大数据学习课程[1]_hadoop2.6.0完全分布式集群环境和伪分布式集群搭建</title>
<link href="http://yoursite.com/2018/07/06/DayDayUP_%E5%A4%A7%E6%95%B0%E6%8D%AE%E5%AD%A6%E4%B9%A0%E8%AF%BE%E7%A8%8B%5B1%5D_hadoop2.6.0%E5%AE%8C%E5%85%A8%E5%88%86%E5%B8%83%E5%BC%8F%E9%9B%86%E7%BE%A4%E7%8E%AF%E5%A2%83%E5%92%8C%E4%BC%AA%E5%88%86%E5%B8%83%E5%BC%8F%E9%9B%86%E7%BE%A4%E6%90%AD%E5%BB%BA/"/>
<id>http://yoursite.com/2018/07/06/DayDayUP_大数据学习课程[1]_hadoop2.6.0完全分布式集群环境和伪分布式集群搭建/</id>
<published>2018-07-06T08:41:08.730Z</published>
<updated>2018-07-06T08:38:44.000Z</updated>
<content type="html"><![CDATA[<ol><li>环境说明</li></ol><hr><p>系统 ：Centos6.5<br>软件版本： hadoop2.6.0 jdk1.8<br>集群状态：<br>master: www 192.168.78.110<br>slave1: node1 192.168.78.111<br>slave2: node2 192.168.78.112<br>hosts 文件<br>192.168.78.110 www<br>192.168.78.111 node1<br>192.168.78.112 node2<br>确保三台机器之间互ping 主机名能ping通<br><a id="more"></a></p><ol start="2"><li>下载 hadoop2.6.0 和jdk1.8</li></ol><hr><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[root@www ~]# wget http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm?AuthParam=1446899640_8da8d9b13f8bbe63b3bc0bc80b730f55 //下载后将.rpm后面的乱码去掉</span><br><span class="line">[root@www ~]# wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.2/hadoop-2.6.2.tar.gz</span><br></pre></td></tr></table></figure><ol start="3"><li>配置java环境</li></ol><hr><h1 id="3-1-安装jdk"><a href="#3-1-安装jdk" class="headerlink" title="3.1 安装jdk"></a>3.1 安装jdk</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"># rpm -ivh jdk-8u45-linux-i586.rpm</span><br></pre></td></tr></table></figure><h1 id="3-2-配置java环境变量"><a href="#3-2-配置java环境变量" class="headerlink" title="3.2 配置java环境变量"></a>3.2 配置java环境变量</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[root@www ~]# vimx /etc/profile</span><br><span class="line">#set java environment</span><br><span class="line">export JAVA_HOME=/usr/java/jdk1.8.0_45 //注意若下载了其他版本，注意变通</span><br><span class="line">export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar</span><br><span class="line">export PATH=$PATH:$JAVA_HOME/bin</span><br><span class="line">export JAVA_HOME CLASSPATH PATH</span><br><span class="line">[root@www ~]# source !$</span><br></pre></td></tr></table></figure><h1 id="3-3-测试java环境"><a href="#3-3-测试java环境" class="headerlink" title="3.3 测试java环境"></a>3.3 测试java环境</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@www ~]# java -version</span><br></pre></td></tr></table></figure><p><code>java version "1.8.0_65"Java(TM) SE Runtime Environment (build 1.8.0_65-b17)Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)</code><br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[root@www ~]# javac -version</span><br></pre></td></tr></table></figure></p><p><code>javac 1.8.0_65</code></p><ol start="4"><li>安装hadoop</li></ol><hr><h1 id="4-1-解压安装"><a href="#4-1-解压安装" class="headerlink" title="4.1 解压安装"></a>4.1 解压安装</h1><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[root@www opt]# tar -xzvf hadoop-2.6.2.tar.gz </span><br><span class="line">[root@www opt]# mkdir /opt/hadoop</span><br><span class="line">[root@www src]# mv hadoop-2.6.2 /opt/hadoop</span><br><span class="line">[root@www src]# cd /opt/hadoop/hadoop-2.6.2</span><br><span class="line">[root@www hadoop-2.6.2]# ls</span><br><span class="line">bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share</span><br></pre></td></tr></table></figure><p>4.2 添加hadoop用户<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[root@www hadoop-2.6.2]# useradd hadoop</span><br><span class="line">[root@www hadoop-2.6.2]# passwd hadoop</span><br><span class="line">[root@www hadoop-2.6.2]# chown -R hadoop:hadoop /opt/hadoop</span><br></pre></td></tr></table></figure></p><p>4.3 修改hadoop配置文件<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[root@www hadoop-2.6.2]# su - hadoop //切换为hadoop用户</span><br><span class="line">[hadoop@www ~]$ mkdir -p ~/hadoop/tmp ~/dfs/data ~/dfs/name //这些目录后期要用</span><br><span class="line">[hadoop@www ~]$ ls</span><br><span class="line">dfs hadoop</span><br><span class="line">[hadoop@www ~]$ cd /opt/hadoop/hadoop-2.6.2/</span><br></pre></td></tr></table></figure></p><p>4.3.1 配置 hadoop-env.sh文件–>修改JAVA_HOME<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/hadoop-env.sh </span><br><span class="line"># The java implementation to use.</span><br><span class="line">export JAVA_HOME=/usr/java/jdk1.8.0_65</span><br></pre></td></tr></table></figure></p><p>4.3.2 配置 yarn-env.sh 文件–>>修改JAVA_HOME<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/yarn-env.sh </span><br><span class="line"># The java implementation to use.</span><br><span class="line">export JAVA_HOME=/usr/java/jdk1.8.0_65</span><br></pre></td></tr></table></figure></p><p>4.3.3 配置slaves文件–>>增加slave节点<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/slaves </span><br><span class="line">node1</span><br><span class="line">node2</span><br></pre></td></tr></table></figure></p><p>4.3.4 配置 core-site.xml文件–>>增加hadoop核心配置（hdfs文件端口是9000、file:/home/hadoop/opt/hadoop-2.6.0/tmp、）<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/core-site.xml </span><br><span class="line"><?xml version="1.0" encoding="UTF-8"?></span><br><span class="line"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></span><br><span class="line"><!--</span><br><span class="line"> Licensed under the Apache License, Version 2.0 (the "License");</span><br><span class="line"> you may not use this file except in compliance with the License.</span><br><span class="line"> You may obtain a copy of the License at</span><br><span class="line"></span><br><span class="line"> http://www.apache.org/licenses/LICENSE-2.0</span><br><span class="line"></span><br><span class="line"> Unless required by applicable law or agreed to in writing, software</span><br><span class="line"> distributed under the License is distributed on an "AS IS" BASIS,</span><br><span class="line"> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span><br><span class="line"> See the License for the specific language governing permissions and</span><br><span class="line"> limitations under the License. See accompanying LICENSE file.</span><br><span class="line">--></span><br><span class="line"></span><br><span class="line"><!-- Put site-specific property overrides in this file. --></span><br><span class="line"></span><br><span class="line"><configuration></span><br><span class="line"> <property></span><br><span class="line"> <name>fs.defaultFS</name></span><br><span class="line"> <value>hdfs://www:9000</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>io.file.buffer.size</name></span><br><span class="line"> <value>131072</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>hadoop.tmp.dir</name></span><br><span class="line"> <value>file: /home/hadoop/hadoop/tmp</value></span><br><span class="line"> <description>Abasefor other temporary directories.</description></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>hadoop.proxyuser.spark.hosts</name></span><br><span class="line"> <value>*</value></span><br><span class="line"> </property></span><br><span class="line"><property></span><br><span class="line"> <name>hadoop.proxyuser.spark.groups</name></span><br><span class="line"> <value>*</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure></p><p>4.3.5 配置 hdfs-site.xml 文件–>>增加hdfs配置信息（namenode、datanode端口和目录位置）<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/hdfs-site.xml</span><br><span class="line"><configuration></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>dfs.namenode.secondary.http-address</name></span><br><span class="line"> <value>www:9001</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.datanode.data.dir</name></span><br><span class="line"> <value>file:/home/hadoop/dfs/data</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.namenode.name.dir</name></span><br><span class="line"> <value>file:/home/hadoop/dfs/name</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.replication</name></span><br><span class="line"> <value>3</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.webhdfs.enabled</name></span><br><span class="line"> <value>true</value></span><br><span class="line"> </property></span><br><span class="line"><property></span><br><span class="line"> <name>dfs.namenode.checkpoint.dir</name></span><br><span class="line"> <value>file:///home/hadoop/hadoop/hdfs/namesecondary</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure></p><p>4.3.6 配置 mapred-site.xml 文件–>>增加mapreduce配置（使用yarn框架、jobhistory使用地址以及web地址）<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml</span><br><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/mapred-site.xml</span><br><span class="line"><configuration></span><br><span class="line"><property></span><br><span class="line"> <name>mapreduce.framework.name</name></span><br><span class="line"> <value>yarn</value></span><br><span class="line"></property></span><br><span class="line"><property></span><br><span class="line"> <name>mapreduce.jobhistory.address</name></span><br><span class="line"> <value>www:10020</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>mapreduce.jobhistory.webapp.address</name></span><br><span class="line"> <value>www:19888</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>mapreduce.jobtracker.staging.root.dir</name></span><br><span class="line"> <value>/home/hadoop/hadoop</value></span><br><span class="line"> </property></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure></p><p>4.3.7 配置 yarn-site.xml 文件–>>增加yarn功能<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/yarn-site.xml</span><br><span class="line"><configuration></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.nodemanager.aux-services</name></span><br><span class="line"> <value>mapreduce_shuffle</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name></span><br><span class="line"> <value>org.apache.hadoop.mapred.ShuffleHandler</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.resourcemanager.address</name></span><br><span class="line"> <value>www:8032</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.resourcemanager.scheduler.address</name></span><br><span class="line"> <value>www:8030</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.resourcemanager.resource-tracker.address</name></span><br><span class="line"> <value>www:8035</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.resourcemanager.admin.address</name></span><br><span class="line"> <value>www:8033</value></span><br><span class="line"> </property></span><br><span class="line"> <property></span><br><span class="line"> <name>yarn.resourcemanager.webapp.address</name></span><br><span class="line"> <value>www:8088</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure></p><p>4.3.8 将所有文件（hadoop2.6.0和hosts）复制到node1 和node2 上<br>4.4.1 设置ssh免密码登陆<br>在三台服务器上分别执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www ~]$ ssh-keygen -t rsa //直接回车不用设置密码</span><br><span class="line">[hadoop@node2 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]</span><br><span class="line">[hadoop@node2 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]</span><br><span class="line">[hadoop@node2 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]</span><br></pre></td></tr></table></figure></p><p>4.4.2 测试ssh免密码登录<br>在三台服务器上分别执行<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@node2 ~]$ ssh www</span><br><span class="line">[hadoop@node2 ~]$ ssh node1</span><br><span class="line">[hadoop@node2 ~]$ ssh node2</span><br></pre></td></tr></table></figure></p><ol start="5"><li>验证hadoop<br>5.1 格式化namenode<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ ./bin/hadoop namenode -format</span><br><span class="line">[hadoop@node1 hadoop-2.6.2]$ ./bin/hadoop namenode -format</span><br><span class="line">[hadoop@node2 hadoop-2.6.2]$ ./bin/hadoop namenode -format</span><br></pre></td></tr></table></figure></li></ol><p>5.2 启动hadoop<br>启动所有<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ ./sbin/start-all.sh //任意一台执行即可</span><br></pre></td></tr></table></figure></p><p>正确的进程情况<br>master：<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ jps</span><br><span class="line">7136 ResourceManager</span><br><span class="line">6993 SecondaryNameNode</span><br><span class="line">6819 NameNode</span><br><span class="line">7399 Jps</span><br></pre></td></tr></table></figure></p><p>slave:<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@node1 hadoop-2.6.2]$ jps</span><br><span class="line">3186 Jps</span><br><span class="line">3064 NodeManager</span><br><span class="line">2974 DataNode</span><br></pre></td></tr></table></figure></p><p>6 运行wordcount程序<br>6.1 创建目录和文件<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@node1 hadoop-2.6.2]$ mkdir input</span><br><span class="line">[hadoop@node1 hadoop-2.6.2]$ touch input/test.log</span><br><span class="line">[hadoop@node1 hadoop-2.6.2]$ <span class="built_in">echo</span> <span class="string">"hello world hello hadoop"</span> > input/test.log </span><br><span class="line">[hadoop@node1 hadoop-2.6.2]$ cat input/test.log </span><br><span class="line">hello world hello hadoop</span><br></pre></td></tr></table></figure></p><p>6.2 在hdfs创建/input目录</p><pre><code>[hadoop@node1 hadoop-2.6.2]$ ./bin/hadoop fs -mkdir /input</code></pre><p>6.3 将test.log文件copy到hdfs /input目录</p><pre><code>[hadoop@www hadoop-2.6.2]$ ./bin/hadoop fs -put input/ /</code></pre><p>6.4 查看hdfs上是否有test.log文件</p><pre><code>[hadoop@www hadoop-2.6.2]$ ./bin/hadoop fs -ls /input</code></pre><p><code>15/11/08 17:59:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 1 items-rw-r--r-- 2 hadoop supergroup 25 2015-11-08 17:59 /input/test.log</code><br>6.5 执行wordcount程序</p><pre><code>[hadoop@www hadoop-2.6.2]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar wordcount /input /output</code></pre><p>6.6 查看结果</p><pre><code>[hadoop@www hadoop-2.6.2]$ ./bin/hadoop fs -cat /output/part-r-00000</code></pre><p><code>15/11/08 18:07:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hadoop 1 hello 2 world 1</code></p><ol start="7"><li>伪分布式集群环境的搭建<br>只需修改namenode的两个文件<br>7.1etc/hadoop/hdfs-site.xml<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/hdfs-site.xml </span><br><span class="line"><configuration></span><br><span class="line"></span><br><span class="line"><property></span><br><span class="line"><name>dfs.namenode.secondary.http-address</name></span><br><span class="line"> <value>www:9001</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.datanode.data.dir</name></span><br><span class="line"> <value>file:///home/hadoop/dfs/data</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.namenode.name.dir</name></span><br><span class="line"> <value>file:///home/hadoop/dfs/name</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.replication</name></span><br><span class="line"> <value>1</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"> <property></span><br><span class="line"> <name>dfs.webhdfs.enabled</name></span><br><span class="line"> <value>true</value></span><br><span class="line"> </property></span><br><span class="line"><property></span><br><span class="line"> <name>dfs.namenode.checkpoint.dir</name></span><br><span class="line"> <value>file:///home/hadoop/hadoop/hdfs/namesecondary</value></span><br><span class="line"> </property></span><br><span class="line"></span><br><span class="line"></configuration></span><br></pre></td></tr></table></figure></li></ol><p>7.2 etc/hadoop/slaves<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ vimx etc/hadoop/slaves </span><br><span class="line">``` </span><br><span class="line">7.3 格式化namenode</span><br></pre></td></tr></table></figure></p><p>[hadoop@www hadoop-2.6.2]$ ./bin/hadoop namenode -format<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">7.4 启动</span><br></pre></td></tr></table></figure></p><p>[hadoop@www hadoop-2.6.2]$ ./sbin/start-all.sh<br><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">7.5 查看进程</span><br><span class="line"></span><br><span class="line">``` </span><br><span class="line">[hadoop@www hadoop-2.6.2]$ jps</span><br><span class="line">4048 NameNode</span><br><span class="line">4545 NodeManager</span><br><span class="line">4130 DataNode</span><br><span class="line">4459 ResourceManager</span><br><span class="line">5469 Jps</span><br><span class="line">4286 SecondaryNameNode</span><br></pre></td></tr></table></figure></p><p>7.6 上传文件</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ ./bin/hadoop fs -put input/ /</span><br></pre></td></tr></table></figure><p>7.7 运行wordcount</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar wordcount /input /output</span><br></pre></td></tr></table></figure><p>7.8 查看执行结果</p><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[hadoop@www hadoop-2.6.2]$ ./bin/hadoop fs -cat /output/part-r-00000</span><br></pre></td></tr></table></figure>]]></content>
<summary type="html">
<ol>
<li>环境说明</li>
</ol>
<hr>
<p>系统 ：Centos6.5<br>软件版本： hadoop2.6.0 jdk1.8<br>集群状态：<br>master: www 192.168.78.110<br>slave1: node1 192.168.78.111<br>slave2: node2 192.168.78.112<br>hosts 文件<br>192.168.78.110 www<br>192.168.78.111 node1<br>192.168.78.112 node2<br>确保三台机器之间互ping 主机名能ping通<br>
</summary>
<category term="大数据" scheme="http://yoursite.com/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="hadoop" scheme="http://yoursite.com/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/hadoop/"/>
<category term="大数据" scheme="http://yoursite.com/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/"/>
<category term="hadoop" scheme="http://yoursite.com/tags/hadoop/"/>
</entry>
</feed>