前言
Cloudera与Hortonworks合并后，再去用CDH、CDP与HDP的Cloudera Manager与Ambari总是没安逸了。可白piao的CDH最高版本6.3.2，组件相对来说也比较古老，生产环境的CDH也陆续要脱保了。甲方爸爸不差钱可以交保护费，我等穷人嘛，白piao一时爽，一直白piao一直爽，能白piao的机会当然是不能放过。笔者没那么勤奋，懒得部署Apache组件了【需要考虑的兼容性、编译等问题很多】，图省心省力部署了一套USDP2的双集群。
硬件配置双路e5 2696v3 256G内存主机
主板：x99f8d CPU：e5 2696v3 *2 【36核72线程】 内存条：DDR4 ECC 32G *8 【256G】 显卡：GT730 【2G显存】 散热器：ta 120ex *2 【单风扇】 SSD：MX500 2T *2 【4T】 HDD：NAS拆的酷狼 4T 电源：GX1000 【1000W】 机箱：614PC 【标配2风扇+套装3风扇】 显示器：13.3寸便携 【2K】 键鼠：笔记本淘汰的一套 【USB口】 网卡：USB无线网卡花销1w左右，省着点8k也够。
虚拟机准备制作7台虚拟机
安装包下载USDP官网，含下载路径及操作指南
安装部署 准备路径 [root@zhiyong1 ~]# cd .. [root@zhiyong1 /]# mkdir /data [root@zhiyong1 /]# mkdir -p /opt/usdp-srv/ 上传文件 将httpd-rpms.tar.gz 、mirror.tgz、epel.tgz放置到/data 将usdp-01-master-privatization-free-2.X.tar.gz放置到/opt/usdp-srv/ [root@zhiyong1 usdp-srv]# ll 总用量 7409020 -rw-r--r--. 1 root root 7586834393 3月 1 21:38 usdp-01-master-privatization-free-2.X.tar.gz [root@zhiyong1 usdp-srv]# tar -zxvf usdp-01-master-privatization-free-2.X.tar.gz其中的一些安装包：
usdp/repository/2.0.0.0/influxdb-1.8.0.tar.gz usdp/repository/2.0.0.0/usdpmonitor-1.0.0.tar.gz usdp/repository/2.0.0.0/alertmanager-0.21.0.tar.gz usdp/repository/2.0.0.0/prometheus-2.18.1.tar.gz usdp/repository/2.0.0.0/grafana-6.5.1.tar.gz usdp/repository/2.0.0.0/node_exporter-1.0.0.tar.gz usdp/repository/2.0.0.0/phoenix-5.0.0.tar.gz usdp/repository/2.0.0.0/tez-0.10.0.tar.gz usdp/repository/2.0.0.0/spark-3.0.1.tar.gz usdp/repository/2.0.0.0/zkui-2.0.0.tar.gz usdp/repository/2.0.0.0/elasticsearch-7.8.0.tar.gz usdp/repository/2.0.0.0/zookeeper-3.4.13.tar.gz usdp/repository/2.0.0.0/zeppelin-0.9.0.tar.gz usdp/repository/2.0.0.0/tez-0.10.0-dependency.tar.gz usdp/repository/2.0.0.0/kibana-7.8.0.tar.gz usdp/repository/2.0.0.0/sqoop-1.4.7.tar.gz usdp/repository/2.0.0.0/hadoop-3.1.1.tar.gz usdp/repository/2.0.0.0/kafka-2.11-2.0.1.tar.gz usdp/repository/2.0.0.0/flume-1.9.0.tar.gz usdp/repository/2.0.0.0/kafkaeagle-2.0.6.tar.gz usdp/repository/2.0.0.0/neo4j-3.5.28.tar.gz usdp/repository/2.0.0.0/hbase-2.1.10.tar.gz usdp/repository/2.0.0.0/hue-4.8.0.tar.gz usdp/repository/2.0.0.0/flink-1.13.2.tar.gz usdp/repository/2.0.0.0/ranger-2.1.0.tar.gz usdp/repository/2.0.0.0/hive-3.1.2.tar.gz usdp/repository/2.0.0.0/dolphinscheduler-2.0.0.tar.gz可以大致看出版本不算太老。还算全面，当然用得上用不上另说，我可以不用，但不希望用的时候没有。。。白piao版少几个组件，Kylin、Atlas、Livy、Alluxio之类的，只好自己慢慢手动安装。Flink也老了点，体验不到1.14流批一体的欢乐。Kerberos认证暂时也没有自动化部署的方式。
准备安装 [root@zhiyong1 usdp-srv]# cd usdp/ [root@zhiyong1 usdp]# ll 总用量 4 drwxr-xr-x. 2 root root 33 12月 29 15:36 agent drwxr-xr-x. 2 root root 136 12月 31 14:39 bin drwxr-xr-x. 2 root root 65 12月 29 14:48 config drwxr-xr-x. 2 root root 137 12月 29 14:46 jmx_exporter drwxr-xr-x. 3 root root 21 12月 29 14:48 license drwxr-xr-x. 2 root root 6 12月 29 14:48 logs drwxr-xr-x. 2 root root 35 12月 29 14:46 recommend drwxr-xr-x. 7 root root 73 12月 29 14:48 repair drwxr-xr-x. 3 root root 21 12月 29 14:46 repository drwxr-xr-x. 2 root root 4096 12月 29 15:34 scripts drwxr-xr-x. 2 root root 34 12月 29 15:35 server drwxr-xr-x. 2 root root 29 12月 29 14:48 sql drwxr-xr-x. 4 root root 32 12月 29 15:30 templated drwxr-xr-x. 2 root root 6 12月 29 14:48 verify drwxr-xr-x. 2 root root 79 12月 31 14:49 versions [root@zhiyong1 usdp]# cd ./repair/ [root@zhiyong1 repair]# ll 总用量 8 drwxr-xr-x. 2 root root 4096 12月 29 15:51 bin drwxr-xr-x. 2 root root 105 12月 29 15:43 config drwxr-xr-x. 2 root root 4096 12月 30 10:47 packages drwxr-xr-x. 2 root root 189 12月 29 14:48 remove drwxr-xr-x. 2 root root 23 12月 30 10:45 sbin [root@zhiyong1 repair]# cd ./config/ [root@zhiyong1 config]# ll 总用量 12 -rwxr-xr-x. 1 root root 723 12月 29 15:44 repair-host-info-add.properties -rwxr-xr-x. 1 root root 507 12月 29 15:44 repair-host-info.properties -rwxr-xr-x. 1 root root 666 12月 29 15:44 repair.properties [root@zhiyong1 config]# vim repair.properties修改为：
# Set the YUM source host IP yum.repo.host.ip=192.168.88.100 #The Host information for installing the NMAP service namp.server.ip=192.168.88.100 namp.server.port=22 namp.server.password=123456 # The Host information for installing the NTP service(Master) ntp.master.ip=192.168.88.100 # Install MySQL machine node information mysql.ip=192.168.88.100 mysql.host.ssh.port=22 mysql.host.ssh.password=123456 # Set the MYSQL database login password mysql.password=123456 # The total number of machines needed to be repaired. repair.host.num=7 # The total number of added machines needed to be repaired. repair.add.host.num=0 # Common Settings. repair.log.dir=./logs修改配置;
[root@zhiyong1 config]# vim repair-host-info.properties修改为：
# 1.Please provide the information of hosts needed to be repaired in the format specified below # 2.usdp.ip.i(eg:i=1,2,3.....): # 3.usdp.password.i: # 4.usdp.ssh.port.i: # 5.usdp.ssh.port.hostname.i: usdp.ip.1=192.168.88.100 usdp.password.1=123456 usdp.ssh.port.1=22 usdp.ssh.port.hostname.1=zhiyong1 usdp.ip.2=192.168.88.101 usdp.password.2=123456 usdp.ssh.port.2=22 usdp.ssh.port.hostname.2=zhiyong2 usdp.ip.3=192.168.88.102 usdp.password.3=123456 usdp.ssh.port.3=22 usdp.ssh.port.hostname.3=zhiyong3 usdp.ip.4=192.168.88.103 usdp.password.4=123456 usdp.ssh.port.4=22 usdp.ssh.port.hostname.4=zhiyong4 usdp.ip.5=192.168.88.104 usdp.password.5=123456 usdp.ssh.port.5=22 usdp.ssh.port.hostname.5=zhiyong5 usdp.ip.6=192.168.88.105 usdp.password.6=123456 usdp.ssh.port.6=22 usdp.ssh.port.hostname.6=zhiyong6 usdp.ip.7=192.168.88.106 usdp.password.7=123456 usdp.ssh.port.7=22 usdp.ssh.port.hostname.7=zhiyong7 执行修复脚本 [root@zhiyong1 config]# cd /opt/usdp-srv/usdp/repair/sbin/ [root@zhiyong1 sbin]# ll 总用量 116 -rwxr-xr-x. 1 root root 117392 12月 29 15:44 repair.sh [root@zhiyong1 sbin]# bash repair.sh initAll等到出现：
SUCCESS: All encryption-free login links have been repaired successfully SUCCESS: All closing firewall links have been repaired successfully SUCCESS: All closing swap links have been repaired successfully SUCCESS: set hostname links have been repaired successfully SUCCESS: Set ntp have been repaired successfully SUCCESS: Set libxslt devel have been repaired successfully SUCCESS: Set psmisc have been repaired successfully SUCCESS: Set mysql-client links have been repaired successfully SUCCESS: Set mysql-python have been repaired successfully SUCCESS: All transparent_hugepage links have been repaired successfully SUCCESS: Set JDK links have been repaired successfully SUCCESS: Set xdg-utils links have been repaired successfully SUCCESS: Set redhat-lsb links have been repaired successfully SUCCESS: Set python-devel links have been repaired successfully SUCCESS: Set cyrus-sasl links have been repaired successfully SUCCESS: Set python36-devel links have been repaired successfully SUCCESS: Set gcc-c++ links have been repaired successfully SUCCESS: Set Cython links have been repaired successfully SUCCESS: Set Six links have been repaired successfully SUCCESS: Set websocket-client links have been repaired successfully SUCCESS: Set ecdsa links have been repaired successfully SUCCESS: Set pytest-runner links have been repaired successfully SUCCESS: Set krb5-devel links have been repaired successfully The USDP deployment environment of all nodes has been repaired successfully . Please proceed to the next step说明脚本执行成功。
刷新环境变量在7台机器都手动执行：
[root@zhiyong1 sbin]# source /etc/profile 配置MySQL [root@zhiyong1 sbin]# vim /opt/usdp-srv/usdp/config/application-server.yml修改这一段：
datasource: type: com.zaxxer.hikari.HikariDataSource # driver-class-name: org.gjt.mm.mysql.Driver driver-class-name: com.p6spy.engine.spy.P6SpyDriver url: jdbc:p6spy:mysql://192.168.88.100:3306/db_udp?useUnicode=true&characterEncoding=utf-8&useSSL=false username: root password: 123456 启动集群 [root@zhiyong1 sbin]# cd /opt/usdp-srv/usdp/ [root@zhiyong1 usdp]# bin/start-udp-server.sh BASE_PATH: /opt/usdp-srv/usdp/bin JMX_PATH: /opt/usdp-srv/usdp/jmx_exporter ln -s /opt/usdp-srv /data/usdp-srv ln -s /opt/usdp-srv/srv/udp /srv/ ln -s /data/var/log/udp /var/log/ REPAIR_PATH: /opt/usdp-srv/usdp/repair UDP_PATH: /opt/usdp-srv/usdp REPAIR_BIN_PATH: /opt/usdp-srv/usdp/repair/bin REPAIR_SBIN_PATH: /opt/usdp-srv/usdp/repair/sbin PACKAGES_PATH: /opt/usdp-srv/usdp/repair/packages nmap-6.40-19.el7.x86_64 nmap exists UDP Server is running with: 52552 Done.集群启动成功。此时Server的可用内存：
[root@zhiyong1 usdp]# free -m total used free shared buff/cache available Mem: 31994 2631 216 13 29146 28813 Swap: 0 0 0Worker的可用内存：
[root@zhiyong2 packages]# free -m total used free shared buff/cache available Mem: 15867 243 8921 11 6701 15175 Swap: 0 0 0 打开Web UI浏览器：
http://192.168.88.100/login即可创建用户。都写Zhiyong1。
软件版本 安装组件点下一步安装组件：
确认并联通节点： 检查环境是否正确：
选择推荐方案A做全量配置：
可以看到： 智能推荐： 有一些默认配置： 下一步等待安装完成： 都绿了就可以完成：
可以看到集群运行状态暂时良好【用不了多久就会JVM堆栈内存溢出导致组件宕掉】： 【建议：16G是能体验的最低配置，不搞HA，只启动最基础的HDFS、HBASE、KAFKA等服务的备用集群可以这么搞，主集群32G内存是组件全开不宕的最低要求】
默认的密码USDP官网部署说明
测试 [root@zhiyong2 bin]# pwd /opt/usdp-srv/srv/udp/2.0.0.0/hdfs/bin [root@zhiyong2 bin]# ./hadoop fs -ls /zhiyong-1/user/hive Found 1 items drwxrwxrwx - hadoop supergroup 0 2022-03-01 23:08 /zhiyong-1/user/hive/warehouse可以访问到数据。
再添加一套集群在上方可以找到切换集群的选项，然后和之前的操作一致。部署后： 当前资源占用： 不推荐128G以下内存的机器搭建Vmware双集群，撑死一个就够了。 笔者的虚拟机全部放在SSD以加快运行速度。
最后可以打快照，压缩存档。安装一次需要40分钟+。
内存溢出组件全开时，用不了多久就会发生JVM堆内存溢出。笔者之后将zhiyong2->zhiyong4的内存提升到32G，zhiyong5->zhiyong7则只开启必要的Zookeeper、HDFS、HBase、Kafka做备用集群。稳定性有了少许提高。这2套集群还是很吃内存的。
致谢感谢UCloud如此大方地让我等穷人可以白piao大数据集群管理工具USDP。对于新手而言，大数据组件的安装相当友好。安装部署相较CDH、CDP与HDP容易太多了。
新版本已经取消了授权操作，更是简化了操作步骤。
对于大数据开发攻城狮而言，也可以抽出更多精力放在Java/Scala/SQL开发而不是组件兼容性适配、源码编译这些琐碎的小事上。
 1.本站遵循行业规范，任何转载的稿件都会明确标注作者和来源；2.本站的原创文章，会注明原创字样,如未注明都非原创，如有侵权请联系删除！；3.作者投稿可能会经我们编辑修改或补充；4.本站不提供任何储存功能只提供收集或者投稿人的网盘链接。 |
标签： #usdp搭建 #部署USDP大数据套件
