docker部署kafka zookeeper模式集群
单机模式链接：https://blog.csdn.net/wsdhla/article/details/133032238
kraft集群模式链接：部署Kafka_kafka 部署-CSDN博客
zookeeper选举机制举例：
目前有5台服务器，每台服务器均没有数据，它们的编号分别是1,2,3,4,5,按编号依次启动，它们的选择举过程如下：
- 服务器1启动，给自己投票，然后发投票信息，由于其它机器还没有启动所以它收不到反馈信息，服务器1的状态一直属于Looking(选举状态)。
- 服务器2启动，给自己投票，同时与之前启动的服务器1交换结果，由于服务器2的编号大所以服务器2胜出，但此时投票数没有大于半数，所以两个服务器的状态依然是LOOKING。
- 服务器3启动，给自己投票，同时与之前启动的服务器1,2交换信息，由于服务器3的编号最大所以服务器3胜出，此时投票数正好大于半数，所以服务器3成为领导者，服务器1,2成为小弟。
- 服务器4启动，给自己投票，同时与之前启动的服务器1,2,3交换信息，尽管服务器4的编号大，但之前服务器3已经胜出，所以服务器4只能成为小弟。
- 服务器5启动，后面的逻辑同服务器4成为小弟。
Docker 自定义静态网络
Docker默认提供了3种网络模式，生成容器时不指定网络模式下默认使用bridge桥接模式。每次容器启动时ip会发生变更，无法达到我们集群的效果。
创建自定义网络模式，指定网段。
docker network create --subnet=172.18.0.0/16 net-kafka若报错：Error response from daemon: Pool overlaps with other one on this address space
说明指定的网络地址段已存在
1、Docker部署zookeeper集群
zk使用官方镜像：
1.1、主机创建目录
[zk01]
D:/mysoft/dev/docker-zk-cluster/zk01/conf
D:/mysoft/dev/docker-zk-cluster/zk01/data
D:/mysoft/dev/docker-zk-cluster/zk01/datalog[zk02]
D:/mysoft/dev/docker-zk-cluster/zk02/conf
D:/mysoft/dev/docker-zk-cluster/zk02/data
D:/mysoft/dev/docker-zk-cluster/zk02/datalog[zk03]
D:/mysoft/dev/docker-zk-cluster/zk03/conf
D:/mysoft/dev/docker-zk-cluster/zk03/data
D:/mysoft/dev/docker-zk-cluster/zk03/datalog1.2、修改配置文件
分别拷贝到zk01、zk02、zk03
[zoo.cfg]dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true
server.1=172.18.0.11:2888:3888;2181
server.2=172.18.0.12:2888:3888;2181
server.3=172.18.0.13:2888:3888;21811.3、指定ip创建启动容器
docker run -d --name zk01 -p 2181:2181 --network net-kafka --ip 172.18.0.11 -v D:/mysoft/dev/docker-zk-cluster/zk01/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk01/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk01/datalog:/datalog zookeeperdocker run -d --name zk02 -p 2182:2181 --network net-kafka --ip 172.18.0.12 -v D:/mysoft/dev/docker-zk-cluster/zk02/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk02/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk02/datalog:/datalog zookeeperdocker run -d --name zk03 -p 2183:2181 --network net-kafka --ip 172.18.0.13 -v D:/mysoft/dev/docker-zk-cluster/zk03/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk03/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk03/datalog:/datalog zookeeperzk01可以正常启动，zk02和zk03启动失败
警告：这个正常，是因为节点启动时选举机制的原因
zk02：
WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888
zk03：
WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
异常：
zk02：
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count.
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As I'm leaving the listener thread, I won't be able to participate in leader election any longer: 172.18.0.11:3888
zk03：
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count.
ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As I'm leaving the listener thread, I won't be able to participate in leader election any longer: 172.18.0.11:3888
原因是没有配置quorumListenOnAllIPs，默认为false，需要设置为true
[zoo.cfg]dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true
quorumListenOnAllIPs=true
server.1=172.18.0.11:2888:3888;2181
server.2=172.18.0.12:2888:3888;2181
server.3=172.18.0.13:2888:3888;2181quorumListenOnAllIPs：
当设置为true时，ZooKeeper服务器将在所有可用IP地址上侦听来自其对等方的连接，而不仅是在配置文件的服务器列表中配置的地址。它会影响处理ZAB协议和快速领导者选举协议的连接。默认值为false。
有一些WARN:
### 1、只启动zk01：
# zk01：
01:21:17.746 [QuorumConnectionThread-[myid=1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)01:21:20.904 [QuorumConnectionThread-[myid=1]-1] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)### 2、 只启动zk01、zk02：
# zk01：
01:21:46.504 [QuorumConnectionThread-[myid=1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)# zk02：
01:21:43.350 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.01:21:48.824 [QuorumConnectionThread-[myid=1]-2] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)### 2、 启动zk01、zk02、zk03：
# zk01正常：
01:31:34.579 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:31:34.579 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:31:34.580 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:31:34.584 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:31:34.584 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:31:34.585 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:31:34.585 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:31:34.585 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:31:34.585 [QuorumConnectionThread-[myid=1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
01:31:34.586 [QuorumConnectionThread-[myid=1]-21] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
01:31:34.585 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:31:34.586 [QuorumConnectionThread-[myid=1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
01:31:34.586 [QuorumConnectionThread-[myid=1]-22] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
01:31:52.173 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - node0 scavenging sessions
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597==dftMaxIdleSec=-1 scavenging sessions
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597==dftMaxIdleSec=-1 scavenging session ids []
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.DefaultSessionCache@57d7f8ca[evict=-1,removeUnloadable=false,saveOnCreate=false,saveOnInactiveEvict=false] checking expiration on []# zk02：
01:35:15.331 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:35:15.332 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:35:15.333 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:35:15.334 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:35:15.334 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:35:15.335 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:35:15.335 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:50246
01:35:15.335 [QuorumConnectionThread-[myid=1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:50246
01:35:15.335 [QuorumConnectionThread-[myid=1]-27] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:35:15.335 [QuorumConnectionThread-[myid=1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
01:35:15.336 [QuorumConnectionThread-[myid=1]-28] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
01:35:34.600 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:49750
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:49750
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:35:59.199 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:56956
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:56956
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.# zk03：
01:36:59.201 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:36:59.201 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:36:59.202 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:36:59.202 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:36:59.203 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:36:59.203 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:36:59.203 [QuorumConnectionThread-[myid=1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:2)
01:36:59.203 [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:36:59.203 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:36:59.203 [QuorumConnectionThread-[myid=1]-15] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2)
01:36:59.204 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:35192
01:36:59.204 [QuorumConnectionThread-[myid=1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 --> sid:3)
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:35192
01:36:59.204 [QuorumConnectionThread-[myid=1]-16] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3)
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:36:59.205 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:35956
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:35956
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:46420
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:46420
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid=1, electionAddr=[/172.18.0.11:3888]}
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
找到原因了，因为data/myid没有修改，目前3个节点的配置全部为1，分别改为1、2、3就正常了。
启动zkui
docker run -d --name zkui -p 9090:9090 --network net-kafka -e ZKLIST=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 maauso/zkui访问localhost:9090，登录之后报错：KeeperErrorCode = ConnectionLoss for /
删除data\version-2\snapshot.0依然不行
stat is not executed because it is not in the whitelist.
envi is not executed because it is not in the whitelist.
4lw.commands.whitelist=*
16:49:58.466 [NIOWorkerThread-10] DEBUG org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /172.18.0.2:42978 (no session established for client)
16:49:58.499 [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181] DEBUG org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /172.18.0.2:42980
16:49:58.500 [NIOWorkerThread-11] WARN org.apache.zookeeper.server.NIOServerCnxn - Close of session 0x0
java.io.IOException: ZooKeeperServer not runningat org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:569)at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:349)at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at java.base/java.lang.Thread.run(Unknown Source)好像改为zk的data/myid配置，上面的问题就没有了。
2、Docker部署kafka集群
kafka没有官方镜像，使用wurstmeister/kafka镜像
2.1、主机创建目录
[kafka01]
D:/mysoft/dev/docker-kafka-cluster/kafka01/config
D:/mysoft/dev/docker-kafka-cluster/kafka01/data
D:/mysoft/dev/docker-kafka-cluster/kafka01/logs
D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts[kafka02]
D:/mysoft/dev/docker-kafka-cluster/kafka02/config
D:/mysoft/dev/docker-kafka-cluster/kafka02/data
D:/mysoft/dev/docker-kafka-cluster/kafka02/logs
D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts[kafka03]
D:/mysoft/dev/docker-kafka-cluster/kafka03/config
D:/mysoft/dev/docker-kafka-cluster/kafka03/data
D:/mysoft/dev/docker-kafka-cluster/kafka03/logs
D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts2.2、修改配置文件
分别拷贝到kafka01、kafka02、kafka03
[server.properties]broker.id=1
advertised.listeners=PLAINTEXT://172.18.0.21:9092
# port=9092
# log.dirs=/kafka/kafka-logs-1ddafafe54d0
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:21812.3、指定ip创建启动容器
# docker run -d --name kafka01 --restart=always --net=host -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc/hosts -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME=192.168.180.42 -e JMX_PORT=9999 -e HOST_IP=127.0.0.1 -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_ZOOKEEPER_CONNECT=127.0.0.1:2181 -e KAFKA_BROKER_ID=1 -e KAFKA_HEAP_OPTS="-Xmx6G -Xms6G" wurstmeister/kafka
# docker run -d --name kafka01 --restart=always --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME=172.18.0.21 -e KAFKA_ADVERTISED_PORT=9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=1 -e KAFKA_LOG_DIRS=/opt/kafka/logs wurstmeister/kafkadocker run -d --name kafka01 -p 9192:9092 --restart=always --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=1 wurstmeister/kafkadocker run -d --name kafka02 -p 9292:9092 --restart=always --network net-kafka --ip 172.18.0.22 -v D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka02/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka02/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka02/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.22:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=2 wurstmeister/kafkadocker run -d --name kafka03 -p 9392:9092 --restart=always --network net-kafka --ip 172.18.0.23 -v D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka03/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka03/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka03/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.18.0.23:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID=3 wurstmeister/kafkakafka容器异常，一直自动反复重启，UP状态维持1分钟左右
[Configuring] 'advertised.listeners' in '/opt/kafka/config/server.properties'
[Configuring] 'port' in '/opt/kafka/config/server.properties'
Excluding KAFKA_HOME from broker config
[Configuring] 'log.dirs' in '/opt/kafka/config/server.properties'
[Configuring] 'listeners' in '/opt/kafka/config/server.properties'
Excluding KAFKA_VERSION from broker config
[Configuring] 'zookeeper.connect' in '/opt/kafka/config/server.properties'
[Configuring] 'broker.id' in '/opt/kafka/config/server.properties'
log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/bin/../config/log4j.properties].
java.io.FileNotFoundException: /opt/kafka/bin/../config/log4j.properties (No such file or directory)at java.base/java.io.FileInputStream.open0(Native Method)at java.base/java.io.FileInputStream.open(Unknown Source)at java.base/java.io.FileInputStream.<init>(Unknown Source)at java.base/java.io.FileInputStream.<init>(Unknown Source)at java.base/sun.net.www.protocol.file.FileURLConnection.connect(Unknown Source)at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(Unknown Source)at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala:25)at kafka.Kafka$.<clinit>(Kafka.scala:30)at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Ignoring configuration file [file:/opt/kafka/bin/../config/log4j.properties].
log4j:WARN No appenders could be found for logger (kafka.utils.Log4jControllerRegistration$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.指定-e KAFKA_LOG_DIRS也依然不可以，手动拷贝一份log4j.properties到映射的目录D:/mysoft/dev/docker-kafka-cluster/kafka01/logs下问题就解决了。
但是出现另外一个异常：
[2023-12-23 15:20:50,136] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,204] INFO Socket connection established, initiating session, client: /172.18.0.21:39814, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,273] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,749] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,811] INFO Socket connection established, initiating session, client: /172.18.0.21:49980, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,914] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,268] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,356] INFO Socket connection established, initiating session, client: /172.18.0.21:40344, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,446] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,617] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2023-12-23 15:20:52,817] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,855] INFO Socket connection established, initiating session, client: /172.18.0.21:39828, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:53,036] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2023-12-23 15:20:53,037] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:53,076] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2023-12-23 15:20:53,222] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTINGat kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)at kafka.server.KafkaServer.startup(KafkaServer.scala:191)at kafka.Kafka$.main(Kafka.scala:109)at kafka.Kafka.main(Kafka.scala)
[2023-12-23 15:20:53,290] INFO shutting down (kafka.server.KafkaServer)
[2023-12-23 15:20:53,375] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2023-12-23 15:20:53,440] INFO shut down completed (kafka.server.KafkaServer)
[2023-12-23 15:20:53,515] ERROR Exiting Kafka. (kafka.Kafka$)
[2023-12-23 15:20:53,590] INFO shutting down (kafka.server.KafkaServer)回头看zk的控制台，发现报错java.io.IOException: ZooKeeperServer not running，跟用zkui客户端连zk集群一样的错，通过修改zk的data/myid配置，之后，3个kafka服务都能正常启动了，而且也不会反复自动重启了。
# kafka01
[2023-12-25 02:08:01,176] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)# kafka02
[2023-12-25 02:09:56,391] INFO [KafkaServer id=2] started (kafka.server.KafkaServer)# kafka03
[2023-12-25 02:10:14,479] INFO [KafkaServer id=3] started (kafka.server.KafkaServer)虽然kafka服务在启动的初始阶段，仍然有一下异常，但是目前没有发现有什么影响：
[2023-12-25 00:52:15,882] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:15,932] INFO Socket connection established, initiating session, client: /172.18.0.21:49568, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:15,984] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:17,040] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:17,094] INFO Socket connection established, initiating session, client: /172.18.0.21:59734, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:17,148] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:17,979] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:18,033] INFO Socket connection established, initiating session, client: /172.18.0.21:50098, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:18,088] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:19,308] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:19,357] INFO Socket connection established, initiating session, client: /172.18.0.21:49580, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:19,403] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:19,974] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:20,027] INFO Socket connection established, initiating session, client: /172.18.0.21:59746, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:20,075] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:20,549] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [2023-12-25 00:52:20,812] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:20,860] INFO Socket connection established, initiating session, client: /172.18.0.21:50110, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:21,016] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [2023-12-25 00:52:21,017] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [2023-12-25 00:52:21,062] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [2023-12-25 00:52:21,164] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTINGat kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)at kafka.server.KafkaServer.startup(KafkaServer.scala:191)at kafka.Kafka$.main(Kafka.scala:109)at kafka.Kafka.main(Kafka.scala) [2023-12-25 00:52:21,213] INFO shutting down (kafka.server.KafkaServer) [2023-12-25 00:52:21,274] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [2023-12-25 00:52:21,319] INFO shut down completed (kafka.server.KafkaServer) [2023-12-25 00:52:21,371] ERROR Exiting Kafka. (kafka.Kafka$) [2023-12-25 00:52:21,425] INFO shutting down (kafka.server.KafkaServer)
启动sheepkiller/kafka-manager
# docker run -d --name kafka-manager -p 9000:9000 --network net-kafka --ip 172.18.0.31 -e ZK_HOSTS=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-managerdocker run -d --name kafka-manager -p 9000:9000 --network net-kafka -e ZK_HOSTS=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-manager
或者使用provectuslabs/kafka-ui:latest
# docker run -d --name kafka-ui -p 8080:8080 --network net-kafka --ip 172.18.0.32 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-uidocker run -d --name kafka-ui -p 8080:8080 --network net-kafka -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui
3、完整配置
3.1 zk完整配置
zk01/conf/zoo.cfg
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true
quorumListenOnAllIPs=true
server.1=172.18.0.11:2888:3888;2181
server.2=172.18.0.12:2888:3888;2181
server.3=172.18.0.13:2888:3888;2181
4lw.commands.whitelist=*zk01/data/myid
1zk02/conf/zoo.cfg
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true
quorumListenOnAllIPs=true
server.1=172.18.0.11:2888:3888;2181
server.2=172.18.0.12:2888:3888;2181
server.3=172.18.0.13:2888:3888;2181
4lw.commands.whitelist=*zk02/data/myid
2zk03/conf/zoo.cfg
dataDir=/data
dataLogDir=/datalog
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
standaloneEnabled=true
admin.enableServer=true
quorumListenOnAllIPs=true
server.1=172.18.0.11:2888:3888;2181
server.2=172.18.0.12:2888:3888;2181
server.3=172.18.0.13:2888:3888;2181
4lw.commands.whitelist=*zk03/data/myid
33.2 kafka完整配置
kafka01/config/server.properties
broker.id=1
advertised.listeners=PLAINTEXT://172.18.0.21:9092
port=9092
log.dirs=/kafka/kafka-logs-7725cecb8a4f
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181kafka02/config/server.properties
broker.id=2
advertised.listeners=PLAINTEXT://172.18.0.22:9092
port=9092
log.dirs=/kafka/kafka-logs-2b3d2a64f445
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181kafka03/config/server.properties
broker.id=3
advertised.listeners=PLAINTEXT://172.18.0.23:9092
port=9092
log.dirs=/kafka/kafka-logs-d9101a867f99
listeners=PLAINTEXT://0.0.0.0:9092
zookeeper.connect=172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:21814、问题
SpringBoot配置：
spring:kafka:bootstrap-servers: localhost:9192,localhost:9292,localhost:9392报错，与单机版错误一致
2023-12-26 15:27:29.296 INFO 11904 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.8.1
2023-12-26 15:27:29.298 INFO 11904 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka commitId: 839b886f9b732b15
2023-12-26 15:27:29.298 INFO 11904 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1703575649293
2023-12-26 15:27:29.302 INFO 11904 --- [ main] o.a.k.clients.consumer.KafkaConsumer : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Subscribed to topic(s): test-topic032023-12-26 15:27:30.191 INFO 11904 --- [ntainer#0-0-C-1] org.apache.kafka.clients.Metadata : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Cluster ID: o6TS0_kER-KgmyB7NWjjEw
2023-12-26 15:27:30.194 INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Discovered group coordinator 172.18.0.23:9092 (id: 2147483644 rack: null)
2023-12-26 15:27:30.200 INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] (Re-)joining group
2023-12-26 15:27:38.804 INFO 11904 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-test-consumer-group01-1, groupId=test-consumer-group01] Group coordinator 172.18.0.23:9092 (id: 2147483644 rack: null) is unavailable or invalid due to cause: null.isDisconnected: true. Rediscovery will be attempted.相关文章：
docker部署kafka zookeeper模式集群
单机模式链接：https://blog.csdn.net/wsdhla/article/details/133032238 kraft集群模式链接：部署Kafka_kafka 部署-CSDN博客 zookeeper选举机制举例： 目前有5台服务器，每台服务器均没有数据，它们的编号分别是1,2,3,4,5…...
Apache Flink连载（二十一）：Flink On Yarn运行原理-Yarn Application模式
🏡 个人主页：IT贫道_大数据OLAP体系技术栈,Apache Doris,Clickhouse 技术-CSDN博客 🚩 私聊博主：加入大数据技术讨论群聊，获取更多大数据资料。 🔔 博主个人B栈地址：豹哥教你大数据的个人空间-豹哥教你大数据个人主页-哔哩哔哩视频 目录 1. 任务提交命令...
《深入理解C++11：C++11新特性解析与应用》笔记五
第五章 提高类型安全 5.1 强类型枚举 5.1.1 枚举：分门别类与数值的名字 具名枚举类型一般声明类似：enum Gender { Male, Female }。 匿名枚举类型可以使用三种方式实现： 第一种方式时宏，比如 #define Male 0 #define Femal…...
Y9000P + ubuntu22.04 配置Anaconda+pycharm +pytorch
Anaconda3 的安装及使用方法安装 Anaconda3 Anaconda3 是 Anaconda 的具体版本 Anaconda3 中的 Python 解释器默认使用的是 Python3.x 版本，而不是 Python2.x 版本 Python2.x 版本中，字符串是以 ASCII 编码处理的，而在 Python3.x 版本中&am…...
使用Ubuntu编译FFmpeg生成Android动态库/静态库
环境 我这里使用windows里的wsl2的ubuntu，使用物理机或者vmware，vbox之类的安装的ubuntu理论上也可以. gcc编译使用的环境如下: Ndk使用17 FFmpeg使用4.0.2. clang编译使用的环境如下: Ndk使用21.4 FFmpeg使用6.1 FFmpeg下载地址:https://ffmpeg.org/…...
【AIGC-图片生成视频系列-2】八仙过海，各显神通：AI生成视频相关汇总剖析
最近「图片生成视频系列」层出不穷，我拜读并结合实践（对，就是手撕代码，有开源就撕），并对以下几篇文章的相似点以及关键点稍微做个总结： 一. 生成视频中图像的一致性 在图像生成视频的这个过程…...
SpringBoot集成RabbitMq消息队列【附源码】
1. 项目背景 要啥项目背景，就是干！！！ SpringBoot版本：2.7.12 2. Rabbit MQ安装 这里讲解使用docker安装RabbitMQ，如果在windows下面安装RabbitMQ，参考下文 【笑小枫的按步照搬系列】Window…...
MySQL数据库的安装与环境配置
下载 下载MySQL8 安装 解压 配置MySQL环境变量 系统环境变量path D:\ProgramFiles\mysql-8.0.20-winx64\bin 1.点击属性 2.点击高级系统设置 3.点击环境变量 4.在系统变量中找到path 注意这里不是用户变量 5.新建后输入解压的地址 MySQL初始化和启动 以管理员身份运行cmd…...
【广州华锐互动】VR科技科普展厅平台：快速、便捷地创建出属于自己的虚拟展馆
随着科技的不断进步，虚拟现实(VR)技术已经在许多领域取得了显著的成果。尤其是在展馆设计领域，VR科技科普展厅平台已经实现了许多令人瞩目的新突破。 VR科技科普展厅平台是广州华锐互动专门为企业和机构提供虚拟展馆设计和制作的在线平台。通过这个平台&…...
XML Extension Supplement
LEGAL ISSUES, COMPANY POLICIES AND STANDARDS Web Services A Web service is a software system designed to support interoperable machine-to-machine interaction over a network. URI和URL URI，全称是统一资源标识符（Uniform Resource Ident…...
手拉手Springboot获取yml配置文件信息
环境介绍 技术栈 springboot3 软件 版本 mysql 8 IDEA IntelliJ IDEA 2022.2.1 JDK 17 Spring Boot 3.1.7 配置文件说明：启动配置文件优先级：properties高于yml 配置文件application.yml yml是 JSON 的超集，简洁而强大…...
行人重识别(ReID)基础知识入门
这里写目录标题 1、ReID技术概述1.1 基本原理1.2 实现流程1.3 重识别存在的技术挑战 2、训练数据格式介绍 1、ReID技术概述 1.1 基本原理 ReID，全称Re-identification，目的是利用各种智能算法在图像数据库中找到与要搜索的目标相似的对象。ReID是图像检…...
【音视频 ffmpeg 学习】 跑示例程序 持续更新中
环境准备 在上一篇文章 把mux.c 拷贝到main.c 中 使用 attribute(unused) 消除警告 __attribute__(unused)/** Copyright (c) 2003 Fabrice Bellard** Permission is hereby granted, free of charge, to any person obtaining a copy* of this software and associated docu…...
前端axios与python库requests的区别
当涉及到发送HTTP请求时，Axios和Python中的requests库都是常用的工具。下面是它们的详细说明： Axios： Axios是一个基于Promise的HTTP客户端，主要用于浏览器和Node.js环境中发送HTTP请求。以下是Axios的一些特点和用法࿱…...
达梦数据库文档
1：达梦数据库(DM8)简介 达梦数据库管理系统是武汉达梦公司推出的具有完全自主知识产权的高性能数据库管理系统，简称DM。达梦数据库管理系统目前最新的版本是8.0版本，简称DM8。 DM8是达梦公司在总结DM系列产品研发与应用经验的基础上…...
CorelDRAW2024新功能有哪些？CorelDRAW2024最新版本更新怎么样？
CorelDRAW2024新功能有哪些？CorelDRAW2024最新版本更新怎么样？让我们带您详细了解！ CorelDRAW Graphics Suite 是矢量制图行业的标杆软件，2024年全新版本为您带来多项新功能和优化改进。本次更新强调易用性，包括更强大…...
基于Mapify的在线艺术地图设计
地图是传递空间信息的有效载体，更加美观、生动的地图产品也是我们追求目标。 那么，我们如何才能制出如下图所示这样一幅艺术性较高的地图呢？今天我们来一探究竟吧！ 按照惯例，现将网址给出： https://www.m…...
mxxWechatBot微信机器人V2版本文档说明
大家伙，我是雄雄，欢迎关注微信公众号：雄雄的小课堂。 先看这里 一、前言二、mxxWechatBot流程图三、怎么使用？ 一、前言 经过不断地探索与研究，mxxWechatBot正式上线，届时全面开放使用。 mxxWechatBot&am…...
红队打靶练习:MISDIRECTION: 1
信息收集 1、arp ┌──(root㉿ru)-[~/kali] └─# arp-scan -l Interface: eth0, type: EN10MB, MAC: 00:0c:29:69:c7:bf, IPv4: 192.168.12.128 Starting arp-scan 1.10.0 with 256 hosts (https://github.com/royhills/arp-scan) 192.168.12.1 00:50:56:c0:00:08 …...
Jmeter吞吐量控制器总结
吞吐量控制器(Throughput Controller) 场景： 在同一个线程组里, 有10个并发, 7个做A业务, 3个做B业务,要模拟这种场景,可以通过吞吐量模拟器来实现。 添加吞吐量控制器 用法1: Percent Executions 在一个线程组内分别建立两个吞吐量控制器, 分别放业务A和业务B …...
【XML】TinyXML 详解（二）：接口详解
【C】郭老二博文之：C目录 1、XML测试文件（laoer.xml） <?xml version"1.0" standalone"no" ?> <!-- Hello World !--> <root><child name"childName" id"1"><c_child…...
【机器学习】人工智能概述
人工智能（Artificial Intelligence，简称AI）是一门研究如何使机器能够像人一样思考、学习和执行任务的学科。它是计算机科学的一个重要分支，涉及机器学习、自然语言处理、计算机视觉等多个领域。 人工智能的概念最早可以追溯到20世…...
flink 实时写入 hudi 参数推荐
数据湖任务并行度计算...
传统项目基于tomcat cookie单体会话升级分布式会话解决方案
传统捞项目基于servlet容器 cookie单体会话改造分布式会话方案 ##引入redis,spring-session依赖 <!--redis依赖 --><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-redis</artifactId>&…...
Unity 关于json数据的解析方式（LitJson.dll插件）
关于json数据的解析方式（LitJson.dll插件） void ParseItemJson(){TextAsset itemText Resources.Load<TextAsset>("Items");//读取Resources中Items文件，需要将Items文件放到Resources文件夹中string itemJson itemText.te…...
智能监控平台/视频共享融合系统EasyCVR海康设备国标GB28181接入流程
TSINGSEE青犀视频监控汇聚平台EasyCVR可拓展性强、视频能力灵活、部署轻快，可支持的主流标准协议有国标GB28181、RTSP/Onvif、RTMP等，以及支持厂家私有协议与SDK接入，包括海康Ehome、海大宇等设备的SDK等。平台既具备传统安防视频监控的能力&…...
expdp到ASM 文件系统 并拷贝
1.创建asm导出数据目录 sql>select name,total_mb,free_mb from v$asm_diskgroup; 确认集群asm磁盘组环境 asmcmd>cd DGDSDB asmcmd>mkdir dpbak asmcmd>ls -l sql>conn / as sysdba create directory expdp_asm_dir as DGDSDB/dpbak; create directory expdp_l…...
【2023】通过docker安装hadoop以及常见报错
💻目录 1、准备2、安装镜像2.1、创建centos-ssh的镜像2.2、创建hadoop的镜像 3、配置ssh网络3.1、搭建同一网段的网络3.2、配置host实现互相之间可以免密登陆3.3、查看是否成功 4、安装配置Hadoop4.1、添加存储文件夹4.2、添加指定配置4.3、同步数据 5、测试启动5.1…...
Baumer工业相机堡盟工业相机如何通过NEOAPI SDK获取相机当前实时帧率（C++）
Baumer工业相机堡盟工业相机如何通过NEOAPI SDK获取相机当前实时帧率（C） Baumer工业相机Baumer工业相机的帧率的技术背景Baumer工业相机的帧率获取方式CameraExplorer如何查看相机帧率信息在NEOAPI SDK里通过函数获取相机帧率（C） …...
SpringBoot项目部署及多环境
1、多环境 2、项目部署上线 原始前端 / 后端项目宝塔Linux容器容器平台 3、前后端联调 4、项目扩展和规划 多环境 程序员鱼皮-参考文章 本地开发：localhost（127.0.0.1） 多环境：指同一套项目代码在把不同的阶段需要根据实际…...
WebGL以及wasm的介绍以及简单应用
简介 下面主要介绍了WebGL和wasm,是除了html,css,js以外Web标准所支持的另外两个大件 前者实现复杂的图形处理,后者提供高效的代码迁移以及代码执行效率 WebGL 简介 首先,浏览器里的游戏是怎么做到这种交互又显示不同的画面的? 试想用我们的前端三件套实现一下.好像可以…...
JS和TS的基础语法学习以及babel的基本使用
简介 本文主要介绍了一下js和ts的基础语法,为前端开发zuo JavaScript 更详细的 JavaScript 学习资料：https://developer.mozilla.org/zh-CN/docs/Web/JavaScript 简介 定位 : JavaScript 是一种动态语言，它包含类型、运算符、标准内置（ bu…...
Centos安装Composer
今天分享下如何在centos系统里安装composer 一、下载composer curl -sS https://getcomposer.org/installer | php二、移动或复制composer到环境下可执行 cp composer.phar /usr/local/bin/composer三、测试看是否安装成功 composer -V四、全局安装 curl -sS https://getc…...
面试题：从 MySQL 读取 100w 数据进行处理，应该怎么做？
文章目录 背景常规查询流式查询MyBatis 流式查询接口为什么要用流式查询？ 游标查询OptionsResultType注意：原因： 非流式查询和流式查询区别： 背景 大数据量操作的场景大致如下： 数据迁移数据导出批量处理数据 在实际…...
销售转行上位机编程：我的学习与职业经历分享
同学们好，我是杨工，原先是一名销售。 通过在华山编程培训中心学习，成功转行上位机编程，对此我想分享学习和职业经历。 在职业生涯的早期，我并没有考虑将技术融入到我的工作中。然而，在几次创业的失败后&a…...
分库分表之Mycat应用学习一
1 为什么要分库分表 1.1 数据库性能瓶颈的出现 对于应用来说，如果数据库性能出现问题，要么是无法获取连接，是因为在高并发的情况下连接数不够了。要么是操作数据变慢，数据库处理数据的效率除了问题。要么是存储出现问题…...
Windows下Qt使用MSVC编译出现需要转为unicode的提示
参考 Qt5中文编码问题解决办法_qt5设置编码-CSDN博客 致敬 提示：warning: C4819: 该文件包含不能在当前代码页(936)中表示的字符。请将该文件保存为 Unicode 格式以防止数据丢失。 出现此问题，应该是Unix格式下代码的编码格式是UTF-8，注意不…...
【数值分析】乘幂法，matlab实现
乘幂法 一种求实矩阵 A {A} A 的按模最大的特征值，及其对应的特征向量 x i {x_i} xi 的方法，只能求一个。特别适合于大型稀疏矩阵。 一个矩阵的特征值和特征向量可以通过矩阵不断乘以一个初始向量得到。 每次乘完之后要规范化，防止上溢或…...
视频监控EasyCVR如何通过设置sei接口，实现在webrtc视频流中添加画框和文字？
安防视频监控系统基于视频综合管理平台EasyCVR视频系统，采用了开放式的网络结构，可以提供实时远程视频监控、视频录像、录像回放与存储、告警、语音对讲、云台控制、平台级联、磁盘阵列存储、视频集中存储、云存储等丰富的视频能力，具备权限管…...
智能三维数据虚拟现实电子沙盘
一、概述 易图讯科技（www.3dgis.top）以大数据、云计算、虚拟现实、物联网、AI等先进技术为支撑，支持高清卫星影像、DEM高程数据、矢量数据、无人机倾斜摄像、BIM模型、点云、城市白模、等高线、标高点等数据融合和切换，智能三维数…...
【SpringCloud】-GateWay源码解析
GateWay系列 【SpringCloud】-GateWay网关 一、背景介绍 当一个请求来到 Spring Cloud Gateway 之后，会经过一系列的处理流程，其中涉及到路由的匹配、过滤器链的执行等步骤。今天我们来说说请求经过 Gateway 的主要执行流程和原理是什么吧 二、正文 …...
华为无线ac双链路冷备和热备配置案例
所谓的冷备和热备，冷备就是不用vrrp和hsb协议同步ap和用户信息，主的断了等七十五秒后，备的capwap和ap连接上去。 双链路冷备不用vrrp和hsb 双链路热备份只用hsb同步ap和用户信息，不用vrrp，两个ac可以不用在同一个二层…...
VSCode Python开发环境配置
目录 1 插件安装2 Debug和测试配置常见问题 1 插件安装 1.1 基础编译插件，Python、Pylance 1.2 修改语言服务器类型，进入用户配置页面搜索Python: Language Server，选择Pylance（一定要修改可以提供很多语法提示） 1…...
浅谈【GPU和CPU】
GPU和显卡的区别 GPU（Graphics Processing Unit，图形处理器）通常指的就是显卡。显卡是一种安装在计算机中的扩展卡，主要用于图形和图像处理任务。 GPU作为显卡的核心组件，负责处理图形渲染、图像处理、视频解码和其他…...
啥是构造器？
当我们new一个对象时就是在引用构造器 构造器又叫做构造函数 构造函数一般分为无参构造函数与有参构造函数 假设我们创建一个pet类，这个类里面就会有一个看不见的自动生成的无参构造函数 如果pet类里没有这个隐形的无参构造，我们new一个对象时就会报错…...
Linux基础知识学习2
tree命令的使用 可以看到dir2目录下的这些文件，要想显示dir2的具体结构，可用tree命令 mv命令 它可以实现两个功能 1.将文件移动到另一个目录中 2.对某一个文件进行重命名 1.将文件移动到另一个目录中 这里将dir1中的2.txt移动到他的子目录dir3中 执行…...
Grafana二进制部署并配置prometheus数据源
1、获取grafna二进制安装包 https://grafana.com/grafana/download?pggraf&plcmtdeploy-box-1 grafana官网下载地址 [rootambari-hadoop1 ~]# cd /opt/module/grafana/ [rootambari-hadoop1 grafana]# pwd /opt/module/grafana2、在安装自己的安装目录执行 wget https:…...
时序预测 | Matlab实现SSA-CNN-BiLSTM麻雀算法优化卷积双向长短期记忆神经网络时间序列预测
时序预测 | Matlab实现SSA-CNN-BiLSTM麻雀算法优化卷积双向长短期记忆神经网络时间序列预测 目录 时序预测 | Matlab实现SSA-CNN-BiLSTM麻雀算法优化卷积双向长短期记忆神经网络时间序列预测预测效果基本介绍程序设计参考资料 预测效果 基本介绍 MATLAB实现SSA-CNN-BiLSTM麻雀算…...
Java中的单元测试
单元测试 单元测试概述: 单元测试是指在软件开发中对软件的最小可测试单元进行测试和验证的过程。最小可测试单元通常是指函数、方法或者类，单元测试可以保证开发人员的代码正确性，同时也方便后期维护和修改。单元测试的主要目的是检测代码的正确性&am…...
143.【Nginx-02】
Nginx-02 (五)、Nginx负载均衡1.负载均衡概述2.负载均衡的原理及处理流程(1).负载均衡的作用 3.负载均衡常用的处理方式(1).用户手动选择(2).DNS轮询方式(3).四/七层负载均衡(4).Nginx七层负载均衡指令 ⭐(5).Nginx七层负载均衡的实现流程 ⭐ 4.负载均衡状态(1).down (停用)(2)…...