MAC-MQ安装配置

学习资料

• 使用Docker部署Kafka单机版本

使用Dcoker部署Kafka单机版

环境配置

工具	说明	适配
Docker	Client/Engine 27.1.1	Dcoker CE *
Docker Desktop	4.33.0	4.0.x
Zookeeper	zookeeper:3.8（Docker Image ）	zookeeper:3.x（Docker Image ）
Kafka	wurstmeister/kafka:2.13-2.8.1（Docker Image）	wurstmeister/kafka:2.x（Docker Image）

1.手动部署Kafka

拉取镜像：docker pull

# 拉取zookeeper镜像
docker pull zookeeper:3.8

# 拉取kafka镜像
docker pull wurstmeister/kafka:2.13-2.8.1

创建数据卷，方便数据持久化

# 创建zookeeper数据卷
docker volume create zookeeper_vol

# 创建kafka数据卷
docker volume create kafka_vol

创建zookeeper容器

​ 创建zookeeper-test容器，同时挂载数据卷和并指定端口映射（2181）

docker run -d --name zookeeper-test -p 2181:2181 \
--env ZOO_MY_ID=1 \
-v zookeeper_vol:/data \
-v zookeeper_vol:/datalog \
-v zookeeper_vol:/logs \
zookeeper

创建kafka容器

​ 创建kafka-test容器，同时挂载数据卷和并指定端口映射（9092），并将zookeeper-test链接到该容器，使Kafka可以成功访问到zookeeper-test，Kafka相关参数通过环境变量（—env）设置

docker run -d --name kafka-test -p 9092:9092 \
--link zookeeper-test \
--env KAFKA_ZOOKEEPER_CONNECT=zookeeper-test:2181 \
--env KAFKA_ADVERTISED_HOST_NAME=localhost \
--env KAFKA_ADVERTISED_PORT=9092  \
--env KAFKA_LOG_DIRS=/kafka/logs \
-v kafka_vol:/kafka  \
wurstmeister/kafka

查看docker desktop内容

2.Docker Compose 部署Kafka

创建Dcoker Compose 配置文件

​ 使用Docker Compose可以将一系列创建及映射资源（网络、数据卷等）操作放在配置文件中，并且可以通过depends_on参数指定容器的启动顺序，通过environment参数指定Kafka需要的基本参数信息 ​

​ 创建kafka-group.yml，保存以下信息

version: '3'

name: kafka-group

services:
  zookeeper-test:
    image: zookeeper
    ports:
      - "2181:2181"
    volumes:
      - zookeeper_vol:/data
      - zookeeper_vol:/datalog
      - zookeeper_vol:/logs
    container_name: zookeeper-test

  kafka-test:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "localhost"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper-test:2181"
      KAFKA_LOG_DIRS: "/kafka/logs"
    volumes:
      - kafka_vol:/kafka
    depends_on:
      - zookeeper-test
    container_name: kafka-test

volumes:
  zookeeper_vol: {}
  kafka_vol: {}

启动容器组

# 启动Kafka容器组
docker compose -f kafka-group.yml up -d

# output
WARN[0000] /Users/holic-x/dev/MQ/kafka/kafka-group.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 3/0
 ✔ Network kafka-group_default         Cr...                               0.0s 
 ✔ Volume "kafka-group_zookeeper_vol"  Created                             0.0s 
 ✔ Volume "kafka-group_kafka_vol"      Created                             0.0s 
 ⠋ Container zookeeper-test            Creat...                            0.0s 
Error response from daemon: Conflict. The container name "/zookeeper-test" is already in use by container "7b0308641a726debcdc8039f6fe94b0dc08f60a88c276c9ab02a4d6e5f7a7b79". You have to remove (or rename) that container to be able to reuse that name.

3.Kafka 消息测试

启动Kafka Producer

​ 新开一个命令窗口，执行以下命令，启动Kafka Producer，向topic:test发送消息

# 进入容器
docker exec -it kafka-test /bin/bash

# 进入Kafka bin目录
cd /opt/kafka/bin

# 启动Producer
sh kafka-console-producer.sh --broker-list localhost:9092 --topic test

启动Kafka Consumer

​ 新开一个命令窗口，执行以下命令，启动Kafka Consumer，订阅来自topic:test的消息

# 进入容器
docker exec -it kafka-test /bin/bash

# 进入Kafka bin目录
cd /opt/kafka/bin

# 启动Consumer
sh kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test

收发消息测试

​ 在Producer窗口发送消息，随后则可在Consumer查看到接收的消息

4.常见问题

​ 如果碰到ZooKeeper、Kafka容器无法正常启动，可以删除数据卷以及容器后重新进行创建

# 停用&删除容器
docker stop zookeeper-test kafka-test
docker rm zookeeper-test kafka-test

# 删除数据卷
docker volume rm zookeeper_vol kafka_vol
docker volume rm kafka-group_kafka_vol kafka-group_zookeeper_vol