安装部署

添码座原创大约 2 分钟

Flume的官方镜像在执行docker pull apache/flume命令时出现下面的错误。

Using default tag: latest
Error response from daemon: manifest for apache/flume:latest not found: manifest unknown: manifest unknown

官方也没有任何tags可用，而且它本身也只是一个数据采集工具而已，所以使用本机部署的方式安装可能更适合Flume。

提前准备虚拟机，安装好Java环境。

先下载并解压Flume安装包。

> cd /home/work
> wget https://dlcdn.apache.org/flume/1.11.0/apache-flume-1.11.0-bin.tar.gz
> tar xvf apache-flume-1.11.0-bin.tar.gz
> mv apache-flume-1.11.0-bin flume-1.11.0

修改flume-1.11.0/conf/flume-env.sh.template配置文件。

> cd /home/work/flume-1.11.0/conf/
> cp flume-env.sh.template flume-env.sh
> vi flume-env.sh

# 取消注释，并且修改JAVA_HOME为正确的路径
export JAVA_HOME=/usr/local/java/jdk1.8.0_401

配置一个Agent。

> cd /home/work/flume-1.11.0/conf/
> cp flume-conf.properties.template flume.conf
> vi flume.conf

# 指定source和sink到channel
a1.sources = r1
a1.channels = c1
a1.sinks = k1


# 配置source
a1.sources.r1.type = netcat
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 13790


# 配置channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


# 配置sink
a1.sinks.k1.type = logger


# 绑定source和sink到channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Flume官方文档中提供了关于Source、Channel和Sink的详细说明和相应实例。

启动Flume并执行下面的命令。

> cd /home/work/flume-1.11.0/
> nohup ./bin/flume-ng agent -n a1 -c conf -f conf/flume.conf -Dflume.root.logger=INFO,console > /dev/null 2>/home/work/logs/flume-error.log &
# 或者
> nohup ./bin/flume-ng agent --name a1 --conf conf --conf-file conf/flume.conf -Dflume.root.logger=INFO,console > /dev/null 2>/home/work/logs/flume-error.log &

启动后，执行jps -m命令可以看到多出了一个Application进程。

151155 Application -n a1 -f conf/flume.conf
62437 SecondaryNameNode
62854 NodeManager
62039 NameNode
62218 DataNode
62699 ResourceManager

然后按照官方的示例，通过telnet命令连接调试。

> telnet 172.16.185.176 13790
Trying 172.16.185.176...
Connected to 172.16.185.176.
Escape character is '^]'.
Hello Flume!
OK

如果不设置的话，日志文件默认存放在${FLUME_HOME}目录下，文件名为flume.log。

> cd /home/work/flume-1.11.0/
> tail -f flume.log
INFO  [main] (org.apache.flume.node.Application.startAllComponents:207)  - Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3514a4c0 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
INFO  [main] (org.apache.flume.node.Application.startAllComponents:214)  - Starting Channel c1
INFO  [main] (org.apache.flume.node.Application.startAllComponents:229)  - Waiting for channel: c1 to start. Sleeping for 500 ms
INFO  [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:119)  - Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
INFO  [lifecycleSupervisor-1-0] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:95)  - Component type: CHANNEL, name: c1 started
INFO  [main] (org.apache.flume.node.Application.startAllComponents:241)  - Starting Sink k1
INFO  [main] (org.apache.flume.node.Application.startAllComponents:252)  - Starting Source r1
INFO  [lifecycleSupervisor-1-1] (org.apache.flume.source.NetcatSource.start:155)  - Source starting
INFO  [lifecycleSupervisor-1-1] (org.apache.flume.source.NetcatSource.start:166)  - Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:13790]
INFO  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.LoggerSink.process:95)  - Event: { headers:{} body: 48 65 6C 6C 6F 20 46 6C 75 6D 65 21 0D          Hello Flume!. }

感谢支持

更多内容，请移步《超级个体》。