(For more resources related to this topic, see here.)

A typical configuration might look something as follows:

To use the Avro Source, you specify the type property with a value of avro. You need to provide a bind address and port number to listen on:

collector.sources=av1
collector.sources.av1.type=avro
collector.sources.av1.bind=0.0.0.0
collector.sources.av1.port=42424
collector.sources.av1.channels=ch1
collector.channels=ch1
collector.channels.ch1.type=memory
collector.sinks=k1
collector.sinks.k1.type=hdfs
collector.sinks.k1.channel=ch1
collector.sinks.k1.hdfs.path=/path/in/hdfs

Here we have configured the agent on the right that listens on port 42424, uses a memory channel, and writes to HDFS. Here I’ve used the memory channel for brevity of this example configuration. Also, note that I’ve given this agent a different name, collector, just to avoid confusion.

The agents on the left—feeding the collector tier—might have a configuration similar to this. I have left the sources off this configuration for brevity:

client.channels=ch1
client.channels.ch1.type=memory
client.sinks=k1
client.sinks.k1.type=avro
client.sinks.k1.channel=ch1
client.sinks.k1.hostname=collector.example.com
client.sinks.k1.port=42424

The hostname value, collector.example.com, has nothing to do with the agent name on that machine, it is the host name (or you can use an IP) of the target machine with the receiving Avro Source. This configuration, named client, would be applied to both agents on the left assuming both had similar source configurations.

Since I don’t like single points of failure, I would configure two collector agents with the preceding configuration and instead set each client agent to round robin between the two using a sink group. Again, I’ve left off the sources for brevity:

client.channels=ch1
client.channels.ch1.type=memory
client.sinks=k1 k2
client.sinks.k1.type=avro
client.sinks.k1.channel=ch1
client.sinks.k1.hostname=collectorA.example.com
client.sinks.k1.port=42424
client.sinks.k2.type=avro
client.sinks.k2.channel=ch1
client.sinks.k2.hostname=collectorB.example.com
client.sinks.k2.port=42424
client.sinkgroups=g1
client.sinkgroups.g1=k1 k2
client.sinkgroups.g1.processor.type=load_balance
client.sinkgroups.g1.processor.selector=round_robin
client.sinkgroups.g1.processor.backoff=true

Summary

In this article, we covered tiering data flows using the Avro Source and Sink. More information on this topic can be found in the book Apache Flume: Distributed Log Collection for Hadoop.

Resources for Article :

Further resources on this subject:

• Supporting hypervisors by OpenNebula [Article]
• Integration with System Center Operations Manager 2012 SP1 [Article]
• VMware View 5 Desktop Virtualization [Article]