7 min read

(For more resources related to this topic, see here.)

Tracing RabbitMQ

Tracing the execution of a program is a convenient way to figure out what is really happening under the hood when reasoning about a particular behavior leads to no firm conclusion. Usually, the buck stops at the border where the application interacts with external resources such as the RabbitMQ broker. The good news is that RabbitMQ provides two tools that can be of tremendous help when it comes to tracing the interactions with a broker.

The first of these tools is Tracer, an AMQP-aware network proxy that can be placed between a RabbitMQ client and a broker in order to gain insight into the interactions that are happening between each other. Tracer is available as part of the Java client download available at http://www.rabbitmq.com/download.html.

After installation, Tracer can be started with the following:

runjava.sh com.rabbitmq.tools.Tracer [listenPort] [connectHost] [connectPort].



All the parameters are optional. If left blank, Tracer will start a local proxy listening on port 5673 and connect to a local RabbitMQ on port 5672. Since you’re happy with the defaults, you start Tracer with just the following command line:




$ ./runjava.sh com.rabbitmq.tools.Tracer




Now you can run the integration tests you’ve just created through this proxy. Do you remember that we made the connection information configurable on these tests? The approach is going to pay off now as we will configure them to go through the proxy port instead of directly hitting the RabbitMQ broker. You do this by running the following command line:


$ mvn -Pintegration_tests -Dtest.rmq.addresses=localhost:5673 verify




The output of Tracer is very verbose as it includes the complete details of the AMQP operations. Hereafter only the columns that show the channel ID, interaction direct (-> is client to broker and <- is opposite), and the name of the operation is reproduced, with the interactions related to the subscriber highlighted:


ch#0 <- <connection.start> ch#0 -> <connection.start-ok> ch#0 <- <connection.tune> ch#0 -> <connection.tune-ok> ch#0 -> <connection.open> ch#0 <- <connection.open-ok> ch#1 -> <channel.open> ch#1 <- <channel.open-ok> ch#1 -> <queue.declare> ch#1 <- <queue.declare-ok> ch#1 -> <channel.close> ch#1 <- <channel.close-ok>() ch#1 -> <channel.open> ch#1 <- <channel.open-ok> ch#1 -> <basic.consume> ch#1 <- <basic.consume-ok> ch#2 -> <channel.open> ch#2 <- <channel.open-ok> ch#2 -> <basic.publish> ch#2 -> <channel.close> ch#1 <- <basic.deliver> ch#2 <- <channel.close-ok> ch#1 -> <basic.cancel> ch#1 <- <basic.cancel-ok> ch#1 -> <channel.close> ch#1 <- <channel.close-ok> ch#0 -> <connection.close> ch#0 <- <connection.close-ok>




You can see the operations from the client and the responses from the broker, typically being named after the operation and suffixed with -ok. In essence, the following is the AMQP synopsis of the test code you’re running:


    

  • Establish a connection
    • 

  • Open a channel, use it to declare the test queue, and close it
    • 

  • Open a channel, use it to consume a queue
    • 

  • Open a channel, use it to publish the test message, and close it
    • 

  • Receive the message delivery, cancel the consumer, and close its channel
    • 

  • Close the connection
    • 



Notice how the connection start and tune operations are initiated by the broker as a response to establishing the connection to it. Also, notice that the channel number gets reused after being closed; it may seem that the same channel #1 has been used for creating the test queue and subscribing to it, but that’s not the case since this channel has been explicitly closed. Only its identifier has been reused.


Tracer is a very powerful tool to easily gain a deep understanding of the AMQP protocol and the usage your applications make of it. However, it requires you to insert a proxy between a client and the broker it connects to. Fear not if this is an issue; RabbitMQ has more than one trick in its bag of tracing tools.


Drinking at the Firehose


RabbitMQ offers the possibility of spying on all message publications and delivery operations that happens in a particular virtual host of a broker. This feature is called the Firehose tracer. When activated on a virtual host, a copy of all published and all delivered messages is sent to the amq.rabbitmq.trace exchange (which is automatically created in every virtual host).


The routing key used for messages published to the amq.rabbitmq.trace exchange is publish.<exchange_name> for publication events and deliver.<queue_name> for message deliveries. The original message body is carried to the copies sent to this exchange. Extra information about the original publication or delivery event are added in a set of headers, including exchange_name for the name of the exchange where the message was originally published or redelivered if the message has been delivered more than once.


You want to use the Firehose when running the integration tests to see the exchanged messages from the broker’s standpoint. Before activating the Firehose on RabbitMQ, you need first to create a client application that will subscribe to the exchange and print out the messages that come to it. For this, you create the following Python script:


#!/usr/bin/env python import amqp connection = amqp.Connection(host=’localhost’, userid=’ccm-dev’,

password=’coney123′, virtual_host=’ccm-dev-vhost’) channel = connection.channel() EXCHANGE = ‘amq.rabbitmq.trace’ QUEUE = ‘firehose-queue’ channel.queue_declare(queue=QUEUE, durable=False,

auto_delete=True, exclusive=True) channel.queue_bind(queue=QUEUE, exchange=EXCHANGE, routing_key=’#’) def handle_message(message): print message.routing_key, ‘->’, message.properties, message.body print ‘——————————–‘ channel.basic_consume(callback=handle_message, queue=QUEUE, no_ack=True) print ‘ [*] Waiting for messages. To exit press CTRL+C’ while channel.callbacks: channel.wait() channel.close() connection.close()




After starting this script, you turn the Firehose on by running the following command line:


$ sudo rabbitmqctl -p ccm-dev-vhost trace_on Starting tracing for vhost “ccm-dev-vhost” … …done.




Now you can run the integration tests again, this time on the standard port since no proxying is needed with the Firehose:


$ mvn -Pintegration_tests verify




Let’s now look at the following output of the Firehose consumer Python script:


publish. -> {‘application_headers’: {u’node’:

u’rabbit@pegasus’, u’exchange_name’: u”, u’routing_keys’:

[u’amq.gen-vTMWL–04lap8s8JPbX5gA’], u’properties’: {}}}

93b56787-b4f5-41e1-8c6f-d5f9b64275ca ——————————– deliver.amq.gen-vTMWL–04lap8s8JPbX5gA ->

{‘application_headers’: {u’node’: u’rabbit@pegasus’,

u’exchange_name’: u”, u’redelivered’: 0, u’routing_keys’:

u’amq.gen-vTMWL–04lap8s8JPbX5gA’], u’properties’: {}}}

93b56787-b4f5-41e1-8c6f-d5f9b64275ca ——————————–




As you can see, the publication to the default exchange (remember, its name is an empty string) and the delivery to the automatically named test queue are clearly visible. All the details that concern them are readily available in the message properties.


Keep in mind that running the Firehose is taxing for the RabbitMQ broker, so when you’re done with your tracing session, shut it down with the following:


$ sudo rabbitmqctl -p ccm-dev-vhost trace_off Stopping tracing for vhost “ccm-dev-vhost” … …done.




The Firehose will come handy when tracing what’s happening between your different applications and your RabbitMQ brokers in depth. Keep in mind that using unique message IDs, as you’ve learned throughout this article, will help you a lot when the time comes to perform forensics analysis and trace the progression of messages across your complete infrastructure.


Summary


In this article, you have discovered powerful tracing tools to peek deeper under the hood of the AMQP protocol and the RabbitMQ broker.


Resources for Article:




Further resources on this subject:







 











 
Packt
  






LEAVE A REPLY 





Please enter your comment!





Please enter your name here