UH Message Broker

Owned by Julio Polo
Last updated: Dec 13, 2023

Table of Contents

Overview

The UH Message Broker provides a messaging service for UH applications. It uses RabbitMQ behind the scenes, which implements the AMQP protocol.

Applications can publish messages that may be of interest to others to the UH Message Broker. Typically, these are XML or JSON messages, and the publisher is often the authoritative source of the data being published. For example, Banner could publish a <dropCourse> message that MyUH and Laulima listen to in order to update a class list.

Another advantage of messaging is that there is no need to pre-establish a relationship between applications to define a custom format for the data being exchanged, the delivery method, the credentials to be used during the transfer, and so on. The publisher only needs to define the messages it publishes, and the rest is all managed by the UH Message Broker.

Environments

Setting

Value

Comments

Host

esb.hawaii.edu (production)
esb-test.its.hawaii.edu (test)

ESB stands for Enterprise Service Bus, our eventual goal

Port

5671

See TLS setting in table below

General Info

  • Producers publish messages to an exchange
    • There are several types of exchanges: direct, fanout and topic.
    • We expect most systems of record will be publishing to a topic exchange:
      • Messages in a topic exchange have a message routing key, e.g.
        • person.add
        • affiliation.add
        • affiliation.delete
      • This allows each consumer to specify the type of message it wants, e.g.
        • affiliation.# to get only affiliation messages
  • Consumers get messages from queues that are bound to an exchange
    • Typically, there is a one-time set up of a queue where:
      • The queue is created
      • The type of messages it receives are defined via bindings, for example:
        • queue is bound to foo-exchange with routing key # (queue gets all messages published to the foo-exchange)
        • queue is bound to bar-exchange with routing key person.# (queue only gets person messages published to the bar-exchange)

Broker Settings

ItemDesired SettingComments
User, Password(app-specific)You application will need an account to connect to the UH Message Broker
Vhost(app-specific, e.g. uhims)Vhost stands for virtual host, a logical host name that makes it easier to move services should we migrate to another physical host.
Exchange(app-specific and usually only needed if publishing, e.g. uhims-exchange)

This is where messages are published.

Each consumer typically has its own queue hanging off an exchange in order to get messages that are published to that exchange.

RabbitMQ client

The oldest Java client you should use is 3.6.6.

Although we expect older clients to work, we recommend that you upgrade to the latest client


TLS

Use TLS 1.2 or 1.3

Your code should verify the cert chain presented by our broker.

Your code should verify that our server cert is for the hostname of the broker to which you connected.

Please refer to TLS and RabbitMQ Java Client.

The RabbitMQ client will display this warning unless you implement peer and hostname verification:

WARN [localhost-startStop-1] com.rabbitmq.client.TrustEverythingTrustManager.<init> SECURITY ALERT: this trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but offers no protection against man-in-the-middle attacks. Please see https://www.rabbitmq.com/ssl.html to learn more about peer certificate verification.


AMQP heartbeat60 seconds or less.

You need to set the AMQP heartbeat on your client settings to 60 seconds so that it matches the server's expectations. Otherwise, the server may think that you are no longer connected.

Here's a Java example: https://www.rabbitmq.com/heartbeats.html#using-heartbeats-in-java

This smaller heartbeat value will generate network traffic every 60s, thus preventing network devices from dropping your connection when it is idle.

Publish confirmsStrongly recommended if you publish messages.If you are publishing messages,  you should enable publish confirms or risk not getting an error if a message was not successfully published.
Handling dropped broker connections

If your RabbitMQ client does not already do this for you, you should have code that handles dropped connections to the broker.  The code should repeatedly attempt to reconnect until it's successful, and if applicable, retry the interrupted operation.


Don't worry too much about this if you connect-consume-and-disconnect on a daily or hourly basis.

WARNING: if you have your own code that reconnects, check whether your version of the RabbitMQ client is also capable of reconnecting.  You should only have one entity perform the handling of lost connections, otherwise you may end up with multiple re-connections for each connection that was lost.

This is how the RabbitMQ client recovers from network failures:
https://www.rabbitmq.com/api-guide.html#recovery

Note: It is possible to have a lost connection in a "zombie" state while at the same time having a newly created recovery connection.  To speed up the clean up of the zombie connection, you should set the recovery retry interval to 3 times the heartbeat value.   To further reduce the likelihood of zombies, you could set the heartbeat to 30 seconds and the recovery retry interval to 90s.  See also https://groups.google.com/forum/#!topic/rabbitmq-users/7AZz4Nr0_Rk

High availability

The broker should come back within 16 seconds if any of its nodes fails.

If you set your dropped connection retry interval to 16 seconds, that should result in a successful re-connection after any dropped connection.

The load balancer health check is every 5 seconds, with 3 failures triggering the switch to another cluster node.
Consumer prefetch

If you are experiencing a connection reset after reading over a thousand messages without ack, try adding this line to your code:

channel.basicQos(10);

Please refer to https://www.rabbitmq.com/consumer-prefetch.html

If your application uses basicConsume or similar (as opposed to basicGet) to read thousands of messages *before* eventually acknowledging them, you may want to consider adding this line of code.

UHIMS Events

The UH Identity Management Service (UHIMS) publishes messages to the UH Message Broker.  Consuming UHIMS Events can be a convenient way to receive identity, affiliation, and contact information about the students, faculty and staff that your application serves.

Note that UHIMS Events uses a topic exchange.

Download Client Software

To interact with the UH Message Broker, download the RabbitMQ client of your choice.

Typical Sequence for Producer

  1. One-time set up:
    1. Obtain info about the exchange you will publish to or ask us to create a new one
  2. Create a broker connection
  3. Obtain a channel using that connection
  4. Publish message (with a message routing key) to the exchange using that channel

See Publishing Messages to the UH Message Broker

Typical Sequence for Consumer

  1. One-time set-up:
    1. Find out which exchange has the messages you are interested in consuming.
    2. Ask us to create a queue for you on that exchange.
      1. We typically use these settings: durable is true, everything else is NOT set: exclusive owner, auto delete, message TTL, auto expire, max length, dead letter exchange, dead letter routing key, arguments.
    3. Tell us if you are interested in some or all messages being published to that exchange.
      1. We will create appropriate bindings for your queue.  For example, binding with person.# means your queue will only receive messages whose routing key begins with 'person'
  2. Create a broker connection
  3. Obtain a channel using that connection
  4. Consume message(s) from the queue with autoAck set to false
    1. Setting autoAck to false requires the consumer to explicitly acknowledge consumption after successfully processing the message.
    2. See "Message acknowledgement" section in http://www.rabbitmq.com/tutorials/tutorial-two-java.html for more info.
  5. Acknowledge the consumed message(s) after they've been successfully processed.
    1. If the consumer crashes, and the channel, connection or network connection is lost, the unacknowledged messages automatically go back to the queue.
    2. If the consumer does not crash, but is not successful in processing the message (e.g. its database is unavailable), it can put back the unacknowledged messages back in the queue by issuing a recover call
    3. The ack and recover calls are described at http://www.rabbitmq.com/amqp-0-9-1-quickref.html

See Consuming Messages from the UH Message Broker