Use Message Broker with other products on CloudFerro Cloud
A queue is useful when the system accepting work and the system processing it need to run independently. You can keep the data in Object Storage, send small work requests through RabbitMQ, and choose VMs, containers, workflows, or functions to carry out the processing.
Choose where the work runs
These are examples you can implement with application code and the products’ connection settings:
OpenStack virtual machines and Private Object Storage: a producer stores input files in a bucket and sends their object keys in messages. Workers on VMs receive the jobs, download the files, and save results to Object Storage.
Managed Kubernetes: run message consumers in containers. Several worker replicas can share a task queue, letting you increase processing capacity as the backlog grows.
Workflows: an Airflow task publishes work requests after a scheduled data-preparation stage. Independent workers consume those requests, so the schedule and the worker deployment can evolve separately.
Functions: configure a RabbitMQ trigger to run function code for incoming messages, such as validating metadata. A function can also publish a result through a RabbitMQ output.
Sherlock: queue translation or summarization requests and let a worker call the Sherlock API. The worker needs a Sherlock project API key in addition to its RabbitMQ credentials; the broker does not call the model itself.
Try Process queued text jobs locally and with Sherlock on CloudFerro Cloud for a downloadable example. It starts with local text processing and then translates queued requests through Sherlock, saving a result before acknowledging each delivery.
VM and Kubernetes applications connect with a RabbitMQ client, the broker endpoint, and credentials. Functions uses the trigger and output configuration in My Portal. Message Broker can also be used independently of Functions and Message Streaming.
Prepare the integration
For a VM, Kubernetes consumer, or Airflow task, the connection requirements are the same as for another application: the broker’s AMQPS endpoint, a RabbitMQ account, the correct vhost, and access from the worker’s public outbound IP range. The application also needs permission to read or write any input and output storage it uses. RabbitMQ credentials do not grant access to Object Storage or provision the worker’s computing resources.
Pass the object key and enough information for the worker to find the input, rather than assuming a message transfers the input file. For example, an image-processing request can name the input bucket, object key, and operation. The worker obtains that object using its own storage credentials and records the result where the application expects it.
Functions requires a broker and a queue that already exist. Its trigger configuration uses the broker, queue, vhost, and RabbitMQ credentials; prepare the exchange as well if the integration uses one. Check that the function and messaging resource are in the same region. Region selection is stored separately for each service, so changing the Functions selector does not change the broker selector.
Use Test configuration for the RabbitMQ integration, then send a test event and inspect the function result and logs. A connection test and a successful business operation are separate checks.
What to do next
Translate dataset descriptions with Functions and Sherlock on CloudFerro Cloud combines a RabbitMQ trigger, a Python function, Sherlock translation, and results in Object Storage.
Connect an application and send a first message on CloudFerro Cloud checks the client endpoint and credentials.
Manage RabbitMQ queues and access on CloudFerro Cloud prepares the vhost, queue, and access rules.
To use a queue with a function, continue with Configure function triggers and outputs on CloudFerro Cloud.
Workflows user guide on CloudFerro Cloud starts the separate path for scheduled workflows. Publishing RabbitMQ messages from an Airflow task requires your task’s application code and connection settings.
Managed Notebooks on CloudFerro Cloud prepares a notebook environment for interactive experiments with a publisher or worker.
Container Registry user guide on CloudFerro Cloud explains where to store a worker container image. Deploy that image using Kubernetes on OpenStack Magnum when the application is ready to run in Kubernetes.