Translate dataset descriptions with Functions and Sherlock on CloudFerro Cloud

A dataset catalogue may need the same description in several languages. Copying each paragraph into a chat window becomes repetitive when descriptions arrive regularly or change over time. A function can send each description to Sherlock and save the translations alongside the original in Object Storage.

The example starts with a single HTTP request that translates a short description into Polish. You then add a RabbitMQ queue and translate the same source into Polish, German, and French. Each language has its own job, so you can inspect or repeat one translation without submitting the others again.

How the products work together

The processing sequence is:

Publisher on your desktop, VM, or notebook
    |-- uploads the English description --> Object Storage: inputs/
    |-- publishes one job per language ---> Message Broker: translation-jobs
                                               |
                                        RabbitMQ trigger
                                               |
                                            Functions
                                               |-- reads the source object
                                               |-- asks Sherlock to translate it
                                               `-- saves JSON --> Object Storage: translations/

Object Storage holds the source and completed results. Message Broker holds small requests containing the source object key and target language. Functions runs the processing code, while Sherlock generates the translation. Uploading an object alone does not start this example: the publisher explicitly sends the queue messages after uploading the source.

You can use the direct HTTP request without a broker or bucket. Add the queued stage when you need to process several descriptions, submit work from another application, or keep the publisher independent of the time taken by the model.

Prerequisites

No. 1 An active Functions namespace

Follow Prepare your first function on CloudFerro Cloud and Create and deploy a function on CloudFerro Cloud to prepare portal access and a namespace. This example uses the Python HTTP template with Python 3.13. Keep the namespace name and selected region available; the example function name is translate-descriptions.

Use Configure function resources and scaling on CloudFerro Cloud for secrets, environment variables, resource limits, and timeout settings. Test a function and manage its endpoints on CloudFerro Cloud explains the portal test panel and endpoint access. The function needs outbound HTTPS access to Sherlock and, for queued jobs, your S3 endpoint.

No. 2 A Sherlock API key and a suitable chat model

Create a project key using Authenticate to Sherlock AI with API keys on CloudFerro Cloud. Select a model available to that project using Models endpoint on CloudFerro Cloud, and check that it handles the languages you need. Keep the exact model ID and sufficient project quota for the test requests. The handler uses the Sherlock chat completion endpoint on CloudFerro Cloud.

No. 3 An S3 bucket and credentials for the queued stage

Prepare an S3-compatible bucket, endpoint, signing region, and access keys using How to access object storage from CloudFerro Cloud using boto3. Replace REPLACE_WITH_YOUR_BUCKET below with a bucket you control. The example keeps inputs and outputs in separate prefixes within that bucket.

The publisher needs permission to write below inputs/ and read results below translations/. The function needs permission to read both prefixes and write below translations/. Allow the result-existence check to distinguish a missing object from denied access; with S3 permissions this normally also requires bucket-list permission for the result prefix. A Sherlock key grants no storage access. The bucket remains private.

No. 4 A Message Broker resource for the queued stage

Prepare a broker in the same region as the function using Prepare access and choose a broker on CloudFerro Cloud. Follow Manage RabbitMQ queues and access on CloudFerro Cloud to prepare a vhost, a durable queue named translation-jobs, and RabbitMQ accounts. The publisher must be able to inspect and publish to that queue; the trigger account must be able to consume from it. Use accounts scoped to this example’s vhost.

Obtain the AMQPS hostname and credentials, and allow the publisher’s outbound IP using Connect an application and send a first message on CloudFerro Cloud. The managed integration must also be able to reach the broker. Configure function triggers and outputs on CloudFerro Cloud covers the trigger fields and connection test. Check the region separately in Functions and Message Broker.

No. 5 A Linux desktop or VM for the publisher

The commands below use Bash and Python 3.13 or later. Prepare an isolated environment with the official Python venv instructions. Install the downloadable requirements in that environment; they provide boto3 for Object Storage and Pika for RabbitMQ. The optional local HTTP test also uses curl; install it through your Linux package manager or the official curl downloads. The portal runs the deployed handler separately from this computer.

Download the example

Download the complete example files and extract them into an empty working directory. The bundle contains:

File

Purpose

func.py and __init__.py

The function handler and its Python template entry point.

submit_job.py

Upload the source, prepare job files, publish them, and retrieve results.

sample-description.txt

An invented dataset description for the exercise.

requirements.txt

Packages for the publisher and local checks.

test_func.py and test_local_flow.py

Offline checks of validation, error handling, result reuse, and HTTP processing.

run_local.py

A development HTTP server bound to the local computer.

build_function.py

Rebuild the function source archive after changing the handler.

The separate function source archive contains function/func.py, function/__init__.py, tests/test_func.py, and pyproject.toml. It follows the Python HTTP template’s new() and handle() interface and includes boto3 as a dependency. Upload this source archive to Functions; keep the publisher and local test tools on your computer.

Create the function and test one translation

  1. Open your namespace in Functions, select Create Function, and choose Upload ZIP. Upload translation-function.zip and select Python 3.13 with a compatible Python HTTP template version from the catalogue.

  2. Name the function translate-descriptions. Use container port 8080 and function path /. Start with a maximum of one replica to keep the first test small; this setting alone does not prevent concurrent requests.

  3. Configure SHERLOCK_MODEL as an environment variable containing your selected model ID. Add SHERLOCK_API_KEY through the function’s secrets settings. Keep the key out of the source archive, request body, and logs.

  4. Set a function timeout that accommodates the model request and storage operations. The handler uses a 45-second timeout for the Sherlock HTTP request and short timeouts for S3 calls; the function’s overall timeout needs additional margin within the limits offered by the service. Choose a suitable flavor, build the version, wait for the build to succeed, and select Deploy.

  5. Open Test and send an HTTP POST with content type application/json and this body:

    {
  "text": "Dataset S2-DEMO-001 contains Sentinel-2 observations acquired on 2026-09-15. The spatial resolution is 10 m.",
  "target_language": "Polish"
}

  6. Check for HTTP 200 and a response containing target_language, translation, and model. An illustrative translation is:

    Zbiór danych S2-DEMO-001 zawiera obserwacje Sentinel-2 pozyskane
2026-09-15. Rozdzielczość przestrzenna wynosi 10 m.

The wording varies with the model. Check that S2-DEMO-001, Sentinel-2, 2026-09-15, and 10 m retain their meaning and values. This direct request returns the translation in the HTTP response; it does not create a storage object. Each new direct request calls Sherlock again.

Configure storage and prepare a job

Add the following configuration to the function before sending storage jobs. Use the endpoint and signing region supplied for your Object Storage service; the signing region is not necessarily the region label shown in My Portal.

Setting

Value and purpose

S3_ENDPOINT_URL

Your HTTPS S3 endpoint, without a bucket or object path.

AWS_DEFAULT_REGION

The S3 signing region required by that endpoint.

TRANSLATION_BUCKET

The existing bucket for inputs and completed translations.

AWS_ACCESS_KEY_ID

Store the function’s storage access key as a secret.

AWS_SECRET_ACCESS_KEY

Store the matching storage secret key as a secret.

AWS_SESSION_TOKEN

Add as a secret only if your storage credentials require a session token.

Apply the configuration and deploy the updated version if the portal requires a new deployment. The function always uses this configured bucket; a queue message cannot choose a different bucket or endpoint.

On your Linux computer, change to the directory containing the extracted example files and install the publisher packages:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Set the publisher’s storage configuration. Replace all REPLACE_… values before running the commands. The hidden prompts keep credential values out of the command text stored in shell history:

export S3_ENDPOINT_URL='https://REPLACE_WITH_YOUR_S3_HOST'
export AWS_DEFAULT_REGION='REPLACE_WITH_YOUR_S3_SIGNING_REGION'
export TRANSLATION_BUCKET='REPLACE_WITH_YOUR_BUCKET'
read -rsp 'Storage access key: ' AWS_ACCESS_KEY_ID; echo
export AWS_ACCESS_KEY_ID
read -rsp 'Storage secret key: ' AWS_SECRET_ACCESS_KEY; echo
export AWS_SECRET_ACCESS_KEY

If the credentials include a session token, set AWS_SESSION_TOKEN using the same hidden-prompt pattern. The publisher does not need the Sherlock API key: the function makes the model request.

Prepare one Polish translation job:

python submit_job.py prepare \
  --file sample-description.txt \
  --dataset-id demo-001 \
  --languages Polish

This uploads the source under inputs/demo-001/ and writes a local JSON job file. The object key contains a SHA-256 checksum of the input, and the job ID includes its first 12 characters. With the supplied sample file, the job is:

{
  "job_id": "demo-001-pl-837e928fad8f",
  "source_key": "inputs/demo-001/837e928fad8f74145bbc7c67cd7a7a145cb21002d776a233a2dc8e495e19ca51/en.txt",
  "source_sha256": "837e928fad8f74145bbc7c67cd7a7a145cb21002d776a233a2dc8e495e19ca51",
  "target_language": "Polish"
}

The file contains no credentials or full source text. source_key locates the input, while source_sha256 lets the handler check that the retrieved content is the expected version. job_id determines the result filename. Changing the sample text, including its line endings, changes the checksum and generated filename; use the filename printed by the script.

Before adding RabbitMQ, paste the generated JSON into the function’s Test panel and send it as an HTTP POST with content type application/json. A successful response contains status: completed and a result key such as translations/demo-001-pl-837e928fad8f.json. This checks the function’s storage access and Sherlock call independently of the messaging connection.

Retrieve the saved result from your computer:

python submit_job.py result demo-001-pl-837e928fad8f.json

The result JSON contains the translated text, language, model ID, source key, checksum, and job ID. Its fingerprint identifies the complete job request for later duplicate checks. Confirm that the translation is present before continuing.

Connect the RabbitMQ trigger

  1. Open Triggers and output for translate-descriptions and select Add trigger.

  2. Choose Message Broker. Select your broker and enter the prepared vhost, translation-jobs queue, and trigger account credentials. Match the existing queue exactly, including case.

  3. Run Test configuration, save the trigger, and check that the integration is active. If the form offers retry or dead-letter settings, configure bounded retries and a destination for failed jobs before using a larger batch.

  4. Keep the function private for this test. The managed trigger delivers the messages; the desktop publisher connects to RabbitMQ and does not need a public function endpoint. No function output integration is needed because the handler writes the result to Object Storage itself.

The publisher sends JSON with content type application/json. The handler accepts a JSON job directly, a binary HTTP CloudEvent whose body is that job, or a structured CloudEvent with the job in data. It does not decode data_base64 or a string containing another JSON document. Check the delivery format if the trigger connects but the function rejects the event.

Publish jobs for three languages

Set the connection details on the publisher computer. RABBITMQ_HOST is the hostname from the AMQPS endpoint, without amqps://, credentials, or a path. The example uses port 5671, TLS certificate verification, and the default vhost /; replace the vhost or port if your connection details differ.

export RABBITMQ_HOST='REPLACE_WITH_YOUR_BROKER_HOST'
export RABBITMQ_PORT='5671'
export RABBITMQ_VHOST='/'
export RABBITMQ_QUEUE='translation-jobs'
read -rp 'RabbitMQ username: ' RABBITMQ_USER
export RABBITMQ_USER
read -rsp 'RabbitMQ password: ' RABBITMQ_PASSWORD; echo
export RABBITMQ_PASSWORD

Prepare a fresh dataset ID for the queued test so that its Polish translation has not already been completed by the direct storage test:

python submit_job.py prepare \
  --file sample-description.txt \
  --dataset-id demo-002 \
  --languages Polish German French

python submit_job.py publish \
  demo-002-pl-837e928fad8f.json \
  demo-002-de-837e928fad8f.json \
  demo-002-fr-837e928fad8f.json

The publisher checks that the queue exists, enables publisher confirms, and sends persistent messages through RabbitMQ’s default exchange with translation-jobs as the routing key. It reports each broker confirmation. A confirmation means the broker accepted the message; it does not mean that Sherlock has completed the translation.

Inspect the queue and the function’s logs, then retrieve the three results:

python submit_job.py result demo-002-pl-837e928fad8f.json
python submit_job.py result demo-002-de-837e928fad8f.json
python submit_job.py result demo-002-fr-837e928fad8f.json

Allow time for the trigger delivery, a possible cold start, and model processing. Three JSON objects below translations/ are the completion evidence. A queue count of zero alone is insufficient: failed messages may have been removed or sent to a dead-letter destination, depending on the integration’s configuration.

The function sends HTTP 200 only after saving a completed result. Check the managed trigger’s response handling and retry settings with a controlled failed job before relying on automatic recovery. Publisher confirms and the function’s HTTP response are separate parts of the delivery process.

Repeat a job and understand the limits

Publish the same Polish job again:

python submit_job.py publish demo-002-pl-837e928fad8f.json

For a completed, unchanged job, the handler finds the existing result and responds with status: already_completed without another model request. You can observe this response directly by submitting the same job JSON through Test. Reusing the job ID with a different language, source key, or checksum returns HTTP 409 instead of overwriting the completed result.

This result check does not guarantee that Sherlock runs exactly once. Two simultaneous deliveries can both start before a result exists. A failure after the model answers but before the result is saved can also lead to another paid model call on retry. For production workloads that need stronger coordination, add a shared job-state store with an atomic claim and recovery policy. A limit of one replica is not a lock.

Completed results remain tied to their job IDs even if you later change SHERLOCK_MODEL or the translation prompt. Use a new dataset ID or job ID when you want to regenerate translations. The example accepts only short UTF-8 text, up to 12,000 bytes, and requests at most 2,048 output tokens. Split longer descriptions if a response is truncated; the handler rejects incomplete completions rather than saving them as finished translations.

Check the code locally

Run the supplied checks from the extracted example directory:

python -m unittest -v test_func test_local_flow

These checks use a simulated storage service and deterministic Sherlock responses. The local flow test uploads an input into that test store, prepares three jobs, sends them through a real loopback HTTP server, retrieves the results, and repeats each request. It verifies processing behavior without spending Sherlock quota. It does not connect to RabbitMQ, deploy a function, or assess translation quality.

To call a real Sherlock model from the local development server, configure the model and key in the terminal running the server:

export SHERLOCK_MODEL='REPLACE_WITH_YOUR_MODEL_ID'
read -rsp 'Sherlock API key: ' SHERLOCK_API_KEY; echo
export SHERLOCK_API_KEY
python run_local.py

From another terminal, send a direct request:

curl --fail-with-body http://127.0.0.1:8080/ \
  -H 'Content-Type: application/json' \
  --data '{"text":"Dataset S2-DEMO-001 has a spatial resolution of 10 m.","target_language":"Polish"}'

The local server makes a real Sherlock request with the key from its environment. Storage jobs additionally require the storage settings described above. This server is for local development and listens only on the same computer; stop it with Ctrl+C when you finish.

After editing the handler, run python build_function.py to regenerate translation-function.zip. Upload the new archive, build a new function version, and deploy it explicitly. Re-run the direct and queued checks against that version.

Troubleshooting

The build cannot install or import boto3. Check that you uploaded the function source archive and selected a compatible Python HTTP runtime and template. The archive’s project dependencies include boto3; the publisher’s virtual environment does not install packages into the deployed function.

HTTP 400, 413, or 415. Check the job fields, target language, request content type, and source size. This example accepts Polish, German, and French. Send the job as a JSON object, including when it is carried in a CloudEvent.

HTTP 503. A required environment variable or secret is missing or still contains a placeholder. Direct translation requires the Sherlock key and model; storage jobs also require the bucket and storage connection settings.

HTTP 502 mentioning Sherlock. Check the reported upstream HTTP status, model availability, project quota, and outbound connectivity. An authentication failure, rate limit, timeout, or incomplete completion does not produce a saved translation. The handler returns a short diagnostic without exposing the upstream response body or API key.

HTTP 502 reporting processing failure. Check the storage credentials, bucket, endpoint, and permissions. The function logs the exception class, not the source text or secret values. A denied result lookup is an error, not evidence that the result is missing. If source retrieval fails, confirm that the publisher’s upload completed and the requested object still exists.

HTTP 409. The job ID was already completed for different input, or the source checksum no longer matches the object. Prepare a new job from the intended source and use its generated ID. Do not edit a stored input in place while jobs refer to it.

Jobs remain in the queue or no result appears. Check the trigger’s connection test, active consumer, region, vhost, queue name, and account permissions. Inspect function errors and any dead-letter destination. Resolve the failure before resubmitting the same job files; do not create a second queue with a similar name as a workaround.

Use the example in a larger application

A Managed Notebook can prepare descriptions interactively, publish selected jobs, and compare the returned translations. A Workflows task can run the publisher after a scheduled catalogue export, then check for the expected result keys before publishing the updated catalogue. Both use application code and credentials; these connections are not enabled automatically by creating the function.

For a larger catalogue, Managed Databases can hold dataset IDs, target languages, job states, and result object keys while Object Storage keeps the longer text. Applications can use that state to show pending, completed, and failed translations. Use a transaction or another atomic operation if the database also coordinates concurrent workers.

You can adapt the same flow to create a short summary or extract structured metadata by changing the prompt, result schema, and validation. Translation quality still needs review before the generated text becomes public catalogue content, especially where a term, number, or unit changes the meaning.

What to do next

Still need help?
If this article doesn’t answer all of your questions, our support team will do their best to help you.
Contact support