Vision model endpoint on CloudFerro Cloud
The vision model endpoint is used when a request contains both text and an image. It allows a vision-capable model to analyse the attached image and answer the prompt that accompanies it. Use this endpoint for image description, screenshot analysis, visual inspection, document review, and other workflows where the model needs to combine visual input with written instructions.
Sherlock AI exposes vision requests through the OpenAI-compatible chat completion endpoint. The image is passed inside the messages list as a Base64-encoded data URL.
Endpoint
Send a POST request to the chat completions endpoint:
https://api-sherlock.cloudferro.com/openai/v1/chat/completions
The request must include the Sherlock AI API key in the Authorization header. The text prompt and the image are sent together in the request body.
Prerequisites
No. 1 A configured client and project API key
Complete the client installation and API-key configuration in Get started with the Sherlock API. Have SHERLOCK_API_KEY set in the terminal where you will run the examples, with curl or the OpenAI Python client available for your chosen method.
No. 2 A vision-capable model and a local image
Choose a vision-capable model using AI models and verify its ID with the models endpoint. Have a readable JPEG image on your computer; the examples also explain the change needed for a PNG file.
Important
If no vision-capable model is returned, the project currently cannot run image analysis requests and the model must be enabled first.
Configure the model and image path
Keep SHERLOCK_API_KEY from the client setup. Set the returned vision model ID and local image path in the same terminal.
In Bash on Linux or macOS:
export SHERLOCK_VISION_MODEL="paste-available-vision-model-id-here"
export SHERLOCK_IMAGE_PATH="/full/path/to/your/image.jpg"
In Windows PowerShell:
$env:SHERLOCK_VISION_MODEL="paste-available-vision-model-id-here"
$env:SHERLOCK_IMAGE_PATH="C:\full\path\to\your\image.jpg"
Image input format
The image is sent as a Base64-encoded data URL. For a JPEG file, the value has the following form:
data:image/jpeg;base64,<base64-encoded-image>
For PNG files, use data:image/png;base64, instead. Make sure that the MIME type matches the file you attach; otherwise, the model may reject the request or fail to process the image correctly.
Send a vision request with Python
The following script reads the image path from SHERLOCK_IMAGE_PATH, encodes the file as Base64, and sends it together with a text prompt to the model stored in SHERLOCK_VISION_MODEL.
import base64
import os
from pathlib import Path
from openai import OpenAI
BASE_URL = "https://api-sherlock.cloudferro.com/openai/v1"
api_key = os.environ["SHERLOCK_API_KEY"]
vision_model = os.environ["SHERLOCK_VISION_MODEL"]
image_path = Path(os.environ["SHERLOCK_IMAGE_PATH"])
client = OpenAI(
api_key=api_key,
base_url=BASE_URL,
)
def encode_image(path: Path) -> str:
"""Return the image file encoded as a Base64 string."""
if not path.is_file():
raise FileNotFoundError(f"Image file not found: {path}")
with path.open("rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image(image_path)
messages = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant that analyses images.",
}
],
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what you see in this image.",
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": "high",
},
},
],
},
]
chat_response = client.chat.completions.create(
model=vision_model,
messages=messages,
)
print(chat_response.choices[0].message.content)
The example assumes that SHERLOCK_IMAGE_PATH points to a JPEG file. If you use a PNG file, change the data URL prefix in the image_url value to data:image/png;base64,.
Install Base64 support for curl examples
The curl example uses the base64 command to encode the image before sending the request. On many Linux systems, this command is already installed as part of coreutils. On Debian or Ubuntu, install it with:
sudo apt update
sudo apt install coreutils
Verify that the command is available:
base64 --version
Send a vision request with curl
The following shell example reads the image path from SHERLOCK_IMAGE_PATH, encodes the local image, inserts it into the JSON request body, and sends the request to the chat completions endpoint.
#!/usr/bin/env bash
set -euo pipefail
API_KEY="${SHERLOCK_API_KEY:?Set SHERLOCK_API_KEY first}"
VISION_MODEL="${SHERLOCK_VISION_MODEL:?Set SHERLOCK_VISION_MODEL first}"
IMAGE_PATH="${SHERLOCK_IMAGE_PATH:?Set SHERLOCK_IMAGE_PATH first}"
BASE64_IMAGE=$(base64 "$IMAGE_PATH" | tr -d '\n')
curl https://api-sherlock.cloudferro.com/openai/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d @- <<EOF
{
"model": "$VISION_MODEL",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,$BASE64_IMAGE",
"detail": "high"
}
}
]
}
]
}
EOF
The command assumes that SHERLOCK_API_KEY, SHERLOCK_VISION_MODEL, and SHERLOCK_IMAGE_PATH are already set in the same terminal session. It also assumes that the input image is a JPEG file. For PNG files, change the data URL prefix to data:image/png;base64,.
Use vision prompts effectively
Vision requests work best when the prompt clearly states what the model should inspect. For general understanding, ask for a description of the image. For operational use, specify the expected output, such as a list of visible objects, a summary of a screenshot, detected issues, or a structured answer.
Avoid sending sensitive images unless your workflow is designed to handle them securely. Treat screenshots, documents, maps, credentials, identity documents, and internal dashboards as sensitive data, especially when they contain names, addresses, tokens, project identifiers, or other private information.
Troubleshooting
If the request returns an authentication error, check that SHERLOCK_API_KEY is set in the same terminal session where you run Python or curl. If the model is rejected, verify that SHERLOCK_VISION_MODEL contains a model ID returned by the models endpoint for your project.
If the image is not processed, check that SHERLOCK_IMAGE_PATH points to an existing file, that the current user can read the file, and that the data URL uses the correct MIME type. If the curl example fails during Base64 encoding, run base64 –version and verify that coreutils is installed.
What to do next
Continue with the OCR Playground guide.
Continue with the consumption and events guide.