Vision model endpoint on CloudFerro Cloud
The vision model endpoint is used when a request contains both text and an image. It allows a vision-capable model to analyse the attached image and answer the prompt that accompanies it. Use this endpoint for image description, screenshot analysis, visual inspection, document review, and other workflows where the model needs to combine visual input with written instructions.
Sherlock AI exposes vision requests through the OpenAI-compatible chat completion endpoint. The image is passed inside the messages list as a Base64-encoded data URL.
Endpoint
Send a POST request to the chat completions endpoint:
https://api-sherlock.cloudferro.com/openai/v1/chat/completions
The request must include the Sherlock AI API key in the Authorization header. The text prompt and the image are sent together in the request body.
Prerequisites
Before you start, make sure that you have access to a Sherlock AI project, that you have created and copied an API key, and that a vision-capable model is enabled for your project. You also need an image file available on your local system.
Store the API key, the model ID, and the image path in environment variables. This keeps the examples reusable and avoids repeating local file paths in the article.
On Linux or macOS, use:
export SHERLOCK_API_KEY="paste-your-api-key-here"
export SHERLOCK_VISION_MODEL="paste-available-vision-model-id-here"
export SHERLOCK_IMAGE_PATH="/full/path/to/your/image.jpg"
On Windows PowerShell, use:
$env:SHERLOCK_API_KEY="paste-your-api-key-here"
$env:SHERLOCK_VISION_MODEL="paste-available-vision-model-id-here"
$env:SHERLOCK_IMAGE_PATH="C:\full\path\to\your\image.jpg"
Select a vision-capable model
Before sending a vision request, check which models are available in your Sherlock AI project. Model availability may change, and a model shown in older examples may no longer be enabled for your project.
Use the models endpoint:
curl https://api-sherlock.cloudferro.com/openai/v1/models \
-H "Authorization: Bearer $SHERLOCK_API_KEY"
You can also list model IDs with Python:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["SHERLOCK_API_KEY"],
base_url="https://api-sherlock.cloudferro.com/openai/v1",
)
models = client.models.list()
for model in models.data:
print(model.id)
Copy a vision-capable model ID returned for your project and store it in SHERLOCK_VISION_MODEL.
Important
If no vision-capable model is returned, the project currently cannot run image analysis requests and the model must be enabled first.
Image input format
The image is sent as a Base64-encoded data URL. For a JPEG file, the value has the following form:
data:image/jpeg;base64,<base64-encoded-image>
For PNG files, use data:image/png;base64, instead. Make sure that the MIME type matches the file you attach; otherwise, the model may reject the request or fail to process the image correctly.
Send a vision request with Python
Install the OpenAI client library if it is not already available in your Python environment:
pip install openai
The following script reads the image path from SHERLOCK_IMAGE_PATH, encodes the file as Base64, and sends it together with a text prompt to the model stored in SHERLOCK_VISION_MODEL.
import base64
import os
from pathlib import Path
from openai import OpenAI
BASE_URL = "https://api-sherlock.cloudferro.com/openai/v1"
api_key = os.environ["SHERLOCK_API_KEY"]
vision_model = os.environ["SHERLOCK_VISION_MODEL"]
image_path = Path(os.environ["SHERLOCK_IMAGE_PATH"])
client = OpenAI(
api_key=api_key,
base_url=BASE_URL,
)
def encode_image(path: Path) -> str:
"""Return the image file encoded as a Base64 string."""
if not path.is_file():
raise FileNotFoundError(f"Image file not found: {path}")
with path.open("rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image(image_path)
messages = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are a helpful assistant that analyses images.",
}
],
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe what you see in this image.",
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}",
"detail": "high",
},
},
],
},
]
chat_response = client.chat.completions.create(
model=vision_model,
messages=messages,
)
print(chat_response.choices[0].message.content)
The example assumes that SHERLOCK_IMAGE_PATH points to a JPEG file. If you use a PNG file, change the data URL prefix in the image_url value to data:image/png;base64,.
Install Base64 support for curl examples
The curl example uses the base64 command to encode the image before sending the request. On many Linux systems, this command is already installed as part of coreutils. On Debian or Ubuntu, install it with:
sudo apt update
sudo apt install coreutils
Verify that the command is available:
base64 --version
Send a vision request with curl
The following shell example reads the image path from SHERLOCK_IMAGE_PATH, encodes the local image, inserts it into the JSON request body, and sends the request to the chat completions endpoint.
#!/usr/bin/env bash
set -euo pipefail
API_KEY="${SHERLOCK_API_KEY:?Set SHERLOCK_API_KEY first}"
VISION_MODEL="${SHERLOCK_VISION_MODEL:?Set SHERLOCK_VISION_MODEL first}"
IMAGE_PATH="${SHERLOCK_IMAGE_PATH:?Set SHERLOCK_IMAGE_PATH first}"
BASE64_IMAGE=$(base64 "$IMAGE_PATH" | tr -d '\n')
curl https://api-sherlock.cloudferro.com/openai/v1/chat/completions \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d @- <<EOF
{
"model": "$VISION_MODEL",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What is in this image?"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,$BASE64_IMAGE",
"detail": "high"
}
}
]
}
]
}
EOF
The command assumes that SHERLOCK_API_KEY, SHERLOCK_VISION_MODEL, and SHERLOCK_IMAGE_PATH are already set in the same terminal session. It also assumes that the input image is a JPEG file. For PNG files, change the data URL prefix to data:image/png;base64,.
Use vision prompts effectively
Vision requests work best when the prompt clearly states what the model should inspect. For general understanding, ask for a description of the image. For operational use, specify the expected output, such as a list of visible objects, a summary of a screenshot, detected issues, or a structured answer.
Avoid sending sensitive images unless your workflow is designed to handle them securely. Treat screenshots, documents, maps, credentials, identity documents, and internal dashboards as sensitive data, especially when they contain names, addresses, tokens, project identifiers, or other private information.
Troubleshooting
If the request returns an authentication error, check that SHERLOCK_API_KEY is set in the same terminal session where you run Python or curl. If the model is rejected, verify that SHERLOCK_VISION_MODEL contains a model ID returned by the models endpoint for your project.
If the image is not processed, check that SHERLOCK_IMAGE_PATH points to an existing file, that the current user can read the file, and that the data URL uses the correct MIME type. If the curl example fails during Base64 encoding, run base64 –version and verify that coreutils is installed.