Troubleshooting AI Experiments on CloudFerro Cloud
A failed sign-in or a missing result can leave you unsure which part of the setup needs attention. The cause may lie in access permissions, the instance, the Python connection, or the storage used for saved files.
The tables below match common messages and symptoms to checks for each part of the setup. A final section lists the details an administrator or support will need if the problem persists.
Sign-in and access
Problem |
Check |
|---|---|
Access denied with a valid platform account |
Your sign-in email must be on the instance’s member list. Ask an instance administrator to add you. |
You created the instance, but MLflow UI returns Forbidden |
Open the instance’s three-dot menu, choose Details, and check Members. A new instance can have an empty member list. Add the account that will administer it through Add Member, enable membership, select Admin, and save. Ask your project administrator if those controls are unavailable. |
Access denied immediately after membership was added |
Sign out and sign in again. Use a private window or clear the browser cache if necessary. |
The MLflow UI button is unavailable |
Check the instance status in My Experiments. The button becomes active when the instance is Running. |
The instance URL does not open |
Copy the endpoint again from the instance details in My Portal. |
A saved role change has not taken effect |
Role lookups are cached for about two minutes. Sign out and back in to fetch the current role. |
Sign-in error messages
Browser and API access can return the following messages. Each entry preserves the service’s error text so you can match it to the response you received.
Forbidden: You are not a member of this MLflow instance.
The captured browser response continues: Ask the instance administrator to add you to the instance members in the MLOps portal (instance details > Add Member). The instance can be Running while the account trying to enter it has no membership.
Configure membership in My Portal, then open MLflow UI again.
Unauthorized: Missing User Identity. Please contact the administrator. (401)
The request reached the instance without a valid session or token. Sign in again in the browser. For Python or CLI access, obtain a fresh token.
Forbidden: You are not authorized to access this MLflow instance. Please contact your administrator to request access. (403)
Sign-in succeeded, but your email is not on the instance member list. Ask an instance administrator to add you.
Service Unavailable: Authentication service temporarily unavailable. Please try again later. (503)
The instance could not verify your access with the platform. The authorization service may be temporarily unavailable, or your project IAM role may lack mlops:GetMlflowRole. Wait a few minutes and try again. If colleagues can sign in while you still cannot, ask your project administrator to check for a role granting that action, such as AIExperimentsViewer or AIExperimentsAdmin. Sign out and back in after it is granted.
Instance creation and storage
Problem |
Check |
|---|---|
Create remains disabled |
Complete the form and pass Check connection in Object storage. |
The storage check fails |
Verify the endpoint, region, bucket name, and matching key pair. Check that those credentials can upload and download a test object in the bucket. |
The instance remains Creating |
Provisioning takes a few minutes. Contact support if it exceeds 30 minutes. |
Error, Creating error, or Deleting error appears |
Hover over the status for details and use Contact Support. |
The instance is Paused |
Contact support to resume it. |
The instance is Partially operational |
Hover over the status for details about degraded components and contact support. |
Python and token errors
The same HTTP status code can occur at different stages. Match the operation, the endpoint that returned the response, and the full error message. The identity service issues and refreshes tokens; the MLflow tracking URI accepts those tokens when your client sends requests.
Initial sign-in
Use these checks when requesting a token from the identity service with your username, password, and TOTP code.
Error or symptom |
Check |
|---|---|
400 invalid_grant from the token endpoint |
The identity service rejected the sign-in request. Check the response details, re-enter your username and password, and use a fresh TOTP code. The error alone does not identify which login field failed. If sign-in still fails, verify the token endpoint and client configuration with your administrator. |
401 invalid_client from the token endpoint |
Check that the helper uses client ID mlflow-cli-authservice. If the ID is correct, verify the token endpoint and client configuration with your administrator. |
401 Invalid user credentials from the token endpoint |
Check the username, password, and TOTP code. A missing or expired TOTP code is one possible cause, even when the password is correct. If MLFLOW_TOTP contains an old value, unset it so the helper prompts for a fresh code. |
Token refresh
Use this check when initial sign-in succeeded, but a later request to refresh the token fails.
Error or symptom |
Check |
|---|---|
400 invalid_grant: Token is not active from the token endpoint during refresh |
The refresh token was rejected. The token or SSO session may have expired; the message alone does not prove that the session idle timeout elapsed. Use the current |
Requests to MLflow
Use these checks when your client sends a request to the MLflow tracking URI with an access token.
Error or symptom |
Check |
|---|---|
401 from the tracking URI |
Check that the client sends an access token. The token may be expired, revoked, malformed, or otherwise invalid. Obtain a fresh token before retrying. |
INTERNAL_ERROR: Response: {} from MLflow |
The instance rejected the token. Obtain a fresh token from the documented identity endpoint; contact your administrator if the problem persists. |
403 from the tracking URI |
Check instance membership and permission for the requested resource. Use an experiment you own or one you can edit. |
Python environment and connectivity
Error or symptom |
Check |
|---|---|
Connection or SSL errors |
Verify the tracking URI and identity endpoint with your administrator. Check that the network permits HTTPS to them. |
No module named ‘get_token’ |
Keep get_token.py beside the example script, or make it available on the notebook’s Python import path. |
No module named ‘mlflow’ or ‘requests’ |
Activate the environment where the packages were installed. Use the same environment as the notebook kernel. |
A missing environment variable is reported |
Set MLFLOW_TRACKING_URI, MLFLOW_USERNAME, and MLFLOW_PASSWORD in the shell that launches the program. Gateway calls also need MLFLOW_GATEWAY_ENDPOINT. |
Runs, artifacts, and gateway calls
Problem |
Check |
|---|---|
first-experiment does not appear |
Confirm that the script completed and that the browser is open on the instance identified by MLFLOW_TRACKING_URI. Check the experiment name and refresh the list. |
The UI shows demo traces instead of the calculation’s runs |
Switch from GenAI to Model training, then open first-experiment. MLflow Demo Experiment is sample data loaded by Start Demo; it does not contain results from first_experiment.py. |
The details page shows INACTIVE but the instance is Running |
Check which field carries the label. INACTIVE beside High Availability (HA) means HA is disabled; it does not describe the instance’s running status. |
A total is logged but result.json is missing |
Check the script’s artifact-upload error. The run may contain records sent before the upload failed. Verify bucket availability and storage credentials. |
Runs or artifacts fail to upload |
Check HTTPS connectivity to the instance, the token, and the role used for the experiment. For artifacts, also check the bucket and whether its credentials have changed. |
Artifacts from an existing run are unavailable |
Verify the bucket, retained objects, and current key pair, then contact your platform administrator. Lifecycle rules or authorized artifact deletion may have removed files. |
The UI is slow or requests time out |
The service may be undersized for the load. Size and HA cannot be changed in the UI; discuss a change with your platform administrator or support. |
A gateway endpoint returns an error |
Confirm its name and provider configuration with an administrator. Calling it requires USER, DATA_SCIENTIST, ML_ENGINEER, or ADMIN; AUDITOR cannot invoke endpoints. |
Getting help
Contact CloudFerro support through your usual support channel. Include the instance URL, the approximate time, the operation you attempted, and the error message. For a failed run, include its run ID when available. Leave passwords, access keys, secret keys, and tokens out of the report.
What to do next
The setup guides cover Your first AI Experiments instance on CloudFerro Cloud, Object storage for AI Experiments on CloudFerro Cloud, and Connect Python and the CLI to AI Experiments on CloudFerro Cloud. Permission changes are described in AI Experiments members and roles on CloudFerro Cloud; the example’s expected results are in Run and compare your first MLflow runs on CloudFerro Cloud.