Troubleshoot Katib experiments on CloudFerro Cloud

Katib tries different training settings, called hyperparameters, and compares the measured results. A single set of settings runs as a Trial inside an Experiment. The cases below cover rejected submissions, missing logs, unfinished Trials, and a documented access-control limitation.

Prerequisites

No. 1 A running instance and the assigned namespace

Follow Open an existing Kubeflow notebook on CloudFerro Cloud to open Central Dashboard and select your team’s namespace. Have the namespace name and the affected resource name available.

No. 2 Katib access and the affected Experiment

The instance must include Katib. Have the Experiment name, Trial name when available, and its YAML configuration ready. Changing or submitting experiments requires the intended working role; the View-role limitation below must be considered when restricting access.

No. 3 Kubernetes access for command-line checks

For the commands below, install kubectl and obtain the namespace’s kubeconfig as described in Create namespaces and manage team access on CloudFerro Cloud. Select the intended cluster context and confirm permission to inspect the named resources. Ask an administrator to run checks outside your role.

The CREATE form rejects trial parameters

When creating a Katib Experiment through the Kubeflow Dashboard / Katib UI, the form-based CREATE button may fail for experiment configurations that define trialTemplate.trialParameters with reference fields.

Symptoms

The user fills in the experiment form with a valid configuration, including:

  • an objective,

  • an algorithm,

  • hyperparameters such as lr and momentum,

  • a trial template that uses trialParameters,

  • trialParameters[].reference values that point to the configured hyperparameter names.

After clicking CREATE, the Experiment is rejected by the Katib admission webhook with an error similar to:

500 validation error: name and reference must be specified

In older versions, this may have appeared as a less useful or silent error, for example:

[200] [object Object]

Cause

The issue is specific to the form-based CREATE workflow.

Although the trial template YAML may contain valid trialParameters entries with both name and reference, the form-based CREATE path rebuilds the trialParameters list from the hyperparameter fields entered in the form.

Those form fields contain the hyperparameter names, but they do not preserve the reference field. As a result, the UI overwrites the valid YAML-defined trialParameters and submits an Experiment where the reference values are missing.

Katib then rejects the request because each trial parameter must include both:

name: ...
reference: ...

Impact

Experiments that require trialTemplate.trialParameters[].reference cannot be reliably created through the form-based CREATE button workflow.

This commonly affects Experiments where the trial template references hyperparameters using placeholders such as:

${trialParameters.lr}
${trialParameters.momentum}

with trial parameters similar to:

trialParameters:
  - name: lr
    reference: lr
  - name: momentum
    reference: momentum

Suggested workaround

Use the Edit and submit YAML option in the Katib UI instead of the form-based CREATE button.

The Edit and submit YAML workflow preserves the trialParameters.reference fields correctly and allows the Experiment to be created successfully.

Before submitting, verify that the YAML contains valid trialParameters entries under spec.trialTemplate, for example:

spec:
  trialTemplate:
    trialParameters:
      - name: lr
        reference: lr
      - name: momentum
        reference: momentum

The reference value should match the corresponding hyperparameter name defined in the Experiment search space.

User guidance

Until the form-based CREATE workflow is fixed, users should avoid creating Katib Experiments with trialParameters.reference through that path.

Instead:

  1. Configure or prepare the Experiment.

  2. Open Edit and submit YAML.

  3. Confirm that each trialParameters entry includes both name and reference.

  4. Submit the Experiment from the YAML editor.

This avoids the UI code path that rewrites trialParameters and prevents the webhook validation error.

Logs are missing after a Trial finishes

Trial details, metrics, YAML, the live loss curve, and log streaming work while a Trial is running, but after completion the Logs tab may display [object Object] instead of historical logs or a clear status message.

Set .spec.trialTemplate.retain: true when post-run debugging is required: retain defaults to false, so Katib otherwise cleans up the completed Trial’s resources, including the worker resource used to locate its pod logs. Completed-Trial logs should remain available while the retained pod exists (or when a separate centralized logging system stores them); if they are unavailable, the UI should report that clearly rather than exposing a raw object.

View access allows changes to experiments

The documented platform limitation allows users with View access to create and remove Katib experiments. Do not rely on that role alone to protect experiments from changes. Check the behavior on your deployed version with the platform administrator.

Cause

Katib is not properly respecting SubjectAccessReviews and always uses Katib UI service account

Workaround

There is no workaround available. If some experiments should be protected from other people a separate namespace should be created with limited access.

A Trial remains Running after training

In the deployment tested for these notes, StdOut and File metrics collectors could leave a Trial unfinished. The affected Katib version was not recorded; diagnose the observed behavior before applying this workaround to another deployment.

Observed behavior:

  • Katib Experiment/Trial starts correctly.

  • The training container finishes successfully.

  • The metric is printed to stdout or written to file.

  • The metrics-logger-and-collector sidecar may stay in Running.

  • As a result, Trial/Experiment may not finish correctly.

This should not be treated as a GPU, CUDA, NVIDIA device plugin, PyTorch, Spark, or Kueue issue if the same behavior appears on a minimal CPU-only Katib test.

Workaround for the observed behavior:

Use the Push collector instead of StdOut or File.

Example:

metricsCollectorSpec:
  collector:
    kind: Push

Install kubeflow-katib in the training container. In the training code, set acc to the measured accuracy and report it with:

import kubeflow.katib as katib
katib.report_metrics({"accuracy": acc})

To test, update your existing, complete Experiment manifest with the collector setting and training code above. The fragments above are not a complete Experiment. Set the namespace and Experiment name to match that manifest, then submit it:

NAMESPACE="<your-kubeflow-namespace>"
EXPERIMENT_NAME="<name-in-your-manifest>"
kubectl --namespace "$NAMESPACE" apply -f katib-push.yaml
kubectl --namespace "$NAMESPACE" get experiment,trials,pods -w

Use Ctrl+C to stop watching; this does not stop the Experiment. Inspect its details with:

kubectl --namespace "$NAMESPACE" describe experiment "$EXPERIMENT_NAME"

Expected result:

  • Experiment finishes correctly.

  • Trials do not stay stuck in Running.

  • Training pods finish as Completed.

  • Metrics are visible in Katib.

  • No metrics-logger-and-collector sidecar remains running forever.

If the same symptom persists with Push, retain the Experiment and Trial names and the relevant events for the administrator.

What to do next

Restrict workspace membership using the namespace guide in Prerequisite No. 3. Upstream references cover Trial retention, metrics collectors, and the reported Katib authorization issue.

Still need help?
If this article doesn’t answer all of your questions, our support team will do their best to help you.
Contact support