Troubleshooting and known limitations on CloudFerro Cloud
If something fails, first identify whether the problem is the instance, a notebook, a pipeline task, or a Katib Experiment. An instance can be RUNNING while one workload is still pending. The symptoms below point to the relevant diagnostic procedure.
Before asking for help
Record the instance, namespace, workload name, status, and relevant event or error text.
For a pipeline, include the run and task names; for Katib, include the Experiment and Trial.
Give support the deployed version when available. Do not include secret values, access tokens, or kubeconfig contents.
These notes include behavior observed on particular deployments. The Istio workaround names KFP 2.5.0; some Katib observations have no recorded version. Match the symptom and consult the administrator rather than assuming every release has the same limitation.
Katib rejects the CREATE form
Missing name or reference errors can come from the form rebuilding the trial parameters. Check the YAML and use Edit and submit YAML for the affected workflow.
Katib logs disappear after a Trial
Completed Trial resources may have been removed. Configure retention before the run when you need post-run pod logs.
A pipeline has no startup logs
A container that never starts cannot produce application logs. Check pod events for missing secrets, keys, or mounting errors.
A pipeline runs an older image
A mutable tag may refer to a cached image on the worker. Compare the pod’s image digest with the registry before changing the pull policy.
A pipeline reuses earlier outputs
Execution caching can skip a task even when the run is successful. Check the cache indicator and choose task defaults or a run-level override.
A notebook or pipeline waits for a volume
An RWO volume already mounted on another node can cause a multi-attach error. Inspect pod events and coordinate volume use.
Python packages disappear after restart
Packages installed in container system directories are lost when the container is replaced. Use the mounted workspace for a user-site or Conda environment.
View access does not protect Katib experiments
The documented limitation allows a viewer to change Katib experiments. Restrict namespace membership where experiments need protection.
KFP 2.5.0 cannot reach Model Registry
On the documented deployment, a shared task template can omit an Istio sidecar. Apply the annotation to every task when the pipeline needs Istio communication.
A Katib Trial stays Running after training
A metrics-collector sidecar can keep the Trial unfinished in the observed deployment. Test the Push collector with explicit metric reporting.