Troubleshoot pipeline startup and reused results on CloudFerro Cloud
A pipeline can appear stuck before its container starts, or finish quickly because it reused an earlier result. Begin with the symptom below: missing startup logs, an unexpected image version, cached step outputs, or Model Registry access errors need different checks.
Prerequisites
No. 1 A running instance and the assigned namespace
Follow Open an existing Kubeflow notebook on CloudFerro Cloud to open Central Dashboard and select your team’s namespace. Have the namespace name and the affected resource name available.
No. 2 Kubernetes access for command-line checks
For the commands below, install kubectl and obtain the namespace’s kubeconfig as described in Create namespaces and manage team access on CloudFerro Cloud. Select the intended cluster context and confirm permission to inspect the named resources. Ask an administrator to run checks outside your role.
No. 3 The pipeline definition and relevant client tools
To change and compile Python examples, use the kfp SDK compatible with the deployed service. The image-policy and Istio examples also need kfp-kubernetes. Keep the pipeline source, affected run, and task names available. Changing secrets or submitting a replacement run requires the appropriate namespace permissions.
A task cannot start and has no logs
When a pipeline task references a missing secret, the wrong secret name, or a key that does not exist in the secret, its container cannot start. The Pipelines UI may show the task as pending or failed but provide no useful logs.
This is expected because container logs are created only after the container starts. Secret validation, environment-variable resolution, and volume mounting happen before startup, so these failures are reported as Kubernetes pod events instead of application logs.
Troubleshooting
Set the namespace and pod name used by the commands:
NAMESPACE="<pipeline-namespace>"
POD_NAME="<pipeline-task-pod>"
If the pod name is unknown, list the most recently created pods in the pipeline namespace:
kubectl get pods \
--namespace "$NAMESPACE" \
--sort-by=.metadata.creationTimestamp
Describe the affected pod:
kubectl describe pod "$POD_NAME" \
--namespace "$NAMESPACE"
Review the Events section at the end of the output. Common messages include:
secret "<name>" not foundcouldn't find key <key> in Secret <namespace>/<name>CreateContainerConfigErrorMountVolume.SetUp failed
Query the pod events directly if the relevant message is difficult to find in the description:
kubectl get events \
--namespace "$NAMESPACE" \
--field-selector "involvedObject.name=$POD_NAME" \
--sort-by=.lastTimestamp
You can also inspect recent namespace events when the pod name is unavailable or the pod has already been removed:
kubectl get events \
--namespace "$NAMESPACE" \
--sort-by=.lastTimestamp
Verify the secret
After identifying the referenced secret and key from the event, verify that the secret exists in the same namespace as the pipeline task:
SECRET_NAME="<secret-name>"
kubectl get secret "$SECRET_NAME" \
--namespace "$NAMESPACE"
kubectl describe secret "$SECRET_NAME" \
--namespace "$NAMESPACE"
kubectl describe secret shows the available key names and value sizes without printing the secret values. Confirm that:
The secret exists in the pipeline namespace.
The pipeline references the exact secret name.
Every referenced key exists with the expected spelling and capitalization.
The pipeline’s service account is allowed to use the secret when platform policy restricts secret access.
Do not print, decode, or include secret values in troubleshooting output or support tickets.
Apply the fix
Correct the secret name or key in the pipeline definition, or create the missing secret in the correct namespace. Then submit a new pipeline run. An existing pod with an invalid secret reference generally cannot be repaired in place.
If the Pipelines UI has no logs and the task container never started, use kubectl describe pod and Kubernetes events first. Retrying the pipeline without correcting the secret reference will produce the same failure.
A task runs an older container image
When executing Kubeflow Pipelines, the pipeline may successfully pull an image from Harbor and complete its run, but use an outdated, locally cached version of the container image. This occurs when developers push updated code to an existing mutable tag (e.g., :dev, :v1, or a branch name), but the Kubernetes node executing the pipeline pod already has a previous version of that tag cached locally.
This limitation stems from how Kubernetes handles the container imagePullPolicy by default when a policy is not explicitly defined in the KFP component:
:latest tags: Kubernetes defaults to
imagePullPolicy: Always. It will always check Harbor for a new digest.Specific tags other than :latest (e.g., :v1.0): Kubernetes defaults to
imagePullPolicy: IfNotPresent. An omitted tag is treated as latest, with Always as the default.
When IfNotPresent is active, the Kubelet checks the node’s local cache first. If an image with the requested name and tag already exists on that specific node, Kubernetes bypasses Harbor entirely and spins up the container using the cached layers. It does not verify if the image in Harbor has been updated.
Workarounds and Best Practices
To resolve this limitation, users must adopt one of the following approaches:
Solution A: Enforce imagePullPolicy: Always (Immediate Fix)
You can explicitly override the Kubernetes default behavior within the Kubeflow Pipeline definition. This forces the Kubelet to contact Harbor and compare image digests before every execution, ensuring the newest version is pulled.
Example using KFP Python SDK (v2):
from kfp import dsl, kubernetes
@dsl.component
def my_ml_component():
pass
@dsl.pipeline(name='my-pipeline')
def my_pipeline():
task = my_ml_component()
# Force Kubernetes to check Harbor for updates
kubernetes.set_image_pull_policy(task, 'Always')
Solution B: Adopt Immutable Image Tagging (Long-Term Best Practice)
Use a unique tag or an image digest for each version of the code. Because the tag is entirely new, Kubernetes will never find it in the IfNotPresent cache and will pull the new image.
Git Commit SHA: Tag images with the short SHA (e.g.,
my-image:a1b2c3d).CI/CD Pipeline ID: Use the automated build number (e.g.,
my-image:build-402).
Verification via CLI
If you suspect a pipeline has executed a stale image, do not rely on the Kubeflow UI. You must check the exact image digest running in the pod using kubectl:
Identify the pod name in your Kubeflow namespace.
Describe the pod and look for the specific
Image IDdigest:
kubectl describe pod <pod-name> -n <your-kubeflow-namespace> | grep -i "Image ID"
Compare the resulting SHA256 digest against the latest digest shown in Harbor. If the digests do not match, the node used a stale local cache.
A run reuses previous task outputs
When executing a pipeline multiple times with the same parameters, subsequent runs may complete almost instantly without actually executing the code. Instead of spinning up new pods, Kubeflow reuses the outputs from a previous successful run.
Execution caching can reuse a previous successful result when the component inputs and definition match. This is separate from the worker node’s container-image cache. In the run graph, a step whose results were retrieved from the cache is marked with a green “arrow from cloud” icon. Users should look for this icon instead of relying only on the step’s successful status or green coloring, which can otherwise be mistaken for a fresh execution. Users often assume they are validating logic or benchmarking performance, but are actually just viewing recycled outputs.
Kubeflow Pipelines implements execution caching by calculating a hash based on a component’s inputs, container image, and parameters. If the orchestration engine finds an exact hash match from a previous successful execution, it skips pod creation entirely and passes the old output artifacts forward.
A task’s default caching setting is recorded in the compiled pipeline. The Python client can override caching for a submitted run with enable_caching; the available dashboard controls depend on the deployed version. If an external dependency changes (like data residing in an external database or an API response) but the KFP component parameters remain identical, the hash does not change, and the pipeline will silently serve stale data.
Workarounds and Best Practices
For a task-level default, change the pipeline definition, recompile it, and upload the new version. An authenticated Python client can instead submit a run with enable_caching=False to override the compiled settings for that run.
Solution A: Disable Caching at Compile Time (Immediate Fix) You can explicitly disable caching at the pipeline or component level using the KFP SDK. This forces the orchestration engine to execute the step regardless of previous run history. You must recompile and upload the new pipeline version after making this change.
Example using KFP Python SDK (v2):
from kfp import dsl
@dsl.component
def my_ml_component():
pass
@dsl.pipeline(name='my-pipeline')
def my_pipeline():
task = my_ml_component()
# Force Kubeflow to execute this step every time
task.set_caching_options(enable_caching=False)
Solution B: Implement Cache-Busting Parameters (Workflow Strategy)
If you want to leave caching enabled globally but force specific runs to execute fresh, inject a unique, changing variable (like a timestamp, Git commit hash, or unique run ID) into the component as an input parameter. Because the input parameter changes every time, the resulting hash will be unique, naturally bypassing the cache mechanism.
Model Registry access fails on KFP 2.5.0
In pipelines compiled and executed on KFP 2.5.0, you must add the following annotation: sidecar.istio.io/inject: "true" to every pipeline step if any of the steps require communication via Istio (e.g., with the Model Registry).
In this version of KFP, tasks may share a single, common template. This template can be generated based on the alphabetically first task. If this first task lacks the sidecar.istio.io/inject: "true" annotation, subsequent tasks may also run without the correct sidecar, even if the annotation was explicitly added to them.
In practice, this can lead to errors such as airbag / access denied when communicating with the Model Registry.
Example:
from kfp import kubernetes
def add_istio_sidecar(task):
kubernetes.add_pod_annotation(
task,
annotation_key="sidecar.istio.io/inject",
annotation_value="true",
)
return task
Next, you need to call this function for every single task:
add_istio_sidecar(name_task)
add_istio_sidecar(register_task)
add_istio_sidecar(verify_task)
Rule: if a pipeline uses Model Registry or another service that requires an Istio sidecar, add sidecar.istio.io/inject: "true" to all pipeline steps, including technical steps that do not call that service directly.
This workaround records the observed behavior on KFP 2.5.0. Check the deployed version and the generated pod annotations when diagnosing another installation.
What to do next
For a volume attachment failure, use Troubleshoot notebooks and volume attachments on CloudFerro Cloud. Upstream references describe execution caching, Kubernetes image pull policies, and KFP Kubernetes extensions.