Troubleshooting
The following section highlights common issues seen when installing and using the HyperFlex CSI integration. The information provided includes symptoms to help diagnose the issue as well as a solution to resolve the issue.
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
The following section highlights common issues seen when installing and using the HyperFlex CSI integration. The information provided includes symptoms to help diagnose the issue as well as a solution to resolve the issue.
Symptom 1: Running the command “kubectl get pods [-n <namespace>]” shows that the HXCSI pods are showing a status of “ImagePullBackOff”.
Symptom 2: Running the command “kubectl describe pod <csi-pod_name>” shows a message containing the following error: “Error: ErrImagePull” and “Back-off pulling image…”
Solution:
Solution 1: Ensure the HXCSI container image name provided to the hxcsi-setup script is correct
Solution 2: Ensure the HXCSI container image exists, either directly within docker on each Kubernetes worker node or on the local container image registry.
Solution 3: Ensure that the “imagePullPolicy” lines in the following YAML files generated by the hxcsi-setup script are set to “IfNotPresent”: csi-attacher-hxcsi.yaml, csi-nodeplugin-hxcsi.yaml, csi-provisioner-hxcsi.yaml
Solution 4: Ensure that the following images are loaded on the local container image registry on each Kubernetes node: csi-attacher-3.0.2-cisco1.tar,csi-node-driver-registrar-2.0.1-cisco1.tar, csi-resizer-1.0.1-cisco1.tar, csi-provisioner-2.0.4-cisco1.tar
Volume delete fails due to stale volume attachments present even after NodeUnpublish succeeds and the volume is unmounted. This happens when the delete volumeattachment kubernetes api is lost during etcd leader election. Even after nodeUnpublish completes and the volume is successfully unmounted from the node, delete volume fails.
The log in the external-provisioner appears as:
volume deletion failed: persistentvolume <pv-name> is still attached to node <node-name>.
The log in the external-attacher appears as:
<Volume-attachment> is already attached.
Solution:
Delete the stale volumeattachments using the following command:
kubectl delete volumeattachments <VA-name>
After few seconds when the provisioner retries, pv is deleted.
Alternatively, you can delete manually using the following command:
kubectl delete pv <pv-name>
Application pods that get stuck in a ContainerCreating state or multi attach error state and are not able to mount the volumes. This may happen during the remove or delete a k8s worker node from the cluster and pods migrate to a new worker node.
The recommended way to delete a K8s worker node is by using the following commands:
kubectl drain <node-name>
kubectl delete node <node-name>
For more information, see How to gracefully remove a node from Kubernetes?.
A pod in Running state may get recreated after deleting using
the kubectl delete pod
command and then get stuck in
Terminating state on deleting its namespace.
Instead of using the kubectl delete pod
command on any running pod,
the following best practices way is recommended:
Note the node name on which pod to be deleted is running on.
kubectl get pods -o wide --all-namespaces
Cordon off the node on which the pod is running.
kubectl cordon <node-name>
Delete the pod.
kubectl delete pod <pod-name>
Check that the deleted pod is scheduled on a different node using the following command:
kubectl get pods -o wide --all-namespaces
Uncordon the cordoned node.
kubectl uncordon <node-name>