Introduction
This document describes how to address FailedGetResourceMetric events generated by istiod pods.
Problem
After deploying Subscriber Microservices Infrastructure (SMI) cluster using 2023.01 Cloud Native Deployment Platform (CNDP) release it has been noticed istiod pods keep generating FailedGetResourceMetric events:
cloud-user@ndce-smi-1:~$ kubectl get events -A -o yaml
apiVersion: v1
items:
- apiVersion: v1
count: 467057
eventTime: null
firstTimestamp: "2023-12-31T03:57:42Z"
involvedObject:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: istiod
namespace: istio-system
resourceVersion: "4374"
uid: c85a05a1-b9ec-4241-8347-0b3c9eec1193
kind: Event
lastTimestamp: "2024-03-22T14:42:49Z"
message: 'unable to get metrics for resource cpu: unable to fetch metrics from resource
metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)'
metadata:
creationTimestamp: "2023-12-31T03:57:42Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:count: {}
f:firstTimestamp: {}
f:involvedObject:
f:apiVersion: {}
f:kind: {}
f:name: {}
f:namespace: {}
f:resourceVersion: {}
f:uid: {}
f:lastTimestamp: {}
f:message: {}
f:reason: {}
f:source:
f:component: {}
f:type: {}
manager: kube-controller-manager
operation: Update
time: "2023-12-31T03:57:42Z"
name: istiod.17a5ce7b410bceb6
namespace: istio-system
resourceVersion: "55452121"
selfLink: /api/v1/namespaces/istio-system/events/istiod.17a5ce7b410bceb6
uid: 9616bdc6-90d1-4475-b6af-6d41e185b434
reason: FailedGetResourceMetric
reportingComponent: ""
reportingInstance: ""
source:
component: horizontal-pod-autoscaler
type: Warning
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Analysis
The issue appears to be due to Horizontal Pod Autoscaler (HPA) is enabled in istiod pods during the deployment. Since HPA requires Metrics API to determine CPU utilization for scaling pods up/down, and Metrics API is not currently used in CNDP managed Kubernetes cluster, then this is a design flaw in istiod deployment.
Solution
HPA has been removed starting 2023.03 CNDP release onwards.
Feedback