[BUG] Failed to check for stale credentials

[srmars]

Components and versions
App version 1.7.3

Describe the bug

After upgrading the akv2k8s to 1.73, noticed pod getting restarted with error “ main.go:267] “failed to get credentials” error=“Failed to check for stale credentials …….. dial 10.0.1.80: connect : connection refused”

But after 1 or 2 restart the pod running without any issues.

To Reproduce

Create deployment with environment injection and delete pod or rollout restart

Expected behavior
No restart to the pod

[srmars]

Any update on this issue, please share if there is any workaround

[c4tz]

I also had this problem the day before yesterday, using the Helm chart. The only difference was that I got a timeout instead of a refused connection.

I tried multiple things:

• Downgrading the chart to 2.7.1 (App version 1.7.1) and 2.6.0 (App version 1.6.0)
• curling the URL logged from the same namespace and from another namespace. Always had no problem and got 200 back
• Setting the loglevel to debug first and trace afterwards, but this didn't change the output at all

Then, at some point it randomly started working again. But this was after I had undone all my changes and about an hour had passed after that.

During my whole investigation, the controller was able to sync secrets to k8s. It was just the env injector having trouble. Because of this, I don't think there was an issue with Azure at the time and (as mentioned before), I also tested the network connectivity.

[srmars]

It’s one of the blocker for the upgrade, for now we have downgraded the versions, it would great if we get a fix for this

[c4tz]

We had this problem again last week, but it turned out to be a faulty node which was missing some network connectivity.

When I replaced it with a new one, the akv2k8s timeouts were gone and everything worked normally. So this might not be a bug in this repo at all.

[PrestonCS]

We struggled with this issue for quite a while. It was never a complete blocker since things usually worked after one or two pod restarts, but it was definitely frustrating.

After some digging, I was able to trace the root cause in our case to Istio and Envoy startup timing. Specifically, the application container would sometimes start before the Envoy proxy was fully ready to accept traffic, causing the Azure Key Vault injection or retrieval to fail intermittently.

If you’re running a service mesh (like Istio), this is something worth checking.

For Istio, enabling the following setting completely resolved the issue for us:

ProxyConfig.holdApplicationUntilProxyStarts

This ensures that your application containers won’t start until the Envoy sidecar is fully up and ready to proxy traffic. Since enabling it, we’ve seen consistent and stable behavior with akv2k8s.

Hope this helps someone else!