Unable to start new statefulset due to readiness probe

[bjaworski3]

Describe the bug
When setting a Kubernetes readiness probe, Varnish will never start when creating a new stateful set or the pod count goes to 0. The first pod will wait for the front end service to get a ready endpoint, but the ready endpoint depends on the server being active.

The readiness probe will work properly if you disable it on the first deploy, then enable it after that. The stateful set will however get into a bad state if all pods are restarted.

My settings could be wrong, but my understanding is that frontend-service should be set to the service in front of the Varnish cache?

W0824 17:09:56.720436       1 endpoints_watch.go:50] service 'api-varnish' has no endpoints

To Reproduce
Steps to reproduce the behavior:

1. Create a new statefulset with a liveness and readiness probe:

    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 9102
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 9102
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5

2. The first pod will never become ready because it is watching its own frontend service and waiting for it to be active.

Expected behavior
On first deploy or when pod count goes to 0, the first pod should become ready by either ignoring the front end service check or skipping it. This would allow subsequent pods to start up

Environment:

• Kubernetes version: 1.16
• kube-httpcache version: v0.3.2 (Though this issue existed pre 0.3.2 if you set the readiness probe to port 80)

[martin-helmich]

My settings could be wrong, but my understanding is that frontend-service should be set to the service in front of the Varnish cache?

True. However, this setting is "only" needed when you run kube-httpcache as a cluster (either as a self-routed sharded cluster or as a HA cluster using the build-in signaller component). Still, this is an obvious design flaw that we should've though about before. 🤦‍♂️ Thanks for the report.

[martin-helmich]

Since the readiness probe is enabled by default, this will break any and all existing deployments. 🤦 I've decided to revert #35 for the time being.

[caiohasouza]

Hello,

I'm sending requests to port 9102 but i'm receiving connection timeout, this feature is not working?

Regards

[mittwald-machine]

There has not been any activity to this issue in the last 14 days. It will automatically be closed after 7 more days. Remove the stale label to prevent this.

[caiohasouza]

Someone?

[martin-helmich]

@caiohasouza As mentioned in my earlier comment, we decided to roll back the PR in which the readiness/health endpoints were added. PRs for fixing this issue are welcome.

[caiohasouza]

@martin-helmich right, and is not possible (or recommended) you add the readiness/health feature but disabled by default?

[mittwald-machine]

There has not been any activity to this issue in the last 14 days. It will automatically be closed after 7 more days. Remove the stale label to prevent this.