In our initial setup, an upstream via kubernetes service exposed pods which is not ideal as then load balancing happens inside of service rather than APISIX. We can however use the service to discover pods and embed into upstream definition dynamically.

Update APISIX to enable discovery plugin

Modify values.yaml as follows and redeploy:

Enable plugin

Find discovery section of values.yaml. Set enabled to true and registry to include kubernetes as follows:

  discovery:
    enabled: true

    registry:
      kubernetes: {}

Service Account

Find serviceAccount section and set create to true. Also set create under rbac to true like so:

  serviceAccount:
    create: true
    annotations: {}
    name: ""

  rbac:
    create: true

NGINX Configuration section

Find nginx section, env and modify as follows:

      envs:
        - KUBERNETES_SERVICE_HOST
        - KUBERNETES_SERVICE_PORT

Update APISIX

Now that all changes are in place:

envsubst '$APISIX_ADMIN_TOKEN $APISIX_VIEWER_TOKEN' < kubernetes/apisix/values.yaml | helm upgrade --install apisix apisix/apisix --create-namespace  --namespace apisix -f -

Verify plugin works as expected

This took a while to find out: various AI tools claimed discovery plugin was not initialising by checking logs and sent to various stray paths. Instead, use the following:

curl http://127.0.0.1:9090/v1/discovery/kubernetes/dump -H "X-API-KEY: $APISIX_VIEWER_TOKEN"

It will list, for each service, all discovered nodes and their IPs. Dead-handy for debugging.

Configure upstream and route

Upstream

Use this to configure an upstream:

 curl http://127.0.0.1:9180/apisix/admin/upstreams/1 \
  -H "X-API-KEY: $APISIX_ADMIN_TOKEN" \
  -X PUT -d '{
  "type": "roundrobin",
  "discovery_type": "kubernetes",
  "service_name": "ramblings/ramblings-app:8080",
  "pass_host": "rewrite",
  "upstream_host": "ramblings-app.ramblings.svc.cluster.local"
}'

and then verify using:

curl http://127.0.0.1:9090/v1/upstreams \
-H "X-API-KEY: $APISIX_VIEWER_TOKEN"

You expect to see the upsteam with nodes populated with actual pod IPs etc.

Route

To create,

curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $APISIX_ADMIN_TOKEN" -X PUT -i -d '{
        "name": "1",
        "status": 1,
        "id": "1",
        "enable_websocket": false,
        "priority": 0,
        "uri": "/deck-api/*",
        "methods": [
          "GET",
          "POST",
          "PUT",
          "DELETE"
        ],
        "upstream_id": "1"
}'

upstream_id needs to match upstream defined above.

To verify,

curl http://127.0.0.1:9090/v1/routes \
-H "X-API-KEY: $APISIX_VIEWER_TOKEN"

Test