In our initial setup, an upstream via kubernetes service exposed pods which is not ideal as then load balancing happens inside of service rather than APISIX. We can however use the service to discover pods and embed into upstream definition dynamically.
Update APISIX to enable discovery plugin
Modify values.yaml as follows and redeploy:
Enable plugin
Find discovery section of values.yaml.
Set enabled to true and registry to include kubernetes as follows:
  discovery:
    enabled: true
    registry:
      kubernetes: {}
Service Account
Find serviceAccount section and set create to true. Also set create under rbac to true like so:
  serviceAccount:
    create: true
    annotations: {}
    name: ""
  rbac:
    create: true
NGINX Configuration section
Find nginx section, env and modify as follows:
      envs:
        - KUBERNETES_SERVICE_HOST
        - KUBERNETES_SERVICE_PORT
Update APISIX
Now that all changes are in place:
envsubst '$APISIX_ADMIN_TOKEN $APISIX_VIEWER_TOKEN' < kubernetes/apisix/values.yaml | helm upgrade --install apisix apisix/apisix --create-namespace  --namespace apisix -f -
Verify plugin works as expected
This took a while to find out: various AI tools claimed discovery plugin was not initialising by checking logs and sent to various
stray paths. Instead, use the following:
curl http://127.0.0.1:9090/v1/discovery/kubernetes/dump -H "X-API-KEY: $APISIX_VIEWER_TOKEN"
It will list, for each service, all discovered nodes and their IPs. Dead-handy for debugging.
Configure upstream and route
Upstream
Use this to configure an upstream:
 curl http://127.0.0.1:9180/apisix/admin/upstreams/1 \
  -H "X-API-KEY: $APISIX_ADMIN_TOKEN" \
  -X PUT -d '{
  "type": "roundrobin",
  "discovery_type": "kubernetes",
  "service_name": "ramblings/ramblings-app:8080",
  "pass_host": "rewrite",
  "upstream_host": "ramblings-app.ramblings.svc.cluster.local"
}'
and then verify using:
curl http://127.0.0.1:9090/v1/upstreams \
-H "X-API-KEY: $APISIX_VIEWER_TOKEN"
You expect to see the upsteam with nodes populated with actual pod IPs etc.
Route
To create,
curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $APISIX_ADMIN_TOKEN" -X PUT -i -d '{
        "name": "1",
        "status": 1,
        "id": "1",
        "enable_websocket": false,
        "priority": 0,
        "uri": "/deck-api/*",
        "methods": [
          "GET",
          "POST",
          "PUT",
          "DELETE"
        ],
        "upstream_id": "1"
}'
upstream_id needs to match upstream defined above.
To verify,
curl http://127.0.0.1:9090/v1/routes \
-H "X-API-KEY: $APISIX_VIEWER_TOKEN"