Cert Manager

Want to configure a wilcard certificate using LetsEncrypt on selected domain. This assumes use of Cloudflare for DNS management. We also want to keep our traditonal APISIX deployment (as we will use API calls for other experiments) which requires us to create a synchronisation job that pushes cert to APISIX, service account etc. As usual, scripts are heavily generated using AI.

Install Cert manager

microk8s enable cert-manager

Create API token in CF

Navigate to Account API Tokens (under api-tokens URL) and create a DNS token. Take note of the token as it will not be visible later through Cloudflare UI (like mist API tokens).

Store the token as K8s secret

Run

kubectl create secret generic cloudflare-api-token-secret \
  --from-literal=api-token='<<YOUR_CLOUDFLARE_API_TOKEN>>' \
  -n cert-manager

replacing <<YOUR_CLOUDFLARE_API_TOKEN>> with the actual token.

Configure cert-manager to download certs

Run

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns01
spec:
  acme:
    email: <<MYEMAIL>>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-dns01-account-key
    solvers:
    - dns01:
        cloudflare:
          email: <<CF-LOGIN>
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
EOF

replacing:

  • «MYEMAIL>> with your email, and
  • «CF-LOGIN» using your login for Cloudflare.

Define the certificate

Run

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <<WILDCARD-NAME>>
  namespace: default
spec:
  secretName: <<WILDCARD-DOMAIN-TLS-NAME>>
  dnsNames:
    - "*.<<WILDCARD-DOMAIN>>"
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
EOF

replacing:

  • «WILDCARD-DOMAIN» with your domain, and
  • «WILDCARD-DOMAIN-TLS-NAME» the name of the secret.
  • «WILDCARD-NAME» the name used to reference wildcard in APISIX SSL config.

Certificate synchronisation pre-requisites

Define secret to be used to upload SSL cert to APISIX

Load admin key for APISIX. Note that ideally, a separate account should be created but sa this is all pure development, we will take some shortcuts.

cat << EOF | envsubst | kubectl apply -f - 
apiVersion: v1
kind: Secret
metadata:
  name: apisix-admin-secret
  namespace: default
stringData:
  admin-api-key: $APISIX_ADMIN_TOKEN
EOF

Create service account

Create service account for synchronising cert as downloaded by cert-manager to APISIX.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-sync
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-sync
  namespace: default
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cert-sync
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cert-sync
subjects:
- kind: ServiceAccount
  name: cert-sync
  namespace: default
EOF

Enable SSL with APISIX

In values.yaml, make sure that ssl is enabled e.g.

  apisix: 
    
    ssl:
      enabled: true

Also, we will fix nodePort on service so we can address it from OCI LB as below:

    tls:
      servicePort: 443
      nodePort: 31640

Update apisix deployment as here.

Define cronjob pushing cert-manager downloaded cert to APISIX

Run

kubectl apply -f - <<'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: apisix-wildcard-cert-sync
  namespace: default
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cert-sync
          restartPolicy: OnFailure
          dnsPolicy: ClusterFirst
          containers:
          - name: cert-sync
            image: heyvaldemar/aws-kubectl
            env:
              - name: SECRET_NAME
                value: <<WILDCARD-DOMAIN-TLS-NAME>>
              - name: DOMAIN
                value: "*.<<WILDCARD-DOMAIN>>""
              - name: APISIX_ADMIN
                value: http://apisix-admin.apisix.svc.cluster.local:9180
              - name: API_KEY
                valueFrom:
                  secretKeyRef:
                    name: apisix-admin-secret
                    key: admin-api-key
            command:
              - /bin/bash
              - -c
              - |
                set -euo pipefail

                echo "🔄 Syncing certificate for domain: ${DOMAIN}"

                CERT=$(kubectl get secret "${SECRET_NAME}" -n default -o jsonpath='{.data.tls\.crt}' | base64 -d)
                KEY=$(kubectl get secret "${SECRET_NAME}" -n default -o jsonpath='{.data.tls\.key}' | base64 -d)

                # Build JSON payload safely (requires jq in image)
                jq -n \
                  --arg cert "$CERT" \
                  --arg key "$KEY" \
                  --arg domain "$DOMAIN" \
                  '{cert: $cert, key: $key, snis: [$domain]}' > /tmp/payload.json

                echo "➡️ Pushing certificate to APISIX (endpoint: /apisix/admin/ssls/)..."
                HTTP_CODE=$(curl -s -o /tmp/resp.json -w "%{http_code}" \
                  -X PUT "${APISIX_ADMIN}/apisix/admin/ssls/<<WILDCARD-NAME>>" \
                  -H "X-API-KEY: ${API_KEY}" \
                  -H "Content-Type: application/json" \
                  -d @/tmp/payload.json || true)

                echo "HTTP status: ${HTTP_CODE}"
                echo "Response body:"
                cat /tmp/resp.json
EOF

replacing:

  • «WILDCARD-DOMAIN-TLS-NAME»
  • «WILDCARD-DOMAIN»

Note: this specific step required lots of debug and AI tools did not really help a lot.

Update the route to use SSL

Run

curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $APISIX_ADMIN_TOKEN" -X PUT -i -d '{
        "name": "1",
        "status": 1,
        "id": "1",
        "enable_websocket": false,
        "priority": 0,
        "uri": "/deck-api/*",
        "host": "deck-api.<<WILDCARD-DOMAIN>>",
        "methods": [
          "GET",
          "POST",
          "PUT",
          "DELETE"
        ],
        "upstream_id": "1"
}'

replacing «WILDCARD-DOMAIN».

Test

To test, map the service port to 9443 and then run the following:

curl -X POST https://deck-api.<<WILDCARD-DOMAIN>>:9443/deck-api/deck/create -H 'Content-Type: application/json'  -d '{
"deckType": "standard52"
}' --resolve deck-api.<<WILDCARD-DOMAIN>>:9443:127.0.0.1

Enable external SSL access:

We took a shortcut here using NodePort service and using OCI manually configured LoadBalancer as here:

Add details

Select Public visibility Reserved IP address Under Choose networking select your network. Under Subnet select your public subnet

Choose backends

On Add backends - select all 4 nodes Health Check: TCP Port: 31640

Configure listener

Specify the type of traffic your listener handles: TCP Port 433 Uncheck Use SSL - as we will let APISIX manage it.

Manage Logging

Whatever suits but select Request ID and use WWWWW header name.

Open external access to 443

For TCP traffic for ports 443, open external access from all IPs (0.0.0.0/0).

Map DNS records for your api

Map <<WILDCARD-DOMAIN>> using A records to external IPs of your LB e.g. *.api A 1.1.1.1 where 1.1.1.1 is your external IP.

Do NOT proxy as CF SSL cert will not cover it.
Note: we do have a certificate on APISIX so we can upload it to CF.

And that is it. All requests should now be hitting APISIX from a public IP.

Issues

Some of the issues with this setup are as follows:

  • DNS records should take the advantage of CF Proxying which in turn would utilise their DDOS protection.
  • Use DNS CNAME records to VMs rather than actual IPs to reduce management overhead.
  • No monitoring - one would want alerts on certificate replacement.
  • Not use full-scale admin account for synchronising certs.
  • IPs reported in APISIX logs are internal.