Troubleshooting Operator Issues
Describe Resources
You can use kubectl describe
on the custom resources for the ngrok operator to check their statuses if they are not behaving properly or are not accessible.
This is a good first step
Check Kubernetes Events
If something is not right with the operator you can check the events for the operator namespace.
kubectl get events -n ngrok-operator --sort-by='.lastTimestamp'
Example Output:
LAST SEEN TYPE REASON OBJECT MESSAGE
3m24s Normal Scheduled pod/ngrok-operator-manager-657c5b69d7-f7vvg Successfully assigned ngrok-operator/ngrok-operator-manager-657c5b69d7-f7vvg to k3d-ngrok-operator-server-0
3m25s Normal ScalingReplicaSet deployment/ngrok-operator-manager Scaled up replica set ngrok-operator-manager-657c5b69d7 to 1
3m25s Normal SuccessfulCreate replicaset/ngrok-operator-manager-657c5b69d7 Created pod: ngrok-operator-manager-657c5b69d7-f7vvg
3m24s Normal Started pod/ngrok-operator-manager-657c5b69d7-f7vvg Started container ngrok-operator
3m24s Normal Created pod/ngrok-operator-manager-657c5b69d7-f7vvg Created container ngrok-operator
3m24s Normal Pulled pod/ngrok-operator-manager-657c5b69d7-f7vvg Container image "docker.io/ngrok/ngrok-operator:0.15.0" already present on machine
3m15s Normal ScalingReplicaSet deployment/ngrok-operator-manager Scaled down replica set ngrok-operator-manager-74b76d8f6d to 0 from 1
3m15s Normal SuccessfulDelete replicaset/ngrok-operator-manager-74b76d8f6d Deleted pod: ngrok-operator-manager-74b76d8f6d-2679x
3m15s Normal Killing pod/ngrok-operator-manager-74b76d8f6d-2679x Stopping container ngrok-operator
2m55s Normal Updating boundendpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc Updating BoundEndpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc
2m55s Normal Updated boundendpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305 Updated BoundEndpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305
2m55s Normal Updated boundendpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc Updated BoundEndpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc
2m55s Normal Status boundendpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc Successfully reconciled status
2m55s Normal Updated boundendpoint/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc Updated Services
2m55s Normal Updated service/ngrok-22680758-eb09-576e-a7ac-dc3728458dfc Updated Upstream Service
2m55s Normal Updating boundendpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305 Updating BoundEndpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305
2m55s Normal LeaderElection lease/ngrok-operator-leader ngrok-operator-manager-657c5b69d7-f7vvg_72fb437b-38ad-444c-a9d8-d37349f2f677 became leader
2m55s Normal Status boundendpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305 Successfully reconciled status
2m55s Normal Updated boundendpoint/ngrok-012050e0-8f06-560c-953e-c9d675d41305 Updated Services
2m55s Normal Updated service/ngrok-012050e0-8f06-560c-953e-c9d675d41305 Updated Upstream Service
2m55s Normal Updating kubernetesoperator/ngrok-operator Updating KubernetesOperator/ngrok-operator
2m55s Normal Status kubernetesoperator/ngrok-operator Successfully reconciled status
2m55s Normal Updated kubernetesoperator/ngrok-operator Updated KubernetesOperator/ngrok-operator
Kubernetes events usually expire after an hour, so you might want to restart the operator to re-create the events while troubleshooting if the events happened more than an hour ago.
Check Operator Errors
To quickly check the operator logs for only errors, you can use a command such as the following. You can replace the name/namespace if you installed the operator somewhere other than the default location.
k logs -n ngrok-operator deploy/ngrok-operator-manager | grep "ERROR"
Resources not getting deleted
If you marked an ngrok custom resource for deletion but it is not getting deleted, there may be unresolved finalizers on it. You can refer to the finalizers page for more information about how the operator uses finalizers and how to clean up resources with dangling finalizers.
Cloud endpoints not getting created from Ingress / Gateway API
If you're applying Ingress
and/or Gateway API resources and do not see any CloudEndpoint
or AgentEndpoint
custom resources created by the operator, here are a few steps to identify the reason.
Ingress uses ngrok IngressClass?
The ngrok IngressClass
is required to use Ingress
resources. Check that your Ingress
uses the ngrok IngressClass
like the following example.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-ingress
namespace: default
spec:
ingressClassName: ngrok
rules:
- host: example-ingress.ngrok.io
http:
paths:
- backend:
service:
name: example-service
port:
number: 80
path: /
pathType: Prefix
Gateway API uses the ngrok GatewayClass?
The ngrok GatewayClass
is required to use Gateway API resources. If you do not already have it installed, you can create it with:
kubectl apply -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: ngrok
spec:
controllerName: ngrok.com/gateway-controller
EOF
Then, make sure your Gateway
uses the GatewayClass
like the following example:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: example-gateway
namespace: default
spec:
gatewayClassName: ngrok
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: example-gateway.ngrok.io
name: example
port: 443
protocol: HTTPS
Check Manager Pod Logs
The "manager" pods are responsible for performing the translation from Ingress
and Gateway API into native ngrok resources.
Check the manager pods for any errors to see if there is a problem with your configuration.
kubectl logs -n ngrok-operator deploy/ngrok-operator-manager | grep "ERROR"
Endpoints not showing up on the dashboard
If you have CloudEndpoint
and/or AgentEndpoint
resources that are not working, and you can't tell what's wrong with them using kubectl describe
, then check the logs for any errors processing them.
Check for AgentEndpoint
Error Logs
AgentEndpoint
resources are handled by the "agent" pods. Check these for any errors to see if there was an issue processing your AgentEndpoint
.
Common errors are an invalid traffic policy configuration that may prevent your endpoint from being created.
kubectl logs -n ngrok-operator deploy/ngrok-operator-agent | grep "ERROR"
Check for CloudEndpoint
Error Logs
CloudEndpoint
resources are handled by the "manager" pods. Check these for any errors to see if there was an issue processing your CloudEndpoint
.
Common errors are an invalid traffic policy configuration that may prevent your endpoint from being created.
kubectl logs -n ngrok-operator deploy/ngrok-operator-manager | grep "ERROR"
Resource Limits
If you've added several resources such as domains and endpoints, you may encounter a limit for the resource you are trying to create depending on your payment plan. If you are on the pay-as-you-go plan, then you will not encounter this issue and you will be able to create resources freely and only be billed for the resources you create. Check with the pricing and limits page for information about each plan and any limits associated with it.
Still Need Help?
If you aren't able to identify and resolve your issue, head over to the support page to reach out to ngrok support. You can also head over to the operator GitHub repo and see if anyone else encountered the same problem or file a new issue.