Deployed with Pulumi and ran into some CRD & RBAC issues

[persinac]

I didn't want to create an issue, because it could be part of how pulumi works with helm charts. However, while going through the k8s ingress tutorial, I encountered some issues where I had to manually create some CRDs and set RBAC with the newly created CRDs - even though all of that should have been encapsulated in the helm chart.

CRDs in question:

• TCPEdge
• HTTPEdge
• IPPolicy

Then created the respective viewer + editor roles and bindings for the CRDs.

Kubernetes Version: v1.26.4
My version of helm:

version.BuildInfo{Version:"v3.13.1", GitCommit:"3547a4b5bf5edb5478ce352e18858d8a552a4110", GitTreeState:"clean", GoVersion:"go1.20.8"}

Version of k8s ingress controller chart:

apiVersion: v2
appVersion: 0.10.1
dependencies:
- name: common
  repository: https://charts.bitnami.com/bitnami
  tags:
  - bitnami-common
  version: 2.x.x
description: A Kubernetes ingress controller built using ngrok.
home: https://ngrok.com
icon: https://ngrok.com/static/img/brand/ngrok-favicon.svg
keywords:
- ngrok
- networking
- ingress
- edge
- api gateway
name: kubernetes-ingress-controller
sources:
- https://github.com/ngrok/kubernetes-ingress-controller
version: 0.12.1

I'm rather new to k8s and helm, but I'd like to start a discussion here on how to test that the helm chart is built and then builds the proper resources for a successful deployment with some IaC provider. I can easily test the pulumi side, but it's the helm chart that I'll need help with getting started. If something comes of the investigation - I'd also like to create a proper issue and attempt to address it.

[bobzilladev]

Hello there! Sorry to hear you're having issues getting the ingress controller working under Pulumi. Is there any further info you've come across while investigating? Hopefully any errors or logs from Pulumi's systems or the ngrok ingress controller that might be helpful in figuring out what might be going on have turned up. It may be that Pulumi is doing something specific that requires a certain configuration to be set to get the usual behavior to start working.

Thanks for bringing this up, it is something we haven't run into before.

cc: @Abdiramen on our team

[persinac]

No sweat! Thanks for responding. I'll attempt to get logs and more info this weekend.

[persinac]

Alright, good news - there's no issue. Pulumi and ngrok are working as expected. What happened to me is that the k8s tutorial helm command:

helm install ngrok-ingress-controller ngrok/kubernetes-ingress-controller \
  --namespace ngrok-ingress-controller \
  --create-namespace \
  --set credentials.apiKey=$NGROK_API_KEY \
  --set credentials.authtoken=$NGROK_AUTHTOKEN

Seems to pick the correct ngrok ingress controller version => kubernetes-ingress-controller v0.12. However, when translated to a pulumi chart reference:

ingress_controller_chart = Chart(
    "ngrok-ingress-controller",
    ChartOpts(
        chart="ngrok-ingress-controller",
        fetch_opts=FetchOpts(
            repo="https://ngrok.github.io/kubernetes-ingress-controller"
        ),
        namespace=namespace_ngrok_ingress_controller_name,
        ...
    ),
...

It actually picks the old version...
Chart.yaml:

apiVersion: v2
appVersion: 0.2.0
description: A Kubernetes ingress controller built using ngrok.
home: https://ngrok.com
icon: https://blog.ngrok.com/_next/image?url=%2Fassets%2Fblog%2Fplaceholder.png&w=1920&q=75
keywords:
- ngrok
- networking
- ingress
- edge
- api gateway
name: ngrok-ingress-controller
sources:
- https://github.com/ngrok/ngrok-ingress-controller
version: 0.4.0

crds:

PS C:\Users\...\AppData\Local\Temp\helm3922193291\ngrok-ingress-controller\templates\crds> dir


    Directory: C:\Users\...\AppData\Local\Temp\helm3922193291\ngrok-ingress-controller\templates\crds


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2024-01-07   3:44 PM           3571 ingress.k8s.ngrok.com_domains.yaml
-a----        2024-01-07   3:44 PM           2307 ingress.k8s.ngrok.com_tunnels.yaml

Changing my pulumi Chart to:

ingress_controller_chart = Chart(  
    "kubernetes-ingress-controller",  
    ChartOpts(  
        chart="kubernetes-ingress-controller",  
...

has resolved the CRD creation issue.

Chart.yaml:

apiVersion: v2
appVersion: 0.10.1
dependencies:
- name: common
  repository: https://charts.bitnami.com/bitnami
  tags:
  - bitnami-common
  version: 2.x.x
description: A Kubernetes ingress controller built using ngrok.
home: https://ngrok.com
icon: https://ngrok.com/static/img/brand/ngrok-favicon.svg
keywords:
- ngrok
- networking
- ingress
- edge
- api gateway
name: kubernetes-ingress-controller
sources:
- https://github.com/ngrok/kubernetes-ingress-controller
version: 0.12.1

CRDs:

PS C:\Users\...\AppData\Local\Temp\helm188085061\kubernetes-ingress-controller\templates\crds> dir


    Directory: C:\Users\...\AppData\Local\Temp\helm188085061\kubernetes-ingress-controller\templates\crds


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2024-01-07   4:00 PM           3653 ingress.k8s.ngrok.com_domains.yaml
-a----        2024-01-07   4:00 PM          59734 ingress.k8s.ngrok.com_httpsedges.yaml
-a----        2024-01-07   4:00 PM           4033 ingress.k8s.ngrok.com_ippolicies.yaml
-a----        2024-01-07   4:00 PM          48274 ingress.k8s.ngrok.com_ngrokmodulesets.yaml
-a----        2024-01-07   4:00 PM           4688 ingress.k8s.ngrok.com_tcpedges.yaml
-a----        2024-01-07   4:00 PM           5914 ingress.k8s.ngrok.com_tlsedges.yaml
-a----        2024-01-07   4:00 PM           2531 ingress.k8s.ngrok.com_tunnels.yaml

The old version has already been reported: https://github.com/ngrok/kubernetes-ingress-controller/issues/310 . So no further action is needed here. Regarding progress on issue #310 , though, would it be possible to take action on that?

[bobzilladev]

Ah, I'm glad you found the issue. It's difficult to remove the old naming without breaking existing users, but we can look into making it more obvious that is deprecated though. Thank you for the feedback!

[bobzilladev]

I think this has been resolved so going to close out the ticket, feel free to open another if there are other questions or issues! Thanks again for writing in.