Allow for provisioning NGINX as DaemonSet

[sjberman]

Problem: With the initial provisioner implementation, the NGINX gateway could only be installed as a Deployment. Users may want to install NGINX as a DaemonSet.

Solution: Allow for provisioning the NGINX gateway as a DaemonSet.

Testing: Deployed NGINX as a DaemonSet

Closes #3120

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NGINX can be provisioned as a DaemonSet

[codecov]

Codecov Report

Attention: Patch coverage is 75.81699% with 37 lines in your changes missing coverage. Please review.

Project coverage is 86.78%. Comparing base (f9fa98c) to head (d3d722a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/mode/static/provisioner/provisioner.go	57.89%	23 Missing and 9 partials ⚠️
internal/mode/static/provisioner/handler.go	16.66%	0 Missing and 5 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3397   +/-   ##
=======================================
  Coverage   86.78%   86.78%           
=======================================
  Files         129      129           
  Lines       14928    15011   +83     
  Branches       62       62           
=======================================
+ Hits        12955    13028   +73     
- Misses       1823     1832    +9     
- Partials      150      151    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[salonichf5]

I noticed you have tested for deploying NGINX as daemonSet but have you tested for the migration from daemonSet to deployment ? and vice versa?

[sjberman]

@salonichf5 Yes, it does work. Though when migrating, the previous object is left behind without being cleaned up. Let me actually look into how best we can clean that up, it wasn't apparently obvious to me at first.

Though I'm not sure how common of a use case this would be.

[salonichf5]

@salonichf5 Yes, it does work. Though when migrating, the previous object is left behind without being cleaned up. Let me actually look into how best we can clean that up, it wasn't apparently obvious to me at first.

Though I'm not sure how common of a use case this would be.

yeah could be an edge case. Do you think it could lead network issues if we have two NGINX one of type deployment and daemonSet? If not, we can ignore this. I haven't manually tested this so I am not sure about the impact.

[sjberman]

@salonichf5 I pushed a change that fixes this.

[salonichf5]

lgtm