Skip to content

feature: adding private repo capability #243

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vhbfernandes wants to merge 4 commits into
base: master
Choose a base branch
from
Open

feature: adding private repo capability #243

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fcfe308
chore: removing deprecated ioutil package
vhbfernandes Nov 16, 2023
d6fea4f
docs: updating docs to add new private repo feature
vhbfernandes Nov 16, 2023
f8e9196
feature: adding env to allow for private repo usage
vhbfernandes Nov 16, 2023
3b00739
chore: checking error
vhbfernandes Jan 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,13 @@ kubeconform -kubernetes-version 3.8.0 -schema-location 'https://raw.githubuserc
Summary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0 Skipped: 0
```

### Schemas behind private GitHub repos

By setting the environment variable `GITHUB_TOKEN=x` you can use schemas that are behind a private repository.
See [getting a token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) for info on how to get one command line token.
> Tip: if using GitHub CLI, `gh auth token` will give a valid token.


## Integrating Kubeconform in the CI

`Kubeconform` publishes Docker Images to Github's new Container Registry (ghcr.io). These images
Expand Down
6 changes: 3 additions & 3 deletions pkg/cache/ondisk.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"crypto/sha256"
"encoding/hex"
"fmt"
"io/ioutil"
"io"
"os"
"path"
"sync"
Expand Down Expand Up @@ -37,12 +37,12 @@ func (c *onDisk) Get(resourceKind, resourceAPIVersion, k8sVersion string) (inter
return nil, err
}

return ioutil.ReadAll(f)
return io.ReadAll(f)
}

// Set adds a JSON schema to the schema cache
func (c *onDisk) Set(resourceKind, resourceAPIVersion, k8sVersion string, schema interface{}) error {
c.Lock()
defer c.Unlock()
return ioutil.WriteFile(cachePath(c.folder, resourceKind, resourceAPIVersion, k8sVersion), schema.([]byte), 0644)
return os.WriteFile(cachePath(c.folder, resourceKind, resourceAPIVersion, k8sVersion), schema.([]byte), 0644)
}
17 changes: 11 additions & 6 deletions pkg/registry/http.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"crypto/tls"
"errors"
"fmt"
"io/ioutil"
"io"
"log"
"net/http"
"os"
Expand All @@ -13,13 +13,13 @@ import (
"github.com/yannh/kubeconform/pkg/cache"
)

type httpGetter interface {
Get(url string) (resp *http.Response, err error)
type httpDoer interface {
Do(*http.Request) (resp *http.Response, err error)
}

// SchemaRegistry is a file repository (local or remote) that contains JSON schemas for Kubernetes resources
type SchemaRegistry struct {
c httpGetter
c httpDoer
schemaPathTemplate string
cache cache.Cache
strict bool
Expand Down Expand Up @@ -72,8 +72,13 @@ func (r SchemaRegistry) DownloadSchema(resourceKind, resourceAPIVersion, k8sVers
return url, b.([]byte), nil
}
}
req, _ := http.NewRequest("GET", url, nil)

@yannh yannh Nov 18, 2023

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check the error? 🙏


resp, err := r.c.Get(url)
if token, exist := os.LookupEnv("GITHUB_TOKEN"); exist {

@yannh yannh Nov 18, 2023

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having some thoughts on how this will work when using multiple schema registries. When using the default registry plus one private one, we will be passing a GITHUB_TOKEN for the default registry too - does this work? This also won't work for multiple registries that might require different GITHUB_TOKENS, though I might be ok with that simplification for the sake of UI.
Also imagine you have a GITHUB_TOKEN set because you use Github, and run kubeconform against non-Github schema registries - the Github token will be passed 🤔

@vhbfernandes vhbfernandes Nov 20, 2023

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the first question, if you pass the token to github in a non-authenticated it works (and also all of them will fail if you have an invalid token)
As for using it with the token set and non-github http registries, might be an issue tho... How about if I make a new registry, github-specific and derive it from the http one?
To make it work with generic http registries I think we'll need to change the UI, to get the header name for example 🤔

@vhbfernandes vhbfernandes Nov 21, 2023

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can make it 2 env vars?
KUBECONFORM_AUTH_TOKEN and KUBECONFORM_AUTH_HEADER? It will still be limited to usage on every registry, but might work... I don't see a way of making it per-registry without changing the interface, maybe introducing a config file or sequential parameters

@vhbfernandes vhbfernandes Nov 22, 2023

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh another option might be creating a private registry struct implementing the registerer, and passing the values like -private-schema-location xxx -private-schema-header xxx -private-schema-auth xxx (order should be mandatory tho)

req.Header.Add("Authorization", fmt.Sprintf("token %s", token))
}

resp, err := r.c.Do(req)
if err != nil {
msg := fmt.Sprintf("failed downloading schema at %s: %s", url, err)
if r.debug {
Expand All @@ -99,7 +104,7 @@ func (r SchemaRegistry) DownloadSchema(resourceKind, resourceAPIVersion, k8sVers
return url, nil, fmt.Errorf(msg)
}

body, err := ioutil.ReadAll(resp.Body)
body, err := io.ReadAll(resp.Body)
if err != nil {
msg := fmt.Sprintf("failed parsing schema from %s: %s", url, err)
if r.debug {
Expand Down
32 changes: 16 additions & 16 deletions pkg/registry/http_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,29 @@ package registry
import (
"bytes"
"fmt"
"io/ioutil"
"io"
"net/http"
"strings"
"testing"
)

type mockHTTPGetter struct {
httpGet func(string) (*http.Response, error)
type mockHTTPDoer struct {
httpDo func(*http.Request) (*http.Response, error)
}

func newMockHTTPGetter(f func(string) (*http.Response, error)) *mockHTTPGetter {
return &mockHTTPGetter{
httpGet: f,
func newMockHTTPDoer(f func(*http.Request) (*http.Response, error)) *mockHTTPDoer {
return &mockHTTPDoer{
httpDo: f,
}
}
func (m mockHTTPGetter) Get(url string) (resp *http.Response, err error) {
return m.httpGet(url)
func (m mockHTTPDoer) Do(req *http.Request) (resp *http.Response, err error) {
return m.httpDo(req)
}

func TestDownloadSchema(t *testing.T) {
for _, testCase := range []struct {
name string
c httpGetter
c httpDoer
schemaPathTemplate string
strict bool
resourceKind, resourceAPIVersion, k8sversion string
Expand All @@ -34,7 +34,7 @@ func TestDownloadSchema(t *testing.T) {
}{
{
"error when downloading",
newMockHTTPGetter(func(url string) (resp *http.Response, err error) {
newMockHTTPDoer(func(req *http.Request) (resp *http.Response, err error) {
return nil, fmt.Errorf("failed downloading from registry")
}),
"http://kubernetesjson.dev",
Expand All @@ -47,10 +47,10 @@ func TestDownloadSchema(t *testing.T) {
},
{
"getting 404",
newMockHTTPGetter(func(url string) (resp *http.Response, err error) {
newMockHTTPDoer(func(req *http.Request) (resp *http.Response, err error) {
return &http.Response{
StatusCode: http.StatusNotFound,
Body: ioutil.NopCloser(strings.NewReader("http response mock body")),
Body: io.NopCloser(strings.NewReader("http response mock body")),
}, nil
}),
"http://kubernetesjson.dev",
Expand All @@ -63,10 +63,10 @@ func TestDownloadSchema(t *testing.T) {
},
{
"getting 503",
newMockHTTPGetter(func(url string) (resp *http.Response, err error) {
newMockHTTPDoer(func(req *http.Request) (resp *http.Response, err error) {
return &http.Response{
StatusCode: http.StatusServiceUnavailable,
Body: ioutil.NopCloser(strings.NewReader("http response mock body")),
Body: io.NopCloser(strings.NewReader("http response mock body")),
}, nil
}),
"http://kubernetesjson.dev",
Expand All @@ -79,10 +79,10 @@ func TestDownloadSchema(t *testing.T) {
},
{
"200",
newMockHTTPGetter(func(url string) (resp *http.Response, err error) {
newMockHTTPDoer(func(req *http.Request) (resp *http.Response, err error) {
return &http.Response{
StatusCode: http.StatusOK,
Body: ioutil.NopCloser(strings.NewReader("http response mock body")),
Body: io.NopCloser(strings.NewReader("http response mock body")),
}, nil
}),
"http://kubernetesjson.dev",
Expand Down
4 changes: 2 additions & 2 deletions pkg/registry/local.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package registry
import (
"errors"
"fmt"
"io/ioutil"
"io"
"log"
"os"
)
Expand Down Expand Up @@ -47,7 +47,7 @@ func (r LocalRegistry) DownloadSchema(resourceKind, resourceAPIVersion, k8sVersi
}

defer f.Close()
content, err := ioutil.ReadAll(f)
content, err := io.ReadAll(f)
if err != nil {
msg := fmt.Sprintf("failed to read schema at %s: %s", schemaFile, err)
if r.debug {
Expand Down