Skip to content

Commit 16436be

Browse files
tanmoysrttanmoysrt
authored
feat: redesign architecture and move to graphql (#109)
* chore: remove old codebase * feat: graphql initiated * feat: types updated * chore: database models updated * feat: enums updated * chore: code restructure * chore: update driver files * chore: cleanup code * feat: graphql routes for git credentials added * chore: remove unnecessary code * feat: graphql endpoints for image repository credentials added * feat: added functions for test git credential againsta a repo * feat: script added to generate graphql codes for gqlgen * chore: rename service name of application to just name * chore: update docs * feat: updated db design to remove json field * feat: git credential operations added * feat: image registry credential operations added * feat: new graphql schemas added * feat: functions added for persistent volume * feat: resolvers added for persistent volume * chore: format code * fix: bug fix persistent volume is exist check * feat: add resolvers and operations for applications * chore: update resolvers * chore: soft deletion for application added * chore: application create function added * chore: application create validation added * feat: enums added for types * chore: application create and fix bug fixed and resolvers added * chore: move to id from int to uint by scalar type to eliminate type casting * chore: if id is zero for a object return blank object * feat: sub resolvers added * chore: resolvers and domain related gql schema added * feat: for all deletion operation return boolean * feat: added functions to update applications * feat: resolvers added for domain, ingress rule and redirect rule * feat: checks added for duplicate mounting paths * feat: packages renamed * feat: init pubusb module * feat: added local pubsub functions * fix: stuck on channel full has been resolve * chore: make bufferlength configurable * fix: concurrent read write from same map fixed by mutex * fix: race conditions * feat: remote pubsub added * feat: updated code to ensure no duplicate channel closure in remote * feat: updated code for example * chore: cleanup * chore: delete dump.rdb * feat: function for register function against a queue added for local task queue * feat: function for task enqeue has been added * feat: added start consumers function to interface * feat: local task queue using go channels has been added * feat: added code for consumer in remote queue * feat: added code for publish tasks in rabbitmq * chore: cleanup * feat: added amqp connection establishment checks in consumers as well' * chore: load local task queue and pubsub client * feat: boilerplates added for worker * feat: application build worker completed * chore: code cleanup * chore: deploy application worker has been added * chore: ssl generation worker has been added * chore: ingress rule apply worker has been added * chore: redirect rule apply worker has been added * feat: ingress and redirect rule deletion workers added * feat: cronjobs added * feat: push deployment logs to redis * feat: subscription endpoint added for deployment log * feat: subscription endpoint added for runtime log * feat: added code upload route * feat: publish tasks to queue has been completed * feat: ssl generate push to queue added * feat: application deletion added * chore: run workers with server * fix: application creation bugs fixed * feat: bug fixed application creation and log * feat: bug fixed application deletion * chore: remove debug statement * fix: ingress rule issue fixed * feat: cronjobs added * fix: source code based application deployment issues fixed * fix: docker config generator added and tested * fix: image build from git issues fixed * chore: todo added * chore: todo added * chore: remove junk codes
1 parent f855766 commit 16436be

File tree

118 files changed

+23880
-3666
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

118 files changed

+23880
-3666
lines changed

‎.gitignore

+3-1
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,6 @@ dev.log
1212
.images
1313
.idea
1414
node_modules
15-
swiftwave
15+
swiftwave
16+
dump.rdb
17+
ignore_*

‎container_manager/image.go

+21-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import (
1010
)
1111

1212
/*
13-
`CreateImage` builds a Docker image from a Dockerfile and returns a scanner to read the build logs.
13+
CreateImage builds a Docker image from a Dockerfile and returns a scanner to read the build logs.
1414
It takes the Dockerfile content as a string, a map of build arguments, the path to the code directory, and the name of the image to be built.
1515
It returns a scanner to read the build logs and an error if any.
1616
*/
@@ -48,3 +48,23 @@ func (m Manager) CreateImage(dockerfile string, buildargs map[string]string, cod
4848
scanner := bufio.NewScanner(response.Body)
4949
return scanner, nil
5050
}
51+
52+
// PullImage pulls a Docker image from a remote registry and returns a scanner to read the pull logs.
53+
func (m Manager) PullImage(image string) (*bufio.Scanner, error) {
54+
// Pull the image
55+
scanner, err := m.client.ImagePull(m.ctx, image, types.ImagePullOptions{})
56+
if err != nil {
57+
return nil, errors.New("failed to pull the image")
58+
}
59+
return bufio.NewScanner(scanner), nil
60+
}
61+
62+
// ExistsImage checks if a Docker image exists locally.
63+
func (m Manager) ExistsImage(image string) bool {
64+
// Check if the image exists locally
65+
_, _, err := m.client.ImageInspectWithRaw(m.ctx, image)
66+
if err != nil {
67+
return false
68+
}
69+
return true
70+
}

‎container_manager/volume.go

+5-8
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,10 @@ package containermanger
22

33
import (
44
"errors"
5-
"fmt"
6-
75
"github.com/docker/docker/api/types/volume"
86
)
97

10-
// Create a new volume, return id of the volume
8+
// CreateVolume : Create a new volume, return id of the volume
119
func (m Manager) CreateVolume(name string) error {
1210
_, err := m.client.VolumeCreate(m.ctx, volume.CreateOptions{
1311
Name: name,
@@ -18,7 +16,7 @@ func (m Manager) CreateVolume(name string) error {
1816
return nil
1917
}
2018

21-
// Remove a volume by id
19+
// RemoveVolume : Remove a volume by id
2220
func (m Manager) RemoveVolume(id string) error {
2321
err := m.client.VolumeRemove(m.ctx, id, true)
2422
if err != nil {
@@ -27,14 +25,13 @@ func (m Manager) RemoveVolume(id string) error {
2725
return nil
2826
}
2927

30-
// Check if volume exists
28+
// ExistsVolume : Check if volume exists
3129
func (m Manager) ExistsVolume(id string) bool {
32-
d, err := m.client.VolumeInspect(m.ctx, id)
33-
fmt.Println(d)
30+
_, err := m.client.VolumeInspect(m.ctx, id)
3431
return err == nil
3532
}
3633

37-
// Fetch all volumes
34+
// FetchVolumes Fetch all volumes
3835
func (m Manager) FetchVolumes() ([]string, error) {
3936
volumes, err := m.client.VolumeList(m.ctx, volume.ListOptions{})
4037
if err != nil {

‎dev.linux.sh

+2
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,10 @@ fi
3535
echo -e "${GREEN}Fetching required images...${NC}"
3636
sudo docker pull ghcr.io/swiftwave-org/dashboard:develop
3737
sudo docker pull haproxytech/haproxy-debian:2.9
38+
sudo docker pull postgres:14
3839
sudo docker save -o ./.images/swiftwave-dashboard.tar ghcr.io/swiftwave-org/dashboard:develop
3940
sudo docker save -o ./.images/haproxy-debian.tar haproxytech/haproxy-debian:2.9
41+
sudo docker save -o ./.images/postgres.tar postgres:14
4042

4143
# build docker image
4244
echo "${GREEN}Building docker image...${NC}"

‎dev/Dockerfile

+4-4
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ FROM docker:dind
44
USER root
55

66
# Install packages
7-
RUN apk update --no-cache && apk add --no-cache gcc g++ make sqlite iproute2 net-tools bind-tools bash apache2-utils sudo shadow redis
7+
RUN apk update --no-cache && apk add --no-cache gcc g++ make git sqlite iproute2 net-tools bind-tools bash apache2-utils sudo shadow redis htop
88

99
ARG NEWUSER=user
1010
# Create non-root user
@@ -20,9 +20,9 @@ RUN addgroup docker
2020
RUN addgroup ${NEWUSER} docker
2121

2222
# Download and install Go
23-
RUN wget https://go.dev/dl/go1.21.0.linux-amd64.tar.gz
24-
RUN tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz
25-
RUN rm go1.21.0.linux-amd64.tar.gz
23+
RUN wget https://go.dev/dl/go1.21.3.linux-amd64.tar.gz
24+
RUN tar -C /usr/local -xzf go1.21.3.linux-amd64.tar.gz
25+
RUN rm go1.21.3.linux-amd64.tar.gz
2626
RUN echo "export PATH=\$PATH:/usr/local/go/bin" >> /home/${NEWUSER}/.bashrc
2727

2828

‎dev/docker-compose.dev.yml

+15
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,18 @@
11
version: '3.5'
22
services:
3+
postgres:
4+
image: postgres:14
5+
environment:
6+
POSTGRES_DB: swiftwave_development
7+
POSTGRES_PASSWORD: postgres
8+
PGDATA: /var/lib/postgresql/data/pgdata
9+
volumes:
10+
- postgres_data:/var/lib/postgresql@14/data/pgdata
11+
ports:
12+
- target: 5432
13+
published: 5432
14+
protocol: tcp
15+
mode: host
316
haproxy:
417
image: "haproxytech/haproxy-debian:2.9"
518
volumes:
@@ -28,6 +41,8 @@ services:
2841
published: 1212
2942
protocol: tcp
3043
mode: host
44+
volumes:
45+
postgres_data:
3146
networks:
3247
swarm_network:
3348
external: true

‎dev/start.sh

+2-3
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ echo "$haproxy_cfg" >"$SWIFTWAVE_HAPROXY_FOLDER/haproxy.cfg"
124124
# Pull docker images
125125
sudo docker load --input /app/.images/swiftwave-dashboard.tar
126126
sudo docker load --input /app/.images/haproxy-debian.tar
127+
sudo docker load --input /app/.images/postgres.tar
127128
# Start the services
128129
sudo docker stack deploy -c /app/dev/docker-compose.dev.yml $STACK_NAME
129130

@@ -143,9 +144,7 @@ export ADMIN_PASSWORD="$SWIFTWAVE_ADMIN_PASSWORD_HASH"
143144
export CODE_TARBALL_DIR="$SWIFTWAVE_APP_TARBALL_FOLDER"
144145
export SWARM_NETWORK="$SWARM_NETWORK"
145146
export HAPROXY_SERVICE_NAME="swiftwave_haproxy"
146-
export DATABASE_TYPE="sqlite"
147-
export SQLITE_DATABASE="/app/.data/gorm.db"
148-
export POSTGRESQL_URI=""
147+
export POSTGRESQL_DSN="host=localhost user=postgres password=postgres dbname=swiftwave_development port=5432 sslmode=disable TimeZone=Asia/Kolkata"
149148
export REDIS_ADDRESS="127.0.0.1:6379"
150149
export REDIS_PASSWORD=""
151150
export ACCOUNT_EMAIL_ID="$SWIFTWAVE_ADMIN_EMAIL"

‎docker_config_generator/utils.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -132,15 +132,15 @@ func (m Manager) generateConfigFromSourceCodeDirectory(directory string) (Docker
132132
}
133133

134134
// Generate DockerConfig from custom dockerfile. If GenerateConfigFromGitRepository fails to detect service, this function will be used.
135-
func (m Manager) GenerateConfigFromCustomDocker(dockerfile string) DockerFileConfig {
135+
func (m Manager) GenerateConfigFromCustomDocker(dockerfile string) (DockerFileConfig, error) {
136136
dockerConfig := DockerFileConfig{}
137137
dockerConfig.DetectedService = "Custom Dockerfile"
138138
dockerConfig.DockerFile = dockerfile
139139
dockerConfig.Variables = ParseBuildArgsFromDockerfile(dockerfile)
140140
if dockerConfig.Variables == nil {
141141
dockerConfig.Variables = map[string]Variable{}
142142
}
143-
return dockerConfig
143+
return dockerConfig, nil
144144
}
145145

146146
// DefaultArgs returns default arguments for a service.

‎generate_graphql.sh

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/usr/bin/env sh
2+
3+
cd ./swiftwave_service && go run github.com/99designs/gqlgen generate

‎go.mod

+19-15
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,56 @@
11
module github.com/swiftwave-org/swiftwave
22

3-
go 1.20
3+
go 1.21
4+
5+
toolchain go1.21.3
46

57
require (
8+
github.com/99designs/gqlgen v0.17.39
69
github.com/docker/docker v24.0.7+incompatible
710
github.com/go-git/go-git/v5 v5.10.0
811
github.com/go-redis/redis/v8 v8.11.5
9-
github.com/gorilla/websocket v1.5.0
12+
github.com/hashicorp/go-set v0.1.14
1013
github.com/labstack/echo/v4 v4.11.2
1114
github.com/mholt/acmez v1.2.0
12-
github.com/vmihailenco/taskq/v3 v3.2.9
15+
github.com/rabbitmq/amqp091-go v1.9.0
16+
github.com/vektah/gqlparser/v2 v2.5.10
1317
gopkg.in/yaml.v3 v3.0.1
1418
gorm.io/driver/postgres v1.5.4
15-
gorm.io/driver/sqlite v1.5.4
1619
gorm.io/gorm v1.25.5
1720
)
1821

1922
require (
2023
dario.cat/mergo v1.0.0 // indirect
2124
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
2225
github.com/acomagu/bufpipe v1.0.4 // indirect
26+
github.com/agnivade/levenshtein v1.1.1 // indirect
27+
github.com/cespare/xxhash/v2 v2.2.0 // indirect
2328
github.com/cloudflare/circl v1.3.3 // indirect
29+
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
2430
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
31+
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
2532
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
2633
github.com/go-git/go-billy/v5 v5.5.0 // indirect
2734
github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
2835
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
36+
github.com/gorilla/websocket v1.5.0 // indirect
37+
github.com/hashicorp/golang-lru/v2 v2.0.3 // indirect
2938
github.com/jackc/pgpassfile v1.0.0 // indirect
3039
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
3140
github.com/jackc/pgx/v5 v5.4.3 // indirect
41+
github.com/mitchellh/mapstructure v1.5.0 // indirect
3242
github.com/pjbgf/sha1cd v0.3.0 // indirect
43+
github.com/russross/blackfriday/v2 v2.1.0 // indirect
3344
github.com/skeema/knownhosts v1.2.0 // indirect
45+
github.com/sosodev/duration v1.1.0 // indirect
46+
github.com/urfave/cli/v2 v2.25.5 // indirect
47+
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
3448
golang.org/x/time v0.3.0 // indirect
3549
)
3650

3751
require (
38-
github.com/bsm/redislock v0.7.2 // indirect
39-
github.com/capnm/sysinfo v0.0.0-20130621111458-5909a53897f3 // indirect
40-
github.com/cespare/xxhash/v2 v2.2.0 // indirect
4152
github.com/containerd/containerd v1.7.2 // indirect
42-
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
43-
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
4453
github.com/emirpasic/gods v1.18.1 // indirect
45-
github.com/go-redis/redis_rate/v9 v9.1.2 // indirect
46-
github.com/hashicorp/golang-lru v0.5.4 // indirect
4754
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
4855
github.com/joho/godotenv v1.5.1
4956
github.com/kevinburke/ssh_config v1.2.0 // indirect
@@ -53,8 +60,6 @@ require (
5360
github.com/opencontainers/runc v1.1.8 // indirect
5461
github.com/sergi/go-diff v1.3.1 // indirect
5562
github.com/sirupsen/logrus v1.9.0 // indirect
56-
github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
57-
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
5863
github.com/xanzy/ssh-agent v0.3.3 // indirect
5964
gopkg.in/warnings.v0 v0.1.2 // indirect
6065
)
@@ -71,7 +76,6 @@ require (
7176
github.com/labstack/gommon v0.4.0 // indirect
7277
github.com/mattn/go-colorable v0.1.13 // indirect
7378
github.com/mattn/go-isatty v0.0.19 // indirect
74-
github.com/mattn/go-sqlite3 v1.14.17 // indirect
7579
github.com/moby/term v0.5.0 // indirect
7680
github.com/morikuni/aec v1.0.0 // indirect
7781
github.com/opencontainers/go-digest v1.0.0 // indirect
@@ -82,7 +86,7 @@ require (
8286
go.uber.org/atomic v1.11.0 // indirect
8387
go.uber.org/multierr v1.11.0 // indirect
8488
go.uber.org/zap v1.24.0 // indirect
85-
golang.org/x/crypto v0.14.0
89+
golang.org/x/crypto v0.14.0 // indirect
8690
golang.org/x/mod v0.12.0 // indirect
8791
golang.org/x/net v0.17.0 // indirect
8892
golang.org/x/sys v0.13.0 // indirect

0 commit comments

Comments
 (0)