diff --git a/devops/01 - git/README.md b/devops/01 - git/README.md index 1fc3f573..6510463b 100644 --- a/devops/01 - git/README.md +++ b/devops/01 - git/README.md @@ -15,14 +15,45 @@ 1. What command can I use to view the commit history? -1. What command can I use to undo the last commit? +`git log [] [] [[--] …]` -1. What command can I use to create a new branch and a new tag? +2. What command can I use to undo the last commit? -1. How do I exclude a file / folder from a commit? +- `git reset --soft HEAD~1` - return to the previous commit, saving changes as uncommitted local modifications +- `git reset --hard HEAD~1` - return to the previous commit, removing changes -1. In case of a merge conflict, what commands can be used to resolve it? +3. What command can I use to create a new branch and a new tag? -1. `*` What are pre-commit hooks and post-commit hooks, and what are they for? +- `git branch ` - create branch +- `git tag ` - create new tag -1. `*` How do I change the last commit without adding a new commit? +4. How do I exclude a file / folder from a commit? + +`git update-index --assume-unchanged ` + +5. In case of a merge conflict, what commands can be used to resolve it? + +- `git log --merge` - view commits that cause conflict +- `git diff` - identify differences between states +- `git checkout` - undo the changes to the file or change branch +- `git reset --mixed` - undo changes to the working directory and staging area +- `git merge --abort` - exit the merge process and return back to the state before the merging began +- `git reset` - reset the conflicted files to their original state + +6. `*` What are pre-commit hooks and post-commit hooks, and what are they for? + +- pre-commit hooks - scripts, executed every time you run `git commit` before entering message, for identifying simple issues before submission +- post-commit hooks - scripts, executed immediately after the commit-msg and used for notification purpose + +7. `*` How do I change the last commit without adding a new commit? + +- change the last commit and commit message +``` +git add +git commit --amend -m "message" +``` +- change the last commit without changing commit message +``` +git add +git commit --amend --no-edit +``` diff --git a/devops/02 - dockerfile/Dockerfile b/devops/02 - dockerfile/Dockerfile index f3258b32..bad8db1c 100644 --- a/devops/02 - dockerfile/Dockerfile +++ b/devops/02 - dockerfile/Dockerfile @@ -1,8 +1,11 @@ -FROM golang:1.15 +FROM golang:1.16-alpine as builder WORKDIR /usr/src/app -COPY main.go . -COPY go.mod . -COPY .env . -RUN go get -d github.com/joho/godotenv +COPY main.go go.mod ./ +RUN echo "PORT=8888" > .env +RUN go get -d github.com/joho/godotenv RUN CGO_ENABLED=0 GOOS=linux go build -o simple-webpage . + +FROM alpine +WORKDIR /app +COPY --from=builder /usr/src/app /app CMD ["./simple-webpage"] diff --git a/devops/02 - dockerfile/README.md b/devops/02 - dockerfile/README.md index 7fddb4e8..91f793e1 100644 --- a/devops/02 - dockerfile/README.md +++ b/devops/02 - dockerfile/README.md @@ -12,12 +12,23 @@ You need to optimize the Dockerfile by correcting or adding steps. 1. What is Docker? Which technology is it based on? +Docker is a tool for developing, building and running applications, separating the application from your infrastructure. Docker is based on containerization. It allows to package and run an application in an isolated environment - container. + 2. Look at the Docker file – what would you change in it? +- separate commands into several stages, to decrease the size of the final image +- use lighter base image instead of golang:1.15 +- add EXPOSE + 3. How do I pass variables to the Docker file when building and running the container? +- `docker build --build-arg =` +- `docker run -e = [:] env` + 4. Why do we need multistage build ? +Multistage build allows to reduce the size of container by leaving behind everything that is not required in the final image. + ## Tasks * Dockerfile - generate .env file inside Dockerfile, provide value of port at build step. @@ -26,4 +37,14 @@ You need to optimize the Dockerfile by correcting or adding steps. * Compare size of docker images with and without multistage build. +image without multistage build +![](./screenshots/initial-size.png) + +image with multistage build +![](./screenshots/size-after-multistage.png) + * Write down all commands which you have used. + +- `docker build -t provectus:latest .` - to build the image +- `docker images provectus:latest` - to see the information about container +- view Dockerfile to see all the changes \ No newline at end of file diff --git a/devops/02 - dockerfile/screenshots/initial-size.png b/devops/02 - dockerfile/screenshots/initial-size.png new file mode 100644 index 00000000..2b5e3a9e Binary files /dev/null and b/devops/02 - dockerfile/screenshots/initial-size.png differ diff --git a/devops/02 - dockerfile/screenshots/size-after-multistage.png b/devops/02 - dockerfile/screenshots/size-after-multistage.png new file mode 100644 index 00000000..de390690 Binary files /dev/null and b/devops/02 - dockerfile/screenshots/size-after-multistage.png differ diff --git a/devops/03 - docker-compose/README.md b/devops/03 - docker-compose/README.md index 2f2a4f23..6ad33f43 100644 --- a/devops/03 - docker-compose/README.md +++ b/devops/03 - docker-compose/README.md @@ -26,22 +26,70 @@ Docker compose with 3 applications (frontend + backend + DB). 1. What is the difference between Docker Compose and dockerfile? Why do I need Docker Compose? +Dockerfile is a text file that contains commands to build an image. Docker Compose is a tool to build and run multi-container dockers applications. +We need Docker Compose here because we want to put database, the app itself and ngnix server in separate containers. + 2. How do I parameterize compose for different environments? +- Put `${VARIABLE}` in docker-compose.yaml instead of desired value, where `VARIABLE` is the key from `.env` file. +- Use `export VARIABLE=new` in command line to set new value of `VARIABLE`. +- Use `docker-compose --env-file [path].env.dev up` in command line to substitute default env file by `[path].env.dev`. + 3. What types of entities are used in docker-compose and for what purpose? +- version - version of the Compose file format +- services - list of containers we want to run +- image - pull image for the container +- volumes - map files from host to container +- networks - specify custom network for service +- depends-on - dependency between services +- ports - define ports for the service + 4. `*` How to output application logs? +- `docker logs` - see information logged by a running container +- `docker service logs` - see information logged by all containers participating in a service +- set up logging from docker-compose.yaml: +``` +logging: + driver: + options: +``` + 4. `*` How to copy\upload a file from host machine to the container? +- using command line + +`docker cp : ` + +- using docker-compose.yml + +``` +volumes: + - : +``` + 5. `*` How to save file changes made inside the container? +- copy a file from the container to the local file system via command line + +`docker cp : ` + +- commit the changes made to the container + +`docker commit [:]` ## Tasks * Docker-compose has a bug - investigate it! What would you improve? +I could not find any bugs + * Docker-compose with an environment file. Create 2 different environment files for docker-compose +I created two environment files: `.env` and `.env.dev`. By default docker uses values from `.env` file, +but we can use the following command to use values from another environment file: +`docker-compose --env-file .env.dev up` + * `*` Change the `docker-compose.yml` to run through dockerstack with code reuse (don't repeat yourself) diff --git a/devops/03 - docker-compose/example/.env b/devops/03 - docker-compose/example/.env new file mode 100644 index 00000000..3574670b --- /dev/null +++ b/devops/03 - docker-compose/example/.env @@ -0,0 +1,3 @@ +DB_TAG="9.6.5" +NGINX_TAG="1.13.5" +NGINX_PORT="8080:80" \ No newline at end of file diff --git a/devops/03 - docker-compose/example/.env.dev b/devops/03 - docker-compose/example/.env.dev new file mode 100644 index 00000000..3574670b --- /dev/null +++ b/devops/03 - docker-compose/example/.env.dev @@ -0,0 +1,3 @@ +DB_TAG="9.6.5" +NGINX_TAG="1.13.5" +NGINX_PORT="8080:80" \ No newline at end of file diff --git a/devops/03 - docker-compose/example/docker-compose.yml b/devops/03 - docker-compose/example/docker-compose.yml index 917b4f65..b45bac05 100644 --- a/devops/03 - docker-compose/example/docker-compose.yml +++ b/devops/03 - docker-compose/example/docker-compose.yml @@ -1,7 +1,7 @@ version: '3' services: db: - image: "postgres:9.6.5" + image: "postgres:$DB_TAG" volumes: - "dbdata:/var/lib/postgresql/data" env_file: @@ -20,9 +20,9 @@ services: depends_on: - db nginx: - image: "nginx:1.13.5" + image: "nginx:$NGINX_TAG" ports: - - "8080:80" + - $NGINX_PORT volumes: - ./conf.d:/etc/nginx/conf.d networks: diff --git a/devops/04 - bash/README.md b/devops/04 - bash/README.md index 3782c2ef..264b3e57 100644 --- a/devops/04 - bash/README.md +++ b/devops/04 - bash/README.md @@ -2,10 +2,30 @@ * Docker on your local machine to launch testing environment ## Questions -* Mention the advantages and disadvantages of bash scripts -* What types of variables are used in bash? -* What is pipes in shell script and how to use it? -* How to show unique values of the second column from the CSV comma-separated file and sort results with the alphabet? +1. Mention the advantages and disadvantages of bash scripts + +Bash scripts are easy to write and understand, they allow to automate frequently used commands. +They may be executed in any Unix-like operating systems without any modifications. + +But bash scripts are not suited for large and complex tasks, also they really syntax sensitive - any extra space may lead to the error. + +2. What types of variables are used in bash? + +- System-Defined Variables (BASH, BASH_VERSION, COLUMNS, LOGNAME, HOME, OSTYPE, PWD, USERNAME) +- User-Defined Variables + + +3. What is pipes in shell script and how to use it? + +Pipe is a form of redirection; it is used to send the output of one command to another for further processing. + +`command_1 | command_2 | command_3 | .... | command_N` + +Output of `command_1` goes to `command_2` and so on. + +4. How to show unique values of the second column from the CSV comma-separated file and sort results with the alphabet? + +`cat file.csv | cut -f2 -d , | sort | uniq` ## Task Create script which provide aggregated information about analysises and they datasets. Information about each analysis/deployment could be retrieved from `awscli.sh`: @@ -55,3 +75,7 @@ If you use OS different than macOS/Linux need to use Docker to launch your scrip ``` bash docker run -it -rm -v :/src -w /src --entrypoint bash cfmanteiga/alpine-bash-curl-jq -- ``` + +The code for the task is in `script.sh` + +To run the code use the following command `./script.sh get_information `, where `some-analysis` is one of the folowing: `users, billings, usage reports`. diff --git a/devops/04 - bash/script.sh b/devops/04 - bash/script.sh new file mode 100755 index 00000000..567f4ffe --- /dev/null +++ b/devops/04 - bash/script.sh @@ -0,0 +1,35 @@ +#! /bin/bash + +get_information() { + datasetID=$(./awscli.sh get_analysis $2 | jq '.datasetsIds' | tr '[]' ' ' | tr '","' ' ') + IDS=(${datasetID// / }) + count=$(./awscli.sh get_analysis $1 |xargs| sed 's/.*datasetsIds\: \[ \(.*\)\].*/\1/'|wc -w) + cat << EOF + +{ + "analysises": { + "$2": { + "datasetsCount": $count + "datasets": [ +EOF +i=0 + + while [ $i -le $((count-1)) ] + do + singleID="${IDS[i]}" + cat << EOF + { + "arn": "$(bash ./awscli.sh get_dataset $singleID | jq '.datasetArn' | tr -d '"' | tr -d '"arn:')", + "name": "$(./awscli.sh get_dataset $singleID | jq '.name' | tr -d '"')" + } +EOF + i=$(( $i + 1 )) +done + cat << EOF + ] + } + } +} +EOF +} +$1 $@ \ No newline at end of file diff --git a/devops/05 - cloud-ops/README.md b/devops/05 - cloud-ops/README.md index aa9d9773..659bbc45 100644 --- a/devops/05 - cloud-ops/README.md +++ b/devops/05 - cloud-ops/README.md @@ -13,10 +13,27 @@ https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRan * Create an AWS account and protect the root user with MFA https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#id_root-user_manage_mfa +![](./screenshots/mfa.png) + * Create an EC2 instance, connect to it through SSH and install apache or nginx manually, terminate it then. +![](./screenshots/aws-instance.png) +``` +alice@alice-book:~/provectus$ chmod 400 devops.pem +alice@alice-book:~/provectus$ ssh -i "devops.pem" ubuntu@ec2-34-217-50-254.us-west-2.compute.amazonaws.com +``` +![](./screenshots/ssh-connect.png) +``` +ubuntu@ip-172-31-24-2:~$ sudo apt update +ubuntu@ip-172-31-24-2:~$ sudo apt install apache2 +ubuntu@ip-172-31-24-2:~$ sudo ufw app list +ubuntu@ip-172-31-24-2:~$ sudo ufw allow 'Apache' +``` +![](./screenshots/apache-status.png) * Create an EC2 instance, provision software (apache/nginx) using Cloudformation, validate the installation, finally delete the stack. Provide the resulting template as outcome of this task. + + Output is in the `ec2apache.yaml` file. * (*) Given that we have a properly written Cloudformation template come up with an idea how we can create and update a cloudformation stack based on this template automatically. Try to implement it. @@ -25,6 +42,20 @@ https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRan 1. What the benefits of the cloud computing? +- scalability - cloud computer service provider can handle scaling up or down your operation for you +- better collaboration across teams - wherever your team is allocated, you all have access to your service +- reduce cost - it is cheaper to use the resources of your cloud computing service provider rather than purchase expensive systems and equipment +- data availability (fault tolerance) - cloud-based services provide quick data recovery + 2. What is the Infrastructure As Code? Pros/cons of it? +It is the process of managing and provisioning infrastructure using config files. +It uses declarative definitions, rather than manual processes. + +Pros: +- administrators and operators may automate the configuration process, which allows to improve productivity +- automation may reduce the risk associated with human error +Cons: +- it may be hard to figure out how to configure such infrastucture +- monitoring may be challenging and maybe you will have to find new tools for this diff --git a/devops/05 - cloud-ops/ec2apache.yaml b/devops/05 - cloud-ops/ec2apache.yaml new file mode 100644 index 00000000..b34d827d --- /dev/null +++ b/devops/05 - cloud-ops/ec2apache.yaml @@ -0,0 +1,35 @@ +Resources: + Ec2Instance: + Type: AWS::EC2::Instance + Properties: + InstanceType: t2.micro + ImageId: ami-090717c950a5c34d3 + SecurityGroups: !Ref SSHSecurityGroup + KeyName: !Ref 'KeyName' + Tags: + - Key: Name + Value: Cloudformation-task + - Key: Type + Value: Server Instance + UserData: + Fn::Base64: !Sub | + #!/bin/bash + yum update -y # update existing packages + yum install -y httpd # install apache web server + systemctl start httpd + systemctl enable httpd + echo "Hello World" > /var/www/html/index.html + SSHSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Allow ssh + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: 0.0.0.0/0 +Parameters: + KeyName: + Description: Name of an existing EC2 KeyPair to connect via ssh. + Type: AWS::EC2::KeyPair::KeyName + ConstraintDescription: must be the name of an existing EC2 KeyPair. diff --git a/devops/05 - cloud-ops/screenshots/apache-status.png b/devops/05 - cloud-ops/screenshots/apache-status.png new file mode 100644 index 00000000..c7a72b46 Binary files /dev/null and b/devops/05 - cloud-ops/screenshots/apache-status.png differ diff --git a/devops/05 - cloud-ops/screenshots/aws-instance.png b/devops/05 - cloud-ops/screenshots/aws-instance.png new file mode 100644 index 00000000..12c92e36 Binary files /dev/null and b/devops/05 - cloud-ops/screenshots/aws-instance.png differ diff --git a/devops/05 - cloud-ops/screenshots/mfa.png b/devops/05 - cloud-ops/screenshots/mfa.png new file mode 100644 index 00000000..8e99d149 Binary files /dev/null and b/devops/05 - cloud-ops/screenshots/mfa.png differ diff --git a/devops/05 - cloud-ops/screenshots/ssh-connect.png b/devops/05 - cloud-ops/screenshots/ssh-connect.png new file mode 100644 index 00000000..0b6663d9 Binary files /dev/null and b/devops/05 - cloud-ops/screenshots/ssh-connect.png differ