Done-11-10-2021

devops/01 - git/README.md

@@ -15,14 +15,45 @@
 
 1. What command can I use to view the commit history?
 
-1. What command can I use to undo the last commit?
+`git log [<options>] [<revision range>] [[--] <path>…]`
 
-1. What command can I use to create a new branch and a new tag?
+2. What command can I use to undo the last commit?
 
-1. How do I exclude a file / folder from a commit?
+- `git reset --soft HEAD~1` - return to the previous commit, saving changes as uncommitted local modifications
+- `git reset --hard HEAD~1` - return to the previous commit, removing changes
 
-1. In case of a merge conflict, what commands can be used to resolve it?
+3. What command can I use to create a new branch and a new tag?
 
-1. `*` What are pre-commit hooks and post-commit hooks, and what are they for?
+- `git branch <branch_name>` - create branch
+- `git tag <tag_name>` - create new tag
 
-1. `*` How do I change the last commit without adding a new commit?
+4. How do I exclude a file / folder from a commit?
+
+`git update-index --assume-unchanged <path>`
+
+5. In case of a merge conflict, what commands can be used to resolve it?
+
+- `git log --merge` - view commits that cause conflict
+- `git diff` - identify differences between states
+- `git checkout` - undo the changes to the file or change branch
+- `git reset --mixed` - undo changes to the working directory and staging area
+- `git merge --abort` - exit the merge process and return back to the state before the merging began
+- `git reset` - reset the conflicted files to their original state
+
+6. `*` What are pre-commit hooks and post-commit hooks, and what are they for?
+
+- pre-commit hooks - scripts, executed every time you run `git commit` before entering message, for identifying simple issues before submission
+- post-commit hooks - scripts, executed immediately after the commit-msg and used for notification purpose 
+
+7. `*` How do I change the last commit without adding a new commit?
+
+- change the last commit and commit message
+```
+git add <new_file>
+git commit --amend -m "message"
+``` 
+- change the last commit without changing commit message
+```
+git add <new_file>
+git commit --amend --no-edit
+``` 

devops/02 - dockerfile/Dockerfile

@@ -1,8 +1,11 @@
-FROM golang:1.15
+FROM golang:1.16-alpine as builder
 WORKDIR /usr/src/app
-COPY main.go .
-COPY go.mod .
-COPY .env .
-RUN go get -d github.com/joho/godotenv 
+COPY main.go go.mod ./
+RUN echo "PORT=8888" > .env
+RUN go get -d github.com/joho/godotenv
 RUN CGO_ENABLED=0 GOOS=linux go build -o simple-webpage .
+
+FROM alpine
+WORKDIR /app
+COPY --from=builder /usr/src/app /app
 CMD ["./simple-webpage"]

devops/02 - dockerfile/README.md

@@ -12,12 +12,23 @@ You need to optimize the Dockerfile by correcting or adding steps.
 
 1. What is Docker? Which technology is it based on?
 
+Docker is a tool for developing, building and running applications, separating the application from your infrastructure. Docker is based on containerization. It allows to package and run an application in an isolated environment - container.
+
 2. Look at the Docker file – what would you change in it?
 
+- separate commands into several stages, to decrease the size of the final image
+- use lighter base image instead of golang:1.15
+- add EXPOSE
+
 3. How do I pass variables to the Docker file when building and running the container?
 
+- `docker build --build-arg <variable>=<value>`
+- `docker run -e <variable>=<value> <image>[:<tag>] env`
+
 4. Why do we need multistage build ?
 
+Multistage build allows to reduce the size of container by leaving behind everything that is not required in the final image.
+
 ## Tasks
 
 * Dockerfile - generate .env file inside Dockerfile, provide value of port at build step.
@@ -26,4 +37,14 @@ You need to optimize the Dockerfile by correcting or adding steps.
 
 * Compare size of docker images with and without multistage build.
 
+image without multistage build
+![](./screenshots/initial-size.png)
+
+image with multistage build
+![](./screenshots/size-after-multistage.png)
+
 * Write down all commands which you have used.
+
+- `docker build -t provectus:latest .` - to build the image
+- `docker images provectus:latest` - to see the information about container
+- view Dockerfile to see all the changes

devops/03 - docker-compose/README.md

@@ -26,22 +26,70 @@ Docker compose with 3 applications (frontend + backend + DB).
 
 1. What is the difference between Docker Compose and dockerfile? Why do I need Docker Compose?
 
+Dockerfile is a text file that contains commands to build an image. Docker Compose is a tool to build and run multi-container dockers applications.
+We need Docker Compose here because we want to put database, the app itself and ngnix server in separate containers.
+
 2. How do I parameterize compose for different environments?
 
+- Put `${VARIABLE}` in docker-compose.yaml instead of desired value, where `VARIABLE` is the key from `.env` file.
+- Use `export VARIABLE=new` in command line to set new value of `VARIABLE`.
+- Use `docker-compose --env-file [path].env.dev up` in command line to substitute default env file by `[path].env.dev`.
+
 3. What types of entities are used in docker-compose and for what purpose?
 
+- version - version of the Compose file format
+- services - list of containers we want to run
+- image - pull image for the container
+- volumes - map files from host to container
+- networks - specify custom network for service
+- depends-on - dependency between services
+- ports - define ports for the service
+
 4. `*` How to output application logs?
 
+- `docker logs` - see information logged by a running container
+- `docker service logs` - see information logged by all containers participating in a service
+- set up logging from docker-compose.yaml:
+```
+logging:
+  driver: 
+  options:
+```
+
 4. `*` How to copy\upload a file from host machine to the container?
 
+- using command line
+
+`docker cp <src-path> <container>:<dest-path> `
+
+- using docker-compose.yml
+
+```    
+volumes:
+  - <src-path>:<dest-path>
+```
+
 5. `*` How to save file changes made inside the container?
 
+- copy a file from the container to the local file system via command line
+
+`docker cp <container>:<src-path> <local-dest-path>`
+
+-  commit the changes made to the container
+
+`docker commit <container_id> <repo>[:<tag>]`
 
 ## Tasks
 
 * Docker-compose has a bug - investigate it! What would you improve?
 
+I could not find any bugs
+
 * Docker-compose with an environment file. Create 2 different environment files for docker-compose
 
+I created two environment files: `.env` and `.env.dev`. By default docker uses values from `.env` file, 
+but we can use the following command to use values from another environment file:
+`docker-compose --env-file .env.dev up`
+
 * `*` Change the `docker-compose.yml` to run through dockerstack with code reuse (don't repeat yourself)
 

devops/03 - docker-compose/example/.env

@@ -0,0 +1,3 @@
+DB_TAG="9.6.5"
+NGINX_TAG="1.13.5"
+NGINX_PORT="8080:80"

devops/03 - docker-compose/example/.env.dev

@@ -0,0 +1,3 @@
+DB_TAG="9.6.5"
+NGINX_TAG="1.13.5"
+NGINX_PORT="8080:80"

devops/03 - docker-compose/example/docker-compose.yml

@@ -1,7 +1,7 @@
 version: '3'
 services:
   db:
-    image: "postgres:9.6.5"
+    image: "postgres:$DB_TAG"
     volumes:
       - "dbdata:/var/lib/postgresql/data"
     env_file:
@@ -20,9 +20,9 @@ services:
     depends_on:
       - db
   nginx:
-    image: "nginx:1.13.5"
+    image: "nginx:$NGINX_TAG"
     ports:
-      - "8080:80"
+      - $NGINX_PORT
     volumes:
       - ./conf.d:/etc/nginx/conf.d
     networks:

devops/04 - bash/README.md

@@ -2,10 +2,30 @@
 * Docker on your local machine to launch testing environment
 
 ## Questions
-* Mention the advantages and disadvantages of bash scripts
-* What types of variables are used in bash?
-* What is pipes in shell script and how to use it?
-* How to show unique values of the second column from the CSV comma-separated file and sort results with the alphabet?
+1. Mention the advantages and disadvantages of bash scripts
+
+Bash scripts are easy to write and understand, they allow to automate frequently used commands. 
+They may be executed in any Unix-like operating systems without any modifications.
+
+But bash scripts are not suited for large and complex tasks, also they really syntax sensitive - any extra space may lead to the error.
+
+2. What types of variables are used in bash?
+
+- System-Defined Variables (BASH, BASH_VERSION, COLUMNS, LOGNAME, HOME, OSTYPE, PWD, USERNAME)
+- User-Defined Variables
+
+
+3. What is pipes in shell script and how to use it?
+
+Pipe is a form of redirection; it is used to send the output of one command to another for further processing.
+
+`command_1 | command_2 | command_3 | .... | command_N`
+
+Output of `command_1` goes to `command_2` and so on.
+
+4. How to show unique values of the second column from the CSV comma-separated file and sort results with the alphabet?
+
+`cat file.csv  | cut -f2 -d , | sort | uniq`
 
 ## Task
 Create script which provide aggregated information about analysises and they datasets. Information about each analysis/deployment could be retrieved from `awscli.sh`:
@@ -55,3 +75,7 @@ If you use OS different than macOS/Linux need to use Docker to launch your scrip
 ``` bash
 docker run -it -rm -v <path-to-this-folder>:/src -w /src --entrypoint bash cfmanteiga/alpine-bash-curl-jq --  <your-script>
 ```
+
+The code for the task is in `script.sh`
+
+To run the code use the following command `./script.sh get_information <some-analysis>`, where `some-analysis` is one of the folowing: `users, billings, usage reports`.

devops/04 - bash/script.sh

@@ -0,0 +1,35 @@
+#! /bin/bash
+
+get_information() {
+        datasetID=$(./awscli.sh get_analysis $2 | jq '.datasetsIds' | tr '[]' ' '  | tr '","' ' ')
+  IDS=(${datasetID// / })
+  count=$(./awscli.sh get_analysis $1 |xargs| sed 's/.*datasetsIds\: \[ \(.*\)\].*/\1/'|wc -w)
+  cat << EOF
+
+{
+        "analysises": {
+                "$2": {
+                        "datasetsCount": $count
+                        "datasets": [
+EOF
+i=0
+
+    while [ $i -le $((count-1)) ]
+           do
+      singleID="${IDS[i]}"
+             cat << EOF
+                                {
+                                        "arn": "$(bash ./awscli.sh get_dataset $singleID | jq '.datasetArn' | tr -d '"' | tr -d '"arn:')",
+                                        "name": "$(./awscli.sh get_dataset $singleID | jq '.name' | tr -d '"')"
+                                }
+EOF
+      i=$(( $i + 1 ))
+done
+    cat << EOF
+      ]
+    }
+        }
+}
+EOF
+}
+$1 $@

devops/05 - cloud-ops/README.md

@@ -13,10 +13,27 @@ https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRan
 
 * Create an AWS account and protect the root user with MFA https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#id_root-user_manage_mfa
 
+![](./screenshots/mfa.png)
+
 * Create an EC2 instance, connect to it through SSH and install apache or nginx manually, terminate it then. 
+![](./screenshots/aws-instance.png)
+```
+alice@alice-book:~/provectus$ chmod 400 devops.pem
+alice@alice-book:~/provectus$ ssh -i "devops.pem" [email protected]
+```
+![](./screenshots/ssh-connect.png)
+```
+ubuntu@ip-172-31-24-2:~$ sudo apt update
+ubuntu@ip-172-31-24-2:~$ sudo apt install apache2
+ubuntu@ip-172-31-24-2:~$ sudo ufw app list
+ubuntu@ip-172-31-24-2:~$ sudo ufw allow 'Apache'
+```
+![](./screenshots/apache-status.png)
 
 * Create an EC2 instance, provision software (apache/nginx) using Cloudformation, validate the installation, finally delete the stack. 
   Provide the resulting template as outcome of this task.
+
+  Output is in the `ec2apache.yaml` file.
 
 * (*) Given that we have a properly written Cloudformation template come up with an idea how we 
   can create and update a cloudformation stack based on this template automatically. Try to implement it.
@@ -25,6 +42,20 @@ https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRan
 
 1. What the benefits of the cloud computing?
 
+- scalability - cloud computer service provider can handle scaling up or down your operation for you
+- better collaboration across teams - wherever your team is allocated, you all have access to your service
+- reduce cost - it is cheaper to use the resources of your cloud computing service provider rather than purchase expensive systems and equipment
+- data availability (fault tolerance) - cloud-based services provide quick data recovery
+
 2. What is the Infrastructure As Code? Pros/cons of it?
 
+It is the process of managing and provisioning infrastructure using config files.
+It uses declarative definitions, rather than manual processes.
+
+Pros:
+- administrators and operators may automate the configuration process, which allows to improve productivity
+- automation may reduce the risk associated with human error
 
+Cons:
+- it may be hard to figure out how to configure such infrastucture
+- monitoring may be challenging and maybe you will have to find new tools for this  

devops/05 - cloud-ops/ec2apache.yaml

@@ -0,0 +1,35 @@
+Resources:
+  Ec2Instance:
+    Type: AWS::EC2::Instance
+    Properties:
+      InstanceType: t2.micro
+      ImageId: ami-090717c950a5c34d3
+      SecurityGroups: !Ref SSHSecurityGroup
+      KeyName: !Ref 'KeyName'
+      Tags:
+        - Key: Name
+          Value: Cloudformation-task
+        - Key: Type
+          Value: Server Instance
+      UserData:
+        Fn::Base64: !Sub |
+          #!/bin/bash
+          yum update -y # update existing packages
+          yum install -y httpd # install apache web server
+          systemctl start httpd
+          systemctl enable httpd
+          echo "Hello World" > /var/www/html/index.html
+  SSHSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupDescription: Allow ssh
+      SecurityGroupIngress:
+        - IpProtocol: tcp
+          FromPort: 22
+          ToPort: 22
+          CidrIp: 0.0.0.0/0
+Parameters:
+  KeyName:
+    Description: Name of an existing EC2 KeyPair to connect via ssh.
+    Type: AWS::EC2::KeyPair::KeyName
+    ConstraintDescription: must be the name of an existing EC2 KeyPair.