Once your code has been built and tested, you can deploy it automatically using GitHub Actions.

Here’s a simplified example:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy
        run: npm deploy

In this example, we use a custom command script for deploy. In your package.json you will have a section for scripts:

  "scripts": {
    "deploy": "echo this is a deploy",
  }

Testing is crucial in CI/CD pipelines. GitHub Actions can automatically run your tests every time someone pushes to your repository.

For example, if you’re using Node.js and Jest for your tests:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Install Dependencies
        run: npm install
      - name: Run tests
        run: npm test

In a real-world scenario, you often need multiple jobs to run different tasks in parallel or sequentially to speed up the process or manage dependencies. In this section, you’ll learn how to set up a workflow with multiple jobs.

Example: npm test and Deployment

Here’s an example YAML configuration file for a GitHub Actions workflow that has two jobs: one for running tests and another for deployment.

name: CI/CD Pipeline

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Install Dependencies
        run: npm install
      - name: Run tests
        run: npm test

  deploy:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Deploy
        run: npm deploy

Understanding the Workflow

1. jobs: Contains two job blocks: test and deploy.
2. runs-on: Specifies the type of runner that the job will run on.
3. steps: The sequence of tasks that will be executed in each job.
4. needs: This key in the deploy job specifies that it needs the test job to complete successfully before it starts.

By defining both jobs in the same YAML file, GitHub Actions will know to run them as part of the same workflow. The needs keyword ensures that the deploy job will only run if the test job completes successfully. We add a layer of protection against bugs making their way into production.

And that’s how you set up a GitHub Actions workflow with multiple jobs. This allows you to make your CI/CD process more robust and maintainable.

Alerting is a key aspect of monitoring; it allows you to know in real-time if something goes wrong or if a certain performance threshold has been met or exceeded. By creating alerts, you make your system more resilient and reduce the time needed to respond to incidents.

Creating Alerts with AWS CloudWatch

AWS CloudWatch provides a feature to set up alerts based on specific conditions or thresholds. These alerts can then be forwarded to other AWS services like SNS for notifications via email, SMS, or other methods.

To create an alert in CloudWatch:

1. Open the CloudWatch console in your AWS Management Console.
2. In the navigation pane, choose “Alarms,” then choose “Create Alarm.”
3. Choose “Select metric” and specify the metric and conditions for your alarm.

# AWS CLI example to create a CPU utilization alarm for an EC2 instance
aws cloudwatch put-metric-alarm --alarm-name cpu-high --metric-name CPUUtilization --namespace AWS/EC2 --statistic Average --period 300 --threshold 70 --comparison-operator GreaterThanThreshold  --dimensions Name=InstanceId,Value=i-12345678 --evaluation-periods 2

Metrics to Monitor in EC2 and RDS

EC2

• CPU Utilization: Measures the compute power required by an EC2 instance.
• Disk Read/Writes: Monitors the read and write operations on the instance storage.
• Network In/Out: Measures the network traffic to and from the instance.

More on EC2 CloudWatch Metrics

RDS

• CPU Utilization: Measures the compute power used by your database instance.
• Database Connections: The number of database connections in use.
• Free Storage Space: Monitors the available storage space of your database instance.

More on RDS CloudWatch Metrics

🎯 1. Meet the criteria

People

• We need at least two trainees to start a module
• A minimum ratio of 1:6 volunteers to qualifying trainees

Skills

• Trainees must have completed the Fundamentals, JS1, JS2, JS3 and Servers modules
• Trainees must have completed the module entry challenge

Space

• A place to meet on Saturdays

Time

• Volunteers and trainees must commit to 5 weeks of Saturday sessions

Money

• The Docs will pay for trainee expenses like food, bus tickets, equipment and childcare

A Plan

• Here’s the plan: it’s this module! When you have all these things, you can start a Cloud module.

📅 2. Set the date!

In any development workflow, especially one that involves deployments, it’s common to have configuration settings that should not be hard-coded in the codebase. This includes API keys, database URLs, and other sensitive information. GitHub Actions allows the use of environment variables and secrets to manage these configurations securely.

Github Environment Variables

Environment variables are key-value pairs that you can create and modify as part of your workflow. They can be accessed in your GitHub Actions YAML file via env context. For example, to set a Node.js environment, you can do:

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      NODE_ENV: production

And then use it in a script like this:

- name: Using Environment Variable
  run: echo Node environment is ${{ env.NODE_ENV }}

Every module must have a tangible product

It’s important to know things, but it’s even more important to show you can do things. That’s why every module at The Docs has a product.

The Cloud module product is a series of resources you will build into your portfolio as you deploy your (existing) portfolio projects. These resources will demonstrate your understanding of cloud computing and your ability to work with cloud technologies.

Read

• What is an incident report?
• What is a runbook?
• What is a case study?

Terraform has a structured workflow:

1. Initialization: Prepare a Terraform working directory with necessary providers using the terraform init command.
2. Planning: Preview infrastructure changes with terraform plan to ensure alignment with your goals.
3. Applying: Implement the desired infrastructure state using terraform apply.
4. Destroy: To dismantle the infrastructure, use terraform destroy.

For further details, dive into Terraform’s official documentation to enhance your understanding and capabilities.

Implementing scalability in the cloud involves various strategies and technologies that help you adapt to the demands of your user base and workloads. Below are some popular methods for achieving scalability in the cloud using AWS services.

Horizontal Scaling and Vertical Scaling

Horizontal Scaling with AWS EC2

In horizontal scaling, additional instances are added to or removed from a resource pool as needed. AWS EC2 offers Auto Scaling Groups that allow you to scale the number of EC2 instances up or down automatically based on pre-defined conditions such as CPU usage or incoming network traffic.

# AWS CLI command to update an existing Auto Scaling group
aws autoscaling update-auto-scaling-group --auto-scaling-group-name my-asg --min-size 1 --max-size 5

Vertical Scaling with AWS RDS

AWS RDS allows you to resize your database instances to better meet your application’s needs. For instance, you could switch from a db.t2.micro to a db.m4.large instance type, thus vertically scaling your database.

# AWS CLI command to modify an RDS instance
aws rds modify-db-instance --db-instance-identifier mydbinstance --db-instance-class db.m4.large

Load Balancing with AWS ELB

AWS Elastic Load Balancer (ELB) distributes incoming traffic across multiple EC2 instances. ELB can be an essential part of both horizontal and vertical scaling strategies.

Elasticity and Provisioning with AWS S3

AWS S3 is designed to scale automatically. It automatically partitions your buckets as they grow, without any need for manual intervention. S3 also offers provisioned capacity for demanding workloads.

State Management in AWS RDS

Managing state in scalable systems is crucial. AWS RDS supports Multi-AZ deployments enhancing high availability and failover support for DB instances.

For more detailed information on AWS scalability strategies, you can refer to the AWS Well-Architected Framework.

By understanding and implementing these AWS-specific concepts, you’ll be better prepared to build scalable and reliable cloud-based applications.

Terraform is an open-source tool created by HashiCorp that allows you to define and provide infrastructure as code (IaC). It uses its own domain-specific language known as HashiCorp Configuration Language (HCL) and can manage infrastructure across multiple cloud service providers.

Why Choose Terraform?

• Provider Agnostic: Terraform isn’t limited to a single cloud. With extensive provider support, you can manage a multi-cloud setup using one framework.

• Immutable Infrastructure: Terraform encourages the creation of unchangeable infrastructure. If updates are needed, the current resources are replaced, ensuring consistency and reducing potential drift.

• Declarative Syntax: Describe what you want with Terraform’s declarative configuration, and let it handle how to achieve that state.

• State Management: Terraform uses a state file to map real-world resources to your configuration, ensuring resources are managed correctly.

Core Concepts in Terraform

• Resource: The primary component in a Terraform configuration. A resource represents an infrastructure object, such as a VM or network.

• Provider: These are plugins that Terraform uses to interact with cloud service APIs, allowing it to manage resources in those clouds.

• Variables: Parameterize your configurations using variables, making your IaC modular and reusable.

• State: Terraform’s state maps configurations to real-world resources, tracking the resources it manages.

What are GitHub Actions?

GitHub Actions is an automation framework that runs in your GitHub repository. It can build, test, and deploy your code right from GitHub.

Key Components

1. Workflows: Orchestrates your CI/CD process.
2. Jobs: Sets of steps that execute sequentially.
3. Steps: Individual tasks within a job.
4. Actions: Pre-built steps that you can use in your jobs.

Setting up a Simple Workflow

To set up a simple GitHub Actions workflow, navigate to your GitHub repository and then to the Actions tab. From there, you can create a new workflow.

Here’s a basic example:

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

This YAML file specifies a single job called build that runs on an ubuntu-latest runner machine. The job has a single step that uses actions/checkout@v3 to download the code from the GitHub repository.

For more information on Github Actions Syntax refer to their documentation: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions

Monitoring is crucial for understanding the behavior of your applications and infrastructure, particularly in cloud environments where many components work together to deliver an application. Monitoring not only helps you to diagnose and fix issues faster, but it also plays a pivotal role in optimizing performance and planning for scalability.

Why is Monitoring Important?

• Performance Tuning: Real-time data helps you understand how well your system is performing and where bottlenecks may be forming.

• Issue Identification: Proactive monitoring can alert you to issues before they impact your users, allowing for quicker resolution.

• Security: Monitoring can help identify unauthorized access or anomalies that could indicate a security breach.

Observability

Observability is the ability to understand the internal state of your system just by examining its outputs. An observable system makes it easier to identify the root causes of failures and debug or optimize code. Observability is often considered the superset of monitoring, logging, and other diagnostic activities.

Monitoring with AWS CloudWatch

AWS CloudWatch is a robust monitoring solution that allows you to collect and track metrics, collect and monitor log files, and set alarms. CloudWatch can monitor AWS resources like EC2 instances, RDS databases, and S3 buckets, as well as custom metrics generated by your applications.

Using AWS CloudWatch, you can create dashboards to visualize metrics, set thresholds for alarms, and even automatically react to changes in your AWS resources.

By setting up monitoring tools and learning the importance of observability, you are taking proactive steps to maintain high availability and performance of your cloud services and applications.

Secrets are similar to environment variables. The difference is that secrets are encrypted and only exposed to selected actions, adding an extra layer of security. Use secrets for storing sensitive data like passwords and API keys.

You can add secrets via GitHub by navigating to your repository, then clicking on Settings -> Secrets -> New Repository Secret. Enter the secret’s name and value, and it will be encrypted and stored securely.

To use a secret, use the secrets context in your YAML file like this:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Accessing secret
        run: echo Accessing secret ${{ secrets.MY_SECRET }}

Replace MY_SECRET with your actual secret’s name, stored in the GitHub repository.

For more information about environment variable refer to their documentation: https://docs.github.com/en/actions/learn-github-actions/variables and for information about secrets refer to: https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions

Install Docker on your computer

This is a requirement. The official website has decent documentation on how to do so. It will push “Docker Desktop” if you’re on a Mac, and “Docker CLI” on Linux. Both are ok for us: we need the underlying program and CLI.

Go over the Docker 101 tutorial

https://www.docker.com/101-tutorial/

Docker 101 is a interesting tutorial from the official website to get yourself familiar with Docker. Go through it.

What is Terraform State?

Terraform uses a state file to keep track of the current status of your infrastructure. This state file maps the resources in your configuration to real-world resources, providing a way to store attributes and manage resource properties. It is crucial to understand how Terraform handles this state, especially as you work on larger projects.

Local vs. Remote State

By default, Terraform stores the state file in the local filesystem. However, for any significant project or team-based work, a remote backend is recommended. Remote backends like AWS S3, Azure Blob Storage, or Google Cloud Storage allow you to store the Terraform state in a centralized, shared, and secure location.

Locking State

State locking prevents others from acquiring the lock and ensures that multiple users can’t make conflicting changes. Locking is automatically handled in some backends like AWS S3 when used in conjunction with a DynamoDB table.

AWS S3 as a Backend with DynamoDB Locking

Here is a basic example of how you can set up AWS S3 as a backend for storing your Terraform state file, and use AWS DynamoDB for state locking:

terraform {
  backend "s3" {
    bucket  = "my-terraform-state-bucket"
    key     = "terraform.tfstate"
    region  = "eu-west-1"
    dynamodb_table = "my-lock-table"
  }
}

This configuration tells Terraform to use an S3 bucket named my-terraform-state-bucket and a DynamoDB table named my-lock-table for state locking.

For more details on various aspects of Terraform state management, you can read the following sections from the official Terraform documentation:

• General State Management
• S3 Backend Configuration
• State Locking with DynamoDB

These resources will give you a comprehensive understanding of how to manage Terraform state effectively, including using remote backends like AWS S3 and state locking mechanisms like DynamoDB.

Vertical Scaling

Also known as “scaling up,” this involves adding more resources like CPU, RAM, or storage to an existing server. While easy to implement, there are physical limits to how much you can scale vertically.

Horizontal Scaling

Also known as “scaling out,” this involves adding more servers to share the application’s load. This is often considered more flexible but can be complex to manage.

📚 Resources

• Scalability
• Vertical vs. Horizontal Scaling

What Are Terraform Modules?

Terraform modules encapsulate a piece of Terraform configuration into a reusable, shareable unit. This allows for better organization and abstraction of Terraform code. More details can be found in the Terraform Modules Documentation.

Why Are Terraform Modules Needed?

• Reuse of Code: Avoid repetition of similar blocks of code in different parts of the project.
• Simplified Configuration: Hide the complexity of your setup by exposing only the most relevant variables.
• Version Control: Modules can be versioned, and you can specify which version to use in your main configuration.

Why Are Modules Good?

1. Modularity: As the name suggests, Terraform modules bring modularity to your infrastructure. You can encapsulate a whole set of functionalities and features into one module.
2. Reusability: Once you’ve written a module for a specific piece of infrastructure, you can reuse it across multiple environments or even multiple projects.
3. Maintainability: Using modules makes your Terraform configuration easier to maintain. If a change is needed for a specific piece of infrastructure, you can make the change in just one place.

For best practices on writing modules, check the Terraform Module Best Practices.

Structure of a Terraform Module

A typical module directory structure looks like this:

my_terraform_module/
|-- main.tf
|-- variables.tf
|-- outputs.tf
|-- README.md

• main.tf contains the core resource declarations.
• variables.tf contains any input variables that the module accepts.
• outputs.tf contains any output variables that the module will produce.
• README.md contains documentation for the module.

Example: An AWS S3 Bucket Module

To help you understand how a Terraform module works, let’s create an example of an AWS S3 bucket module. The example includes two parts:

1. The code for the module itself.
2. How to use the module in a Terraform script.

Code for the Module

First, let’s define the module. Create a new directory for the module and inside it, create a main.tf file with the following content:

provider "aws" {
  region = "eu-west-1"
}

resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  acl    = "private"

  tags = {
    Name        = var.bucket_name
    Environment = var.environment
  }
}

output "bucket_arn" {
  value = aws_s3_bucket.this.arn
}

Create a variables.tf file to declare the input variables:

variable "bucket_name" {
  description = "The name of the bucket"
  type        = string
}

variable "environment" {
  description = "The environment this bucket will be part of"
  type        = string
  default     = "dev"
}

Using the Module

To use this module in a Terraform script, create a main.tf in a different directory and refer to the module like this:

module "s3_bucket" {
  source       = "./path/to/module/directory"
  bucket_name  = "my-new-bucket"
  environment  = "prod"
}

output "new_bucket_arn" {
  value = module.s3_bucket.bucket_arn
}

Run terraform init and terraform apply to create the S3 bucket using your module.

This example should give you a good understanding of how to create a basic Terraform module and how to use it. For more detailed information, you can refer to the Terraform AWS S3 Bucket Documentation and Terraform Modules Documentation.

CI, or Continuous Integration, is a development practice where developers integrate code into a shared repository frequently. This usually happens multiple times a day and is complemented by automated tests.

CD, or Continuous Deployment/Delivery, takes the code changes from a repository, automatically builds and tests them, and then automatically deploys the tested code to a live service.

These practices make it easier to catch bugs earlier and make deployments faster and more reliable.

Infrastructure as Code (IaC) is the practice of managing and provisioning your cloud resources through code, rather than manual operations or one-off scripts. Essentially, it lets you script your infrastructure, the same way you script your application code. In doing so, IaC allows you to apply the principles of software development, such as version control and continuous integration, to your infrastructure.

Why is IaC Needed?

• Consistency: IaC allows for a consistent and standardized environment, which reduces errors that can occur when infrastructure is set up manually.
• Scalability: IaC makes it easier to scale infrastructure up or down in response to demand.
• Version Control: Infrastructure can be treated like any other codebase, enabling you to use version control systems to track changes and rollback if needed.
• Collaboration: Developers and operations can work together more seamlessly. Infrastructure becomes part of the application development process.
• Cost-Efficiency: By automating repetitive tasks, you save both time and resources, thus reducing operational costs.

The Evolution from Manual Operations to IaC

• Manual Operations: Initially, system administrators would manually configure servers and other resources. This was time-consuming and error-prone.
• Scripting: Automating configurations through scripts improved the speed but still lacked standardization and could be hard to manage for complex systems.
• Configuration Management Tools: Tools like Ansible, Chef, and Puppet brought more structure but were often specific to certain types of operations.
• Infrastructure as Code: IaC brings a holistic approach, where the entire environment is coded, versioned, and automated.

Scalability is the capability of a system to handle a growing amount of work or its potential to be enlarged to accommodate that growth. In the context of cloud computing and web applications, scalability often refers to adding more resources or modifying the system architecture to manage increased load.

Let’s consider a hypothetical social media platform as an example. When the platform is new and has a small number of users, a single server might be sufficient to handle the traffic and data. But what happens when the platform goes viral and suddenly attracts millions of users?

• Database Queries: The number of queries hitting the database will grow exponentially.

• File Storage: Photos, videos, and other media files will consume more and more storage.

• Bandwidth: More users mean more data transfer.

To continue providing a seamless user experience, the system needs to scale. This could mean adding more servers, optimizing database queries, or increasing the bandwidth capacity.

So, in tech, scalability isn’t just about making things bigger; it’s about making them more efficient to handle a growing user base or data volume.

• Performance: As your application grows, you’ll need to ensure it remains responsive.

• Cost-Effectiveness: Proper scalability strategies can help you use resources more efficiently, thereby saving costs.

• Availability: Scalability ensures your application can handle increased load without going down.

Scalability Challenges

• Complexity: Scaling often involves re-architecting your application, which can be complex and time-consuming.

• Cost: While cloud resources are generally inexpensive, costs can add up as you scale.

• Management: Increased resources and complexity require better management strategies.

High Availability

High Availability (HA) is closely related to scalability. It’s about ensuring that a service is available as much as possible, even in the face of various kinds of failures (hardware failure, software crashes, etc.). While scalability focuses on accommodating more users, high availability focuses on providing reliable service to the existing user base. In many cases, these two go hand-in-hand because a system that scales well is often better equipped to maintain high availability.

For more information, you can read about High Availability in the AWS Well-Architected Framework.

Now that you understand the basics of Terraform and the need for Infrastructure as Code, let’s try an exercise that doesn’t interact with any cloud providers. We’ll use the null_resource for demonstration and learning purposes.

Example: Using null_resource

This example uses Terraform’s null_resource.

resource "null_resource" "example" {
  provisioner "local-exec" {
    command = "echo Hello, World"
  }
}

In this example, we define a null_resource with the name example. The local-exec provisioner will run the command specified (echo Hello, World) on your local machine when you apply this Terraform configuration.