Showing posts with label EC2. Show all posts
Showing posts with label EC2. Show all posts

Monday, 21 December 2020

Boto3 examples(SGs,DynamoDB,EC2 and Cloudwatch)

 Create a new Dynamo table using Boto3

# Get the service resource.
dynamodb = boto3.resource('dynamodb')

# Create the DynamoDB table.
table = dynamodb.create_table(
    TableName='users',
    KeySchema=[
        {
            'AttributeName': 'username',
            'KeyType': 'HASH'
        },
        {
            'AttributeName': 'last_name',
            'KeyType': 'RANGE'
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'username',
            'AttributeType': 'S'
        },
        {
            'AttributeName': 'last_name',
            'AttributeType': 'S'
        },
    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 5,
        'WriteCapacityUnits': 5
    }
)

# Wait until the table exists.
table.meta.client.get_waiter('table_exists').wait(TableName='users')

# Print out some data about the table.
print(table.item_count)
This method will return a Dynamodb.Table

Once you have a DynamoDB.Table resource you can add new items to the table using DynamoDB.Table.put_item():

table.put_item(
   Item={
        'username': 'janedoe',
        'first_name': 'Jane',
        'last_name': 'Doe',
        'age': 25,
        'account_type': 'standard_user',
    }
)
You can then retrieve the object using DynamoDB.Table.get_item():

response = table.get_item(
    Key={
        'username': 'janedoe',
        'last_name': 'Doe'
    }
)
item = response['Item']
print(item)
You can then update attributes of the item in the table:

table.update_item(
    Key={
        'username': 'janedoe',
        'last_name': 'Doe'
    },
    UpdateExpression='SET age = :val1',
    ExpressionAttributeValues={
        ':val1': 26
    }
)

You can also delete the item using DynamoDB.Table.delete_item():

table.delete_item(
    Key={
        'username': 'janedoe',
        'last_name': 'Doe'
    }
)
With the table full of items, you can then query or scan the items in the table using the DynamoDB.Table.query() or DynamoDB.Table.scan() methods respectively. To add conditions to scanning and querying the table, you will need to import the boto3.dynamodb.conditions.Key and boto3.dynamodb.conditions.Attr classes. The boto3.dynamodb.conditions.Key should be used when the condition is related to the key of the item. The boto3.dynamodb.conditions.Attr should be used when the condition is related to an attribute of the item:

from boto3.dynamodb.conditions import Key, Attr
This queries for all of the users whose username key equals johndoe:

response = table.query(
    KeyConditionExpression=Key('username').eq('johndoe')
)
items = response['Items']
print(items)
Similarly you can scan the table based on attributes of the items. For example, this scans for all the users whose age is less than 27:

response = table.scan(
    FilterExpression=Attr('age').lt(27)
)
items = response['Items']
print(items)
Describing instances
import boto3
ec2 = boto3.client('ec2')
response = ec2.describe_instances()
print(response)

Monitor and unmonitor instances
import sys
import boto3


ec2 = boto3.client('ec2')
if sys.argv[1] == 'ON':
    response = ec2.monitor_instances(InstanceIds=['INSTANCE_ID'])
else:
    response = ec2.unmonitor_instances(InstanceIds=['INSTANCE_ID'])
print(response)

Start and stop instances

import boto3
from botocore.exceptions import ClientError

instance_id = sys.argv[2]
action = sys.argv[1].upper()

ec2 = boto3.client('ec2')


if action == 'ON':
    # Do a dryrun first to verify permissions
    try:
        ec2.start_instances(InstanceIds=[instance_id], DryRun=True)
    except ClientError as e:
        if 'DryRunOperation' not in str(e):
            raise

    # Dry run succeeded, run start_instances without dryrun
    try:
        response = ec2.start_instances(InstanceIds=[instance_id], DryRun=False)
        print(response)
    except ClientError as e:
        print(e)
else:
    # Do a dryrun first to verify permissions
    try:
        ec2.stop_instances(InstanceIds=[instance_id], DryRun=True)
    except ClientError as e:
        if 'DryRunOperation' not in str(e):
            raise

    # Dry run succeeded, call stop_instances without dryrun
    try:
        response = ec2.stop_instances(InstanceIds=[instance_id], DryRun=False)
        print(response)
    except ClientError as e:
        print(e)
		
Reboot instances
import boto3
from botocore.exceptions import ClientError


ec2 = boto3.client('ec2')

try:
    ec2.reboot_instances(InstanceIds=['INSTANCE_ID'], DryRun=True)
except ClientError as e:
    if 'DryRunOperation' not in str(e):
        print("You don't have permission to reboot instances.")
        raise

try:
    response = ec2.reboot_instances(InstanceIds=['INSTANCE_ID'], DryRun=False)
    print('Success', response)
except ClientError as e:
    print('Error', e)
Describe Regions and Availability Zones

ec2 = boto3.client('ec2')

# Retrieves all regions/endpoints that work with EC2
response = ec2.describe_regions()
print('Regions:', response['Regions'])

# Retrieves availability zones only for region of the ec2 object
response = ec2.describe_availability_zones()
print('Availability Zones:', response['AvailabilityZones'])

Create a security group and rules
import boto3
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2')

response = ec2.describe_vpcs()
vpc_id = response.get('Vpcs', [{}])[0].get('VpcId', '')

try:
    response = ec2.create_security_group(GroupName='SECURITY_GROUP_NAME',
                                         Description='DESCRIPTION',
                                         VpcId=vpc_id)
    security_group_id = response['GroupId']
    print('Security Group Created %s in vpc %s.' % (security_group_id, vpc_id))

    data = ec2.authorize_security_group_ingress(
        GroupId=security_group_id,
        IpPermissions=[
            {'IpProtocol': 'tcp',
             'FromPort': 80,
             'ToPort': 80,
             'IpRanges': [{'CidrIp': '0.0.0.0/0'}]},
            {'IpProtocol': 'tcp',
             'FromPort': 22,
             'ToPort': 22,
             'IpRanges': [{'CidrIp': '0.0.0.0/0'}]}
        ])
    print('Ingress Successfully Set %s' % data)
except ClientError as e:
    print(e)
	
Delete a security group
import boto3
from botocore.exceptions import ClientError

# Create EC2 client
ec2 = boto3.client('ec2')

# Delete security group
try:
    response = ec2.delete_security_group(GroupId='SECURITY_GROUP_ID')
    print('Security Group Deleted')
except ClientError as e:
    print(e)

Wednesday, 16 December 2020

IAM role , ALB and EC2 Terraform creatiion

 ALB Terraform:

HTTP and HTTPs are listeners with the default actions.

With ALB rules always use actions like "forward" ,"redirect" and "Fixed response".

Below is the code for ALB.

module "alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 5.0"

  name = "my-alb"

  load_balancer_type = "application"

  vpc_id             = "vpc-abcde012"
  subnets            = ["subnet-abcde012", "subnet-bcde012a"]
  security_groups    = ["sg-edcd9784", "sg-edcd9785"]

  access_logs = {
    bucket = "my-alb-logs"
  }

  target_groups = [
    {
      name_prefix      = "pref-"
      backend_protocol = "HTTP"
      backend_port     = 80
      target_type      = "instance"
    }
  ]

  https_listeners = [
    {
      port               = 443
      protocol           = "HTTPS"
      certificate_arn    = "arn:aws:iam::123456789012:server-certificate/test_cert-123456789012"
      target_group_index = 0
    }
  ]

  http_tcp_listeners = [
    {
      port               = 80
      protocol           = "HTTP"
      target_group_index = 0
    }
  ]

  tags = {
    Environment = "Test"
  }
}

IAM User:

resource "aws_iam_user" "lb" {
  name = "loadbalancer"
  path = "/system/"

  tags = {
    tag-key = "tag-value"
  }
}

resource "aws_iam_access_key" "lb" {
  user = aws_iam_user.lb.name
}

resource "aws_iam_user_policy" "lb_ro" {
  name = "test"
  user = aws_iam_user.lb.name

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:Describe*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

Sunday, 20 September 2020

Terraform - Part 1

 Definition:

Automate the provisioning of resources over the cloud.

To learn this topic , we need to below tools and SW are ready in our system

Terraform:https://www.terraform.io/downloads.html

Visual studio code: https://code.visualstudio.com/download

AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html

Programmatic access from Terraform to AWS console.

Lets create a IAM user to provide access to Terraform.

Click "Next permissions"

Once User created ,Make a note of secret key and access key IDs to configure the access from AWS CLI to the console.

Open command prompt and type AWS to check your CLI access.

type aws configure command


Next download and configure Terraform tool

Source docs.oracle.com

To see the terraform success installation, check as below


Now lets use Visual studio code editor in my case, you case whatever editor your convenient.

Create a directory and with simple file name, in my case sample.tf. Form name must have extension .tf as a naming convention.

Folder structure will be like below


sample.tf

provider "aws"{

region="us-west-2"

}

resource "aws_vpc" "main" {

  cidr_block       = "10.0.0.0/16"

  instance_tenancy = "default"  

  tags = {

    Name = "main"

  }

}

resource "aws_subnet" "subnet1" {

  vpc_id     = aws_vpc.main.id

  cidr_block = "10.0.1.0/24"

   tags = {

    Name = "Subnet1"

  }

}

Once you enter the vpc and subnet code , you need to initialise the terraform by using the below 

Before running to create AWS resources , lets see what we have us-west-2 region.

We have default VPC and subnets as below

VPC:

subnet:


Now execute "terraform apply" command in terminal and give "yes" as you want to approve to create the resources what you have asked.

See in above screen , we can see two resources are created ,lets open our console and check.

Check the resources in "us-west-2" region 

VPC:

subnet



With this we can easily provision the AWS resources with terraform code.
Once it successfully executes the creation of resources, it will create the state form where it has the code to create the resources in AWS.
In our case it will be like below.

Now lets learn deep out resource filed and how we can separate the variables and initialize them .
See vars.tf , how we are separating
execute again by using terraform apply 
This is how we can initialize the variables and execute them.
Now lets create multiple subnets using loops in terraform.

Since we are in us-west-2 region and lets check how many AZs are available at present.
Lets create CIDR blocks dynamically and create subnets against CIDR.
Now the vars.tf looks like below.
variable "region"{
    default="us-west-2"
}
variable "vpc_cidr"{
    default="10.0.0.0/16"
}
variable "subnet_cidr"{
    type=list(string)
    default=["10.0.5.0/24","10.0.2.0/24","10.0.3.0/24","10.0.4.0/24"]
}
variable "azs"{
    type=list(string)
    default=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"]
}

sample.tf like below
provider "aws"{
  region=var.region
}
resource "aws_vpc" "main" {
  cidr_block       = var.vpc_cidr
  instance_tenancy = "default"
  
  tags = {
    Name = "main"
  }
}
resource "aws_subnet" "subnets" {
  count=length(var.azs)
  vpc_id     = aws_vpc.main.id
  cidr_block = element(var.subnet_cidr,count.index)
  
  tags = {
    Name = "Subnet1"
  }
}

apply the terraform and see the resources are created as below.



When you see the above pic, we have subnet names are equal. Lets change the name of the subnet using the count.index

Just change the Name = "Subnet-${count.index+1}". and apply terraform.


As of Now we are hardcoded region and AZs but we can get those also dynamically using data sources.

Just change the code in vars.tf as below
#variable "azs"{
 #   type=list(string)
  #  default=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"]
#}
# Declare the data source
data "aws_availability_zones" "azs" {
  state = "available"
}

update the sample.tf as below
resource "aws_subnet" "subnets" {
  count=length(data.aws_availability_zones.azs.names)
  vpc_id     = aws_vpc.main.id
  cidr_block = element(var.subnet_cidr,count.index)
  
  tags = {
    Name = "Subnet-${count.index+1}"
  }
}

apply the terraform, so here we can see same output as above though we are getting dynamically the data of AZs.

Since we the subnets are created in single AZ like below

Lets create subnets in different AZs as you want.

sample.tf:
count=length(data.aws_availability_zones.azs.names)
  availability_zone=element(data.aws_availability_zones.azs.names,count.index)


Lets apply terraform
If you observe above pic, it created only 3 subnets again and 3 subnets deleted , because it make used the existing one and created the freshly with new AZs as below

Lets see other functions in Terraform.

Map:
Lets see without Map , how we can create ec2 instance using terraform.
vars.tf:
variable "region"{
    default="ap-south-1"
}
variable "ec2_ami"{
    default="ami-76d6f519"
}
provider.tf:
provider "aws"{
    region=var.region
}
ec2-instance.tf:
resource "aws_instance" "web" {
  ami           = var.ec2_ami
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}

Apply terraform and check the EC2 is created.

In above code , we specified the region and AMI , But when we change the region  AMI id will change.
So , to get the AMI id based on the region we are using the Map.

Get the AMIs using Map and get the region using Lookup as below

vars.tf:
variable "region"{
    default="ap-south-1"
}
variable "ec2_ami"{
    type=map
    default={
        ap-south-1="ami-76d6f519"
        us-west-2="ami-e251209a"
    }
    
}
ec2-instance.tf:
resource "aws_instance" "web" {
  ami           = lookup(var.ec2_ami, var.region)
  instance_type = "t2.micro"

  tags = {
    Name = "HelloWorld"
  }
}

See the below there is no instance created in us-west-2 region
but the Ec2 instance is created in ap-south-1.


Thank you for reading 👍👍👍👍👍

Tuesday, 21 July 2020

Jenkins with Github

Jenkins with Github

In previous topic, we have learned how to setup Jenkins on EC2.

Now lets see how we integrate Jenkins with Github.

Usually Jenkins comes with default plugins which will be used up to some extent.

So,here lets integrate Gihub third party plugin for continuous Integration, Basically Github is used to update , merge and get the latest code from repository among the developers.It means multiple developers will work on the same project and merge the changes continuously without issue.

So, Jenkins should have this Github plugin to pull the code from Github repository, Which has the latest code where all the developers merged once after testing of the code.

We no need to install this GitHub plugin , if you already installed git in your Jenkins.So, Lets install Git Plugin.

Lets see how to install Git plugin


  • On Jenkins Home Page , Click on Manage Jenkins

 

  • Select manage Plugins





  • Select on "Available" Tab,Filter with "GIT",




  1. Select checkbox beside Git Plugin: This will install the Git plugin
  2. Select "Install without restart": Once after install of Git , Machine doesn't restart
  3. Download Now and Install aftrer restart: Once after install Git, Machine will restart
  4. Once installation done , you can see the below screen under "installed" tab.



Now Integrate Github plugin with Jenkins job.
Configure a Jenkins Job, For this Click on
New Item-->Enter Item Name (This is jenkins job name)--> Select Freestyle Project -->OK
 

This will redirect to the below page , where you should enter project details like below.


Have my sample Github project , which have entered here and click save.

Once above step done Goto Jenkins dashboard to see the configured jobs.



Now select "Sample Job" click on build now.



You can see the job progress in the Build History.



Once its done successfully, It will show as above else , it will show as "Red" color.

You can click on #1 and select "Console output" to see the logs of the Jenkins Job




Now Your Jenkins Job is integrated with Github Project.

Also If you have Git Project , select Git option as below


Enter Project details and click on save .
Run the jenkins job once you configured to see the Job status.

"Here we configured Github Project and Git project configuring in Jenkins with in EC2 instance".