AWS Command Line Interface (CLI) tool.

What is AWS CLI ?

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

Why do you use AWS CLI ?

We have been using AWS services for a long time. We are communicating those services through API build on different languages. AWS also provides a command line tool to use those services.
Many of us know that and using it some extent but many of us might not know about this tool. It’s very excellent tool.
You don’t need to write a program to access those services. They provide huge command references; you will get them all from here.

Install and Configure AWS CLI

You can download the AWS Command Line Interface from here.
You can also installed using pip command pip install awscli

Once the instalation is done you need to configure it. You will get the details of configuration from here

Usefulness of AWS CLI with an example

Let me give an example how it very much useful. Suppose you are using an Employee DynamoDb table and you want to know the date and time of the last provisioned throughput increase or decrease for this table or the number of provisioned throughput decreases for the table during this UTC calendar day. How can you get that information? There are basically two ways to get it:

• Write a small program using AWS API to get the desired information.
• Login to the AWS console and go to the Metrics tab of that DynamoDB table and find out your desired data. Though sometimes it’s very hard to find some specific data from the console.

But you can get that information by using a simple command in your command prompt without writing a small application!!!!
aws dynamodb describe-table --table-name Employee

Monitoring DynamoDB using Amazon CloudWatch

You can monitor dynamo using CloudWatch service. If you wish to build up your own auto-scaling or only scale down application then you have to use this service to get different Metrics data. For example, you can capture ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ReadThrottleEvents etc using Amazon CloudWatch service.

Here I'm going to show you how to get those data using boto3 python API. Coding structure is almost same for all other API. You can try with any other language.

    import boto3
    import datetime

    cloudwatch = boto3.client('cloudwatch')

    endTime = datetime.datetime.utcnow()
    startTime = endTime - datetime.timedelta(minutes=5)
    consumedWriteCapacityData = cloudwatch.get_metric_statistics(Period=60, 
                        StartTime = startTime, EndTime = endTime,
                        MetricName = 'ConsumedWriteCapacityUnits', Namespace = 'AWS/DynamoDB',
                        Statistics = ['Sum'], Dimensions = [{'Name': 'TableName', 'Value': 'Employee' }])
    for item in consumedWriteCapacityData['Datapoints']:
        print('Time: ', item['Timestamp'], '\t ', item['Sum'])
                

As I am using 60 secods period. So, you have to divide each data point value by 60 to get the actual ConsumedWriteCapacityUnits.

Similarly you can also get WriteThrottleEvents data.

    import boto3
    import datetime

    cloudwatch = boto3.client('cloudwatch')

    endTime = datetime.datetime.utcnow()
    startTime = endTime - datetime.timedelta(minutes=5)

    writeThrottleEventsData = cloudwatch.get_metric_statistics(Period=60, 
                        StartTime = startTime, EndTime = endTime,
                        MetricName = 'WriteThrottleEvents', Namespace = 'AWS/DynamoDB',
                        Statistics = ['Sum'], Dimensions = [{'Name': 'TableName', 'Value': 'Employee' }])
    for item in writeThrottleEventsData['Datapoints']:
        print('Time: ', item['Timestamp'], '\t ', item['Sum'])
                

References:

• http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/dynamo-metricscollected.html
• http://boto3.readthedocs.io/en/latest/reference/services/cloudwatch.html#CloudWatch.Client.get_metric_statistics

The AWS SQS message delete request get success but message does not get deleted from the queue.

Backgraund

We are using AWS SQS service to communicate between web application and background job processor for long running task and when the task completes, we delete the SQS message from the queue. We have a background job processor console application written in C# and a job scheduler which monitor the SQS message and when a new message receive, it start processing with the message. When the same message read by the job scheduler that time it skips the message because the message already in processing. When the message successfully processed, it deletes the message from the SQS queue.

Problem

Recently, we have found that some messages do not get deleted even though the delete request successfully executed. As a result same message is processing multiple times.

Reason of The Problem

If you receive a message more than once, each time you receive it, you get a different receipt handle string value with the message object. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).
As some of our message processor taking much time to process the message, in the meantime our job scheduler read the message multiple times and skip the message. So when message processor completes its operation, it requests to delete the message with ReceiptHandle value of the message object. The delete request get success but message does not get deleted because it tried to delete the message with old ReceiptHandle value.

Solution

When the job scheduler read the same message then we capture the ReceiptHandle value only and associate it with the message and skip the current message.

Sample Code to Generate The Issue

Let's you have a FileProcessingQueue queue configured with Default Visibility Timeout to 1 minutes and there is a message in the queue. Now if you run the test method of SqsQueueRunner class then you will see message does not get deleted.

    public class SqsQueueRunner
    {
        private int count;
        private string firstReceiptHandle = string.Empty;

        public void Test()
        {
            var sqsClient =new AmazonSQSClient(RegionEndpoint.USEast1);
            ProcessMessage(sqsClient);
        }

        private void ProcessMessage(IAmazonSQS sqsClient)
        {
            while (true)
            {
                var request = new GetQueueUrlRequest {QueueName = "FileProcessingQueue"};

                var response = sqsClient.GetQueueUrl(request);

                var url = response.QueueUrl;

                var receiveRequest = new ReceiveMessageRequest
                {
                    QueueUrl = url, MaxNumberOfMessages = 1
                };

                var receivedResponse = sqsClient.ReceiveMessage(receiveRequest);
                if (!receivedResponse.Messages.Any())
                    continue;

                var message = receivedResponse.Messages[0];
                if (firstReceiptHandle == string.Empty)
                    firstReceiptHandle = message.ReceiptHandle;

                if (count < 5)
                {
                    Thread.Sleep(61*1000);
                    Console.WriteLine(count);
                    count++;
                    continue;
                }
                sqsClient.DeleteMessage(new DeleteMessageRequest(url, firstReceiptHandle));
                break;
            }
        }
    }
  

If you delete the message with the most recent ReceiptHandle value, the message will be deleted from the queue.