The AWS SQS message delete request get success but message does not get deleted from the queue.

Backgraund

We are using AWS SQS service to communicate between web application and background job processor for long running task and when the task completes, we delete the SQS message from the queue. We have a background job processor console application written in C# and a job scheduler which monitor the SQS message and when a new message receive, it start processing with the message. When the same message read by the job scheduler that time it skips the message because the message already in processing. When the message successfully processed, it deletes the message from the SQS queue.

Problem

Recently, we have found that some messages do not get deleted even though the delete request successfully executed. As a result same message is processing multiple times.

Reason of The Problem

If you receive a message more than once, each time you receive it, you get a different receipt handle string value with the message object. You must provide the most recently received receipt handle when you request to delete the message (otherwise, the message might not be deleted).
As some of our message processor taking much time to process the message, in the meantime our job scheduler read the message multiple times and skip the message. So when message processor completes its operation, it requests to delete the message with ReceiptHandle value of the message object. The delete request get success but message does not get deleted because it tried to delete the message with old ReceiptHandle value.

Solution

When the job scheduler read the same message then we capture the ReceiptHandle value only and associate it with the message and skip the current message.

Sample Code to Generate The Issue

Let's you have a FileProcessingQueue queue configured with Default Visibility Timeout to 1 minutes and there is a message in the queue. Now if you run the test method of SqsQueueRunner class then you will see message does not get deleted.

    public class SqsQueueRunner
    {
        private int count;
        private string firstReceiptHandle = string.Empty;

        public void Test()
        {
            var sqsClient =new AmazonSQSClient(RegionEndpoint.USEast1);
            ProcessMessage(sqsClient);
        }

        private void ProcessMessage(IAmazonSQS sqsClient)
        {
            while (true)
            {
                var request = new GetQueueUrlRequest {QueueName = "FileProcessingQueue"};

                var response = sqsClient.GetQueueUrl(request);

                var url = response.QueueUrl;

                var receiveRequest = new ReceiveMessageRequest
                {
                    QueueUrl = url, MaxNumberOfMessages = 1
                };

                var receivedResponse = sqsClient.ReceiveMessage(receiveRequest);
                if (!receivedResponse.Messages.Any())
                    continue;

                var message = receivedResponse.Messages[0];
                if (firstReceiptHandle == string.Empty)
                    firstReceiptHandle = message.ReceiptHandle;

                if (count < 5)
                {
                    Thread.Sleep(61*1000);
                    Console.WriteLine(count);
                    count++;
                    continue;
                }
                sqsClient.DeleteMessage(new DeleteMessageRequest(url, firstReceiptHandle));
                break;
            }
        }
    }
  

If you delete the message with the most recent ReceiptHandle value, the message will be deleted from the queue.