DEV Community

Dinesh_gowtham
Dinesh_gowtham

Posted on

SQS Partial Batch Failure: The Default That Silently Drops Your Messages

#aws #serverless #lambda #sqs

We were losing critical messages in our SQS workflow without realizing it, until we dug into the SQS partial batch failure behavior. It turns out, the default configuration was silently dropping failed messages, causing data loss and inconsistencies. This shocking discovery led us to re-evaluate our entire message processing pipeline.

Introduction to SQS Partial Batch Failure

SQS provides two types of batch failure handling: all-or-nothing and partial batch failure. The all-or-nothing approach will fail the entire batch if any message fails, whereas the partial batch failure approach will only fail the specific messages that encounter an error.

import { SendMessageCommand } from '@aws-sdk/client-sqs';

const sqsClient = new SQSClient({ region: 'us-east-1' });
const params = {
  MessageBody: 'Hello, world!',
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
};
const command = new SendMessageCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

When working with SQS, it's crucial to understand the difference between these two modes, as the default all-or-nothing approach can lead to significant data loss if not properly configured.

When using the @aws-sdk/client-sqs package, it's essential to handle potential errors, such as the AWS.SimpleQueueService.NonExistentQueue error, which occurs when the specified queue does not exist.

try {
  // Send message to non-existent queue
  const params = {
    MessageBody: 'Hello, world!',
    QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/non-existent-queue',
  };
  const command = new SendMessageCommand(params);
  await sqsClient.send(command);
} catch (error) {
  if (error.name === 'NonExistentQueue') {
    console.error('The specified queue does not exist.');
  } else {
    console.error(error);
  }
}
Enter fullscreen mode Exit fullscreen mode

The Consequences of Silent Message Drops

The consequences of silent message drops can be severe, leading to data inconsistencies, incorrect business decisions, and ultimately, a loss of customer trust.

import { ReceiveMessageCommand } from '@aws-sdk/client-sqs';

const params = {
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
  MaxNumberOfMessages: 10,
};
const command = new ReceiveMessageCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

Be aware that the default SQS behavior is to silently drop failed messages, unless explicitly configured to handle failures. This can lead to data loss and inconsistencies, making it crucial to implement proper error handling mechanisms.

When using SQS with Lambda, it's essential to configure the Lambda function to handle potential errors, such as the AWS.Lambda.ServiceException error, which occurs when the Lambda function is unable to process the message.

try {
  // Process message
  const message = 'Hello, world!';
  // Simulate processing error
  throw new Error('Processing error');
} catch (error) {
  if (error instanceof Error) {
    console.error('Error processing message:', error.message);
  } else {
    console.error('Unknown error:', error);
  }
}
Enter fullscreen mode Exit fullscreen mode

Configuring SQS for Reliable Message Processing

To configure SQS for reliable message processing, you need to enable partial batch failure handling and set up a dead-letter queue to handle failed messages.

import { CreateQueueCommand } from '@aws-sdk/client-sqs';

const params = {
  QueueName: 'my-queue',
  Attributes: {
    'DeadLetterTargetArn': 'arn:aws:sqs:us-east-1:123456789012:my-dead-letter-queue',
  },
};
const command = new CreateQueueCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

When configuring SQS, it's crucial to consider the trade-offs between message throughput, latency, and data consistency. For example, using a FIFO queue can provide high data consistency, but it's limited to 300 transactions per second.

When using the @aws-sdk/client-sqs package, you can configure the SQS client to use long polling, which can reduce the number of empty receives and improve message processing efficiency.

import { ReceiveMessageCommand } from '@aws-sdk/client-sqs';

const params = {
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
  MaxNumberOfMessages: 10,
  WaitTimeSeconds: 20,
};
const command = new ReceiveMessageCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

Handling Failed Messages with Lambda

To handle failed messages with Lambda, you need to configure the Lambda function to process messages from the dead-letter queue.

import { LambdaClient } from '@aws-sdk/client-lambda';
import { InvokeCommand } from '@aws-sdk/client-lambda';

const lambdaClient = new LambdaClient({ region: 'us-east-1' });
const params = {
  FunctionName: 'my-lambda-function',
  InvocationType: 'Event',
  Payload: JSON.stringify({ message: 'Hello, world!' }),
};
const command = new InvokeCommand(params);
lambdaClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

When using Lambda with SQS, it's essential to consider the potential for duplicate message processing, which can occur when the Lambda function is retried after a failure. To mitigate this, you can use an idempotent Lambda function or implement a deduplication mechanism.

When using the @aws-sdk/client-lambda package, you can configure the Lambda client to use a specific runtime, such as Node.js 22, which can provide improved performance and security.

import { UpdateFunctionConfigurationCommand } from '@aws-sdk/client-lambda';

const params = {
  FunctionName: 'my-lambda-function',
  Runtime: 'nodejs22.x',
};
const command = new UpdateFunctionConfigurationCommand(params);
lambdaClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

Best Practices for SQS and Lambda Integration

To ensure reliable and efficient message processing with SQS and Lambda, follow these best practices:

  • Use a dead-letter queue to handle failed messages
  • Configure the Lambda function to process messages from the dead-letter queue
  • Implement idempotent message processing to prevent duplicate message processing
  • Use long polling to reduce empty receives
  • Monitor SQS and Lambda metrics to detect potential issues 
import { GetQueueAttributesCommand } from '@aws-sdk/client-sqs';

const params = {
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
  AttributeNames: ['All'],
};
const command = new GetQueueAttributesCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

When integrating SQS with Lambda, it's essential to consider the potential for cold starts, which can occur when the Lambda function is invoked after a period of inactivity. To mitigate this, you can use provisioned concurrency or implement a warm-up mechanism.

The Takeaway

Here are the key takeaways from this article:

  • SQS partial batch failure can silently drop failed messages, leading to data loss and inconsistencies
  • Configuring SQS for reliable message processing requires enabling partial batch failure handling and setting up a dead-letter queue
  • Handling failed messages with Lambda requires configuring the Lambda function to process messages from the dead-letter queue
  • Implementing idempotent message processing can prevent duplicate message processing
  • Monitoring SQS and Lambda metrics is crucial to detect potential issues and ensure reliable message processing
  • Using long polling and provisioned concurrency can improve message processing efficiency and reduce costs

When working with SQS and Lambda, it's essential to carefully consider the trade-offs between message throughput, latency, and data consistency. By following best practices and implementing reliable message processing mechanisms, you can ensure that your SQS and Lambda integration is efficient, reliable, and scalable.

import { GetQueueAttributesCommand } from '@aws-sdk/client-sqs';

const params = {
  QueueUrl: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
  AttributeNames: ['All'],
};
const command = new GetQueueAttributesCommand(params);
sqsClient.send(command).then(
  (data) => console.log(data),
  (error) => console.error(error)
);
Enter fullscreen mode Exit fullscreen mode

Transparency notice

The topic was scouted from live AWS and Node.js ecosystem signals, and the content —
including all code examples.

Published: 2026-05-11 · Primary focus: SQS

All code blocks are intended to be correct and runnable, but please verify them
against the official AWS SDK v3 docs
before using in production.

Find an error? Drop a comment — corrections are always welcome.

Top comments (0)