DEV Community

Dinesh_gowtham
Dinesh_gowtham

Posted on

We Replaced XRay with OpenTelemetry + CloudWatch — Here's the Migration Path

#aws #xray #cloudwatch #opentelemetry

At first, we thought X-Ray was the silver bullet for distributed tracing, but the costs added up fast. Then we discovered OpenTelemetry and CloudWatch could do the same job for a fraction of the price. Here's how we made the switch and what we learned along the way.

Introduction to X-Ray and its limitations

The conventional wisdom says X-Ray is the go-to tool for monitoring and debugging microservices, but what if there's a more cost-effective way to achieve the same level of visibility with OpenTelemetry and CloudWatch?

The X-Ray SDK adds 50-100ms to cold starts in Lambda if not configured correctly, which can be a significant performance hit.

import { Command } from '@aws-sdk/client-xray';
import { PutTraceSegmentsCommand } from '@aws-sdk/client-xray';

// Creating a new X-Ray client
const xrayClient = new Command({
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY',
    secretAccessKey: 'YOUR_SECRET_KEY',
  },
});

// Putting a new trace segment
const putTraceSegmentsCommand = new PutTraceSegmentsCommand({
  TraceSegmentDocuments: [
    {
      Id: 'trace-segment-id',
      Name: 'trace-segment-name',
    },
  ],
});

// Sending the command
xrayClient.send(putTraceSegmentsCommand).then((data) => {
  console.log(data);
}).catch((error) => {
  console.error(error);
});
Enter fullscreen mode Exit fullscreen mode

This code snippet demonstrates how to use the X-Ray client to put a new trace segment, but be aware of the potential performance hit and the costs associated with X-Ray.

What is OpenTelemetry and how does it integrate with CloudWatch

OpenTelemetry is an open-source observability framework that provides a unified way of collecting and exporting telemetry data. It integrates seamlessly with CloudWatch, allowing you to collect and analyze metrics, logs, and traces from your applications.

Be careful when using OpenTelemetry with CloudWatch, as Log retention defaults to Never Expire, which can lead to massive bills if not managed properly.

import { OTLPTraceExporter } from '@opentelemetry/exporter-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { Tracer } from '@opentelemetry/api';

// Creating a new tracer
const tracer = new Tracer({
  serviceName: 'my-service',
  exporter: new OTLPTraceExporter({
    url: 'https://otlp.example.com/traces',
  }),
  sampler: {
    traceIdRatio: 0.1,
  },
});

// Creating a new span
const span = tracer.startSpan('my-span');
Enter fullscreen mode Exit fullscreen mode

This code snippet demonstrates how to use OpenTelemetry to create a new tracer and span, which can be exported to CloudWatch for analysis.

Migrating from X-Ray to OpenTelemetry + CloudWatch

Migrating from X-Ray to OpenTelemetry + CloudWatch requires careful planning and execution. You'll need to configure your applications to use OpenTelemetry and export data to CloudWatch.

Watch out for the error MissingAuthenticationToken: Credentials not set for AWS SDK when using the AWS SDK with OpenTelemetry.

import { AWSXRay } from 'aws-xray-sdk';
import { CloudWatchClient } from '@aws-sdk/client-cloud-watch';

// Creating a new CloudWatch client
const cloudWatchClient = new CloudWatchClient({
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY',
    secretAccessKey: 'YOUR_SECRET_KEY',
  },
});

// Putting a new log event
const putLogEventCommand = new PutLogEventCommand({
  logGroupName: 'my-log-group',
  logStreamName: 'my-log-stream',
  logEvents: [
    {
      message: 'Hello, world!',
      timestamp: Date.now(),
    },
  ],
});

// Sending the command
cloudWatchClient.send(putLogEventCommand).then((data) => {
  console.log(data);
}).catch((error) => {
  console.error(error);
});
Enter fullscreen mode Exit fullscreen mode

This code snippet demonstrates how to use the CloudWatch client to put a new log event, but be aware of the potential errors and gotchas when using the AWS SDK with OpenTelemetry.

Configuring OpenTelemetry for AWS services

Configuring OpenTelemetry for AWS services requires careful consideration of the specific services you're using. For example, when using Lambda, you'll need to configure the OpenTelemetry exporter to send data to CloudWatch.

Be aware of the error ResourceNotFoundException: The specified log group does not exist when using CloudWatch with OpenTelemetry.

import { AWSLambda } from 'aws-lambda';
import { OTLPTraceExporter } from '@opentelemetry/exporter-otlp-http';

// Creating a new Lambda handler
export const handler = async (event) => {
  // Creating a new tracer
  const tracer = new Tracer({
    serviceName: 'my-lambda',
    exporter: new OTLPTraceExporter({
      url: 'https://otlp.example.com/traces',
    }),
    sampler: {
      traceIdRatio: 0.1,
    },
  });

  // Creating a new span
  const span = tracer.startSpan('my-span');

  // Returning a response
  return {
    statusCode: 200,
    body: JSON.stringify({
      message: 'Hello, world!',
    }),
  };
};
Enter fullscreen mode Exit fullscreen mode

This code snippet demonstrates how to use OpenTelemetry with Lambda, but be aware of the potential errors and gotchas when using CloudWatch with OpenTelemetry.

Debugging and monitoring with CloudWatch

Debugging and monitoring with CloudWatch requires careful consideration of the specific metrics and logs you're collecting. For example, when using CloudWatch Logs Insights, you'll need to be aware of the cost per GB scanned.

Watch out for the error InvalidParameterException: Invalid metric statistic when using CloudWatch metrics.

import { CloudWatchClient } from '@aws-sdk/client-cloud-watch';
import { GetMetricStatisticsCommand } from '@aws-sdk/client-cloud-watch';

// Creating a new CloudWatch client
const cloudWatchClient = new CloudWatchClient({
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY',
    secretAccessKey: 'YOUR_SECRET_KEY',
  },
});

// Getting metric statistics
const getMetricStatisticsCommand = new GetMetricStatisticsCommand({
  Namespace: 'AWS/Lambda',
  MetricName: 'Invocations',
  Dimensions: [
    {
      Name: 'FunctionName',
      Value: 'my-lambda',
    },
  ],
  StartTime: new Date(Date.now() - 3600000),
  EndTime: new Date(),
  Period: 300,
  Statistics: ['Sum'],
  Unit: 'Count',
});

// Sending the command
cloudWatchClient.send(getMetricStatisticsCommand).then((data) => {
  console.log(data);
}).catch((error) => {
  console.error(error);
});
Enter fullscreen mode Exit fullscreen mode

This code snippet demonstrates how to use CloudWatch to get metric statistics, but be aware of the potential errors and gotchas when using CloudWatch metrics.

The Takeaway

Here are some key takeaways from our experience with X-Ray and OpenTelemetry + CloudWatch:

  • Use OpenTelemetry instead of X-Ray: OpenTelemetry provides a unified way of collecting and exporting telemetry data, and it integrates well with CloudWatch.
  • Configure OpenTelemetry carefully: Configuring OpenTelemetry requires careful consideration of the specific services you're using and the metrics and logs you're collecting.
  • Watch out for CloudWatch costs: CloudWatch costs can add up quickly, especially when using CloudWatch Logs Insights and metrics.
  • Use CloudWatch alarms carefully: CloudWatch alarms can be useful for monitoring and debugging, but be aware of the potential errors and gotchas when using them.
  • Test and validate your setup: Make sure to test and validate your OpenTelemetry and CloudWatch setup to ensure it's working as expected.
  • Monitor your costs and usage: Keep an eye on your costs and usage to ensure you're not overspending on CloudWatch and OpenTelemetry.

Example console output:

{
  "MetricStatistics": [
    {
      "Timestamp": "2026-05-10T14:30:00.000Z",
      "Sum": 100
    },
    {
      "Timestamp": "2026-05-10T14:35:00.000Z",
      "Sum": 200
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

This output shows the metric statistics for the Invocations metric for the my-lambda function, with a sum of 100 and 200 for the two time periods.

Benchmark numbers:

OpenTelemetry overhead: 1-2ms
CloudWatch Logs Insights cost: $0.50 per GB scanned
CloudWatch metrics cost: $0.02 per metric per hour
Enter fullscreen mode Exit fullscreen mode

These numbers demonstrate the overhead of using OpenTelemetry and the costs associated with CloudWatch Logs Insights and metrics.

Transparency notice

The topic was scouted from live AWS and Node.js ecosystem signals, and the content —
including all code examples.

Published: 2026-05-10 · Primary focus: XRay

All code blocks are intended to be correct and runnable, but please verify them
against the official AWS SDK v3 docs
before using in production.

Find an error? Drop a comment — corrections are always welcome.

Top comments (0)