OpenTelemetry Support in Mule Runtime

A trace is a sequence of spans representing a complete operation in a distributed system. Spans can be nested and represent a unit of work or operation. Each span includes:

When tracing a flow execution, each Mule component execution involved in processing the message is represented as a span. The component spans describe the relevant aspects of the execution. They can range from a single span for a simple component, such as the Set Payload, to more complex structures, such as a Batch Job.

The Mule runtime traces complex components, such as Batch Jobs.

The trace shows how the first step of the Batch Job processes each batch record, with each record span containing a child span that represents the unit of work the Logger component performs. Next, it displays the Batch Aggregator span, along with the For Each component and the processors within it, such as Transform Message and Logger. Then, the trace displays the other two batch steps and their respective components. Finally, it shows the Batch On Complete span, along with the Logger component that appears after the Batch Job in the flow.

In this example, one Mule app sends a request to another Mule app:

Distributed tracing captures both flows as part of the same trace because OpenTelemetry is instrumented in Mule and HTTP Connector.

If the execution of the unit of work represented by a span produces an error, the span’s status is set to Error. MuleSoft follows the semantic convention defined by OpenTelemetry that adds one or more exception events to the span:

In this example, when an error is produced, the span corresponding to the Raise Error component has an Error status:

Because this flow lacks error handling, the error propagates and fails the flow execution, causing the Error status to propagate to the flow span. A detailed view of the span also shows the associated exception events.

The HTTP Listener and HTTP Request operations contribute additional attributes to the spans, providing more context for tracing the HTTP interactions.

For flows with an HTTP Listener:

Additional attributes for an HTTP Listener associated flow span:

For flows with an HTTP Request:

Additional attributes for an HTTP Request associated flow span:

OpenTelemetry logging is a sequence of log records with a unified data model, where each record corresponds to a Java log call. Each log record contains:

Any entry added to the Log4j MDC is added to the OpenTelemetry logs as Log Record attributes, so all Log4j MDC data is exported as part of OpenTelemetry logs.

For adding custom attributes on both traces and logs, see Custom Attributes.

Instrumentation scope: `org.mule.runtime.flow.statistics`

These metrics track message throughput and processing errors at the flow level. They reflect the core message and response lifecycle of a Mule application.

These metrics are useful for understanding application throughput and detecting error rate trends over time.

Instrumentation scope: `org.mule.runtime.flow.statistics`

These metrics provide a snapshot of the application’s flow topology. They report how many flows of each category are declared in the application configuration and how many are currently active (started).

Instrumentation scope: `org.mule.runtime.errors`

These metrics break down runtime errors by their Mule error type, providing more granular error visibility than the aggregate execution.errors counter.

The error-type attribute contains the Mule error identifier (e.g., MULE:COMPOSITE_ROUTING, HTTP:NOT_FOUND, MULE:SOURCE_RESPONSE_SEND), enabling per-type trend analysis and alerting.

Instrumentation scope: `org.mule.runtime.diagnostics`

These metrics expose the Mule runtime built in alerting system, tracking operational conditions that may require attention.

Known alert types:

The mule.alert.context attribute provides additional detail about the alert (e.g., Resource: log records, Action: delayed).

Instrumentation scope: `io.opentelemetry.runtime-telemetry`

Standard OpenTelemetry JVM instrumentation metrics providing visibility into the underlying Java Virtual Machine.

Instrumentation scopes: io.opentelemetry.exporters.otlp_grpc_span_exporter, io.opentelemetry.exporters.otlp_grpc_log_exporter, io.opentelemetry.exporters.otlp_grpc_metric_exporter, io.opentelemetry.sdk.metrics, io.opentelemetry.sdk.trace, `io.opentelemetry.sdk.logs`

These metrics show the health and throughput of the Direct Telemetry Stream exporters so you can monitor the export of the telemetry data.

Mule runtime supports exporting all its telemetry data using the OpenTelemetry protocol (OTLP) and binary/protobuf format. You can configure an endpoint for each signal type (traces, logs, and metrics), and data is sent to that endpoint. However, we strongly recommend using a collector because it provides the most flexible and feature-rich architecture for a wide range of use cases and deployment scenarios.

See OpenTelemetry Collector for more information.

Key reasons for using a collector include:

Mule runtime includes predefined configuration templates for exporting traces and logs.

Tracing export setup properties:

Tracing export tuning properties:

Minimal configuration example that exports 50% of the traces to an HTTP OpenTelemetry collector endpoint:

Logging export setup properties:

Logging export tuning properties:

Minimal configuration example that sends logs to an HTTP OpenTelemetry collector endpoint:

Metrics export setup properties:

Metrics export tuning properties:

Mule runtime exports all its metrics using cumulative temporality. Each exported data point represents the total accumulated value of a metric since the application started, not just the delta since the last export.

If an export fails (including after backpressure retries), that export cycle is dropped without being queued or persisted. Because metrics are cumulative, a dropped export doesn’t result in permanent data loss. The next successful export carries the full accumulated state, and the backend service reconstructs the timeline with only a brief gap in resolution.

If the Direct Telemetry Stream exporter is still engaged in a previous metric export cycle when the next interval fires, the new cycle is skipped and a "Exporter busy. Dropping metrics." INFO log message is triggered. To minimize skipped cycles, keep your collector and backend service reachable with low latency.

Sampling helps reduce observability costs. Mule runtime supports Head sampling, which makes the sampling decision as early as possible. In a Mule application, that moment is when a message source is about to trigger a message processing operation, for example, when an HTTP listener has received a request.

Head sampling is a good fit for:

While Head sampling is simple and efficient, it has limitations. It can’t:

Tail sampling supports these advanced cases but is costly and can affect performance. Therefore, Mule runtime only supports Head sampling. If needed, implement Tail sampling in the OpenTelemetry collector.

Regarding sampling algorithms, Mule runtime currently supports these standardized OpenTelemetry algorithms: AlwaysOn, AlwaysOff, ParentBasedAlwaysOn, ParentBasedAlwaysOff, TraceIdRatio, and ParentBasedTraceIdRatio.

Mule runtime tracing is structured around three base tracing levels: OVERVIEW, MONITORING, and DEBUG. Each generated span has an associated tracing level. Flow spans use the OVERVIEW level, flow component spans use the MONITORING level, and internal flow component execution spans (parameter resolution, connection obtention, among others) use the DEBUG level.

In most cases, the default tracing level is the best choice. Use DEBUG for troubleshooting.

Service attributes identify the source of the telemetry data, Mule applications in this case. Mule runtime supports setting the service.name and service.namespace properties to custom values:

Additionally, a service.instance.id identifies the instance of the Mule application and is automatically generated as a unique identifier on each deployment or redeployment.

Adding custom data to the generated spans and log records can help with:

Mule runtime supports adding custom attributes to spans and log records through the Tracing Module. Any logging variable set by the module is also added as an attribute of the span that represents the set-logging-variable operation and the spans and log records that are generated during the flow’s execution, including subflows, flow references, and error handlers.

While useful, custom attributes must be handled with care. Include only relevant information. Too many attributes can affect performance and reach the feature limits:

Backpressure occurs when export queues for traces or logs are full. Backpressure strategies define how Mule runtime handles this event:

Dropping spans or log records results in missing data at the export destination. Blocking threads, on the other hand, can cause performance issues in Mule runtime. For both conditions, Mule runtime generates a WARNING log. Internal statistics track historical occurrences, and the log messages include cumulative totals since the export started. An alert is also triggered and is available at the Mule troubleshooting plugin DIAF files.

Export queue overflow: 2 spans have been dropped. Total spans dropped since the export started: 4

Export queue overflow: 2 spans have been delayed. Total spans delayed since the export started: 4

When errors occur during export requests, Mule runtime uses an exponential backoff with jitter to avoid overwhelming the destination until recovery. Only errors marked as retryable, as defined by OpenTelemetry standards, trigger retry logic. See HTTP transport protocol reference and gRPC transport protocol reference for more information.

Direct telemetry stream supports configuring the export for traces and logs to use mutual TLS authentication, which can be used to secure the communication with an external collector.

CERTIFICATES_PATH strategy is supported for Standalone and Hybrid Standalone deployments. Configure the certificates location when using this strategy:

CERTIFICATES_BASE64 strategy is supported for all the deployments. Encode the certificates in Base64 when using this strategy:

Mule runtime logs are the place to check for connection issues, export errors, and delayed or dropped traces and logs.

Mule diagnostics provides information about delayed or dropped traces and logs and are useful for quick checks. Note that diagnostics are temporal and cover only the last 60 minutes.

Use diagnostics as a starting point and complement your analysis by reviewing the logs.

Use an OpenTelemetry collector configured to report its internal metrics. See Collector internal metrics for more information. Having a collector with that setup can greatly improve the troubleshooting capabilities while moving the hard limits of your particular setup from the 3rd party solution to your own infrastructure where full observability is possible.

Keep standard Log4j logging enabled, or at least a variation that continues to log errors, to allow troubleshooting. Without this, you won’t be able to troubleshoot OpenTelemetry log export failures.

Do a performance test of your application before enabling this feature in a production environment. While our performance tests did not show huge performance costs, Direct telemetry stream is not a zero cost feature and therefore must be tested.

Always consider the limits of your third-party solution and configure your exports accordingly. Mule runtime can’t compensate for poor downstream performance scenarios without compromising its own stability.

The most common cause of export errors, delays, or drops is a bottleneck in the ingestion pipeline (the OpenTelemetry Collector and the third-party solution). Make sure these components operate within acceptable performance limits.

Be aware that tuning changes, such as modifying the queue size, can impact Mule application performance. Test these changes carefully before applying them to production.