Using Anypoint Monitoring

Mule apps and API gateways deployed to Runtime Fabric include native support for Anypoint Monitoring that allows viewing metrics from within Anypoint Platform.

Anypoint Monitoring

Anypoint Monitoring includes tools that provide feedback from Mule flows and components in your Mule app network. Anypoint Monitoring runs as a sidecar in application deployments and gathers metrics for Mule apps and API gateways deployed to Runtime Fabric. Titanium-plan customers have access to additional capabilities, including application log retrieval and search.

Enable Anypoint Monitoring

By default, Runtime Fabric is configured to send application metrics to Anypoint Monitoring. Data is securely transmitted using mutual TLS encryption. For metrics and logs to be sent to Anypoint Monitoring, outbound network connectivity must be available.

Starting with Runtime Fabric agent version 2.6.22, Anypoint Monitoring sidecar is compatible only with HTTP-proxy instead of Monitoring SOCKS5 proxy.

If you need an outbound proxy, configure Anypoint Runtime Fabric to use a SOCKS5 proxy by following the instructions in Configuring Proxies on Runtime Fabric.

To verify outbound network connectivity on Anypoint Runtime Fabric version 1.8.50 or later, ensure that each node in Runtime Fabric can send outbound TCP (Lumberjack) communication using port 443 to the following domains:

US control plane:
dias-ingestor-router.us-east-1.prod.cloudhub.io
us1.ingest.mulesoft.com
EU control plane:
dias-ingestor-router.eu-central-1.prod-eu.msap.io
eu1.ingest.mulesoft.com

If you are using a version of Anypoint Runtime Fabric older than 1.8.50, use port 5044 instead of port 443. Use the following domains:

US control plane: dias-ingestor-nginx.prod.cloudhub.io
EU control plane: dias-ingestor-nginx.prod-eu.msap.io

For further details on Anypoint Monitoring endpoints, refer to configuring your network to support Runtime Fabric.

Metrics

Each Mule app and API gateway replica runs a metrics agent that collects and transmits metrics data to Anypoint Monitoring via secure mutual TLS encryption. Each replica can cache up to 100 megabytes of metrics, after which the metrics data is rotated to provide storage for more recent data. If metrics can’t be sent to Anypoint Monitoring, this typically provides at least four days worth of storage per replica. Custom metrics might consume more storage.

The following Anypoint Runtime Fabric updates do not impact or restore monitoring metrics of applications deployed on Runtime Fabric:

A backup and restore of Runtime Fabric
An upgrade of Runtime Fabric
OS patching of Runtime Fabric nodes

Logs

Anypoint Monitoring provides access to log data for applications deployed to Runtime Fabric. To access logs with Anypoint Monitoring, you need a Titanium subscription.

Each Mule app and API gateway replica can store up to 450 megabytes of log data on disk, after which log data is rotated to provide storage for more recent log data.

Starting with Runtime Fabric agent version 3.0.0, application logs are forwarded exclusively through a sidecar container deployed alongside each application replica. This architecture provides better isolation, improved resource management, and eliminates the need for node-level log forwarding agents.

The log forwarding sidecar can transmit up to 300 kilobytes of log data per second per application replica.

In some cases, logs with multiple lines might not be fully transmitted to Anypoint Monitoring. This can occur if multiple lines of log output are split across two files due to log rotation inside Runtime Fabric.

You can enable log forwarding and set log levels when you deploy applications in Runtime Manager.

Resource and Permission Requirements for Anypoint Monitoring Agent

MuleSoft runs Anypoint Monitoring agents alongside Mule apps to send telemetry to the Anypoint control plane.

Starting with Runtime Fabric agent version 3.0.0, all telemetry (metrics, logs, and traces) is transported through a single sidecar container deployed per application replica. This architecture provides better resource isolation and eliminates the need for node-level DaemonSet deployments.

Depending on the features you use, the sidecar agent consumes corresponding resources for optimal performance:

To enable default Anypoint metrics, the sidecar agent (per application replica) requires 0.05vCPU or Core of CPU allocation and 90MiB memory allocation.
If you opt into forwarding application logs to Anypoint Monitoring, the sidecar agent (per application replica) requires up to 30% of CPU limit configured for application replica and up to 150MiB memory allocation.
If you opt into trace data generation for Mule apps, the sidecar agent (per application replica) requires up to 30% of CPU limit configured for application replica and up to 240MB memory allocation.

Allocate appropriate capacity to the underlying cluster to avoid application deployment failure.

Legacy Architecture (Runtime Fabric Agent Version 2.11.x and Earlier)

For Mule apps deployed or managed using Runtime Fabric agent version 2.11.x or earlier, two different agents were used:

One DaemonSet per underlying node to transport logs to Anypoint Monitoring control plane.
One sidecar per application replica to transport metrics to Anypoint Monitoring control plane.

This architecture has been replaced with the sidecar-only model in version 3.0.0.

Log Forwarding in Runtime Fabric 3.0.0

Starting with Runtime Fabric agent version 3.0.0, the DaemonSet-based log forwarding mechanism has been removed. Log forwarding now occurs exclusively through application sidecar containers, which provides better isolation, improved resource management, and simplified operations.

The sidecar-only architecture offers several benefits:

Each application manages its own log forwarding resources, preventing one application from affecting others.
No need to manage node-level DaemonSets or worry about node-specific configurations.
Log forwarding scales automatically with your application replicas.
Reduced cluster-level permissions and better isolation between applications.

Migration Steps Required Before Upgrading to Runtime Fabric Agent Version 3.0.0

If you are currently using the log forwarder DaemonSet, you must complete these steps before upgrading Runtime Fabric to version 3.0.0. Failure to complete these steps may result in log forwarding disruption.

If the log forwarder DaemonSet is running on your cluster, complete these steps before upgrading:

Enable Application-Scoped Log Forwarding
Redeploy Applications Using Legacy Log Forwarding
Remove the Log Forwarder DaemonSet
Verify Migration

Enable Application-Scoped Log Forwarding

This setting is enabled by default on most Runtime Fabric instances. However, if it was previously disabled, you must re-enable it using the Runtime Fabric API:

curl -X PATCH \
  -H 'Authorization: Bearer {authToken}' \
  -H 'Content-Type: application/json;charset=UTF-8' \
  -H 'Accept: application/json, text/plain, */*' \
  https://anypoint.mulesoft.com/runtimefabric/api/organizations/{orgId}/fabrics/{fabricId}/logforwarding/appScoped?enabled=true

Replace {authToken}, {orgId}, and {fabricId} with your specific values.

Redeploy Applications Using Legacy Log Forwarding

All applications currently forwarding logs through the DaemonSet must be redeployed. This switches them to use the sidecar container for log forwarding. You can trigger a redeployment by making any update to the application in Runtime Manager, or by using the Runtime Manager API.

Remove the Log Forwarder DaemonSet

After all applications are using the new sidecar-based log forwarding, remove the legacy DaemonSet by calling the Runtime Fabric API:

curl -X PATCH \
  -H 'Authorization: Bearer {authToken}' \
  -H 'Content-Type: application/json;charset=UTF-8' \
  -H 'Accept: application/json, text/plain, */*' \
  https://anypoint.mulesoft.com/runtimefabric/api/organizations/{orgId}/fabrics/{fabricId}/logforwarding \
  -d '{"anypointMonitoring": false}'

Replace {authToken}, {orgId}, and {fabricId} with your specific values.

Disabling the DaemonSet doesn’t impact log forwarding to Anypoint Monitoring via the sidecar containers. Your applications will continue to forward logs normally after completing these steps.

Verify Migration

After completing the previous steps, verify that:

All applications are running and forwarding logs successfully
The am-log-forwarder DaemonSet is no longer present in your cluster
Application logs are visible in Anypoint Monitoring

Anypoint Visualizer

Anypoint Visualizer displays views of different aspects of an application network graph. You can use the graph to explore your application network, identify problems, and make decisions.

By default, Anypoint Visualizer is disabled when deploying an application to Mule runtime 4.x. See Enable Header Injection for more information. Due to the impact on performance, enabling Anypoint Visualizer for application deployments with less than 0.2 vCPU is not supported.

(Optional) Disable Anypoint Monitoring for a Mule App

When you deploy a Mule app to Runtime Fabric, Anypoint Monitoring is enabled by default. To disable Anypoint Monitoring, pass the following custom property when deploying your Mule app:

anypoint.platform.config.analytics.agent.enabled=false

For compliance or geospecific requirements if your team or organization cannot send metrics to Anypoint control plane, you can disable Anypoint metrics if you are not contractually obligated to send these metrics for Anypoint usage and billing purposes.

Enable Trace Data Collection

This feature is available only if your organization has the Anypoint Integration Advanced package or a Titanium subscription to Anypoint Platform. For more information, see the Pricing and Packaging documentation.

Traces enable you to understand the path of a transaction in the MuleSoft ecosystem by providing full visibility into the execution path of API calls or application transactions. When you enable trace data collection for an app, Anypoint Platform collects and exposes tracing information from Mule runtime engine in the OpenTelemetry standard, enabling you to export it using the Anypoint Monitoring Telemetry Exporter.

You can enable trace data collection for your Runtime Fabric applications running on Mule version 4.6 or later, and the minimum Runtime Fabric agent version required is 2.6.31 or above.
You can enable trace data collection using only the Runtime Manager UI and API, but you can’t enable trace data collection using the Mule Maven plugin, Anypoint CLI, or Anypoint Studio.

To enable trace data collection for an app:

Click the Monitoring tab.
Select the Enable tracing checkbox.
Click Deploy Application if you are enabling the feature for a new deployment, or Apply Changes if you are editing an existing application.

Once you enable tracing, and redeploy your application, the Runtime Manager backend starts instrumenting traces for your application.