Skip to main content

Logging

This guide covers the essential information you need to effectively configure and manage logging in your Agent Mesh applications. Proper logging configuration is critical for troubleshooting issues, monitoring system behavior, and maintaining reliable production deployments.

Agent Mesh uses Python's built-in logging module to provide flexible and powerful logging capabilities.

This approach provides several advantages:

  • Centralized Control: Single configuration file manages logging for all components
  • Python Native: Built on Python's standard logging module
  • Flexible and Powerful: Full access to Python's logging capabilities
  • Production-Ready: Industry-standard approach used by many Python applications

Configuration​

Agent Mesh supports logging configuration as either a YAML or JSON file. Both formats leverage Python's dictConfig method for advanced logging features.

To provide a logging configuration, set the LOGGING_CONFIG_PATH=path/to/logging_config.yaml environment variable in your .env file or with the export command.

info

While the INI format using Python's fileConfig() is still supported by Agent Mesh, it is not recommended due to its limitations compared to YAML and JSON formats.

Agent/Gateway Specific Logging Configuration

While individual agent and gateway YAML files may contain a log section similar to the example below:

log:
  stdout_log_level: INFO
  log_file_level: INFO
  log_file: my-agent.log

using a dedicated logging configuration file (YAML, JSON) is the recommended approach. The simple log: section configuration has lower precedence and will only be active when a dedicated logging configuration file is not provided.

Default Logging Configuration​

When you run sam init, Agent Mesh automatically generates a configs/logging_config.yaml file in your project directory. This file establishes sensible defaults while remaining easy to customize for your specific needs.

# Python logging configuration version (always 1)
version: 1

# Don't disable existing loggers when this config is loaded
disable_existing_loggers: false

# Formatters control the structure and appearance of log messages
formatters:
  # Simple human-readable format
  simpleFormatter:
    format: "%(asctime)s | %(levelname)-5s | %(threadName)s | %(name)s | %(message)s"

  # Colored simple human-readable format 
  coloredFormatter:
    class: solace_ai_connector.logging.ColoredFormatter
    format: "%(asctime)s | %(levelname)-5s | %(threadName)s | %(name)s | %(message)s"
    
  # JSON format for structured logging
  jsonFormatter:
    "()": pythonjsonlogger.json.JsonFormatter # The python-json-logger package is used for JSON formatting
    format: "%(timestamp)s %(levelname)s %(threadName)s %(name)s %(message)s"
    timestamp: "timestamp" # Generates ISO 8601-formatted timestamps for %(timestamp)s placeholder

# Handlers determine where log messages go
handlers:
  # Stream handler - outputs logs to console (stdout)
  streamHandler:
    class: logging.StreamHandler
    formatter: coloredFormatter
    stream: "ext://sys.stdout"
  
  # Rotating file handler - writes to log files with automatic rotation
  rotatingFileHandler:
    class: logging.handlers.RotatingFileHandler
    formatter: simpleFormatter
    filename: ${LOGGING_FILE_NAME, sam.log}
    mode: a             # Append mode - don't overwrite existing logs
    maxBytes: 52428800  # 50 MB - rotate when file reaches this size
    backupCount: 10     # Keep up to 10 historical log files

# Loggers 
loggers:
  # Keys are logger names used in the application code
  solace_ai_connector:
    level: ${LOGGING_SAC_LEVEL, INFO}
    handlers: []
  
  solace_agent_mesh:
    level: ${LOGGING_SAM_LEVEL, INFO}
    handlers: []
  
  # Special trace logger for detailed troubleshooting. Set to DEBUG to enable.
  sam_trace:
    level: ${LOGGING_SAM_TRACE_LEVEL, INFO}
    handlers: []

# Root logger - applies to all log statements (including those from external libraries) that propagate up to root
# The root logger also specifies handlers for the application
root:
  level: ${LOGGING_ROOT_LEVEL, WARNING}
  handlers:
    - streamHandler
    - rotatingFileHandler
note

The examples in this documentation use YAML format, but examples can be easily converted to JSON if preferred.

Loggers​

Loggers are organized in a hierarchical namespace using dot-separated names, forming a tree structure where child loggers inherit configuration from their parents. When a logger is asked to handle a log record, it propagates the record up through the logger hierarchy until it reaches a logger with handlers configured or reaches the root logger.

Handlers​

Handlers determine where log messages go. The default configuration includes:

  • streamHandler: Outputs logs to the console (stdout) for immediate visibility
  • rotatingFileHandler: Writes logs to files with automatic rotation when size limits are reached.

For complete details on handlers, see Python's supported handlers documentation

Formatters​

Formatters control the structure and appearance of log messages. The default configuration includes:

  • simpleFormatter: Human-readable format including timestamp, level, thread, logger name, and message.
  • coloredFormatter: Similar to simpleFormatter but with color coding for log levels and backend component logs to enhance readability in the console.
  • jsonFormatter: JSON format for log aggregation and analysis tools. See Structured Logging for possible customizations.

Consult Python's documentation for complete details on formatters and available fields.

Understanding Effective Log Levels​

The effective log level for a logger is determined by the most specific configuration in the logger hierarchy. If a logger doesn't have a level explicitly set, it inherits from its parent. The root logger applies to all modules that do not have a logger defined.

For example, if you set the root logger to DEBUG but create a more specific logger for solace_ai_connector at the INFO level, the effective log level for the solace_ai_connector module will be INFO. This means DEBUG level logs from solace_ai_connector will not be handled, as they fall below the effective log level.

Environment Variable Substitution​

All configuration formats (YAML, JSON, and INI) support environment variable substitution using the syntax:

${VARIABLE_NAME, default_value}

Users can use variable names of their choice; the application will look for these environment variables at runtime and substitute their values accordingly. If the environment variable is not set, the provided default value will be used.

Common Configuration Scenarios​

Structured Logging​

Structured logging outputs log messages in JSON format, making them easier to parse, search, and analyze in log aggregation systems like Datadog, Splunk, Elasticsearch, and others.

Enabling structured logging includes two steps.

1- Enable JSON Formatter​

Structured logging is enabled by assigning the jsonFormatter, which is provided in the default logging configuration, to one or more logging handlers in your configuration. This means log messages handled by those specific handlers will be output in structured JSON format.

handlers:
  rotatingFileHandler:
    class: logging.handlers.RotatingFileHandler
    formatter: jsonFormatter  # Changed from simpleFormatter
    filename: ${LOGGING_FILE_NAME, sam.log}
    mode: a
    maxBytes: 52428800
    backupCount: 10
  
  streamHandler:
    class: logging.StreamHandler
    formatter: simpleFormatter  # Kept as simpleFormatter to show handlers can have different formatters
    stream: "ext://sys.stdout"

2- Configure Contextual Info (Optional)​

Log aggregation systems often expect contextual fields to be included in log records for better filtering, grouping, and analysis. For example, contextual fields like service and env can be added to each log record to indicate which service generated the log and the environment it ran in.

To add contextual info to every log record, use python-json-logger's static_fields feature as shown below:

  jsonFormatter:
    "()": pythonjsonlogger.json.JsonFormatter
    format: "%(asctime)s %(levelname)s %(threadName)s %(name)s %(message)s"
    static_fields:
      service: ${SERVICE_NAME, payment-service}
      env: ${ENV, production}

With this jsonFormatter configuration, all JSON log records will automatically include the specified static fields:

{
   "asctime":"2025-10-30 22:25:56,960",
   "levelname":"INFO",
   "threadName":"MainThread",
   "name":"solace_ai_connector.flow",
   "message":"Processing message",
   "service": "payment-service",
   "env": "production"
}

Notice that environment variable substitution can be used for increased flexibility.

Customizing Log Levels​

You can add loggers to control the log level of specific modules or external libraries in your application. This allows you to increase verbosity for troubleshooting specific components while keeping other parts of the system quiet.

For instance, imagine you are troubleshooting an issue with the HTTP SSE gateway, and you also want to see more detailed logs coming from the google_adk external library, you can add the following loggers:

loggers:
  solace_ai_connector:
    level: INFO
    handlers: []
  
  solace_agent_mesh:
    level: INFO
    handlers: []
  
  sam_trace:
    level: INFO
    handlers: []
  
  # Increase verbosity of a specific package
  solace_agent_mesh.gateway.http_sse:
    level: DEBUG
    handlers: []
  
  # Increase verbosity of a specific external library
  google_adk:
    level: INFO
    handlers: []

root:
  level: WARNING
  handlers:
    - streamHandler
    - rotatingFileHandler
Discovering Logger Names

To discover what logger names are available to control, temporarily set the root logger level to DEBUG and run your application. The logger names will appear in each log entry, displayed as a dot-separated hierarchy. You can control verbosity at any level of the logger hierarchy.

Once you've identified the logger names you need, you can:

  1. Set the root logger level back to WARNING to reduce overall verbosity
  2. Add specific logger configurations for the modules/library you want to monitor with increased verbosity

This approach keeps your logs clean while giving you detailed visibility into the specific components you're troubleshooting.

Agent-Specific Log File​

For debugging a specific agent in isolation, you can run your SAM solution across multiple processes, with the agent you want to isolate running by itself with its own dedicated log file. This is particularly useful during development and troubleshooting.

To isolate an agent's logs, run your SAM solution in two separate processes:

# Process 1: Run your other components with default logging
sam run configs/gateways/webui.yaml configs/agents/main_orchestrator.yaml configs/agents/some_other_agents.yaml # logs go to sam.log

# Process 2: Run the isolated agent with its own log file
export LOGGING_FILE_NAME=my_isolated_agent.log && sam run configs/agents/my_agent_with_isolated_logs.yaml # logs go to my_isolated_agent.log

This approach allows you to isolate and analyze a specific agent's behavior without interference from other components in your mesh. Each process writes to its own log file, making debugging much easier.