Logging

Agent Mesh uses Python's built-in logging module to provide flexible and powerful logging capabilities.

Understanding logging configuration is essential for troubleshooting, monitoring system behavior, and maintaining production deployments. This guide covers everything you need to effectively configure and manage logging in your Agent Mesh applications.

Configuration

The logging system is configured through an INI-based configuration file that gives you fine-grained control over log levels, output destinations, and formatting options.

This approach provides several advantages:

• Centralized Control: Single configuration file manages logging for all components
• Python Native: Built on Python's standard logging module
• Flexible and Powerful: Full access to Python's logging capabilities
• Production-Ready: Industry-standard approach used by many Python applications

To provide a logging configuration, set the LOGGING_CONFIG_PATH=path/to/logging_config.ini environment variable in your .env file or with the export command.

INI vs YAML Configuration

While individual agent and gateway YAML files may contain a log: section:

log:
  stdout_log_level: INFO
  log_file_level: INFO
  log_file: my-agent.log

the INI-based configuration is the recommended and preferred approach. YAML configuration has lower precedence and will only be active when INI-based configuration is not enabled.

Default Logging Configuration

When you run sam init, Agent Mesh automatically generates a configs/logging_config.ini file in your project directory. This file establishes sensible defaults while remaining easy to customize for your specific needs.

Understanding the Default Configuration

Here's the default logging_config.ini generated by sam init, with annotations explaining each section:

# Lists all configured loggers in this file
[loggers]
keys=root,solace_ai_connector,solace_agent_mesh,sam_trace

# Root logger - applies to all modules (including external libraries) that do not have a specific logger configured. 
# The root logger also specifies handlers for the application.
[logger_root]
level=${LOGGING_ROOT_LEVEL, WARNING}
handlers=streamHandler,rotatingFileHandler
qualname=root

# Main logger for the solace-ai-connector module
[logger_solace_ai_connector]
level=${LOGGING_SOLACE_AI_CONNECTOR_LEVEL, INFO}
handlers=
qualname=solace_ai_connector

# Main logger for the solace-agent-mesh module
[logger_solace_agent_mesh]
level=${LOGGING_SAM_LEVEL, INFO}
handlers=
qualname=solace_agent_mesh

# Special trace logger for detailed troubleshooting. Set to DEBUG to enable.
[logger_sam_trace]
level=${LOGGING_SAM_TRACE_LEVEL, INFO}
handlers=
qualname=sam_trace

# Lists all configured handlers in this file
[handlers]
keys=streamHandler,rotatingFileHandler

# Rotating file handler - writes to log files with automatic rotation
[handler_rotatingFileHandler]
class=logging.handlers.RotatingFileHandler
formatter=simpleFormatter
args=('sam.log', 'a', 52428800, 10)

# Stream handler - outputs logs to console (stdout)
[handler_streamHandler]
class=StreamHandler
formatter=simpleFormatter
args=(sys.stdout,)

# Lists all configured formatters in this file
[formatters]
keys=simpleFormatter,jsonFormatter

# Simple human-readable format
[formatter_simpleFormatter]
format=%(asctime)s | %(levelname)-5s | %(threadName)s | %(name)s | %(message)s

# JSON format for structured logging
[formatter_jsonFormatter]
class=pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s %(levelname)s %(threadName)s %(name)s %(message)s

Loggers

Loggers are organized in a hierarchical namespace using dot-separated names, forming a tree structure where child loggers inherit configuration from their parents. When a logger is asked to handle a log record, it propagates the record up through the logger hierarchy until it reaches a logger with handlers configured or reaches the root logger.

Handlers

Handlers determine where log messages go. The default configuration includes:

• streamHandler: Outputs logs to the console (stdout) for immediate visibility
• rotatingFileHandler: Writes logs to files with automatic rotation when size limits are reached. The rotating file handler is configured with these parameters (in the args tuple):
  • Filename: sam.log - the base log file name
  • Mode: 'a' - append mode (don't overwrite existing logs)
  • Max Size: 52428800 bytes (50 MB) - rotate when file reaches this size
  • Backup Count: 10 - keep up to 10 historical log files

For complete details on handlers, see Python's supported handlers documentation

Formatters

Formatters control the structure and appearance of log messages:

• simpleFormatter: Human-readable format including timestamp, level, thread, logger name, and message
• jsonFormatter: Structured JSON format for log aggregation and analysis tools

For complete details on formatters and available fields, see Python's LogRecord attributes documentation.

Understanding Effective Log Levels

The effective log level for a logger is determined by the most specific configuration in the logger hierarchy. If a logger doesn't have a level explicitly set, it inherits from its parent. The root logger applies to all modules that do not have a logger defined.

For example, if you set the root logger to DEBUG but create a more specific logger for solace_ai_connector at the INFO level, the effective log level for the solace_ai_connector module will be INFO. This means DEBUG level logs from solace_ai_connector will not be handled, as they fall below the effective log level.

Environment Variable Substitution

The INI configuration supports environment variable substitution using the syntax:

${VARIABLE_NAME, default_value}

Users can use variable names of their choice; the application will look for these environment variables at runtime and substitute their values accordingly. If the environment variable is not set, the provided default value will be used.

Enabling Structured Logging

Structured logging outputs log messages in JSON format, making them easier to parse, search, and analyze with log aggregation tools.

To enable JSON logging, modify the handler configurations to use jsonFormatter instead of simpleFormatter:

[handler_rotatingFileHandler]
class=logging.handlers.RotatingFileHandler
formatter=jsonFormatter  # Changed from simpleFormatter
args=('sam.log', 'a', 52428800, 10)

[handler_streamHandler]
class=StreamHandler
formatter=simpleFormatter  # Kept as simpleFormatter to show handlers can have different formatters
args=(sys.stdout,)

Common Configuration Scenarios

Customizing Log Levels

You can add loggers to control the log level of specific modules or external libraries in your application. This allows you to increase verbosity for troubleshooting specific components while keeping other parts of the system quiet.

# Add to the loggers list
[loggers]
keys=root,solace_ai_connector,solace_agent_mesh,sam_trace,my_logger,google_adk

[logger_root]
level=WARNING
handlers=streamHandler,rotatingFileHandler
qualname=root

[ some loggers were omitted ]

# Increase verbosity of a specific package
[logger_my_logger]
level=DEBUG
handlers=
qualname=solace_agent_mesh.gateway.http_sse

# Increase verbosity of a specific external library
[logger_google_adk]
level=INFO
handlers=
qualname=google_adk

Discovering Logger Names

To discover what logger names are available to control, temporarily set the root logger level to DEBUG and run your application. The logger names will be visible in the actual log output (shown in the logger name field). This is particularly useful for identifying logger names from external libraries you want to control.

Once you've identified the logger names you need, you can:

1. Set the root logger level back to WARNING to reduce overall verbosity
2. Add specific logger configurations for the modules/library you want to monitor with increased verbosity

This approach keeps your logs clean while giving you detailed visibility into the specific components you're troubleshooting.