引言日志是任何生产级应用不可或缺的组成部分。作为从Python转向Rust的开发者我深刻理解良好的日志系统对于应用可观测性的重要性。本文将深入探讨Python日志框架的设计原理和最佳实践帮助你构建高效、可扩展的日志系统。一、logging模块基础1.1 基本配置import logging # 基础配置 logging.basicConfig( levellogging.INFO, format%(asctime)s - %(name)s - %(levelname)s - %(message)s, datefmt%Y-%m-%d %H:%M:%S ) logger logging.getLogger(__name__) logger.debug(Debug message) logger.info(Info message) logger.warning(Warning message) logger.error(Error message) logger.critical(Critical message)1.2 日志级别级别数值用途NOTSET0未设置继承父级别DEBUG10详细调试信息INFO20一般运行信息WARNING30警告信息ERROR40错误信息CRITICAL50严重错误1.3 Logger层次结构import logging # 创建层级Logger root_logger logging.getLogger() app_logger logging.getLogger(myapp) db_logger logging.getLogger(myapp.database) # 设置不同级别 root_logger.setLevel(logging.WARNING) app_logger.setLevel(logging.INFO) db_logger.setLevel(logging.DEBUG) # 子Logger继承父Logger的设置 db_logger.info(Database connected)二、Handler机制2.1 多Handler配置import logging logger logging.getLogger(multi_handler) logger.setLevel(logging.DEBUG) # 控制台Handler console_handler logging.StreamHandler() console_handler.setLevel(logging.INFO) # 文件Handler file_handler logging.FileHandler(app.log) file_handler.setLevel(logging.DEBUG) # 格式化器 formatter logging.Formatter( %(asctime)s - %(name)s - %(levelname)s - %(message)s ) console_handler.setFormatter(formatter) file_handler.setFormatter(formatter) # 添加Handler logger.addHandler(console_handler) logger.addHandler(file_handler) logger.debug(Only in file) logger.info(In file and console)2.2 常用Handler类型import logging # RotatingFileHandler - 按大小轮转 rotating_handler logging.handlers.RotatingFileHandler( app.log, maxBytes1024*1024*5, # 5MB backupCount5 ) # TimedRotatingFileHandler - 按时轮转 timed_handler logging.handlers.TimedRotatingFileHandler( app.log, whenmidnight, backupCount7 ) # SMTPHandler - 邮件告警 smtp_handler logging.handlers.SMTPHandler( mailhost(smtp.example.com, 587), fromaddrloggerexample.com, toaddrs[adminexample.com], subjectApplication Error ) smtp_handler.setLevel(logging.CRITICAL)三、高级配置3.1 字典配置import logging.config LOGGING_CONFIG { version: 1, disable_existing_loggers: False, formatters: { standard: { format: %(asctime)s [%(levelname)s] %(name)s: %(message)s }, detailed: { format: %(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s } }, handlers: { console: { class: logging.StreamHandler, formatter: standard, level: INFO }, file: { class: logging.handlers.RotatingFileHandler, filename: app.log, formatter: detailed, level: DEBUG, maxBytes: 5*1024*1024, backupCount: 5 } }, loggers: { myapp: { handlers: [console, file], level: DEBUG, propagate: False }, myapp.database: { handlers: [file], level: DEBUG, propagate: False } }, root: { handlers: [console], level: WARNING } } logging.config.dictConfig(LOGGING_CONFIG)3.2 YAML配置version: 1 disable_existing_loggers: false formatters: standard: format: %(asctime)s [%(levelname)s] %(name)s: %(message)s detailed: format: %(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s handlers: console: class: logging.StreamHandler formatter: standard level: INFO file: class: logging.handlers.RotatingFileHandler filename: app.log formatter: detailed level: DEBUG maxBytes: 5242880 backupCount: 5 loggers: myapp: handlers: [console, file] level: DEBUG propagate: false root: handlers: [console] level: WARNINGimport yaml import logging.config with open(logging.yaml, r) as f: config yaml.safe_load(f.read()) logging.config.dictConfig(config)四、结构化日志4.1 使用JSON格式import logging import json from pythonjsonlogger import jsonlogger logger logging.getLogger(json_logger) logger.setLevel(logging.DEBUG) handler logging.StreamHandler() formatter jsonlogger.JsonFormatter( %(asctime)s %(levelname)s %(name)s %(message)s ) handler.setFormatter(formatter) logger.addHandler(handler) logger.info(User login, extra{ user_id: 123, ip_address: 192.168.1.1 })4.2 自定义LogRecordimport logging class CustomLogRecord(logging.LogRecord): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.app_name myapp self.environment production class CustomLogger(logging.Logger): def makeRecord(self, *args, **kwargs): return CustomLogRecord(*args, **kwargs) # 使用自定义Logger logging.setLoggerClass(CustomLogger) logger logging.getLogger(custom) logger.info(Test message)五、日志最佳实践5.1 命名规范# 推荐使用模块级别Logger logger logging.getLogger(__name__) # 不推荐使用固定名称 # logger logging.getLogger(myapp)5.2 异常处理日志import logging logger logging.getLogger(exception_logger) try: result 1 / 0 except Exception as e: # 记录异常信息 logger.exception(Error occurred during calculation) # 等同于 # logger.error(Error occurred during calculation, exc_infoTrue)5.3 性能考虑import logging logger logging.getLogger(performance) # 避免不必要的字符串格式化 if logger.isEnabledFor(logging.DEBUG): logger.debug(fProcessing item {item_id} with data {complex_data}) # 使用lazy evaluation logger.debug(Processing item %s with data %s, item_id, complex_data)六、实战构建企业级日志系统6.1 完整配置示例import logging.config import os def setup_logging(): log_dir os.path.join(os.path.dirname(__file__), logs) os.makedirs(log_dir, exist_okTrue) LOGGING_CONFIG { version: 1, disable_existing_loggers: False, formatters: { standard: { format: %(asctime)s [%(levelname)s] %(name)s: %(message)s }, json: { (): pythonjsonlogger.jsonlogger.JsonFormatter, format: %(asctime)s %(levelname)s %(name)s %(message)s } }, handlers: { console: { class: logging.StreamHandler, formatter: standard, level: INFO }, file: { class: logging.handlers.RotatingFileHandler, filename: os.path.join(log_dir, app.log), formatter: standard, level: DEBUG, maxBytes: 5*1024*1024, backupCount: 10 }, json_file: { class: logging.handlers.RotatingFileHandler, filename: os.path.join(log_dir, app.json), formatter: json, level: INFO, maxBytes: 5*1024*1024, backupCount: 10 } }, loggers: { : { handlers: [console, file, json_file], level: DEBUG, propagate: True } } } logging.config.dictConfig(LOGGING_CONFIG)6.2 日志监控集成import logging from logging.handlers import HTTPHandler # 添加远程日志收集 http_handler HTTPHandler( hostlogs.example.com, url/api/logs, methodPOST ) http_handler.setLevel(logging.ERROR) logger logging.getLogger(remote) logger.addHandler(http_handler)七、从Python到Rust的日志迁移7.1 Python logging vs Rust tracingPython版本import logging logger logging.getLogger(app) logger.info(User logged in, extra{user_id: 123})Rust版本use tracing::{info, instrument}; use tracing_subscriber; #[instrument] async fn login(user_id: u64) { info!(User logged in, user_id user_id); } #[tokio::main] async fn main() { tracing_subscriber::fmt::init(); login(123).await; }7.2 优势对比特性Python loggingRust tracing结构化日志第三方支持原生支持性能较好接近原生分布式追踪有限原生支持宏支持无宏实现八、常见问题与解决方案8.1 日志重复输出# 问题重复输出 logger logging.getLogger(myapp) logger.info(Message) # 可能输出多次 # 解决方案禁止传播 logger.propagate False8.2 日志级别不生效# 问题设置了级别但不生效 logger logging.getLogger(myapp) logger.info(Message) # 不输出 # 解决方案检查根Logger级别 logging.basicConfig(levellogging.INFO)8.3 多进程日志冲突# 问题多进程写入同一文件冲突 # 解决方案使用ConcurrentLogHandler from concurrent_log_handler import ConcurrentRotatingFileHandler handler ConcurrentRotatingFileHandler( app.log, maxBytes5*1024*1024, backupCount5 )九、总结Python的logging模块提供了强大而灵活的日志功能。通过合理配置可以构建出满足各种需求的日志系统基础配置设置级别、格式和Handler高级配置使用字典或YAML配置文件结构化日志支持JSON格式便于分析最佳实践命名规范、异常处理、性能优化实战集成构建企业级日志系统通过掌握Python日志框架你可以提升应用的可观测性更好地监控和调试生产环境中的问题。参考资料Python logging文档https://docs.python.org/3/library/logging.htmlpython-json-loggerhttps://pypi.org/project/python-json-logger/tracing cratehttps://crates.io/crates/tracing
Python日志框架设计：从基础到高级配置
引言日志是任何生产级应用不可或缺的组成部分。作为从Python转向Rust的开发者我深刻理解良好的日志系统对于应用可观测性的重要性。本文将深入探讨Python日志框架的设计原理和最佳实践帮助你构建高效、可扩展的日志系统。一、logging模块基础1.1 基本配置import logging # 基础配置 logging.basicConfig( levellogging.INFO, format%(asctime)s - %(name)s - %(levelname)s - %(message)s, datefmt%Y-%m-%d %H:%M:%S ) logger logging.getLogger(__name__) logger.debug(Debug message) logger.info(Info message) logger.warning(Warning message) logger.error(Error message) logger.critical(Critical message)1.2 日志级别级别数值用途NOTSET0未设置继承父级别DEBUG10详细调试信息INFO20一般运行信息WARNING30警告信息ERROR40错误信息CRITICAL50严重错误1.3 Logger层次结构import logging # 创建层级Logger root_logger logging.getLogger() app_logger logging.getLogger(myapp) db_logger logging.getLogger(myapp.database) # 设置不同级别 root_logger.setLevel(logging.WARNING) app_logger.setLevel(logging.INFO) db_logger.setLevel(logging.DEBUG) # 子Logger继承父Logger的设置 db_logger.info(Database connected)二、Handler机制2.1 多Handler配置import logging logger logging.getLogger(multi_handler) logger.setLevel(logging.DEBUG) # 控制台Handler console_handler logging.StreamHandler() console_handler.setLevel(logging.INFO) # 文件Handler file_handler logging.FileHandler(app.log) file_handler.setLevel(logging.DEBUG) # 格式化器 formatter logging.Formatter( %(asctime)s - %(name)s - %(levelname)s - %(message)s ) console_handler.setFormatter(formatter) file_handler.setFormatter(formatter) # 添加Handler logger.addHandler(console_handler) logger.addHandler(file_handler) logger.debug(Only in file) logger.info(In file and console)2.2 常用Handler类型import logging # RotatingFileHandler - 按大小轮转 rotating_handler logging.handlers.RotatingFileHandler( app.log, maxBytes1024*1024*5, # 5MB backupCount5 ) # TimedRotatingFileHandler - 按时轮转 timed_handler logging.handlers.TimedRotatingFileHandler( app.log, whenmidnight, backupCount7 ) # SMTPHandler - 邮件告警 smtp_handler logging.handlers.SMTPHandler( mailhost(smtp.example.com, 587), fromaddrloggerexample.com, toaddrs[adminexample.com], subjectApplication Error ) smtp_handler.setLevel(logging.CRITICAL)三、高级配置3.1 字典配置import logging.config LOGGING_CONFIG { version: 1, disable_existing_loggers: False, formatters: { standard: { format: %(asctime)s [%(levelname)s] %(name)s: %(message)s }, detailed: { format: %(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s } }, handlers: { console: { class: logging.StreamHandler, formatter: standard, level: INFO }, file: { class: logging.handlers.RotatingFileHandler, filename: app.log, formatter: detailed, level: DEBUG, maxBytes: 5*1024*1024, backupCount: 5 } }, loggers: { myapp: { handlers: [console, file], level: DEBUG, propagate: False }, myapp.database: { handlers: [file], level: DEBUG, propagate: False } }, root: { handlers: [console], level: WARNING } } logging.config.dictConfig(LOGGING_CONFIG)3.2 YAML配置version: 1 disable_existing_loggers: false formatters: standard: format: %(asctime)s [%(levelname)s] %(name)s: %(message)s detailed: format: %(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s handlers: console: class: logging.StreamHandler formatter: standard level: INFO file: class: logging.handlers.RotatingFileHandler filename: app.log formatter: detailed level: DEBUG maxBytes: 5242880 backupCount: 5 loggers: myapp: handlers: [console, file] level: DEBUG propagate: false root: handlers: [console] level: WARNINGimport yaml import logging.config with open(logging.yaml, r) as f: config yaml.safe_load(f.read()) logging.config.dictConfig(config)四、结构化日志4.1 使用JSON格式import logging import json from pythonjsonlogger import jsonlogger logger logging.getLogger(json_logger) logger.setLevel(logging.DEBUG) handler logging.StreamHandler() formatter jsonlogger.JsonFormatter( %(asctime)s %(levelname)s %(name)s %(message)s ) handler.setFormatter(formatter) logger.addHandler(handler) logger.info(User login, extra{ user_id: 123, ip_address: 192.168.1.1 })4.2 自定义LogRecordimport logging class CustomLogRecord(logging.LogRecord): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.app_name myapp self.environment production class CustomLogger(logging.Logger): def makeRecord(self, *args, **kwargs): return CustomLogRecord(*args, **kwargs) # 使用自定义Logger logging.setLoggerClass(CustomLogger) logger logging.getLogger(custom) logger.info(Test message)五、日志最佳实践5.1 命名规范# 推荐使用模块级别Logger logger logging.getLogger(__name__) # 不推荐使用固定名称 # logger logging.getLogger(myapp)5.2 异常处理日志import logging logger logging.getLogger(exception_logger) try: result 1 / 0 except Exception as e: # 记录异常信息 logger.exception(Error occurred during calculation) # 等同于 # logger.error(Error occurred during calculation, exc_infoTrue)5.3 性能考虑import logging logger logging.getLogger(performance) # 避免不必要的字符串格式化 if logger.isEnabledFor(logging.DEBUG): logger.debug(fProcessing item {item_id} with data {complex_data}) # 使用lazy evaluation logger.debug(Processing item %s with data %s, item_id, complex_data)六、实战构建企业级日志系统6.1 完整配置示例import logging.config import os def setup_logging(): log_dir os.path.join(os.path.dirname(__file__), logs) os.makedirs(log_dir, exist_okTrue) LOGGING_CONFIG { version: 1, disable_existing_loggers: False, formatters: { standard: { format: %(asctime)s [%(levelname)s] %(name)s: %(message)s }, json: { (): pythonjsonlogger.jsonlogger.JsonFormatter, format: %(asctime)s %(levelname)s %(name)s %(message)s } }, handlers: { console: { class: logging.StreamHandler, formatter: standard, level: INFO }, file: { class: logging.handlers.RotatingFileHandler, filename: os.path.join(log_dir, app.log), formatter: standard, level: DEBUG, maxBytes: 5*1024*1024, backupCount: 10 }, json_file: { class: logging.handlers.RotatingFileHandler, filename: os.path.join(log_dir, app.json), formatter: json, level: INFO, maxBytes: 5*1024*1024, backupCount: 10 } }, loggers: { : { handlers: [console, file, json_file], level: DEBUG, propagate: True } } } logging.config.dictConfig(LOGGING_CONFIG)6.2 日志监控集成import logging from logging.handlers import HTTPHandler # 添加远程日志收集 http_handler HTTPHandler( hostlogs.example.com, url/api/logs, methodPOST ) http_handler.setLevel(logging.ERROR) logger logging.getLogger(remote) logger.addHandler(http_handler)七、从Python到Rust的日志迁移7.1 Python logging vs Rust tracingPython版本import logging logger logging.getLogger(app) logger.info(User logged in, extra{user_id: 123})Rust版本use tracing::{info, instrument}; use tracing_subscriber; #[instrument] async fn login(user_id: u64) { info!(User logged in, user_id user_id); } #[tokio::main] async fn main() { tracing_subscriber::fmt::init(); login(123).await; }7.2 优势对比特性Python loggingRust tracing结构化日志第三方支持原生支持性能较好接近原生分布式追踪有限原生支持宏支持无宏实现八、常见问题与解决方案8.1 日志重复输出# 问题重复输出 logger logging.getLogger(myapp) logger.info(Message) # 可能输出多次 # 解决方案禁止传播 logger.propagate False8.2 日志级别不生效# 问题设置了级别但不生效 logger logging.getLogger(myapp) logger.info(Message) # 不输出 # 解决方案检查根Logger级别 logging.basicConfig(levellogging.INFO)8.3 多进程日志冲突# 问题多进程写入同一文件冲突 # 解决方案使用ConcurrentLogHandler from concurrent_log_handler import ConcurrentRotatingFileHandler handler ConcurrentRotatingFileHandler( app.log, maxBytes5*1024*1024, backupCount5 )九、总结Python的logging模块提供了强大而灵活的日志功能。通过合理配置可以构建出满足各种需求的日志系统基础配置设置级别、格式和Handler高级配置使用字典或YAML配置文件结构化日志支持JSON格式便于分析最佳实践命名规范、异常处理、性能优化实战集成构建企业级日志系统通过掌握Python日志框架你可以提升应用的可观测性更好地监控和调试生产环境中的问题。参考资料Python logging文档https://docs.python.org/3/library/logging.htmlpython-json-loggerhttps://pypi.org/project/python-json-logger/tracing cratehttps://crates.io/crates/tracing
