Reusable Python Code Modules, Part 10 - Monitoring and Analytics
Tracking Performance: Building Reusable Monitoring and Analytics Modules
Monitoring and analytics are critical for understanding the performance and usage of back-end APIs, identifying bottlenecks, and ensuring the system operates efficiently. This guide covers how to structure reusable monitoring and analytics modules in Python using Flask.
Common Libraries and Tools
1. Prometheus
Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability.
Key Features
Metrics Collection: Collects time-series data via a pull model over HTTP.
Alerting: Integrated alerting system
Multi-dimensional Data Model: Data is identified by metric name and key-value pairs
Powerful Query Language (PromQL): Allows for flexible and powerful queries on the collected data
2. Grafana
Grafana is an open-source platform for monitoring and observability, often used in conjunction with Prometheus.
Key Features
Visualization: Create dashboards and visualizations for metrics data
Alerting: Custom alerting rules and notifications
Data Source Integration: Supports various data sources, including Prometheus, Graphite, and Elasticsearch
Extensibility: Supports plugins for additional functionalities
3. ELK Stack (Elasticsearch, Logstash, Kibana)
The ELK Stack is a collection of three open-source products designed to collect, search, and visualize log data in real time.
Key Features
Log Management: Collect, analyze, and visualize log data
Real-time Search and Analysis: Real-time search, analysis, and visualization of log data
Scalability: Can scale horizontally to handle large volumes of data
Extensive Ecosystem: Wide range of plugins and integrations
4. Sentry
Sentry is an error tracking and performance monitoring tool that helps developers monitor and fix crashes in real time
Key Features
Real-Time Error Tracking: Captures and reports errors in real-time
Performance Monitoring: Tracks application performance and identifies bottlenecks
Contextual Information: Provides detailed context for each error
Integration with Various Platforms: Supports integration with many platforms and frameworks
Comparison
Prometheus: Best for time-series metrics collection with integrated alerting and a powerful query language
Grafana: Ideal for visualizing metrics data and creating custom dashboards
ELK Stack: Suitable for log management, real-time search, and analysis
Sentry: Best for real-time error tracking and performance monitoring with detailed context and integrations
Examples
Example 1: Prometheus
Setup:
$ pip install prometheus_clientConfiguration:
from flask import Flask, Response
from prometheus_client import Counter, generate_latest
app = Flask(__name__)
REQUEST_COUNT = Counter('request_count', 'Total request count', ['method', 'endpoint'])
@app.before_request
def before_request():
REQUEST_COUNT.labels(method=request.method, endpoint=request.path).inc()
@app.route('/metrics')
def metrics():
return Response(generate_latest(), mimetype='text/plain')Usage:
@app.route('/hello')
def hello():
return 'Hello, World!'Example 2: Grafana
Setup: No additional setup required for Grafana integration, but Grafana should be configured to visualize Prometheus data.
Configuration:
# Grafana datasource configuration
datasources:
- name: Prometheus
type: prometheus
url: http://localhost:9090
access: proxy
isDefault: trueUsage: Create dashboards in Grafana to visualize metrics collected by Prometheus
Example 3: ELK Stack (Elasticsearch, Logstash, Kibana)
Setup:
# Elasticsearch
docker run -d --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.10.1
# Logstash
docker run -d --name logstash -p 5044:5044 -v logstash.conf:/usr/share/logstash/pipeline/logstash.conf logstash:7.10.1
# Kibana
docker run -d --name kibana -p 5601:5601 kibana:7.10.1Configuration (logstash.conf):
input {
tcp {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
}
stdout { codec => rubydebug }
}Usage:
import logging
import socket
logger = logging.getLogger('flask_app')
handler = logging.StreamHandler(socket.socket(socket.AF_INET, socket.SOCK_STREAM))
handler.setFormatter(logging.Formatter('%(message)s'))
logger.addHandler(handler)
@app.route('/log')
def log():
logger.info('This is a log message')
return 'Logged message'Example 4: Sentry
Setup:
$ pip install sentry-sdkConfiguration:
import sentry_sdk
from flask import Flask
from sentry_sdk.integrations.flask import FlaskIntegration
sentry_sdk.init(
dsn="your_sentry_dsn",
integrations=[FlaskIntegration()],
traces_sample_rate=1.0
)
app = Flask(__name__)Usage:
@app.route('/error')
def trigger_error():
division_by_zero = 1 / 0
@app.errorhandler(Exception)
def handle_exception(e):
sentry_sdk.capture_exception(e)
return 'An error occurred', 500