philiprehberger-log_filter
v0.4.0405,944 downloadsMIT
Pattern-based log filtering with drop, replace, and preset rules
$
gem install philiprehberger-log_filterREADME
philiprehberger-log_filter
Pattern-based log filtering with drop, replace, and preset rules
Requirements
- Ruby >= 3.1
Installation
Add to your Gemfile:
gem "philiprehberger-log_filter"
Or install directly:
gem install philiprehberger-log_filter
Usage
require "philiprehberger/log_filter"
# Build a custom filter chain
filter = Philiprehberger::LogFilter::Filter.new
.drop(/health_?check/i)
.drop(/DEBUG/)
.replace(/password=\S+/, "password=[REDACTED]")
filter.apply("GET /healthcheck 200") # => nil (dropped)
filter.apply("DEBUG some noise") # => nil (dropped)
filter.apply("login password=abc123") # => "login password=[REDACTED]"
filter.apply("GET /api/users 200") # => "GET /api/users 200"
Wrapping a Logger
require "logger"
require "philiprehberger/log_filter"
logger = Logger.new($stdout)
filter = Philiprehberger::LogFilter::Filter.new
.drop(/healthcheck/i)
.replace(/token=\S+/, "token=[REDACTED]")
filtered_logger = Philiprehberger::LogFilter.wrap(logger, filter)
filtered_logger.info("GET /healthcheck 200") # silently dropped
filtered_logger.info("auth token=secret123") # logs "auth token=[REDACTED]"
filtered_logger.info("GET /api/users 200") # logs normally
Using Presets
require "philiprehberger/log_filter"
# Drop health-check noise
filter = Philiprehberger::LogFilter.health_check_filter
filtered_logger = Philiprehberger::LogFilter.wrap(logger, filter)
# Drop static asset requests
filter = Philiprehberger::LogFilter.asset_filter
# Drop bot/crawler traffic
filter = Philiprehberger::LogFilter.bot_filter
Block-Based Drop Rules
require "philiprehberger/log_filter"
filter = Philiprehberger::LogFilter::Filter.new
.drop_if { |msg| msg.length > 1000 } # drop excessively long messages
.drop_if { |msg| msg.count("\n") > 10 } # drop multi-line spam
Sampling
require "philiprehberger/log_filter"
# Only pass through 10% of debug messages
filter = Philiprehberger::LogFilter::Filter.new
.sample(/DEBUG/, rate: 0.1)
filter.apply("DEBUG verbose output") # => nil (90% of the time)
filter.apply("INFO normal message") # => "INFO normal message" (always passes)
Structured Log Support
require "philiprehberger/log_filter"
filter = Philiprehberger::LogFilter::Filter.new
.drop_field("password")
.mask_field("ssn", with: "***")
filter.apply('{"user":"alice","password":"secret","ssn":"123-45-6789"}')
# => '{"user":"alice","ssn":"***"}'
# Non-JSON messages pass through unmodified
filter.apply("plain text log line") # => "plain text log line"
Chaining filters
Compose two filters into a new filter whose apply pipes each event through
the first filter and then through the second. If the first filter drops the
event (returns nil), the second is skipped. Composition is associative, so
a.chain(b).chain(c) works as expected.
require "philiprehberger/log_filter"
redact = Philiprehberger::LogFilter::Filter.new
.replace(/password=\S+/, "password=[REDACTED]")
drop_debug = Philiprehberger::LogFilter::Filter.new
.drop(/DEBUG/)
pipeline = drop_debug.chain(redact)
pipeline.apply("DEBUG noise") # => nil (dropped by first filter)
pipeline.apply("login password=abc123") # => "login password=[REDACTED]"
The chained filter tracks its own stats independently of the source filters.
PII and Secret Redaction Presets
require "philiprehberger/log_filter"
pii = Philiprehberger::LogFilter::Presets.pii
pii.apply("login user=alice@example.com") # => "login user=[REDACTED]"
pii.apply("SSN 123-45-6789 lookup") # => "SSN [REDACTED] lookup"
pii.apply("card 4242424242424242 charged") # => "card [REDACTED] charged"
secrets = Philiprehberger::LogFilter::Presets.secrets
secrets.apply("Authorization: Bearer abc.def-_xyz") # => "Authorization: Bearer [REDACTED]"
secrets.apply("GET /v1?api_key=sk_live_xyz123 200") # => "GET /v1?api_key=[REDACTED] 200"
secrets.apply("AKIAIOSFODNN7EXAMPLE in env") # => "[REDACTED] in env"
Filter Statistics
require "philiprehberger/log_filter"
filter = Philiprehberger::LogFilter::Filter.new
.drop(/DEBUG/)
.replace(/secret/, "[REDACTED]")
filter.apply("DEBUG noise")
filter.apply("has secret data")
filter.apply("normal message")
filter.stats # => { dropped: 1, passed: 2, replaced: 1, sampled: 0 }
filter.reset_stats!
filter.stats # => { dropped: 0, passed: 0, replaced: 0, sampled: 0 }
API
| Class / Method | Description |
|---|---|
Filter.new | Create a new empty filter chain |
Filter#drop(pattern) | Add a regex drop rule; returns self |
Filter#drop_if(&block) | Add a block-based drop rule; returns self |
Filter#replace(pattern, replacement) | Add a replacement rule; returns self |
Filter#sample(pattern, rate:) | Add a sampling rule; only pass rate fraction of matches |
Filter#drop_field(key) | Remove a field from JSON log messages; returns self |
Filter#mask_field(key, with:) | Mask a field value in JSON log messages; returns self |
Filter#truncate(max_length, suffix:) | Truncate outgoing messages longer than max_length and append the suffix; returns self |
Filter#apply(message) | Run all rules; returns transformed string or nil |
Filter#chain(other) | Compose with another filter; returns a new filter piping events through both |
Filter#stats | Return counters: dropped, passed, replaced, sampled |
Filter#reset_stats! | Zero all statistics counters |
Wrapper.new(logger, filter) | Wrap a Logger with a filter |
Presets.health_check | Filter dropping health-check paths |
Presets.assets | Filter dropping static-asset requests |
Presets.bots | Filter dropping bot/crawler traffic |
Presets.pii | Filter redacting emails, SSNs, and credit-card-shaped numbers with [REDACTED] |
Presets.secrets | Filter redacting Bearer tokens, api_key=/access_token= values, and AWS access keys |
LogFilter.wrap(logger, filter) | Convenience wrapper constructor |
LogFilter.health_check_filter | Shortcut for Presets.health_check |
LogFilter.asset_filter | Shortcut for Presets.assets |
LogFilter.bot_filter | Shortcut for Presets.bots |
Development
bundle install
bundle exec rspec
bundle exec rubocop
Support
If you find this project useful: