如何解决Logstash JSON Grok 过滤器问题
我设置了鱿鱼代理以通过 Logstash 将 JSON 格式的日志发送到 Elastic。我正在尝试使用 GROK 过滤来解析日志。过滤器在 Kiabana Grok Debugger 中工作,但在我重新启动 Logstash 时抱怨以下错误
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:squid_logs,:exception=>"LogStash::ConfigurationError",:message=>"Expected one of [ \\t\\r\\n],\"#\",\"
{\",\",\"]\" at line 10,column 62 (byte 137) after filter {\n grok {\n match => {\n
\"message\" => [ \"%{IPV4:vendor_ip}\",\"%{WORD:message}\"",:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'","org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'","org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}
我有以下 GROK 过滤器
"%{IPV4:vendor_ip}","%{WORD:message}": "%{IPV4:clientip}","%{WORD:message}": "%
{DATA:timestamp}","%{WORD:message}": "%{WORD:verb}","%{WORD:message}": "%{DATA:request}","%
{WORD:message}": "%{URIPATHPARAM:path}"
在 Kibana Grok Debugger 中,过滤器可以很好地处理如下消息:
{ "vendor_ip": "x.x.x.x","clientip": "x.x.x.x","timestamp": "2021-04-09T13:58:38+0000","verb": "GET","request": "https://domain","path": "/somepath","httpversion": "HTTP/1.1","response": 200,"bytes": 2518042,"referer": "-","useragent": "Microsoft BITS/7.8","request_status": "HIER_DIRECT","hierarchy_status": "HIER_DIRECT" }
Logstash 配置如下:
input {
beats {
port => 5045
}
}
filter {
grok {
match => {
"message" => [ "%{IPV4:vendor_ip}","%{WORD:message}": "%{DATA:timestamp}","%{WORD:message}": "%{URIPATHPARAM:path}" ]
}
}
}
output {
elasticsearch {
hosts => ["x.x.x.x:9200"]
index => "squid_logs"
}
}
解决方法
使用 grok 过滤器解析 json 消息是错误的方法,没有必要这样做,这将需要大量工作,因为您需要转义消息中的所有双引号,否则会出现配置错误,这就是你的情况。
使用 json 过滤器解析 json 消息
只需在您的管道中使用它:
filter {
json {
source => "message"
}
}
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。