Grok模式在LinuxUbuntu中提取Syslog

如何解决Grok模式在LinuxUbuntu中提取Syslog

如果您有任何 grok模式来提取ubuntu中的系统日志，请提供它。谢谢！

已编辑--- >>

我的系统日志示例->

"Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:23:02Z\",\"tags\":[],\"pid\":1877,\"method\":\"post\",\"statusCode\":200,\"req\":{\"url\":\"/internal/search/es\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"content-length\":\"861\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"origin\":\"http://example.com\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip,deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":65,\"contentLength\":9},\"message\":\"POST /internal/search/es 200 65ms - 9.0B\"}"

我尝试了以下过滤器->

filter {
      grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
            #match => {"syslog_message" => "%{WORD:FILTERED}" }
            #add_field => [ "received_at","%{@timestamp}" ]
            #add_field => [ "received_from","%{host}" ]
            remove_field => ["host","message"]
            
      }

      mutate{

            rename => ["@timestamp","time"]

      }
}

然后我得到以下输出。我想提取系统日志消息部分。

 "time" => 2020-08-20T11:17:57.995Z,"syslog_message" => "message repeated 9 times: [ {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:27:22Z\",\"method\":\"get\",\"req\":{\"url\":\"/api/rollup/indices\",\"responseTime\":31,\"message\":\"GET /api/rollup/indices 200 31ms - 9.0B\"},]","syslog_timestamp" => "Aug 20 15:58:52","path" => "/var/log/syslog","@version" => "1","syslog_hostname" => "amantha-ubuntu-server","syslog_program" => "amantha"

解决方法

我要做的第一件事是在该行上使用gsub来删除"和\。

您可以使用：

mutate {
 gsub => [ "message","[\\\"]","" ]
}

这会让您离开：

Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {type:response,@timestamp:2020-08-20T10:23:02Z,tags:[],pid:1877,method:post,statusCode:200,req:{url:/internal/search/es,headers:{connection:upgrade,host:example.com,content-length:861,kbn-version:7.8.1,user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/84.0.4147.125 Safari/537.36,content-type:application/json,accept:*/*,origin:http://example.com,referer:http://example.com/app/kibana,accept-encoding:gzip,deflate,accept-language:en-US,en;q=0.9,si;q=0.8},remoteAddress:127.0.0.1,userAgent:127.0.0.1,referer:http://example.com/app/kibana},res:{statusCode:200,responseTime:65,contentLength:9},message:POST /internal/search/es 200 65ms - 9.0B}

然后您可以使用以下行在gsub之后获取“ syslog消息”。我真的不知道您是否希望将其进一步分解，但如果愿意的话，很乐意提供帮助。

(?<syslog_timestamp>%{SYSLOGTIMESTAMP}) (?<syslog_hostname>%{SYSLOGHOST}) (?<syslog_program>%{SYSLOGPROG}): {(?<syslog_message>(?<={).*(?=}))}