如何解决使用Kinesis Agent从单行收集的自定义日志写入Kinesis Firehose
所以我有一行日志一起生成。
我想在我的EC2实例上使用Kinesis Agent将这些数据写入Kinesis Firehose。
我已经将自定义matchPattern与自定义fieldNames一起使用,但似乎代理不能从一行中获取多个记录。
我在python脚本生成的伪数据上对此进行了测试。此处的示例:
77.253.46.255 - - [03/Sep/2020:09:46:12 +0000] "PUT /explore HTTP/1.0" 200 4957 "http://www.moore.biz/post.php" "Mozilla/5.0 (Android 2.3.5; Mobile; rv:48.0) Gecko/48.0 Firefox/48.0" 128.206.196.208 - - [03/Sep/2020:09:49:14 +0000] "DELETE /search/tag/list HTTP/1.0" 200 4998 "http://hernandez.com/explore/faq.php" "Mozilla/5.0 (Windows CE) AppleWebKit/533.1 (KHTML,like Gecko) Chrome/29.0.806.0 Safari/533.1" 75.217.144.114 - - [03/Sep/2020:09:51:04 +0000] "GET /wp-content HTTP/1.0" 200 5047 "http://www.mason.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_8) AppleWebKit/536.2 (KHTML,like Gecko) Chrome/33.0.891.0 Safari/536.2" 17.39.118.208 - - [03/Sep/2020:09:52:27 +0000] "PUT /explore HTTP/1.0" 404 5094 "https://ellison-saunders.com/faq/" "Mozilla/5.0 (Android 4.4.2; Mobile; rv:42.0) Gecko/42.0 Firefox/42.0" 9.101.23.95 - - [03/Sep/2020:09:57:04 +0000] "PUT /wp-admin HTTP/1.0" 200 5034 "http://mcmahon-cooper.com/app/main/posts/login.php" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_6_9 rv:3.0; af-ZA) AppleWebKit/535.43.1 (KHTML,like Gecko) Version/5.1 Safari/535.43.1" 43.182.24.1 - - [03/Sep/2020:10:01:55 +0000] "GET /posts/posts/explore HTTP/1.0" 200 4981 "https://pacheco.com/home.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_6 like Mac OS X) AppleWebKit/534.2 (KHTML,like Gecko) FxiOS/12.5b2439.0 Mobile/84P693 Safari/534.2"
我尝试了一些不同的代理配置。这是一个示例:
{
"cloudwatch.emiTMetrics": true,"kinesis.endpoint": "","firehose.endpoint": "https://firehose.eu-west-1.amazonaws.com","flows": [
{
"filePattern": "/tmp/access_log*","deliveryStream": "web-log-ingestion-stream","initialPosition": "START_OF_FILE","dataProcessingOptions": [
{
"optionName": "LOGTOJSON","logFormat": "COMMONAPACHELOG","matchPattern": "([\\d.]+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3})","customFieldNames": ["host","ident","authuser","datetime","request","response"]
}
]
}
]
}
任何建议如何解决此问题?
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。