使用者从avro模式读取两次数据

如何解决使用者从avro模式读取两次数据

我有一个流媒体应用程序，它在其中侦听一些数据，然后通过将数据推送到新主题来转换数据。我都使用avro模式将数据读/写成主题。问题是，当我使用下面的命令从最终目的地消耗数据时。但是，我的数据有点复杂，其中包含一些数组和json，我怀疑我的avro模式可能不适用于我的目的。没有错误或任何东西，我可以看到我最后一个主题上的所有数据，但出于某些原因“ Pets”字段重复，因此我不明白为什么。实际上，我只向avro模式中的现有数据添加了一个新字段（job_id），在转换它时，并没有对其进行大的更改。

./bin/kafka-console-consumer --topic my_topic \
--bootstrap-server localhost:9092 \

这是我拥有的json数据

{
   "Person":{
      "id":"104440","Name":"William","LastName":"Dorsey","archived":false,"Timezone":"America/Los_Angeles","brandCompanyName":"Twitter","brandID":"cf545a7b","creatorID":"1234","currency":"USD","dateCreated":"2020-09-07T02:56:22Z","dateModified":"2020-09-07T02:57:24Z","disabled":false,"endDate":"2020-11-29T19:51:00-08:00","startDate":"2020-08-31T20:55:00-07:00","totalBudget":0
   },"Pets":[
      {
         "Name":"Pawny","Id":"4214","budget":"0","adoptionDate":"2020-09-07T02:56:22Z","year":"2","type":"Golden","gender":"male"
      }
   ],"CreationTime":"1604036638"
}

我的Avro模式

{
  "name": "MyClass","type": "record","namespace": "com.acme.avro","fields": [
    {
      "name": "Person","type": {
        "name": "Person","fields": [
          {
            "name": "id","type": "string"
          },{
            "name": "Name",{
            "name": "LastName",{
            "name": "archived","type": "boolean"
          },{
            "name": "Timezone",{
            "name": "brandCompanyName",{
            "name": "brandID",{
            "name": "creatorID",{
            "name": "currency",{
            "name": "dateCreated","type": "int","logicalType": "date"
          },{
            "name": "dateModified",{
            "name": "disabled",{
            "name": "endDate",{
            "name": "startDate",{
            "name": "totalBudget","type": "int"
          }
        ]
      }
    },{
      "name": "Pets","type": {
        "type": "array","items": {
          "name": "Pets_record","fields": [
            {
              "name": "Name","type": "string"
            },{
              "name": "Id",{
              "name": "budget",{
              "name": "adoptionDate","logicalType": "date"
            },{
              "name": "year",{
              "name": "type",{
              "name": "gender","type": "string"
            }
          ]
        }
      }
    },{
      "name": "CreationTime","type": "string"
    },{
      "name":"jobID","type":"string"
    }
  ]
}

我消费主题时主题中的输出-出于某些原因，pets字段重复了吗？我不知道为什么

{
      "id":"104440","LastName:"Dorsey","totalBudget":0,"adoptionDate":2020-09-07T02:56:22Z","CreationTime":1604036638,"jobID":12512,"pets":[
      {
         "Name":"Pawny","gender":"male"
      }
   ]
}

解决方法

这是因为我在字段名称中使用了大写字母名称……在无休止的循环中徘徊了24小时，如果有人遇到相同的问题，我终于能够弄清楚这一点。请在此处阅读，并将小写名称用作您的字段名称。当我将字段名称更改为“ pet”时。重复项不见了