Kafka Connect自动从Debezium主题中提取主键

如何解决Kafka Connect自动从Debezium主题中提取主键

我能够通过Postgres成功连接Debezium CDC + Kafka。我还能够通过JDBC Kafka连接回postgresql db，在简单的表格书（序列pk，字符串标题，字符串作者）上获取debezium kafka主题。插入，更新，删除可以在此简单表上正常工作。但是，如果我想为多个postgres表设置一个kafka connect主题（debezium主题），该怎么办？我需要以1：1传输数据。下面的示例使用相同的数据库，实际上它将写入其他数据库，因此表名将保持一致。

下面是postgres配置属性文件

name=sink-postgres
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
topics=pokladna.public.books
connection.url=jdbc:postgresql://localhost:5434/postgres
connection.user=postgres
connection.password=postgres
dialect.name=PostgreSqlDatabaseDialect
table.name.format=books_kafka
transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.drop.tombstones=false
auto.create=true
auto.evolve=true
insert.mode=upsert
delete.enabled=true
pk.fields=id
pk.mode=record_key

我有2个问题：

您知道吗，如何将pk.mode和pk.fields设置为在具有约束的复合主键的情况下获取表中的更改？

更新和答案：简单的复合键就可以了（例如，PRIMARY KEY（question_id，tag_id）），只需将其添加到pk.fields列名即可。约束下的键也可以正常工作（例如CONSTRAINT tag_constraint RIMARY KEY（question_id，tag_id））-由于约束名称不同，您无法将同一数据库用于源和目标。

如何从kafka消息中自动提取主列名称？

下面我附上了debezium主题的输出。我本来打算使用SMT extract field by name，但它可以用于示例中显示的简单消息。在以下情况下，我想我必须弄平消息并提取Payload.after.id。 但是，如果每个表中的主键名称不同，该怎么办？

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --to                      
pokladna.public.books --property print.key=true --property key.separator=::::

{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"}],"name":"pokladna.public.books.Key"},"payload":{"id":8}}
::::
{"schema":{"type":"struct","fields":[{"type":"struct","field":"id"},{"type":"string","field":"title"},"optional":true," field":"primary_author"}],"name":"pokladna.public.books.Value","field":"before"},{"type":"struct","field":"primary_author"}],"field":"after"},"fields":[{"type":"string","field":"version"},"field":"connector"},"field":"name"},{"type":"int64","field":"ts_ms"},"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},"field":"db"},"field":"schema"},"field":"table"},"field":"txId"},"field":"lsn"},"field":"xmin"}],"name":"io.debezium.connector.postgresql.Source","field":"source"},"field":"op"},"fields":"type":"string","field":"total_order"},"field":"data_collection_order"}],"optional"                      :true,"field":"transaction"}],"name":"pokladna.public.books.Envelope"},"payload":{"before":null,"after":{"id":8,"title":"jabadyba","primary_author":"jajajaja"},"source":{"version":"1.4.0-SNAPSHOT","connector":"postgresql","name":"pokladna","ts_ms":1603718235441,"snapshot":"false","db":"postgres","schema":"public","table":"books","txId":716,"lsn":30457288,"xmin":null},"op":"c","ts_ms":1603718235812,"transaction":null}}

Kafka Connect自动从Debezium主题中提取主键

如何解决Kafka Connect自动从Debezium主题中提取主键

相关推荐