Kiba 介绍
Kiba 是一个轻量级的 Ruby 的 ETL 框架。
作业定义 xxx.etl:
# declare a ruby method here, for quick reusable logic def parse_french_date(date) Date.strptime(date, '%d/%m/%Y') end # or better, include a ruby file which loads reusable assets # eg: commonly used sources / destinations / transforms, under unit-test require_relative 'common' # declare a pre-processor: a block called before the first row is read pre_process do # do something end # declare a source where to take data from (you implement it - see notes below) source MyCsvSource, 'input.csv' # declare a row transform to process a given field transform do |row| row[:birth_date] = parse_french_date(row[:birth_date]) # return to keep in the pipeline row end # declare another row transform, dismissing rows conditionally by returning nil transform do |row| row[:birth_date].year < 2000 ? row : nil end # declare a row transform as a class, which can be tested properly transform ComplianceCheckTransform, eula: 2015 # before declaring a definition, maybe you'll want to retrieve credentials config = YAML.load(IO.read('config.yml')) # declare a destination - like source, you implement it (see below) destination MyDatabaseDestination, config['my_database'] # declare a post-processor: a block called after all rows are successfully processed post_process do # do something end
执行作业:bundle exec kiba my-data-processing-script.etl
Kiba 官网
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。