如何解决我需要使用带Python的AWS GLUE中的ETL从SQL Server数据库中提取信息
我需要使用来自AWS GLUE的ETL和Python使用ETL从SQL Server数据库中提取信息,这是我想要做的
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import SQLContext
from awsglue.dynamicframe import DynamicFrame
import pymssql
args = getResolvedOptions(sys.argv,['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'],args)
dbread = pymssql.connect('Host','user','pass','DB')
cursorread = dbread.cursor()
cursorread.execute("select tra.codigo_tramo,pozI.descripcion + ' - ' + pozF.descripcion,ope.razon_social as nombre_transportador \
from m_tramo tra,m_pozo pozI,m_pozo pozF,m_operador ope \
where tra.estado <> 'I' and tra.codigo_pozo_ini = pozI.codigo_pozo \
and tra.codigo_pozo_fin = pozF.codigo_pozo \
and tra.codigo_trasportador = ope.codigo_operador")
dbread.close()
consulta = spark.createDataFrame(consulta,['codigo_tramo','descripcion_tramo','codigo_transportador'])
consulta.show()
如何查询该信息?
解决方法
我已经解决了
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext,SparkConf
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import SQLContext
from awsglue.dynamicframe import DynamicFrame
args = getResolvedOptions(sys.argv,['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'],args)
# Construct JDBC connection options
connection_sqlserver_options = {
"url": "","dbtable": "","user": "","password": ""}
# Read from JDBC databases with custom driver
DyF = glueContext.create_dynamic_frame.from_options(connection_type="sqlserver",connection_options=connection_sqlserver_options)
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。