数据工厂：XML复制到SQL的多个集合参考

如何解决数据工厂：XML复制到SQL的多个集合参考

我正在尝试建立一个数据工厂管道，该管道将XML数据提取到Azure SQL数据库中。 XML遵循以下结构：

<schools>
    <school>
        <students>
            <student></student>
            <student></student>
            <student></student>
        </students>
    </school>
    <school>
        <students>
            <student></student>
            <student></student>
            <student></student>
        </students>
    </school>
<schools>

我已经在SQL中设置了多个表来接受此数据。简而言之，有一个学校桌可容纳所有学校，而学生桌则可容纳所有学生。

我在数据工厂中设置了复制任务，并且必须将“集合引用”设置为<school>，以使其遍历学校。如果我不这样做，它只会加载到第一所学校，而忽略其余的学校。

这在学校加载时效果很好。问题在于下一个复制任务，该任务将查看相同的XML，并尝试将所有学校的所有学生复制到students表中。

如果我将集合引用设置为<school>，它将仅复制每所学校的第一个学生，而忽略其余的学生。如果我将集合引用设置为<student>，它将复制第一所学校的所有学生，但忽略其余学校和学生。

我想遍历所有学校和学生，以便加载所有学校的所有学生，但是我看不到任何简便的方法。有什么方法可以为学校和学生设置多个收藏夹引用？

解决方法

XML来自哪里？我将更多地采用ELT方法，并将XML放在登台表中，然后使用Azure SQL数据库的内置XML功能，例如nodes和value方法。简化示例：

使用Web活动和ADF中存储的Proc活动的类似JSON模式：

用于导入XML的示例SQL：

DROP TABLE IF EXISTS dbo.yourXMLStagingTable
DROP TABLE IF EXISTS dbo.student
DROP TABLE IF EXISTS dbo.school
DROP TABLE IF EXISTS #tmp
GO

CREATE TABLE dbo.yourXMLStagingTable (
    rowId       INT IDENTITY PRIMARY KEY,yourXML     XML NOT NULL,dateAdded   DATETIME NOT NULL DEFAULT GETDATE(),addedBy     VARCHAR(50) NOT NULL DEFAULT SUSER_NAME()
    )
GO


CREATE TABLE dbo.school (
    schoolId    INT IDENTITY PRIMARY KEY,schoolName  VARCHAR(50) UNIQUE NOT NULL,)
GO


CREATE TABLE dbo.student (
    studentId   INT IDENTITY(1000,1) PRIMARY KEY,schoolId    INT FOREIGN KEY REFERENCES dbo.school(schoolId),studentName VARCHAR(50) UNIQUE NOT NULL,)
GO


-- Use Data Factory to insert the data into a staging table
-- This is just to generate sample data
INSERT INTO dbo.yourXMLStagingTable ( yourXML )
SELECT '<schools>
    <school schoolName = "school1">
        <students>
            <student name = "student11"></student>
            <student name = "student12"></student>
            <student name = "student13"></student>
        </students>
    </school>
    <school schoolName = "school2">
        <students>
            <student name = "student21"></student>
            <student name = "student22"></student>
            <student name = "student23"></student>
        </students>
    </school>
</schools>'
GO

-- Look at the dummy data
SELECT * FROM dbo.yourXMLStagingTable
GO

-- Dump into a staging table
-- Get the schools
SELECT 
    s.rowId,schools.c.value('@schoolName','VARCHAR(50)') AS schoolName,students.c.value('@name','VARCHAR(50)') AS studentName
INTO #tmp
FROM dbo.yourXMLStagingTable s
    CROSS APPLY s.yourXML.nodes('schools/school') schools(c)
        CROSS APPLY schools.c.nodes('students/student') students(c)


-- Look at the temp data
SELECT 'temp data' s,* FROM #tmp


-- Insert distinct schools data to schools table
INSERT INTO dbo.school ( schoolName )
SELECT DISTINCT schoolName
FROM #tmp


-- Insert distinct student data to student table,maintaining link to schools table
INSERT INTO dbo.student ( schoolId,studentName )
SELECT DISTINCT s.schoolId,t.studentName
FROM #tmp t
    INNER JOIN dbo.school s ON t.schoolName = s.schoolName
GO


-- End result
SELECT 'end result' s,*
FROM dbo.school school
    INNER JOIN dbo.student student ON school.schoolId = student.schoolId