从一组混合形状中有效更新ARSCNFaceGeometry

如何解决从一组混合形状中有效更新ARSCNFaceGeometry

我正在使用ARSCNFaceGeometry，并且需要在我的游戏循环中更新面部模型的混合形状。我当前的解决方案是使用新的ARFaceGeometry来调用ARSCNFaceGeometry.update：

class Face {
    let geometry: ARSCNFaceGeometry
    
    init(device: MTLDevice) {
        guard let geometry = ARSCNFaceGeometry(device: device,fillMesh: true) else {
            fatalError("Could not create ARSCNFaceGeometry")
        }
        self.geometry = geometry
    }
    
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation: NSNumber]) {
        let faceGeometry = ARFaceGeometry(blendShapes: blendShapes)!
        geometry.update(from: faceGeometry)
    }
}

但是，这不适合实时使用，因为仅ARFaceGeometry行大约需要0.01秒（仅供参考，在60fps时，我们的总帧预算为0.0166秒）。

几百次更新后，我们似乎遇到了某种错误，使整个游戏循环速度达到了10-20fps。这是来自探查器的6秒示例，其中ARFaceGeometry耗时2.3秒！：

是否有更有效的方法从一组混合形状中更新现有的ARSCNFaceGeometry？

例如，使用自定义3D模型，我可以更新SCNNode.morpher上的混合形状值。 ARSCNFaceGeometry是否有等效项？

解决方法

Apple开发人员文档说：

要更新在AR会话中主动跟踪的面部的SceneKit模型，请在ARSCNViewDelegate对象的renderer(_:didUpdate:for:)回调中调用此方法，并从回调提供的ARFaceAnchor对象中传递geometry属性。

第一个代码版本

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer,nodeFor anchor: ARAnchor) -> SCNNode? {

        let faceGeo = ARSCNFaceGeometry(device: sceneView.device!)
        let node = SCNNode(geometry: faceGeo)
        node.geometry?.firstMaterial?.fillMode = .lines
        return node
    }

    func renderer(_ renderer: SCNSceneRenderer,didUpdate node: SCNNode,for anchor: ARAnchor) {

        if let faceAnchor = anchor as? ARFaceAnchor,let faceGeo = node.geometry as? ARSCNFaceGeometry {

            if faceAnchor.lookAtPoint.x >=  0 {
                faceGeo.firstMaterial?.diffuse.contents = UIColor.red
            } else {
                faceGeo.firstMaterial?.diffuse.contents = UIColor.cyan
            }

            faceGeo.update(from: faceAnchor.geometry)
            self.facialExrpession(anchor: faceAnchor)

            DispatchQueue.main.async {
                self.label.text = self.textBoard
            }
        }
    }
}

...

extension ViewController {

    func facialExrpession(anchor: ARFaceAnchor) {

        self.textBoard = "PLEASE SMILE!"
        let smileLeft = anchor.blendShapes[.mouthSmileLeft]
        let smileRight = anchor.blendShapes[.mouthSmileRight]

        if ((smileLeft?.decimalValue ?? 0.0) + 
            (smileRight?.decimalValue ?? 0.0)) > 0.5 {
            self.textBoard += "You are smiling"
        }
    }
}

第二个代码版本

import ARKit
import Metal

class BlendShapes: NSObject,ARSCNViewDelegate {

    typealias BlendShapeDict = [ARFaceAnchor.BlendShapeLocation: NSNumber]
    var collectBlendShapes: ((BlendShapeDict) -> Void)?

    let geometry = ARSCNFaceGeometry(device: MTLCreateSystemDefaultDevice()!,fillMesh: true)
    
    func renderer(_ renderer: SCNSceneRenderer,for anchor: ARAnchor) {

        guard let facialAnchor = anchor as? ARFaceAnchor else { return }
        let blendShapes: (Any)? = collectBlendShapes?(facialAnchor.blendShapes)
        
        geometry!.update(from: blendShapes as! ARFaceGeometry)
    }
}

或者，您可以通过使用ARFaceGeometry init（blendShapes :)初始化程序创建面部几何对象并将其传递给此方法来创建，配置和可视化与AR会话无关的面部模型。

,

我无法找到ARFaceGeometry性能问题。要解决此问题，我决定从ARFaceGeometry构建自己的模型。

首先，我从ARFaceGeometry生成了一个模型文件。该文件包括基本几何以及应用每个单独的混合形状时的几何：

let LIST_OF_BLEND_SHAPES: [ARFaceAnchor.BlendShapeLocation] = [
    .eyeBlinkLeft,.eyeLookDownLeft,// ... fill in rest
]

func printFaceModelJson() {
    // Get the geometry without any blend shapes applied 
    let base = ARFaceGeometry(blendShapes: [:])!
    
    // First print out a single copy of the indices.
    // These are shared between the models
    let indexList = base.triangleIndices.map({ "\($0)" }).joined(separator: ",")
    print("indexList: [\(indexList)]")
    
    // Then print the starting geometry (i.e. no blend shapes applied)
    printFaceNodeJson(blendShape: nil)
    
    // And print the model with each blend shape applied
    for blend in LIST_OF_BLEND_SHAPES {
        printFaceNodeJson(blendShape: blend)
    }
}

func printFaceNodeJson(
    blendShape: ARFaceAnchor.BlendShapeLocation?
) {
    let geometry = ARFaceGeometry(blendShapes: blendShape != nil ? [blendShape!: 1.0] : [:])!
    
    let verticies = geometry.vertices.flatMap({ v in [v[0],v[1],v[2]] })
    let vertexList = verticies.map({ "\($0)" }).joined(separator: ",")
    print("{ \"blendShape\": \(blendShape != nil ?  "\"" + blendShape!.rawValue + "\"" : "null"),\"verticies\": [\(vertexList)] }")
}

我离线运行了此代码以生成模型文件（将输出手动手动转换为正确的json）。您还可以使用适当的3D模型文件格式，这可能会导致模型文件更小。

然后对于我的应用程序，我从json模型文件中重建模型：

class ARMaskFaceModel {
    
    let node: SCNNode
    
    init() {
        let data = loadMaskJsonDataFromFile() // implement this!
        
        let elements = [SCNGeometryElement(indices: data.indicies,primitiveType: .triangles)]
        
        // Create the base geometry
        let baseGeometryData = data.blends[0]
        let geometry = SCNGeometry(sources: [
            SCNGeometrySource(vertices: baseGeometryData.verticies)
        ],elements: elements)
        
        node = SCNNode(geometry: geometry)
        
        // Then load each of the blend shape geometries into a morpher
        let morpher = SCNMorpher()
        morpher.targets = data.blends.dropFirst().map({ x in
            SCNGeometry(sources: [
                SCNGeometrySource(vertices: x.verticies)
            ],elements: elements)
        })
        node.morpher = morpher
    }

    /// Apply blend shapes to the model
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation : NSNumber]) {
        var i = 0
        for blendShape in LIST_OF_BLEND_SHAPES {
            if i > node.morpher?.targets.count ?? 0 {
                return
            }
            node.morpher?.setWeight(CGFloat(truncating: blendShapes[blendShape] ?? 0.0),forTargetAt: i)
            i += 1
        }
    }
}

并不理想，但是即使没有任何优化，它也可以正常工作并且性能更好。我们现在一直保持60fps。另外，它也适用于较旧的手机！ （尽管printFaceModelJson必须在支持实时面部跟踪的手机上运行）