Copy/Re-use animation from one instance on another one with the same skeleton

[hannojg]

Is your feature request related to a problem? Please describe.

In my use-case I have a main character model. This model contains all animations for the character.
I now have many other models that are clothes or accessories that have the same skeleton as the character, but don't contain any animations.
When the character equips for example a hat and we run the idle animation of the character, we copy over the idle animation to the hat model. Playing both simultaneously gives the impression of the hat fitting on the character's hat, as it's following all of its animations.

The current app code is written in threejs but we want to migrate to filament, and we're wondering if we can use the same approach with filament.
Here is an example code snippet of how to do this in threejs:

// this._character being the main model with all animations and
// this._accessoryObject being the accessory model we want to copy the animations to
this._character.copyAnimationsTo(
	this._accessoryObject.animations,
	this._accessoryObject.mixer,
);

copyAnimationsTo(animationsMap: { [key: string]: THREE.AnimationAction }, mixer: THREE.AnimationMixer) {
	const animations = this._character.animations
	animations.forEach(animation => {
		const animationAction = mixer.clipAction(animation)
		animationsMap[animation.name] = animationAction
	})
}

Describe the solution you'd like

I haven't thought of any exact solution yet, as I feel it might introduce a new API (if the feature doesn't already exist?) and I don't have any specific API in mind.

Maybe something like:

animator->addInstance(otherAsset->getInstance())

Describe alternatives you've considered

I see that the Animator has (internal) code i.e. addInstance, however thats internal and only usable with an FFilamentInstance.

OS and backend

I think it's not.

[hannojg]

I got it working by making a new public constructor for the Animator:

// Animator.h
public:
Animator(FilamentAsset *asset, FilamentInstance *instance) : Animator(downcast(asset), downcast(instance))
{
}

and then instantiating an animator on my own like this:

Animator* animator = new gltfio::Animator(mainCharacterAsset, hatAsset->getInstance());

And then using this animator instead of hatAsset->getInstance()->getAnimator() to drive the animation.

I am sure the constructor was private for a reason and shouldn't be exposed 😅 But it proves it's possible with filament, which is awesome! Are you open to discuss an API to enable this feature?

[hannojg]

As a side note: I tried to use animator->addInstance with a FilamentInstance (I changed the code for that).
After doing so I was under the impression that the instance would be then controlled by the same animator. However the animator only animated the main instance (and not the one I added with addInstance).

In our use case it would be actually beneficial to use the same animator for all the (different) instances, to save resources. However, I am not sure if thats feasible?

[mrousavy]

I feel like that use-case should be handled fundamentally different though, no? As in; use one model for that specific animation set, then work with that model, e.g. by attaching skins to it (like a hat on the penguin), instead of creating a new model with the hat and copying over the animations.
But I might be wrong in how filament works.

[hannojg]

Note: The copied animator wasn't supporting applyCrossFade correctly, I addressed that in this PR:

• fix: apply applyCrossFade to correct entity #7665

[ninjz]

I haven't yet tried @hannojg's solution but if it works to be able to apply the animation from one model to another with the same bone structure it would be very beneficial to add the public constructor.

My use case I've talked about in this thread: SceneView/sceneview-android#326 I've been pulling my hair out for a while now over this issue. I've been using SceneView in my project which depends on Filament and though I will probably make these changes in my own fork it would be nice to have these changes applied upstream to the main projects.

Is there any particular reason the public constructor shouldn't be added?

[hannojg]

I think I found a better solution to this than copying the whole animator.

The problem with copying the animator is that all animation data will be duplicated. In our case we had a lot and complex animations. This easily starts taking a lot of memory, so that approach is kind of suboptimal.

Our use case is character clothing. The clothings share the same skeleton as the main character.
So a better solution is to apply all transformation of the bones of the character to the bones of the clothing.

In practice I implemented this in three steps:

1. Loop over all entities, their names and their transforms from the main character
2. Loop over all entities of the clothing, and apply the same transformation to entities matching the same name
3. Lastly, update the bone matrices of the clothing using the animator instance/state from the main character

This way we are not duplicating any animation data.

To get point 3) to work I needed to add a simple method to the Animator class. Will open a PR for that soon, really hope we can support this use case 🙏

// Edit: with this PR we are able to get the use case of clothing working:

• feat: updateBoneMatricesForInstance #7859

[ninjz]

@hannojg Thanks for your work on this. This is a really important functionality for my use case.

I'm trying your solution on my Android application but I'm having troubles playing the animation from the animation file to the model.

This is what I'm doing to apply the updateBoneMatricesForInstance call:

inner class FrameCallback : Choreographer.FrameCallback {
    private val startTime = System.nanoTime()
    override fun doFrame(frameTimeNanos: Long) {
        choreographer.postFrameCallback(this)
        modelViewer.render(frameTimeNanos)

        animationAnimator?.apply {
            if (animationCount > 0) {
                val elapsedTimeSeconds = (frameTimeNanos - startTime).toDouble() / 1_000_000_000
                applyAnimation(0, elapsedTimeSeconds.toFloat())
            }
            updateBoneMatricesForInstance(modelViewer.asset!!.instance)
        }
    }
}

And this is what I have for loading the model and the animation:

private fun loadAnimation() {
   val engine = Engine.create()
   val materialProvider = UbershaderProvider(engine)
   val assetLoader = AssetLoader(engine, materialProvider, EntityManager.get())
   val resourceLoader = ResourceLoader(engine, true)

   val buffer = assets.open("animations/jogging.glb").use { input ->
       val bytes = ByteArray(input.available())
       input.read(bytes)
       ByteBuffer.allocateDirect(bytes.size).apply {
           order(ByteOrder.nativeOrder())
           put(bytes)
           rewind()
       }
   }

   val asset = assetLoader.createAsset(buffer);
   asset?.let { asset ->
       resourceLoader.asyncBeginLoad(asset)
       animationAnimator = asset.instance.animator
       asset.releaseSourceData()
   }
}

private fun loadModel() {
   val buffer = assets.open("models/female_masterScene.glb").use { input ->
       val bytes = ByteArray(input.available())
       input.read(bytes)
       ByteBuffer.allocateDirect(bytes.size).apply {
           order(ByteOrder.nativeOrder())
           put(bytes)
           rewind()
       }
   }

   modelViewer.loadModelGlb(buffer)
   modelViewer.transformToUnitCube()
}

What am I missing here?

[hannojg]

Yeah so for the solution you have to sync all entities, as I outlined here:

1. Loop over all entities, their names and their transforms from the main character
2. Loop over all entities of the clothing, and apply the same transformation to entities matching the same name
3. Lastly, update the bone matrices of the clothing using the animator instance/state from the main charac

The PR I put up only concerns the third (3) point. An implementation of the first two could look something like this (I think sharing this is interesting for the full perspective on this issue and maybe the filament maintainers can better understand the use case [because maybe there is a better solution!]):

(Pseudo code):

void AnimatorWrapper::applyAnimationToInstance(FilamentInstance* instanceToSync) {
  // _instance is the FilamentInstance to the master asset containing the animation we want to sync with
  FilamentInstance* masterInstance = _instance;

  const FilamentAsset* asset = instanceToSync->getAsset();
  Engine* engine = asset->getEngine();
  TransformManager& transformManager = engine->getTransformManager();
  Animator* masterAnimator = masterInstance->getAnimator();

  // 👀 STEP 1
  // Get name map for master instanceToSync
  size_t masterEntitiesCount = masterInstance->getEntityCount();
  const Entity* masterEntities = masterInstance->getEntities();
  std::map<std::string, Entity> masterEntityMap;
  for (size_t entityIndex = 0; entityIndex < masterEntitiesCount; entityIndex++) {
    const Entity masterEntity = masterEntities[entityIndex];
    NameComponentManager::Instance masterNameInstance = GlobalNameComponentManager::getInstance()->getInstance(masterEntity);
    if (!masterNameInstance.isValid()) {
      continue;
    }
    auto masterInstanceName = GlobalNameComponentManager::getInstance()->getName(masterNameInstance);
    masterEntityMap[masterInstanceName] = masterEntity;
  }

  // Get name map for instanceToSync
  size_t instanceEntitiesCount = instanceToSync->getEntityCount();
  const Entity* instanceEntities = instanceToSync->getEntities();
  std::map<std::string, Entity> instanceEntityMap;
  for (size_t entityIndex = 0; entityIndex < instanceEntitiesCount; entityIndex++) {
    const Entity instanceEntity = instanceEntities[entityIndex];
    NameComponentManager::Instance instanceNameInstance = GlobalNameComponentManager::getInstance()->getInstance(instanceEntity);
    if (!instanceNameInstance.isValid()) {
      continue;
    }
    auto instanceName = GlobalNameComponentManager::getInstance()->getName(instanceNameInstance);
    instanceEntityMap[instanceName] = instanceEntity;
  }

  // 👀 STEP 2
  // Sync the same named entities:
  for (auto const& [name, masterEntity] : masterEntityMap) {
    auto instanceEntity = instanceEntityMap[name];
    if (instanceEntity.isNull()) {
      continue;
    }

    // Sync the transform
    TransformManager::Instance masterTransformInstance = transformManager.getInstance(masterEntity);
    TransformManager::Instance instanceTransformInstance = transformManager.getInstance(instanceEntity);

    if (!masterTransformInstance.isValid() || !instanceTransformInstance.isValid()) {
      Logger::log(TAG, "Transform instanceToSync is for entity named %s is invalid", name.c_str());
      continue;
    }

    math::mat4f masterTransform = transformManager.getTransform(masterTransformInstance);
    transformManager.setTransform(instanceTransformInstance, masterTransform);
  }

  // 👀 STEP 3
  // Syncing the bones / joints
  masterAnimator->updateBoneMatricesForInstance(instanceToSync);
}