layerMask in android

[mscoder1987]

I'm going crazy trying to control renderable visibility using layerMask on Android's Filament 1.40. The actual runtime behavior is always unexpected and I totally don't understand how this API works. I have two questions:

When calling RenderableManager.Builder to create renderables, who does the layerMask method actually affect - the whole rendering environment or just the created renderable objects?
I created two renderables: renderable1 and renderable2, without setting layerMask initially. In a tap event, I made the following modifications:

engine.renderableManager.setLayerMask(renderable1, 0xff, 0x00)
view.setVisibleLayers(0xff,0XFF)

Based on my understanding, renderable1 should be hidden and only renderable2 displayed. But it seems there was no change. And if I call engine.renderableManager.setLayerMask(renderable1, 0xff, 0x00) right after creating the renderables, and only call view.setVisibleLayers(0xff,0XFF) in the tap event, the situation is even worse - exiting and re-entering the rendering activity causes the app to freeze. The documentation around layerMask is very confusing to me.

[romainguy]

RenderableManager::Builder::layerMask only affects the renderable being currently created.

Are you calling setLayerMask() with an entity instead of an instance maybe? The proper way to call those APIs (like other RenderableManager APIs) is:

renderableManager.setLayerMask(renderableManager.getInstance(myRenderableEntity), 0xff, 0x00)

view.setVisibleLayers(0xff, 0xff) makes all the layers visible, which you don't need to do in your example since you are removing renderable from layer 0 (the default visible layer).

Here's an example from our own sample code:

// Both these renderables are in layer 1; these calls clear layer 0 and set layer 1
rcm.setLayerMask(rcm.getInstance(lightmapCube->getSolidRenderable()), 0x3, 0x2);
rcm.setLayerMask(rcm.getInstance(lightmapCube->getWireFrameRenderable()), 0x3, 0x2);

// mDepthView will only show layer 2 (bitmask: 0100); the two renderables above are not visible
window->mDepthView->getView()->setVisibleLayers(0x4, 0x4);
// mGodView and mOrthoView show layers 1 and 2 (bitmask: 110); the two renderables above are visible
window->mGodView->getView()->setVisibleLayers(0x6, 0x6);
window->mOrthoView->getView()->setVisibleLayers(0x6, 0x6);

[romainguy]

This matches the C++ implementation in that entities and component instances are just ints. Until Project Valhalla is a reality, a version of those APIs that use wrapper types will cause unnecessary allocations and extra pointer dereferences that we don't want to pay for. Note that's why in the source all entity ints are annotated with @entity.

[mscoder1987]

I understand that it is a trade-off. When a framework is spreading in a small scope, performance and efficiency are the top priorities. However, when it is widely adopted or expected to be widely adopted, considerations of usability should rise, in other words, it should be compatible with less experienced programmers, lowering the learning curve. In the development field, the competition of languages and syntax is already minimal, and the competition lies in APIs. If a framework aims to go further, it needs to attract more developers. Based on my observations of colleagues, most people may not have highly advanced technical skills, and due to deadline pressures, their performance requirements may not be as stringent. Simplicity and ease of use to accomplish the tasks at hand are the most important aspects. This demands that the API should be easy to get started with. For a great framework, survival is the first step.

[romainguy]

I completely agree with you overall but in this particular case it is a conscious choice. Doing otherwise would nullify many other things Filament does internally for efficiency. Once a better technical solution becomes available in Java/Kotlin we will absolutely make use of it. In the meantime we can improve the documentation. Hopefully we can customize the javadoc output to automatically take into account the annotations used internally to distinguish entities and component instances.

[mscoder1987]

Thank you for your team's efforts. I will do my best to introduce Filament to people around me. It has the potential to become a great framework, and I really like it. @_@

[romainguy]

Thanks! For more context we've always thought of Filament as a rendering library and not a higher level framework (that's why for instance no file formats are supported in Filament itself and why glTF is but as a separate library). However what you are describing is perfectly suited for something on top of Filament. SceneView is a great example of what we envisioned sitting on top of Filament.

Something in my to-do list btw is to build a useful website for Filament with higher level documentation (explaining entities and instances for instance). Just need to take/find the time 😅