Mesh Audio Editor

Real-time mesh viewer and editor supporting conversion of meshes to rigid body audio models using Linear Modal Analysis/Synthesis.

Features

General features:

• Create/delete meshes and mesh instances
  • Editable mesh primitives (Rect, Circle, Cube, IcoSphere, UVSphere, Torus, Cylinder, Cone)
  • Load .obj and .ply mesh files (via tinyobjloader and tinyply)
• Select meshes or mesh elements (vertices, edges, or faces), with click or box-select, and optional x-ray mode (to ignore occlusion), in real-time with fast GPU acceleration
• Flat/smooth/wireframe mesh rendering
• Translate / rotate / (nonuniformly) scale meshes and instances with numeric inputs or a custom transform gizmo designed to look and behave just like Blender's
• Edit camera with mouse wheel, numeric inputs, or with a custom orientation gizmo designed to look and behave just like Blender's
• Simple camera + scene lighting model, roughly matching Blender visually
• Edit lighting parameters
• Render face/vertex normals as lines for debugging
• Render bounding box wireframes for debugging
• Edge-detection-based silhouette outline of active mesh/instance, embedded into the scene with accurate per-pixel depth
• Fast infinite grid with horizon fade

Audio-specific features:

• Enable/disable audio output and change device and native format/sample rate
• Volume / Mute
• Generate an efficient physical audio model for any mesh. (See Physical audio modeling.)
  • Click on an audio mesh to excite the nearest vertex, or trigger a selected vertex in the audio menu
  • Edit model DSP params in real-time (via an embedded ImGui Faust parameter editor)
  • View/navigate the generated audio graph (via an embedded Faust SVG diagram navigator)
• Load RealImpact datasets, including the object mesh and instanced cylinders for each microphone position.

Noteworthy dev bits:

• Terse and direct usage of Vulkan-Hpp
• Bindless rendering with descriptor-indexed SSBO vertex pulling (not BDA/buffer-reference vertex pulling yet) from contiguous arenas for mesh data (vertices, indices, attributes, selection state, etc.)
• Multi-Draw Indirect rendering (MDI): Each pipeline issues one vkCmdDrawIndexedIndirect per render pass (when used)
• Uses unified CPU/GPU memory when available. For discrete GPUs, minimizes staging transfers by deferring and merging copy ranges per-frame
• GPU-accelerated mouse interactions (no CPU acceleration structures like BVH)
• Half-edge iterators for mesh topology operations
• Shader hot reloading: Edit shader code and recompile/reload GLSL->SPIRV at runtime in the UI
• C++/GLSL structs generated from YAML to keep types in sync w/o duplication

Physical audio modeling

This project supports generating an efficient physical audio model for any mesh using Linear Modal Analysis/Synthesis

The physical audio modeling components were implemented as a final project for PHYS-6260 - Computational Physics at Georgia Tech during my Master's, and this is the final report. Here is the final report paper.

And here is a 36X48 poster:

Impact audio experiments

Below are audio examples synthesized by "striking" modal audio models (by injecting a short wideband pulse at the selected vertex) for various meshes, with comparisons to impact recordings of their real-world counterparts being struck at the same position. The audio recordings and 3D-scanned meshes come from the RealImpact dataset. See the blog post for embedded audio players.

The cylinders shown in the images represent recorded microphone positions, but all recordings come from a single microphone centered near the impacted object, and the modal audio model does not implement any audio wave radiation modeling. All modal audio samples are generated by extracting estimated surface vibrations, as if recorded from a contact microphone.

Object Name	Mesh	Real Impact Audio	Modal Impact Audio
Ceramic Koi Bowl		Impact	Modal
Ceramic Pitcher		Impact	Modal
Glass Cup		Impact	Modal
Iron Mortar		Impact	Modal
Iron Skillet		Impact	Modal
Plastic Scoop		Impact	Modal
Small Swan Ceramic		Impact	Modal

glTF viewer

MeshEditor is also a fully featured glTF 2.0 viewer/importer. glTF scene nodes are mapped to corresponding MeshEditor objects (meshes, armatures, cameras, lights, empty objects), with the scene parenting hierarchy mirroring the glTF node hierarchy. All Khnonos ratified extensions that visually impact the scene are supported (imported, rendered, fully editable), except for the common realtime approximation to sample the refracted IBL behind transmissive objects instead of scene contents behind them (same as Blender). PBR BRDF/lighting equations are taken directly from the reference glTF-Sample-Renderer shaders.

PBR render features that are not needed by the scene (because the feature isn't enabled on any current object's materials, scene lights are not present or enabled, or IBL environment not present (in Solid render mode)) are not compiled. This is updated dynamically as the scene changes or edits are made. (This is implemented with specialization constants, so only the VkPipeline variants (opaque/blend) need to be recreated, and no GLSL->SPIR-V recompilation is needed for the PBR shader. This is fast enough to run synchronously whenever the enabled feature mask changes - usually around 60ms on my machine.)

glTF supported extensions

✅ supported | 🟨 partial | ⬜ not supported

Extension	Status	Notes
KHR_mesh_quantization	✅	Handled in fastgltf parser/import path
EXT_mesh_gpu_instancing	✅	Imported into MeshEditor instances
KHR_lights_punctual	✅	Imported into MeshEditor light entities
KHR_texture_transform	✅
KHR_texture_basisu	✅	KTX2 transcoded via basis_universal. BC7/ETC2/RGBA32 target selected based on device support
KHR_materials_emissive_strength	✅
KHR_materials_unlit	✅
KHR_materials_specular	✅
KHR_materials_sheen	✅
KHR_materials_ior	✅
KHR_materials_dispersion	✅
KHR_materials_transmission	✅
KHR_materials_diffuse_transmission	✅
KHR_materials_volume	✅
KHR_materials_clearcoat	✅
KHR_materials_anisotropy	✅
KHR_materials_iridescence	✅
EXT_lights_image_based	🟨	Imported as Scene IBL when present (used for Rendered view mode)
KHR_animation_pointer	⬜	Too much complexity for now, for an extension that isn't widely used yet
EXT_meshopt_compression	⬜	Not taking on the meshopt dependency for now

Build & run

Install dependencies

• Download and install the latest SDK from https://vulkan.lunarg.com/sdk/home
• Set the VULKAN_SDK environment variable. For example, add the following to your .zshrc file:

  export VULKAN_SDK="$HOME/VulkanSDK/{version}/macOS"

Mac

$ git clone --recursive git@github.com:khiner/MeshEditor.git
$ brew install cmake pkgconfig llvm sdl3 fftw eigen
$ brew link llvm --force

Linux

(Only tested on Ubuntu, and it's been awhile, so I'd honestly be suprised if it works. If you want a Linux build, I'd be happy to get this working!)

$ sudo apt install llvm sld3 libeigen3-dev
$ export PATH="$(llvm-config --bindir):$PATH"

Install GTK (for native file dialogs):

$ sudo apt install build-essential libgtk-3-dev

Clone, clean, build, and run

$ git clone --recurse-submodules git@github.com:khiner/MeshEditor.git
$ cd MeshEditor
$ ./script/Clean # optionally clean first
$ ./script/Build [--release]
$ cd build && ./MeshEditor

Stack

• Vulkan + ImGui + SDL3: Graphics + immediate-mode UI/UX
• glm: Small numeric vector/matrix types + math
• entt: Entity Component System (ECS) for an efficient and scalable mixin-style architectural pattern
• Faust: Functional audio programming language, used to render modal audio models to audio graphs
• miniaudio: Audio stream I/O
• tetgen: Fast conversion of triangular 3D surface meshes into tetrahedral volume meshes
• Spectra Estimate eigenvalues/vectors for modal analysis
• VulkanMemoryAllocator: Efficient Vulkan memory allocation
• nativefiledialog-extended: Native file dialogs (TODO SDL3 now has SDL_Dialog)
• ImPlot: Plotting
• fftw for computing spectrograms (visualized with ImPlot)
• fastgltf for gltf 2.0 scene loading
• basis_universal for KTX2 texture transcoding (KHR_texture_basisu)
• tinyobjloader and tinyply: for .obj and .ply mesh loading
• lunasvg: Render Faust SVGs to bitmaps, parse SVG link bounding boxes, render SVG icons

Development

Build options

Staged buffer transfers: By default, the app assumes unified memory (direct-mapped GPU memory). For discrete GPUs, enable staged transfers:

$ cmake -B build -DMVK_FORCE_STAGED_TRANSFERS=ON .

Update submodules

All submodules are in the lib directory. Currently, no submodules are forked. Here is my process for updating to the tip of all the submodule branches:

$ git submodule update --remote lib/{submodule}
$ git add .
$ git cm -m "Update {submodule} ..."