Was working with the Project64 people on this issue, turns out it's a problem with this plugin, as I tried out Hey You, Pikachu! with NRage instead, and the lag I was having went away.
project64/project64#1337
You can see the lag in this video.
The lag only occurs when the VRU is plugged in, if I use your plugin but only have 1 controller plugged in, there is no lag.