Multirotor toilet bowling detection

[breadoven]

Adds toilet bowling detection for multirotors. If toilet bowling is detected a heading correction is applied which should stop the toilet bowling occurring. A system message is also shown on the OSD.

Detection is based on inconsistency between the required heading to the hold point and the GPS course over ground and also a speed/distance to hold point factor. These are only active when the quad is > 1m away from the hold point and has a speed of at least 1m/s (GPS ground course is unreliable below this speed).

The speed/distance to hold point factor, speed * distance, seems to provide a good indication of the main characteristic of toilet bowling which is rapidly increasing speed and distance from the hold point. From testing this factor is in the low hundreds for normal position hold. When toilet bowling occurs it rapidly rises to 10's of thousands.

Detection can be tuned for sensitivity by a setting that effectively sets the threshold value of the speed/distance to hold point factor. When set to 0 (default) toilet bowling detection is disabled .

This has been tested during position hold on a 5" multirotor and appears to work well. The testing initiated toilet bowling by adding an offset to posControl.actualState.yaw. 45 degs didn't seem to do much, no toilet bowling, but 90 degs immediately causes problems. When toilet bowling was detected and the heading correction applied the toilet bowling stopped immediately. The heading correction was typically within 20 degs of the actual true heading which seems to work OK given even a 45 deg heading error doesn't seem to cause toilet bowling.

Currently only works when holding position. This doesn't include when adjusting position although it appears this should work, needs more testing. Also could be extended to work with RTH etc when on route to home/WP. There is also the possibility to use this detection to flag the magnetometer as unreliable in the IMU preventing its use when Course Over Ground is reliable at higher ground speeds.

[Jetrell]

I like the concept.

Also could be extended to work with RTH etc when on route to home/WP

This would be very useful.
I've seen many logs that show magnetic deviation when a copter tilts at higher bank angles in navigation flight, as a result of the motors magnetic fields intercepting the magnetometer. to a greater degree.

May I ask.. Did you write this as a result of your copters experiencing toilet bowling over the last year ?

[breadoven]

I like the concept.

Also could be extended to work with RTH etc when on route to home/WP

This would be very useful. I've seen many logs that show magnetic deviation when a copter tilts at higher bank angles in navigation flight, as a result of the motors magnetic fields intercepting the magnetometer. to a greater degree.

May I ask.. Did you write this as a result of your copters experiencing toilet bowling over the last year ?

I've had a couple of crashes caused by toilet bowling. The first was close to trees, it hit them before you had a chance to do anything. The second was recently and shouldn't have been a problem given it was high and away from things to hit except it also had a problem with an incorrect Z velocity (caused by the toilet bowling ?) which resulted in it disarming from quite a height. It's not always immediately obvious it's toilet bowling looking through the FPV View and it had held position happily for several minutes beforehand so I assumed the compass was OK (the toilet bowling starting after yawing 180 degs so inconsistent heading all the way around). I ended up hitting RTH and it managed to make it home albeit in a massive sweeping arc then drifted off as it turned to the home heading at which point it actually recovered to a stable landing hover (the heading was actually vaguely correct in that direction). It was trying to land but the Z velocity estimation was screwed up which caused it to climb rapidly when it thought it was descending. This confused the landing detector leading to the disarm. In hindsight I should have just switch out of Nav modes and flown back manually which I tried literally at the point it disarmed. At least with this PR it probably would have stopped the instability and thrown up a message telling you it's got a problem with the compass. Would have just made thing a lot more immediately obvious. Amazingly all the quad suffered was a bent (S shaped) frame spacer post which I think was the only bit that hit a tree branch.

[Jetrell]

The reason I asked. Is because two of my test copters have expereinecd toilet bowling as well over the latter half of the 8.0 master, I tried re-calibrating the compass etc. With varied success. But haven't really pinned down why its happening.. Being that both these copters are well established in the hardware layout, over many releases. With both only having small crashes. Not enough to damage much at all.

[breadoven]

The reason I asked. Is because two of my test copters have expereinecd toilet bowling as well over the latter half of the 8.0 master, I tried re-calibrating the compass etc. With varied success. But haven't really pinned down why its happening.. Being that both these copters are well established in the hardware layout, over many releases. With both only having small crashes. Not enough to damage much at all.

I don't know much about this. But I had wondered ?

The problem in my case seemed to be caused by an old BN-880 compass and possibly a new GPS mount which was a cm lower and closer to the frame than the old mount. The heading was out by around 70 degs going from NW to S when the toilet bowling occurred. Recalibrating afterwards and fiddling with compass alignment settings to account for the fact the new mount was a few degs different to the old one didn't seem to help, the heading was always 30-40 degs off at some point in the circle. A new Walksnail WS-M181compass worked straight away, consistent compass and quicker GPS lock + smaller and lighter.

[sensei-hacker]

The reason I asked. Is because two of my test copters have expereinecd toilet bowling as well over the latter half of the 8.0 master, I tried re-calibrating the compass etc. With varied success. But haven't really pinned down why its happening.. Being that both these copters are well established in the hardware layout, over many releases. With both only having small crashes. Not enough to damage much at all.

Are you willing and able to test this?

Compiled targets can be found here, or if it's more convenient I can compile for whichever board so you don't have to hunt for them.
https://github.com/iNavFlight/inav/actions/runs/14904692673/job/41864265308?pr=10854

[breadoven]

I've been testing this using BEEPER mode with the following bit of extra test code in navigation.c

inav/src/main/navigation/navigation.c

Line 2896 in 9591dee

void updateActualHeading(bool headingValid, int32_t newHeading, int32_t newGroundCourse)

:

void updateActualHeading(bool headingValid, int32_t newHeading, int32_t newGroundCourse)
{
    /* Update heading. Check if we're acquiring a valid heading for the
     * first time and update home heading accordingly.
     */
    // TEST
    if (STATE(MULTIROTOR) && IS_RC_MODE_ACTIVE(BOXBEEPERON)) {
        newHeading = wrap_36000(newHeading + 7000);   // 7000 = 70 deg heading offset, change as required
    } else {
        mcToiletBowlingHeadingCorrection = 0;
    }
    // TEST
    navigationEstimateStatus_e newEstHeading = headingValid ? EST_TRUSTED : EST_NONE;

I've noticed you need some wind to induce toilet bowling. Even with a 70 deg heading error it wouldn't toilet bowl the other day when conditions were calm but it started pretty quickly when there was some wind on another day. Toilet bowling is more reliable with a 90 deg heading error. Which is interesting because I've wondered how accurate the compass really needs to be to avoid problems in Poshold. Seems not that accurate. I struggled to get any instability at 45 degs even with wind.

[Jetrell]

Which is interesting because I've wondered how accurate the compass really needs to be to avoid problems in Poshold. Seems not that accurate.

I figure with MAX_MAG_NEARNESS allowing for 25% magnetic field error. That maybe be why.

Which raises a question I've seen a lot of in the later part of 8.0.. How much is the magnetometer actually doing in 8.0. Because I've noticed some unusual bearing error.
I'm finding when looking over logs, that magless operation doesn't appear to work much worse. And sometimes better.

[Jetrell]

Testing with this firmware.
I lowered the GNSS module in this copter to help simulate magnetic deviation between the motors. Which it did very slightly at nav bank angles greater than 29°... So I set it to 30° for this test.
nav_mc_toiletbowl_detection was set to 3 for this 3".

In the first part of this log (not pictured here to save video file size). I was travel straight in Poshold (Cruise) and released the stick to center. And the copter pulled-up straight and in-line with the course it was traveling on.. I did this again, and the result was the same, with no deviation.
Then I turned the copter around and traveled in the opposite direction. Which incurred a sudden hook to the left by navigation. Seen by the rcCommand[roll] and not rcData[0].
As can be seen, this lead to the toilet bowl effect, until it slowly leveled out after a few swirls.

It is interesting that this also happened after a 180° turn around, as you experienced.. Which leaves the question. What is actually causing it. And why did the copter just hook to the left ? Its not the only copter I've seen this happen with over the last 8 months or so. I doubt there was enough mag deviation to cause that to happen, or it would have happen in the two stops, just before.

Once the copter had settled down to a fixed hover position, while toilet bowling. I assume systems message warning should have appeared ?

Deviation.mp4

If you would like to look at the whole log. Just ask, and I'll DM you it on Discord.

[breadoven]

Not sure it detected that toilet bowling @Jetrell given there was no system message. It also tends to very obviously snap back under control when detection happens. Looks more like the toilet bowling simply faded away by itself.

[Jetrell]

It appears to work okay in Poshold.
However I've seen toilet bowling also occur in a WP mission. Especially when I plotted it to perform a 180° heading change, then a stop.

So it would appear this feature helps to stop toilet bowling once it occurs; in a cases when the user doesn't have their copter setup correctly.. This makes it very helpful addition, with all else being good.
But as we have seen in the 8.0 release. There is still something fundamentally wrong that is causing this deviation to occur in the first place, on otherwise well setup and established copters..

After flashing back to 7.0 (arbitrarily) with these same copters. The toilet bowl effect doesn't occur after a 180° heading reversal.

I had a skim through 8.0 changes that might have added to this. I'd considered the new auto declination tables being incorrect at my location. But after running a re-calibration and using manual mag declination for my location. The issue still persists.
The only other changes are related to GNSS stuff. But I'm not sure specifically on this one. But I can say. There is some strange stuff that randomly occurs with GNSS / FC communications. More so noticeable on M8 and cheaper M10 modules.

[breadoven]

Did you test @Jetrell by inducing a heading error or did this toilet bowling just occur anyway ?

As it is this PR will only work if a hold position is active which for WP missions is Time hold or end of mission hold.

I take it toilet bowling doesn't occur if you do a 180 deg reversal in Poshold mode ?

[Jetrell]

The tests in Poshold where done while the copter was in motion with the pitch stick held forward to make it travel.
It is under this condition while travel e.g. north, I can release stick to bring it to a stop, with no deviation. Then immediately yaw 180° while stationary. And push the stick forward again to travel e.g. south.. This will often induce heading deviation while it's traveling. It occurs in the direction the 180° yaw turn was command just previous. e.g left rotation/ left deviation.

The other times I notice this other more concerning issue occur. Is when the copter is performing an RTH.
And when it reaches with-in 1m of the home location, just prior to landing descent. It will yaw 180°, depending on what direction it was traveling back to home from.
When this occurs, it will descend with a toilet bowling effect... This feature doesn't correct for that, while its descending. And I've only seen this effect since 7.1 to 8.1.

It was using a 3 sec PH_time in the test mission.

However when there is noticeable deviation between the headings seen in the log. This feature does catch it, and snaps it back straight again.
No offense meant by this statement, it feels like its slamming the gate shut on the horse as its bolting.

[breadoven]

Are you sure that's not just a dodgy compass @Jetrell ? The issue I had was the compass was just unreliable. Would work fine most of the time then become sluggish and inaccurate for no obvious reason. Almost seemed location related as if there was some sort of interference in certain locations. Although in the end it wouldn't calibrate no matter what you did.

As for the horses ... this is really only meant to stabilise the quad and prevent it going nuts before you have time to react. It can't really do anything else because the heading correction this applies may in fact only work for the current heading. If you try and turn to some other heading the correction could be wrong because the compass error will have changed. However, switching out of the Nav mode to Acro resets the correction so you could turn to home for instance (if you knew where it was from the FPV view) then try and use Nav modes again in which case any heading correction applied should be good to get home.

[Jetrell]

I wish it was a dodgy compass. It would be easier to replace them LOL. That would save me a lot of time and frustration. But after flashing back to 7.0. Its all good again.

Adding to your other question.
This feature will catch the toilet bowling sometimes. They are the times when a heading deviation between the two can be seen in the logs, and on the OSD.
But the time this feature doesn't catch it. Are the times I can't see any difference in the headings.
Its like it's caused by the heading update correction in navigation code.

We know that during magless copter navigation usage. It updates the IMU heading from GNSS course.
I often see toilet bowling with it too, when the copter banks to slow, and yaws, before RTH landing descent.
And once it gets into the toilet bowl effect, the only way to stop it from occurring is to manually yaw the copter more than 90° from its current heading while it's descending.
I'm wondering if this is because the GNSS heading is all over the place while toilet bowling is occurring. So the IMU heading update becomes untrusted. And those same calculations are effecting magnetometer heading updates to the IMU as well ?

[shota3527]

I am considering using acceleration data from GPS to estimate yaw. If this works well, it may get correct yaw estimation even on toilet bowling.
acceleration data from the GPS had some success on the centrifugal_force_compensation on fw ahrs, Then multirotor`s trust vector is same with acceleration from GPS assuming no lag in gps

[shota3527]

#10854 (comment)
This looks normal to me, except the home arrow is off(yaw is used to get the arrow), I guess the home point is somewhere near the white Tent/building. may be 40-60deg off.

The yaw and the course over ground is different; yaw is where the plane is facing, and CoG　is where it is flying.
If a quad is flying backward, then there will be 180deg Deviation

[Jetrell]

This looks normal to me, except the home arrow is off

In what way do you mean normal ?
The home arrow was correct. Arming occurred near the flight line fence.

Quote for that post :

In the first part of this log (not shown here to save video file size). I was travelling straight in Poshold (Cruise) and released the stick to center. And the copter pulled-up straight and in-line with the course it was traveling on.. I did this again, and the result was the same, with no deviation.

Then I turned the copter around 180degs on its yaw axis and traveled in the opposite direction, then relased the stick. This is where that video starts. Which incurred a sudden hook to the left by navigation. Seen by the rcCommand[roll] and not rcData[0]. it was not a stick response from me, but from the nav controller. This lead to the toilet bowl effect.

And from tests with other copters, it's the 180 deg yaw turn on its axis, that causes this deviation once the copter is commanded to slow or stop.
This can also be seen when climb_first RTH is command by the altitude controller. And the copter climbs and turn 180degs to head back to the home location.. This turning causes the copter to travel in a left swing arc back to home. Then it leads to toilet bowling when the copter is descending to land.
This is more often seen with MC compass-less navigation. But the effect is also seen to a lesser degree with a compass.

[Jetrell]

@shota3527 Here is a video from UAVTech.
It's interesting that he also says the same thing about the quad navigate to the left, even with stick input. While I'd noticed it far more often when the stick is released in Poshold. Or in the case I mentioned in the previous post.
Just to confirm. This is not wind related, which I thought it may have been in the early days after you wrote the changes for compass-less navigation.

[shota3527]

The only problem is that there is a heading/yaw estimation error(difference on estimated yaw vs real) on the multi-rotor, causing the toilet bowl or navigating to the left/right problem when navigation mode is engaged.

In the UAVtech video at 06:03, pay attention to yaw and cog. The quad is flying to keep on an estimated 198 ° course, but it actually flies on a 205 ° course in reality(COG). The estimated yaw is being corrected to 205 °. But the navigation command is to maintain a 198 ° course, so turn left to keep the 198 ° course.

And in Deviation.mp4 vedio, the estimation is already disoriented by mag sensor

The AHRs aim to obtain the best Roll/Pitch/Yaw Estimate using available sensors, and navigation relies heavily on this. However, this does not work well for my yaw estimate. Only a perfect mag or flying absolutely a straight line (facing a different direction is okay) in no wind can get perfect estimation.
And gyros such as the BMI270 or noisy builds have problems; A 180 ° turn cloud results in a 10 ° error in 0.5 seconds. Or the estimation is not good in the first place. Side 5km/h in forward 36km/h movement can result in 8deg estimation error. Or even worse on the climbing phase with 0km/h forward but more wind.

I don't think with evidence. chance of positive estimation error(real is greater than estimated) is statistically significant, it can also cause by other reasons, like most of us mount the mag sensor in same orientation,

[Jetrell]

@shota3527 I get what you mean. Please don't take anything I say as criticism. I'm just trying to put forward the culmination of many tests with many model.

With the position estimator being the sum of GNSS, IMU and Mag data. Its hard not to get incorrect data from loss of satellite precision and mag deviation during banking, turning or slowdowns.. And its also impossible for the IMU not to experience higher levels of vibrations at certain points in the throttle range (unless INAV had BDshot) even on quads and fixedwing that have their props and motors balanced together. Which all my test fleet and cruisers do.

Fly long legs in a mission or RTH generally doesn't see issues.. But the worst cases are in modes like Poshold or WP missions, when the copter is traveling along, then can stop, turn or slow at a fixed point.

I don't think with evidence. chance of positive estimation error(real is greater than estimated

From flashing back to 7.0 and before with multiple copters, using the same hardware. I had noticed the estimator does seem more tolerant over a wide range of tests. Including flying in different temperatures and times of the day to provide the availability of more satellites.
I honestly can't put my finger on the direct cause. But after hundreds of INAV tests per year on multiple copters. I can say something is worse. .Especially on models that don't have higher performance GNSS modules... Its honestly driving me mad, to the point I've taken a break from testing over the last month.

[shota3527]

@Jetrell
I think we need to relax first and break down the problem into multiple hypotheses to solve the issue. For each hypothesis, we can run a targeted experiment to verify it. Testing with all the stuff mixed is impossible to identify the issue.
Assuming MC on 7.0 is the best/better

• Yaw, Attitude(RPY) estimation problem from 7.1, changes are MAX_MAG_NEARNESS and GPS yaw eatimation for multirotor, should be tested without navigation mode
  • for MAX_MAG_NEARNESS, check sensor reading on configurator sensors tab, with battery mounted/removed, armed/not armed. or run with increased MAX_MAG_NEARNESS value to restore to 7.0 behavior
  • for GPS yaw estimation, turn it off for debugging by gps_yaw_weight=0
• position(XYZ) estimation problem from 7.1, perferabley be tested without navigation mode
  • position estimator pos.z, vel.z uses both baro and GPS for more precise estimation, can turn it off for debugging
  • Raise the default position estimator GPS altitude/vertical velocity gain, can revert the gain for debugging
  • Dynamically adjust gps/baro gains by measured vibration in the position estimator, maybe look for virbration value in blackbox log
• GPS problem?
  • changed the gps mode default to air<2G from air<1G, can revert back to 1G for debugging
  • other changes i am not familiar of
• Changes in navigation modes that I am not familiar with
• Other unknown bugs
  • i just found a realflight sitl mag reading have problems
  • ...

[Jetrell]

Dynamically adjust gps/baro gains by measured vibration in the position estimator, maybe look for vibration value in blackbox log

I often check for vibrations in the log data. They are as clean as I can get them, both with hardware and filtering. The only part of the throttle range there is some noise, is between around 35 to 42% due to ESC pwm drive. But this is very minimal. The filters squash that with ease.

changed the gps mode default to air<2G from air<1G, can revert back to 1G for debugging
other changes i am not familiar of

I had tried that early in 2024. And have switch back and forth between those in the dynamic model. And I have stayed with air<1G. The two are practically identical. With no user quantifiable benefits that can be noticed, either good or bad.

I have also more recently drop the GPS serial baud rate down to 57600 from the higher default. Due to the way GNSS comms is a bit touchy, and looses hot fix in a moment. This certainly doesn't help on lower quality modules, when the almanac has to be acquired again.
I would have turned off auto baud to speed up the establishment of the communications. But that feature appears to not work anymore.

Other unknown bugs. i just found a realflight sitl mag reading have problems

That is interesting to hear!

[shota3527]

@Jetrell
the issue bother you is position estimation problem rather than yaw estimation problem?

[Jetrell]

the issue bother you is position estimation problem rather than yaw estimation problem?

I would say yes. It effects all axis's.
When it occurs. It adds to altitude drift at times,. It also causes yaw drift until a strong ground course is obtained . As well as randomly drifting off a few meters on either x or y in Poshold. This XYZ occurrence can be with or without a compass. And I'm not the only person who has reported the same things over the last year or so.

I understand the limitations of what the available sensors can accomplish under challenging conditions. And if I didn't have past software references, when using the same hardware I did back then, as well as today... Then I'd most likely not know any different. And just ignore it. Not that I feel its acceptable navigation performance. Its just that I wouldn't be faced with it every time I go out flying. When I'm trying to assess if other new changes are making any difference.. If you get my meaning.

@breadoven may have something extra to add.

[sensei-hacker]

I think we have two different things being discussed here.

1. The estimation seems to have been better in 7.x than in 8.0.1. Possibly related to about 20 changes done to the GPS / Ublox code, possibly not. That's one issue.

2. A different question is should we merge this PR to detect and deal with toilet bowling?

Can I get a show of hands on question number 2? Should we merge this PR, is it ready?

[shota3527]

A different question is should we merge this PR to detect and deal with toilet bowling?
I object to this PR because the main point is that it includes code to overwrite yaw estimation. Detection is fine, but overwrite yaw estimation is a nasty band-aid

[breadoven]

I did think it might be better to have options for detection only and detection and correction. Don't forget that the correction only applies until the Nav mode is switch off at which point it resets.

The only outstanding test I wanted to do was checking behaviour in Pos Hold cruise mode which I never got around to doing.