This is where the extended Kalman filter comes in (basically just a KF with linearization). I imagine you could also do a KF to include manifold constraints (e.g., rotations) more explicitly but I have never had the need.
If you model your rotation as a quaternion, there is a way to linearize the update process of the quaternion and use a KF. This can work very well and is what most quadrotors I’ve worked with do. However, care is needed to ensure the result gives a valid updated rotation and that you implement the disgustingly messy equations correctly.
If you model your rotation as a quaternion, there is a way to linearize the update process of the quaternion and use a KF. This can work very well and is what most quadrotors I’ve worked with do. However, care is needed to ensure the result gives a valid updated rotation and that you implement the disgustingly messy equations correctly.