With local models you can use control net, which is simply speaking, the model trying to adhere to a given wireframe/openpose. Which is more likely to give you an stable result. I have no experience with it, just wanted to point out that there is tooling that is more advanced.