Stable Diffusion AI Mapping User Manual (5-2): Lock the character pose through the normal map | T Kebang

In the previous tutorial, we saw the use of OpenPose in ControlNet to control the pose of the character. This time we further use the normal map to lock the pose more strongly.

Stronger control than OpenPose

Although OpenPose has successfully controlled many character poses, there are still some character skeletons in the reference pictures that cannot be correctly recognized by OpenPose, resulting in a situation where the generated pictures are not as good as expected. At this time, you can consider using the Normal BAE model with stronger control , use the normal map to solve the problem.

First of all, we need to refer to the previous teaching article “Controlling Character Pose Through ControlNet” to complete the installation procedure of ControlNet first.

Next, you need to go to the ControlNet page on Huggingface, download the “control_v11p_sd15_normalbae.pth” model, and save it to the “extensionssd-webui-controlnetmodels” under the “stable-diffusion-webui” folder, so that the previous set homework.

▲ Using the OpenPose model to process the reference image, you can see that the preview image in the right half column cannot accurately identify the skeleton.

▲ However, using the Normal BAE model for processing can more accurately control the poses of the characters in the generated pictures.

▲ The left side of the reference picture uses the Normal BAE model, which can restore the content of the original picture. The OpenPose model is used on the right.

▲ Switching to another set of reference pictures, the OpenPose model also cannot correctly identify the skeleton.

▲ The Normal BAE model can make the character pose correctly, but the pose of the OpenPose model has been completely out of shape.

▲Readers must install ControlNet by referring to “Controlling character poses through ControlNet”, then go to ControlNet’s Huggingface page, download the “control_v11p_sd15_normalbae.pth” model, and save it to “extensionssd-webui-controlnetmodels” under the “stable-diffusion-webui” folder ” to complete the pre-work.

obediently put on the same pose

The usage of the Normal BAE model is similar to the previously introduced OpenPose model. The main difference between the two is that if you want to use the Normal BAE model, you need to select normal_bae in the Preprocessor and control_v11p_sd15_normalbae in the Model. For other settings, you can refer to the basics mentioned above. concept.

▲ Readers make various settings according to the suggestions in the figure, select normal_bae in Preprocessor, select control_v11p_sd15_normalbae in Model, and then click the “explosion icon” pointed by the green arrow to preview the normal map. In addition, click the “up arrow” to export the resolution of the reference picture to the resolution of the generated picture.

▲ The process of generating pictures also requires the general operation process to fill in positive and negative prompts, set various parameters, and finally press the Generate button to start calculating the picture.

▲ In the test example picture, the Normal BAE model can correctly reflect the hand posture when playing volleyball in the reference picture. The OpenPose model catches the ball.

▲ Although the Normal BAE model adds a volleyball that is not in the reference picture on the right hand part, the skeleton of the character is relatively correct. The OpenPose model has obvious errors in the fingers of the left hand.

▲ The Normal BAE model retains the correct pose, and fully restores the long hair of the character in the reference picture (however, it is hoped that the generated picture will be short hair). Although the OpenPose model correctly generates short hair, the pose of the character is completely wrong.

▲ The poses of this group of reference pictures are relatively complex, and the Normal BAE model can be restored correctly. The skeleton of the right hand of the person in the picture generated by the OpenPose model is obviously wrong.

Although the Normal BAE model has a strong control force, it is because its effect is too strong, sometimes it is not suitable for the situation where you only want to restore the pose in the reference picture one-sidedly, and do not completely lock various details, so readers are using it , you may wish to interactively test the effects of the Normal BAE and OpenPose models, generate a few more pictures, and select the ideal finished product.

(Back to the series article directory