SSIE-Diffusion: Personalized Generative Model for Subject-Specific Image Editing

Large-scale Text-to-Image (T2I) models can generate high-quality images by controlling the synthesis based on textual prompts. However, concepts related to specific subjects are often challenging to accurately describe through text, leading to the model's inability to generate diverse images containing these specific concepts. Current methods exhibit shortcomings, including stringent requirements for input images, poor model generalization, uncontrollable image generation quality, lack of editing flexibility and unbalanced semantic consistency. In this paper, we propose a personalized generation method for targeted subject image editing, referred to as SSIE-Diffusion. We address these challenges by training an image encoder to invert original images into text embeddings, ensuring the controllability and accuracy of concept generation. To mitigate overfitting issues during training, we devise a mask loss. Additionally, building upon the T2I model, we propose fine-tuning the U-Net architecture in the model using a regularized dataset. This involves improving the projection matrices mapping to key and value features in the cross-attention layers, enhancing the visual fidelity of personalized images of the subject, and simultaneously addressing language drift issues to balance semantic consistency. Experimental results demonstrate the effectiveness of our proposed method in editing specific subjects within individual images, surpassing current SOTA methods. Across various scenarios, the generated images exhibit superior visual fidelity and editing flexibility, while maintaining high computational efficiency.