It's part of the reason. Another big issue with current models is that their language understanding is very primitive. Something like "person standing" they can understand, but "one person with a red hat sitting and another standing behind them with a blue shirt" already fails. There will be red and blue things in the image, but they'll be pretty arbitrarily spread across and not assigned to the person the prompt said.

That said, this won't last for long. With Segment Anything we have AI that has a very good understanding of what is in an image, which should make training on arbitrary images much easier, as well as editing. We also have lots of research going into video and plenty of more powerful language models, that just haven't yet integrated into image generation. Even just ControlNet and some inpainting can overcome most of those issues, it just takes a bit more manual work than a text prompt. There is also DraGan, which is an incredible powerful drag&drop approach to AI image editing, but due to using a completely different approach from StableDiffusion it is not yet properly integrated into other tools.