Presumably they wouldn't be the *only* person you were interacting with. It might make it hard to love anyone else though, yeah. I don't think it would be a permanent effect though -- usually people learn to tolerate mild annoyances in a matter of months or years.

Quite aside from, 'what if you subconsciously wanted to be annoyed?' I suspect a significant percentage of the population would be completely unable to form a relationship with someone like Lena, because of ingrained defensiveness and avoidance making her into an enemy instead of a lover.