I've said this many times, and it's fundamental to my photography workflow...to get a really good photograph of someone you need to spend the time befriending them. I don't mean spending hours making conversation, but be genuinely friendly, ask permission and engage your subjects before shoving your lens in their face. There are many benefits to this approach...the subjects are more relaxed, are more amenable to move and shift poses if you deem it necessary, and they'll relate better to the photographer...and believe me, this "special relationship" shows in the end result.
When leading my photo expeditions to India and South East Asia, I ask of the participating members that they make the effort of connecting with whoever they photograph...but a few still don't or can't out of shyness or other reasons. Invariably, those who managed to connect have better photographs...and also have better memories.
Of course, there's a difference in approach if you're photographing in a street or a crowd. In this case, asking for permission may be either superfluous (in the case of candid photography) or unwarranted because of the need to quickly photograph a "decisive" moment or scene. However, if noticed by the subject(s) of the photograph, it's generally better to thank them with a nod of the head and a smile, rather than walking hurriedly away, looking guilty.
This reminds me of how difficult it was to photograph in Marrakesh, for instance. In the famed square of Djema Al Fna, a number of street performers make their living of tips from tourists taking their photographs, and do not take kindly to what they consider "freeloaders". Engaging and befriending people who consider tips for photographs as their income will not get them to pose for free. So when in such situations, I either walk away or pay.
I've seen similar situations in La Paz, Bolivia where one of the cholitas in a marketplace was so annoyed by a photographer in our group, that she pelted her with potatoes...from across the street!
Naturally, there are ways and techniques to still get the photograph, even in dicey situations such as of the performers in Djema Al Fna. Wide lenses will allow you to get close to your subject, place him or her on one side of the frame, and chances are that they won't realize that they're being photographed since the lens isn't directly pointed at them. A long lens may also allow you to photograph in such situations, but at the expense of the intimacy that I prefer with my subjects.
For me, there's no two ways about it...people photography (as opposed to street photography) requires engagement, and the establishment of a relationship (even if a fleeting one) between the photographer and the subject(s)...and most importantly, showing respect and understanding for the culture in which you find yourself.
In situations such as with the Djema Al Fna performers or the La Paz cholitas, I'd rather be up front about it, show respect for the way they earn a living, establish a rapport with one or two of them, and then negotiate a fee. If interested in making ethnographic or environmental portraits, I'd have to photograph them elsewhere and that would probably cost me...but hey, I'd try to get model release out of them as well!