I think you're right about wanting the primary caregiver for comfort.

As an infant/toddler my youngest child would go to Tusk before he came to me, and he would go to battblush before he came to me. Although I swear it is because I was so upset about being pregnant for almost the entire pregnancy that he picked up on those emotions. Munchkin is the the most affectionate kid now, but he didn't want to know me then.