"People who don't know me don't should not have a say in what I do with my life," the singer said on Call Her Daddy