"I don't want to say that to him." 
Why?
"Because he might take it the wrong way." 
How do you know? Has this happened before?
"...No. I just don't want it to happen." 
So you're not saying what needs to be said, because you're afraid of something that didn't even happen.
"..."
I wrote this two days ago: Fear is the most debilitating of all human emotions. Its destructive power surpasses the power of all other emotions put together, both positive and negative. In other words, fear will destroy everything else you may have going for or against you.
Most of our fears sound like the above conversation. We attach negative consequences to neutral circumstances, creating stories in our heads of what might happen with no actual proof of it ever happening. Then we do... nothing. And fail, because we neglected to take (enough) action. And when confronted, the whole story or our failure is imaginary.
Make like a good scientist and test your theories.
Fear is a magnifying glass: it enlarges anything you look at through its lens. And it's keeping you from your success. Is knowing this enough to move you to action?
If not, the goals that fear is blocking are too small.