Why do people hate each other based on what color your skin is?

Isn't it silly to hate someone simply because they were born a certain way?

"No Ma'am I don't wanna work with him, he gets caught up in the wrong crowd, All of his friends are a bunch of big black thugs that smoke and do drugs."

...

Oh

Haha

Guess I'm racist.

I wonder if I was just born that way or if I can blame society.