When people say you can't, here's what they're really saying,
"I don't think I could".
"I'm too afraid to try".
"So... I don't think you should try either."
It's usually a glimpse into their lack of self-confidence, not your ability to accomplish your goal.
These aren't bad people, don't judge them. In my experience, they are well meaning and just trying to protect you. They just see the world differently. Don't get angry and don't listen.
I had a teacher who uses to tell me, "One day when you stand at the pearly gates, you will have to answer for your life. No one else will be there to answer for you. It will just be you."
As a kid that terrified me. As an adult, it gives me a great sense of responsibility.
As Seth Godin said, "Safe is risky and risky is safe".
Go for it.