I know of a few people who go to undergrad for a certain major or focus, and then they pursue culinary to fine-tune their skills and understanding of the culinary world.
One friend specifically was in the hospitality management major (which gives you the whole background and skill set for being in the food/beverage industry). Hospitality Degrees | Learn about Hospitality Jobs on All Culinary Schools
for some ideas if you want to check that out.
Anyway, he graduated with a degree in hospitality management, then he applied to culinary schools to see where he could get in. He ended up for graduate school at the Culinary Institute of America Greystone in Napa, CA.
So, I guess what I'm ultimately suggesting is if this is something you know you want to pursue but you're afraid of changing schools or your major late in the game, there are grad school opportunities you can persue. I think it has a lot to do with whether you still want to be in school after the traditional 4 years of college, and if you are willing to pay for grad school (or the student loans years later).
Ultimately, you do have options. Like other people suggested, culinary classes are offered everywhere, mostly in the larger cities of most states. But I you just need to search for what's available in your area.
Additionally, you could always do something for a summer or for a year as someone else already suggested to see if you're cut out for it and really want to pursue it.
I think, either way, you'll have a really great, beneficial experience in learning what you want, what you're good at, and what ou want to do or not do.