If some seemingly virtuous foods take on what marketers call a “health halo,” you might say that other foods can acquire some pretty nasty “devil horns” too. Among the most-often maligned foods these days: the protein everyone loves to hate on, gluten.

A 2015 poll showed that more than 20 percent of Americans actively seek out gluten-free foods, and, as of 2014, one in four consumers reported believing that “gluten-free is good for everyone.” We’ve all probably seen the influx of “GF” labels on market shelves and waded through the chatter about going gluten-free on our social media feeds.

Though there are good reasons why some people feel better taking gluten-containing wheat, rye, and barley out of their diet, only approximately 1 percent of the population suffers from celiac disease, the medical diagnosis that technically requires the exclusion of gluten. So how did we become convinced that this invisible-to-the-naked-eye ingredient is so detrimental to our health?

There’s been an increased incidence of celiac disease—and that’s been getting a lot of needed attention.

Along with other autoimmune conditions, this inability to digest gluten is on the rise in the U.S., and, when left untreated, can lead to malnutrition, bone loss, infertility, and even death. Understanding gluten’s harm to the health of celiac patients has brought important resources to sufferers.

And in recent years, another, less-fully understood condition has arisen in medical literature: non-celiac gluten sensitivity. This diagnosis of exclusion is made when gluten appears to cause GI and “extraintestinal” symptoms like headache, brain fog, and fatigue, even when tests rule out celiac disease and wheat allergy. Experts believe it may affect anywhere from one to 6 percent of the population.

Still, the rising rate of these conditions is not exclusively to blame for The Great 21st Century Gluten Panic in which we currently live. Many health professionals point to the media as the driving force behind this phenomenon, while others think the vilification of gluten and carbs go hand in hand.

“Many of these foods are high in carbohydrates,” says Karolin Saweres, RDN, “and people tend to think that carbohydrates are bad for you.” A market flooded with gluten-free products—often housed in grocery stores’ health-food section or given their own space on restaurant menus—contributes to the sense that gluten-free equals healthy.

Even if you haven’t been diagnosed with celiac disease or NCGS, you might have heard the buzz about how a gluten-free diet could relieve your IBS, reduce inflammation, or clear your acne. Perhaps these possibilities have intrigued you enough to go GF to treat a non-GI condition or for general health. After all, what’s the worst that could happen? A few eye rolls from friends and some extra dollars on your grocery bill?

There may be big issues with adopting a gluten-free diet when it’s not medically necessary.

In the midst of consumer worries around gluten, many nutrition pros are now fighting to restore its reputation, since removing grains from the diet for any reason other than medical necessity can actually cause—rather than cure—a number of health problems.

To understand the potential negative effects of going gluten-free, it’s important to note that most foods that contain gluten also contain other valuable nutrients. So removing wheat, barley, and rye from your diet may be throwing out the nutritional baby with the bathwater.

Consumed as whole (rather than refined) grains, these plants have long been recognized for their protective effects against heart disease and various types of cancers—and removing them from the diet can unfortunately have the opposite result: A large-scale 2017 study published in the British Medical Journal determined that eating fewer whole grains because of a gluten-free diet could increase the risk of cardiovascular disease. The risk was significant enough for the study authors to conclude that “the promotion of gluten-free diets among people without celiac disease should not be encouraged.”

This impact on heart health has to do in part with the amount of fiber in gluten-containing grains.

Fiber is known for binding to cholesterol, flushing it out of the body before it can clog blood vessels. As we all know, fiber plays other vital roles too, like assisting digestion—and not getting enough of it from grains can quickly lead to constipation. Plus, dietary fiber from grains also benefits our microbiome—the colony of good bacteria in the gut associated with mental health and reduced risk of diabetes, heart disease, and cancer. Removing gluten in grains means you’ll need to add another source of dietary fiber to keep a happy, healthy microbiome.

Going unneccesarily GF can also lead to significant losses of vitamins and minerals. “Studies show gluten-free diets can be deficient in iron, folate, niacin, thiamine, riboflavin, and vitamin B12,” Saweres says. These critical nutrients help the body perform any number of functions, from preventing anemia to warding off disease to creating healthy babies in utero. If you say goodbye to gluten, you might want to consider a vitamin supplement, which may prevent harmful deficiencies.

Additionally, though we may not always realize it, many products like breads and cereals are fortified with added nutrients such as calcium and phosphorous to fill in any gaps in the American diet. “Gluten-free foods, with a few exceptions, have not caught up with this fortification,” writes Peter H.R. Green, M.D., director of the Celiac Disease Foundation at Columbia University, and Rory Jones, in their book, Gluten Exposed. This means that going gluten-free can cause you to miss out on nutrients you wouldn’t normally associate with muffins or pasta.

It’s not just about what’s missing, either: Since gluten serves as the glue that holds breads, cookies, cakes, and snacks together, removing it means it must be replaced with another binder. Many such alternatives are additives and fillers that add fat, sugar, and salt, say Green and Jones: “Depending on how you define ‘going gluten-free,’ you may actually find that one of the things it ‘gives’ is added pounds.” Researchers at Harvard Medical School have confirmed that the closer you stick to a gluten-free diet, the more likely you are to gain weight.

And going GF unnecessarily can also have undesirable side effects not just on the body, but also the mind.

“The most common issue that I see in my practice is the development of disordered eating from unnecessarily restricting any group of foods, including gluten,” says Annie Goldsmith, RDN.

A sense of fear around “forbidden” foods may wreak havoc on dieters’ lifestyle, driving them to avoid social situations and obsess over their every food choice. “If a client suspects gluten sensitivity, the first thing I do is assess their relationship with food,” Goldsmith says. “The gut/brain connection is very powerful, so a disordered relationship with food and the belief that gluten is harmful can itself trigger GI symptoms.”

Certainly, for those who can’t digest gluten due to celiac disease or suffer the disruptive symptoms of non-celiac gluten sensitivity, the benefits of a gluten-free diet outweigh its risks. But for the rest of us, jumping in “for our health” might not be so healthy after all, especially since evidence connecting gluten with other medical conditions is poor. “Gluten is simply not a problem in healthy individuals, and published medical literature supports this,” says gastroenterologist Chad Gonzales, M.D.

If you suspect gluten may be the cause of your health problems, talk to a registered dietitian or gastroenterologist, Gonzales says. They have access to the gold-standard tests that diagnose celiac disease or will guide you through the process of identifying NCGS. Hard-and-fast medical answers can determine whether going gluten-free is the solution to your health problems—or if it may just add more.

Sarah Garone is a nutritionist and freelance writer in Mesa, AZ. Find her sharing (mostly) healthy recipes and down-to-earth nutrition info on her blog, A Love Letter to Food, or on Twitter.