Have you wondered if going gluten free can help you drop unwanted pounds? The simple answer: Don’t count on it. The more nuanced reply: It may depend on how you go about eliminating gluten.

Most experts agree that forgoing gluten just to lose weight—what many nutritionists call the gluten-free “fad”—isn’t a sound idea. “The gluten-free diet is not a weight loss diet,” says Delicious Living advisory board member Rachel Begun, RD, a food and nutrition consultant and gluten-free expert. But here’s the qualifier: “If you follow a gluten-free diet by eating mostly naturally gluten-free foods, then you might experience weight loss—but that’s because you’ve replaced low-nutrient processed foods with fresh, nutrient-dense foods, which is always recommended for weight loss,” she says.

In other words, there’s no evidence that gluten itself is a weight culprit, unless you have a gluten-related disorder like celiac disease, according to the Gluten Intolerance Group (GIG). And beware: If your idea of going gluten free is simply to swap in gluten-free versions for your favorite carb-loaded foods (think pizza, pasta, cookies), you may actually get fewer key nutrients and even gain unhealthy weight.

According to GIG, many gluten-free processed foods are lower in fiber, vitamins, and minerals than conventional versions (which are often fortified). They also may have more calories in the form of fats and sugar, often added to compensate for flavor and texture changes that result when gluten is not used.

However, if you give up eating gluten the right way, by cutting out refined carbs and eating more fresh produce, lean proteins, and gluten-free whole grains, you may very well feel better and have more energy, because you’ll be eating fewer processed, high-calorie foods. Here are four more tips that healthy gluten-free and weight loss diets share.

Focus on nutrient-dense flavor. Some of the healthiest whole foods are delicious and naturally gluten free: fruits, vegetables, fish, poultry, low-fat dairy, beans, seeds, and nuts. Start with high-quality ingredients to make simple, tasty meals. Top a salad with grilled chicken breast, shaved Parmesan, and toasted walnuts; blend up an almond milk and banana smoothie; or make a bean-based, pasta-free minestrone.

Choose healthier packaged foods. Gluten-free packaged foods enjoy a “health halo” that not all deserve, so read labels carefully. Avoid products that list refined white rice flour, potato starch, tapioca flour, or corn as the first (and second and third) ingredient. Look for those made with more wholesome flours such as almond, amaranth, teff, quinoa, and millet, as well as other nutritious ingredients, including ground flaxseed, hemp seed, chia, and nuts.

Eat a balanced breakfast. People who eat a protein-rich breakfast are less likely to indulge in high-calorie snacks later in the day, according to a 2013 study published in The American Journal of Clinical Nutrition. So don’t just reach for a morning bowl of high-sugar cereal, whether it has gluten or not. Instead, accompany an egg with sautéed kale. Or cook certified-gluten-free oats and top with plain Greek yogurt and fruit.

Ditch sugary foods and beverages. Like refined carbs, these foods not only load you with empty calories, but they also cause blood sugar spikes and dips that can increase feelings of hunger and cravings, according to another 2013 study. Avoiding these foods may make it easier to choose better foods that support weight loss. The bottom line: Make health, not weight loss, your primary goal. And keep in mind that if you’re sensitive to gluten, switching to a healthy gluten-free diet might cause you to gain appropriate weight because your body will better absorb nutrients as it heals, says Alice Bast, president of the National Foundation for Celiac Awareness.