If you’ve ever had a roommate (or just lived life as a human in this world), you’ve probably dealt with passive-aggression more times than you’d like. You get real familiar with sentiments like “I guess you could wash dishes that way, I just like them a little cleaner” or “I love that bracelet. Usually gold looks so tacky, but it works for you.” Or the classic “it’s fine,” paired with a tense, angry-eyed smile.

This backhanded aggression is hard to deal with (though there are ways to handle the passive-aggressive people in your life) and can sometimes make you feel a little crazy—it’s not always easy to tell if someone’s trying to be hurtful with a smile on their face or if their words are just triggering your insecurities.

One time, my mom made a simple comment about keeping in touch with our old neighbors and I spent the next few hours crying about how she doesn’t think I love her enough. In that case, she wasn’t making some backhanded dig, but it set off my insecurities about feeling distant from my family.

I’m not saying that everyone’s gone on a crying jag over a totally innocuous statement, but it’s not uncommon to take neutral remarks personally and assume the other person is just being a passive-aggressive nightmare. So I spoke to experts to find out how to determine when someone is being legitimately hurtful in a backhanded way and when our insecurities are simply getting the best of us.

Also, please note, any reaction you have to a person—whether they’re being passive-aggressive or not—is OK. I’m not at all trying to tell you to not get upset or hide your feelings because your feelings are valid no matter what the cause.

Unfortunately, if you’re at work and dealing with a poor communicator or someone who sets off all your worst insecurities, you may not want to be all up in your feelings during office hours. So by figuring out when someone is intending to be hurtful (as opposed to inadvertently causing you pain), you might be able to have more control over your reactions—and you can save your real anger for the people who are purposeful jerks.

Take a Step Back and Get Perspective

When you hear a potentially passive-aggressive comment, empowerment coach Alani Bankhead suggests you take a step back and try to identify the specific behavior that offended you as objectively as possible. Basically, before things get washed away in emotion, it’s good to break down what just happened.

Maybe you just had a passive-aggressive boss, so now you interpret everything your new boss says as passive-aggressive, when, in reality, they might be giving you a simple note. This is backed by the appraisal theory of emotion, which states that we feel emotions based on our appraisal of the situation. This explains why people can react so differently to the same situations.

For example, a dog running away could make an acting teacher storm into the room in tears and take the entirety of class time to weep and talk about a dog psychic that gave her bad news, whereas the same dog running away might make a student in that acting teacher’s class think, Seems like we should still probably have class today? (For the record, my acting teacher’s dog came home that night, even though the dog psychic said she was “with the angels.” Lesson: Don’t spend a lot of money on dog psychics or acting classes.)

Anyway, the appraisal theory helps explain why some things might set you off while they don’t bother anyone else. It also explains that our whole life experience and history influence our day-to-day reactions. Once you become more aware of your appraisals, you can have more control over your reactions.

Bankhead says it’s good to take a look at the situation after you’ve had an emotional response. Quickly replay what they said and try to see if there was any real malice behind it. It’s also helpful to ask for other opinions. If everybody at the office thinks that person is a passive-aggressive a-hole, then you probably don’t need to do a lot of soul-searching to find out if that’s true. But if no one had a problem, it’s good to give your initial reaction a second thought.

Now, I’m not saying to blindly trust everyone else’s judgment here. If you know someone is passive-aggressively manipulating you, it doesn’t matter what anyone else says. But if you’re not sure if something is based on aggression or insecurity, getting a second opinion can help.

See the Signs of Insecurity

Sometimes, it’s hard to see your own insecurity. Either we’ve lived with our own self-negativity for so long it doesn’t register anymore or we’ve never stopped to analyze the things that make us feel insecure.

A report from Harvard and the University of Pennsylvania (as reported in Slate) found that there are lots of clues to insecurity in the way people speak.For example, people who are constantly self-promoting or trying to emphasize their status in a group are often the most insecure.

In the study, students at Penn often referred to their school as “Ivy League,” while Harvard kids usually left that moniker out. Since Harvard is the King of Ivy League schools, the students didn’t need to assert their dominance. But for kids at Penn, a school that most people forget has Ivy League status, their insecurity made them want to shout “I’m Ivy League too” from the rooftops.

So if you notice yourself ever wanting to overtly brag about accomplishments or trying to inflate your status in a situation, it probably means you’re a little insecure about the topic. When I say “I’m a freelance writer,” I always drop a few names of places I’ve worked so people don’t think I’m some random Yelp reviewer with a blog.

It’s not because anybody actually cares about my writing career, but I feel the need to assert my status out of insecurity. Once you really get to know what makes you feel insecure, you’ll immediately know when someone is setting off your insecure alarm and when someone’s being legitimately passive-aggressive.

Know Your Triggers

As you get to know your insecurities, dig deeper into your specific triggers. It will help prepare you for when someone accidentally steps on your emotional minefield.

“When you find yourself dissecting every word, action, tone, and gesture the other person used in the allegedly offensive comment, identify what specifically irritated you,” Bankhead says. “What emotions are you feeling? What do you physically feel in your body? Often times, we have physical reactions but don’t even notice!” After you identify the feeling, see if there’s a root cause to that reaction, Bankhead suggests. Maybe you had a traumatic experience in your life that not everyone knows about. It makes perfect sense for you to have a bigger reaction to something that triggers anything close to that trauma.

Even if you didn’t have a major tragedy in your past, you can still get upset about things that other people brush off. For another personal example, nothing makes me madder than someone telling me to “calm down.” I work hard to regulate my emotions, stay rational in work or public situations, and avoid conflict. So when I’m trying to make a point without a ton of emotion behind it and someone tells me to “calm down,” well then I’m ready to punch a b*tch. Now, I wasn’t told to calm down as a kid, nor did a mugger with a “Calm Down” jacket try to rob me a gunpoint. I just hate hearing that phrase. But I also know that those words are a trigger, so when I hear “calm down,” I have to regulate that rage feeling and take the words at face value.

“Being self-aware is the key to knowing when someone is intentionally mistreating you,” says Sal Raichbach PsyD, LCSW, of Ambrosia Treatment Center. “Self-awareness gives you the ability to take constructive criticism without projecting your insecurities onto the situation.”

Now, if someone is outright aggressive, uses hurtful language, or is utterly insensitive, you don’t need to check yourself. But when someone pushes your buttons without malice, knowing your sore spots can make your life a little easier.

Empathize (Even When You Don’t Want To)

Before you go around town telling everyone what a jerk this passive-aggressive person is, try to empathize. I know, that’s not fun, but sometimes it gives you a better perspective. “Most people are generally trying to do the best they can in life, but it’s 100 percent normal to have conflict,” Bankhead says.

If you’re dealing with a person you can’t avoid (like a boss), try to give them the benefit of the doubt. Assume they meant the best, even if it came out wrong. We all say stupid things sometimes, so it’s nice to cut people a little bit of slack, especially when it’s the first instance of potential passive-aggressive behavior.

Taking a moment to put yourself in their shoes can sometimes resolve the situation entirely. Maybe the other person is stressed, overworked, and just as insecure as you. They said something that wasn’t great, but you can see how that might happen when you empathize with the entirety of the situation.

Or you might try to empathize and find that there’s simply no excuse for their behavior. In that case, they are being passive-aggressive and you should deal with them on those terms.

Take It at Face Value

“If you can’t decide whether someone is being disrespectful or just giving you their opinion, recognize the fact that their lack of communication is likely to blame,” Raichbach says. “Instead of projecting your self-doubt onto the situation, remind yourself that it isn’t your job to teach them how to communicate.” So, if you get an “it’s fine,” it’s totally OK for you to take those words at their literal meaning and go about your day as though everything is totally fine.

Stop Analyzing

There’s a fine line between reflecting and overanalyzing. Raichbach recommends reflecting about what was said, how you felt about it, and if it set off any triggers. But spending much more time thinking about the incident becomes counter-productive.

“It can be tempting to spend hours trying to pick apart what a passive-aggressive means when they say something,” Raichbach says. “Remind yourself that it’s impossible to see what’s going on inside someone else’s head.” When your thoughts start to swirl around a potentially passive-aggressive person, you suffer. That person goes about their day, while you pour over every word, wondering what you did wrong or if they have mean motives behind their sentiments.

“Instead of trying to interpret, move forward by asking them direct questions the next time they voice their opinion,” Raichbach says. Simply ask in a polite tone, “What did you mean by that?” or “Are you upset with something I did?” If the person is really passive-aggressive, they’ll have to either get aggressive or back away from their back-handed response. Or if the person didn’t have any malice behind what they said, they’ll probably apologize and correct the situation. Either way, you’ll immediately know what you’re dealing with and won’t have to spend so much mental energy on the hidden meanings of the other person’s words.

Keep Communicating

“If you complete this process of self-evaluation and empathy and arrive at the decision that the person’s comment was truly meant to be hurtful, then it’s a great opportunity to identify what boundaries were violated and how to address it so that it doesn’t happen again.” No one deserves to be mistreated at work, home, or anywhere. So, if someone said something with malice, they shouldn’t do it again.

Sadly, this can be difficult to approach in work situations, but it’s not impossible. If someone is constantly trying to stab you in the back with a smile, go to HR and talk about some of the hurtful conversations you’ve had. Or, if you can, address the person directly. That’s all easier said than done, but addressing the situation head-on will make your life better in the long run.

We all have insecurities, triggers, and jerks in our life. When you get really self-aware and know all your sore spots, it gives you power. Instead of accidentally getting set off all day, you know what upsets you and when someone is stepping over the line. Rather than get caught in a cycle of worry about what someone really means by “it’s fine,” you can let that go and put your energy elsewhere. “At the end of the day,” Bankhead says, “we all decide how we choose to view the world.”

Amber Petty is an L.A.-based writer and a regular contributor to Greatist. Follow along as she shares her weight-loss journey in her new bi-monthly column, Slim Chance. Take singing lessons from her via Sing A Different Tune and follow her on Instagram @Ambernpetty.