Yeah, those "friends" just sound like dicks. It's best to just do what you're doing and not hang out with them as much.
It's normal for guys to make fun of each other back and forth and be blunt/honest, but there's a line you walk. It's usually just innocent insults that don't continue everyday, and it's usually obvious when it's going too far or too long. When it goes too far or continues too long, then it gets into the "bully" stage, and not meaningful friendships.
If they're guys...
1.) and if don't do favors or any normal friendship stuff and not really "close" like it seems from your example, then it's best to just hang out and talk with them less. In the past, if these guys actually wanted to be friends, I've either had them start being friendlier (in which case I'll start talking/hanging out more with them) or start inviting me to more things since guys don't like to be so "open" and that's like an indirect way (in which case you should turn down a few or every other and then gradually start accepting again). However, if they don't do either of these, then best to just ditch it. You'll feel a lot better; trust me.
2) and they're close friends, then you can just straight up tell them when they say something kind of hurtful like, "Hey man, that's not cool..."
#1 is like the "dominant man" approach as you asked about. A dominant man doesn't need to hang out with guys that make them feel bad because he can find friends elsewhere. No reason to waste energy, emotion, time, etc.
If they're girls, this gets a little bit trickier and something that I'm still working on myself. Here's the reason why it's trickier... If they're girls trying to be like "one of the guys" or a "bro" or something -- sorry girls -- but they just don't get it. They haven't grown up as a guy and in a guy environment, and you have to understand that they're on the outside looking in (much as we are on the outside looking in at their friendships). Most of the time they'll try to mirror guy behavior and say insults at you trying to be one of the guys, but they usually do it way too much. They'll repeat the same insult every time they see you, and then when you stop hanging around them, they'll scratch their heads wondering why and ask you. Another example is flirting with women. Let's say I'm at a bar with some girl/guy friends. I go and flirt with a lady and get turned down. A girl friend trying to be one of the guys will say something like, "Ah, that was terrible! Nice fail." A guy friend will say something more like, "Ah, that was terrible! Oh well, don't sweat it. That's awesome you approached. Can't get any worse right? Go talk to that other hottie over there!" It seems insulting, but it's actually insulting mixed with support and encouragement.
What's been working the most in my experience with these type of women is just correcting. So in this example, she'd say, "Ah, that was terrible! Nice fail." I'd just say in reply, "No, I did fine." Something short and sweet and the opposite of what they said and just said in a flat tone with no emotion. This has worked really well actually, and women learn fast from this.
Now, this is different from a girly girl or girl that you're interested in. Usually girly girls won't say something like the above, and if they do, they see you as lower value and not trying to be your friend really. They just see you as a rung in the social ladder below them. If they're girls that you're interested in, instead you'd say something like "Ah, no big deal" and just brush off any insult and maintain your carefree attitude.
But what I'm wondering is how I can project myself as a dominant man to a new group of friends and have them recognize me as such?
I know a lot of people think of male friendships as this alpha, beta, etc. mentality, but I don't think it's really that way as much nowadays. You'll have some guys as leaders because they're just natural leaders and other guys don't care. For example, when I eat with my buddies, I honestly don't care what restaurant. I may rattle off a couple that I definitely don't want to do. Then you may have someone that's more of a leader (which can change depending on if someone really wants to go to a specific restaurant) and pick one. But just because they decide, doesn't make them the alpha. With friendships, it's more of just like the others don't care that much. And if they do care, then they'll either say something or just not go.
I think this has gone away more because of the huge number of people there are, so you can find people in a common mindset. People don't have to hang around with bullies or jocks or someone they don't have any interests with because there is no forced tribe. You can live fine independently. And because people usually want to be around other people, then most everyone is willing to make some compromises.
It sounds like you just need to find better guys that respect you, or just better guys in general. Friendship should be where you just act yourself, no reason to "game" it really in my opinion. If they scratch your back, scratch their back. If they don't scratch back, then just stop hanging around them.
