...after a while it seemed to devolve into a back and forth of question and answers with eventually one of us becoming tired of the entire thing and leaving the conversation, it was like we were conversing for the sake of conversing and both were being polite to exit.
I find the "juicy" things in conversations are emotionally charged topics. So you have the small talk (ex. weather, sports, work) but a deeper layer is things that have more emotion to them. So basically talking feelings more.
Examples:
Instead of saying what you do for work, say why you find it exciting.
Instead of saying NYC is an interesting city, talk about the time you got flipped off by a taxi driver.
Generally, whereas small talk is "what", talking about feelings is "how" and "why."
I recommend looking back at past conversations and analyzing moments where you felt fully engaged. Also find interesting conversations on youtube (podcasts, interviews, etc). In these, identify engaging moments and replicate in your conversations. Keep an eye out for what and how things are said and received.
In person, you can test saying different things to people and keeping an eye out for when they light up.
So an example is if you talk about dogs and their eyes are glazed over. They barely give you responses. That's a cue to cut that thread and switch topics or leave. But if you suddenly switch to talking about cats and they can't shut up, then you've found a topic that they like.
Each person has different topics that they'll find engaging so that's another dimension to this.
Also, every conversation naturally dies so don't worry if it happens. It can be a lot of things not reflective of you. Maybe they're tired or have to run somewhere. Usually if you get shorter and shorter responses, that's a good indicator that they're reaching the end of wanting to talk.
But something to consider is even the world's best conversationalist will reach a point where the conversation dies.
How do you come up with witty things, pull their legs or just have a good time talking, most conversations I have seem to be pretty boring
As far as joking/teasing, I've found it's a matter of time practicing and calibrating. Don't be afraid to try jokes and failing. Not everyone hits home runs every time when throwing jokes around. But do keep track of what lands well and what doesn't. Also audience plays a huge role as what's funny for some could be offensive for others.
Something that may help is this panel I've watched of an actor from a tv show, Daniel Gillies (brunette guy on right in video). He's pretty funny and good at teasing and working the crowd. I think it'd be great inspiration and reference in trying to sharpen teasing/joking/wittiness.
Note: Audio may be loud, adjust in case.