MLOps, Generative AI, and Beyond with Atul Dhingra

Sheikh Shuvo: Today's guest is Atul Dhingra. Atul is an ML engineer at PayPal, where he's working on driving generative AI into all aspects of a customer's journey. He's had a wide-ranging career that started with a fascination with facial recognition and authentication. This led to a passion for computer vision and how it's applied in different contexts.

Since then, he's worked in healthcare, self-driving cars, and retail. In our chat, we dive into his latest research, common mistakes ML startups make when building their stack, and more.

Atul, thank you so much for joining us. Now, looking at your background, you've worked in so many different industries and different aspects of the machine learning world. Taking a look back on your career, what would you say were some of the inflection points that led to where you are now?

Atul Dhingra: Yeah, that's a good question. So my journey kind of started out with my undergrad. I originally started my undergrad at a place called Delhi College of Engineering. It's back in India, in Delhi. I was doing an engineering degree in instrumentation and control, which is kind of far away from computer science and anything that I'm doing right now, but that was more of a kind of a probative sense of what do I want to do with my career.

And at that point, I explored a lot of options, areas like microcontrollers, vision, image processing, and all. But the first real kind of inflection, the first chapter in my career, was at IIT Delhi where I was a visiting researcher for about four years. I worked on various problems in biometrics, specifically around authentication. Yeah, a fun fact is, how your eyes look at a screen or a page of paper, it's very unique to you.

So that's how my interest piqued into this image processing and pattern recognition world. Published some work from there, but from there on out, I realized I wanted to kind of get more into computer science rather than being in a more electrical, hands-on engineering role. So I moved to Rutgers for my master's degree.

There, I worked on quite a few problems, but the prominent ones were around face verification and identification. And that was what my academic career kind of looked like. After that, I moved into industry, held a few roles, started at a digital health startup, working mostly on medical claims data on how do you audit patients for effective care, right?

So that is how I started out. I worked on other projects, like self-driving vehicles, and more recently, worked on an autonomous checkout startup as well. And recently, I started at PayPal, working on a lot of generative AI initiatives there.

Sheikh Shuvo: With that early start in biometrics and facial verification and identification, what aspects of that interested you the most?

Atul Dhingra: I really liked the fact that you could feed data into an algorithm and it tells you what kind of patterns it finds for you, right? So it was illuminating at that point, like this was early around 2011, 2012, when I just started out and it was kind of magical to me in some sense. I think that is one of the key things that led me down that path, that you could teach an algorithm to do what you are potentially doing. And one of my lifelong side projects has been to do this at a high level — can you replicate most of the senses that a human has? And most of my jobs in industry or academia as well have kind of reflected that.

Sheikh Shuvo: Is this side project you currently have?

Atul Dhingra: Yeah, that's like a 10-year running project. I keep tinkering with it, trying to see where I can improve things. Yeah.

Sheikh Shuvo: Behind you, in a different room, is there like a giant lab with a bunch of sensors everywhere?

Atul Dhingra: No, the beauty of computer vision is everything can be on GitHub.

Sheikh Shuvo: Yeah. Awesome. Cool. So, in working across all of those different companies, and side projects too, they're all very different, with different cultures and sizes. And I imagine very different tech stacks. What would you say in comparing a lot of the startups that you've worked at, as well as the big companies, are the differences in ML architecture and processes that you would highlight?

Atul Dhingra: I think the key difference is at the 0 to 1 stage, where you're kind of thinking about an idea and trying to execute the first MVP. That's, I believe, where startups have an edge, where you have that agility to move fast, break things kind of culture. But when it comes to bigger companies, big tech companies, I feel like the key value lies in how you scale effectively. So when you're at thousands or tens of thousands of scale, that is where bigger companies play a better role. You could imagine this like Instagram coming out with another feature — they'll be able to serve billions of customers much more effectively if you were to start out a photo-sharing app on your own, right? So I think that's the key difference, on agility and moving fast versus kind of putting that scale forward because more is at stake at that point. I think that's a key difference that I've seen over the years.

Sheikh Shuvo: This is an area it seems you've spent quite a bit of thought on. So it seems you publish quite a bit of research as well. And you recently wrote a paper called "Scaling ML Product Startups: A Practitioner's Guide". What experiences led you to write this?

Atul Dhingra: It was more of a thought experiment on how you think about the ML stack overall at different stages of a company or different stages of a product. So I typically, in my head, break it down, and this is kind of drawing more from the AV, self-driving industry. You could imagine the initial MVP is, can you drive on a road in SF? The next scale that you're thinking about is, okay, can I drive around a block in SF? Versus the next big thing is going to be, can I drive autonomously throughout the city, right? So that's, in my head, what you think about different kinds of progressions of a product, if you will. And how does, you know, scaling or how does the ML kind of workflows change overall in these three different buckets? Initially, when you're at an MVP, like zero to one stage, you don't really care about anything. You just need to prove your product-market fit. You need to get out your product. The key challenge lies when you're actually transitioning from this one to five, like five to 10,000 scale, right? How do you take your existing infrastructure, make sure that you're now cost-effective to go to that scale?

Because initially, as you may know, everybody wants to get to the fastest solution possible. That might not be the most economical solution possible, but when you need to start showing scale, how do you go from like zero to one to 10,000 of these builds or these kinds of products, right? So that was mostly the thought process on how do you cut down costs, fixed costs, and variable costs. And yeah, I would like you to read that. And that was a good thought experiment.

Sheikh Shuvo: And in producing that research paper, talking with lots of different companies and getting insights as to what their ML product life cycles are, was it challenging collecting that information?

Atul Dhingra: Yeah, definitely challenging. But, wrote that paper with another person. He was actually one of my managers at an earlier job. So, yeah, it was a team effort to understand how different people do things without divulging their IP. Right?

So it's definitely challenging. But it took us a long way to even compile.

Sheikh Shuvo: You're at PayPal now, tell us about some of the work that you're doing there. What types of projects are keeping you busy?

Atul Dhingra: Right. So at PayPal, I can't go into a lot of specifics, but mostly I've been working in, like, a strategic generative AI unit. We are leading a lot of efforts around how do you enable generative AI tools and technologies in a more safe, secure, and responsible way across the entire customer journey, right?

So that's the kind of charter that we're working with. But I'll share out more as I can at a later stage, but at a high level, that's what I'm doing. Mostly every aspect of like the customer journey and how can we improve that, overall.

Sheikh Shuvo: Okay. Well, the next time I'm on PayPal, I'll think of you. Let's say you are working on an AI feature at a startup or a big company. What does the team of people look like that are on it? What types of talents are on AI teams at these companies? In the news that you read, it's very much focused on just researchers and ML engineers, but there's a wide world of other talents needed there. Could you talk a bit more about what your teams have looked like in different experiences?

Atul Dhingra: I think it kind of depends on where you are as an organization as well. Are you a small startup building out a new feature or are you a big company building out a new feature, right? I'd like to talk a little bit more about the former because that's where my experience lies. So if you're like a startup and trying to build out a new feature, what I typically look for is a small team to start with.

And I've been on a lot of hiring panels as well. I've seen the industry shift into more of a full-stack ML role. Especially for startups, just being a researcher or an engineer or an ML engineer doesn't quite cut it in most places because you want to take your idea and be able to land it in production in some shape or form. You might not have the best scaling ability, but you should be able to provide that MVP to the leadership in some sense so that you're able to take that idea and put it in production. And so, what I'm seeing as a trend happening is, and this you typically see with staff or principal level engineers anyway, where they start to get more acquainted with things that they're not, they don't have the background in. But even at the mid-senior level, what I've been looking at is a lot of people not just having that interest to take this thing to production, but also the industry pushing them in that way that, okay, you need to have the entire set, right?

So if I were to start a team, start small. Have a group of a few full-stack ML engineers where you can ideate a problem and be able to take that from zero to one and they will deliver an MVP feature out of it. And then after that point, you think about, okay, is this valuable enough for you to now build across in a scaling or do you want to build an infra team around it? Or do you want to build another kind of ops team around it? Right. So that's how I would start.

Sheikh Shuvo: Looking at that process of getting something into production, there's obviously a lot more work beyond just the product and engineering, and a challenge I know I've personally run across in the past is, especially when you're working with a lot of folks who might not know the technical details of what you're working on, there might be challenges along the path. What are some of the ways and maybe even tools that you use when you're trying to drive alignment internally at a company?

Atul Dhingra: So I think the key thing that you want to look at is how do you effectively communicate your ideas, right? It's always about being able to understand from a technical perspective what you're trying to build and being able to deliver it to your team, whatever their level may be, in their own level of understanding, right?

You never have two people agree or have the same level of understanding about the same idea, however diverse your team may be, right? So the key challenge is how do you communicate effectively across your team in a way that they understand and also message it in a way... And this is what has actually helped me a lot with my teams: how do you message it in a way that people understand what is the value that they are bringing to the team or they're bringing to the product, right?

So having that end-to-end visibility not just helps the product, but also your people as well, on what they are building and what is the impact on the organization or the company, right? So this is how I think about having different levels of technical ability in a team and how do you deliver a product in such a way.

Sheikh Shuvo: Speaking about teams, now, throughout your career, you've built lots of different teams and managed them as well. Outside of just technical aptitude and experience, what are some of the things that you look for in people that you're recruiting onto your team?

Atul Dhingra: I think one of the key things that I go for is looking for an employee that is better than you in some area that you or your team want to learn, right? So you always want to hire a better employee or a better leader. That's your mantra, right? That kind of goes without saying for senior level folks, but even with, like, if you're hiring a new grad or a newbie, hiring the talent that has an aptitude to learn and grow is what you're looking for in an early stage employee as well.

So, always go for somebody who can kind of put the team together. So I don't know if you've read that book, "Radical Candor". There's this superstar versus rock star mentality where you have a superstar that is excelling, this kind of lead your features, kind of build everything. You have a rock star that is kind of your rock of the team, not deliver as much if they were working alone, but they take the team together.

So that's what you want to find, the right balance of superstars and rock stars in your team, and somebody who can take your team forward as well.

Sheikh Shuvo: As you're looking for these rock stars, do you have any favorite interview questions or techniques that you like to use throughout the process?

Atul Dhingra: I think how people talk kind of says a lot about them. It's mostly about how do you collaborate with other people. The key thing that you're looking at is, can you leave the ego at the door, right? So there are a lot of ways to kind of judge that, but just talking to people, you understand what their nature is and based on what projects they've worked on, how much they use 'we worked on something' versus 'I delivered something'. It's always that kind of middle ground of what they worked on, what they think the team delivered.

Sheikh Shuvo: Going back to ML stacks, a lot of the big companies right now are investing tons of resources into creating different foundational models and lots of AI developer tools. From the position of a startup looking to innovate in the space, what's your general framework for figuring out what types of ML models a startup should be using off the shelf versus trying to develop something on their own?

Atul Dhingra: I think the strategy hasn't changed. Like if you are starting out, the MVP is zero to one. It's always try to get your product-market fit as soon as possible, right? You want to understand how your product does. And for that, if you want to leverage whatever is open source, or do you want to buy an enterprise solution to do that, I think that strategy hasn't changed. What I've seen a recent shift in is because of all this advent of LLMs and generative AI, is how fast down the stack that you go, right? Like, that's where your moat is, either your data or how fast you penetrate that stack downwards. It's not just like a wrapper on top of anything, but if you need to stay in business, how fast you penetrate down that stack. And that's what you see, like bigger companies like Google, Apple, all these bigger folks, they've kind of come down in the stack where they control a lot more functionality so that you have more of a lock-in. You can't escape that environment. You can't escape that ecosystem. So having that agility to think about, okay, I need to buy this solution to test out the product-market fit, but I have a strategy to build this on my own if the costs become too much to bear to show them an upside, that's what you want to think about if you're starting out.

Sheikh Shuvo: Say, within the first 6 months of creating an ML startup, if you can generalize your own experiences in that as an advisor or investor, do you think there are any common mistakes that ML startups make in their early days?

Atul Dhingra: I think one of the key things to look out for, and I think hindsight is always 20/20, right, is hiring the right kind of people initially because those are the people who are going to stick with you for a long time. If initially, you hire people that don't really work well with others, you might be able to deliver the first iteration of your product, but you may never be able to successfully scale or successfully scale not just with the product, but as a company, as a culture, as an organization as a whole.

I think that's one of the key things that I've noticed is hiring the right talent in the beginning so that they can attract the next right talent, if you will. Right. So I think that's one thing that, as a startup starting out, people tend to neglect a lot. I mean, the first 10 employees, you need what needs to be done. But after that, you need to hire people that will take your team to the next level, or take your product to the next level, thinking about scaling, not just the product, but the whole people organization as well.

Sheikh Shuvo: What's next for you in terms of the research that you're working on? Can you give us a teaser of what's coming up next?

Atul Dhingra: Yeah, I've been tinkering around with a lot of ideas. Keep an eye out on arXiv. Mostly been delving into the generative AI side of things. I actually personally moved more to an IC role just to get my hands dirty at this point because, yeah, you can't miss this journey at this point in any case, right?

So, but yeah, a lot of things happening in the generative AI space. Keep an eye out. I'll keep you posted.

Sheikh Shuvo: Cool. Looking at the broader field, then, is there an area of AI tools and research that you're following closely right now, part of it that you're most excited about?

Atul Dhingra: I think one of the kind of underrated things at this point, again, this might be my perspective, is ML security ops. I think that's going to be big, given like this advent of new threats or new types of issues that you're facing around prompt injections, leakage, and so on and so forth. Right. So I think it's going to become big in the coming times.

And if you had asked me this question like five years ago, I've been preaching MLOps is going to be big, but it's in the past now, but I do feel like ML security ops is going to take off, really big. If you want to adopt these generative technologies at a scale where we feel safe using them, not just as a user level but from an enterprise level.

Sheikh Shuvo: Nice. In that space, do you think there are any particularly interesting companies in that space or any market leaders at this point?

Atul Dhingra: I don't know if there's a clear market leader, but there are a lot of startups kind of open-sourcing these preliminary packages to live and get started, right? Like you want to understand what the vulnerabilities are with your models, with your data, with all the pipelines that you have, so that you're at least prepared to make the right calls at the right points in time.

But yeah, I think it's still too early to say who's going to be the leader at this point. But, yeah.

Sheikh Shuvo: Well, the very last question I have for you, Atul, is to time travel a bit. And let's say you could go back in the past to right about when you were entering graduate school, what would you have changed about what you studied then, knowing what you know now?

Atul Dhingra: Looking back, I would say focus, right? I was split too thin on a lot of different things at that point. Everything looked great to explore, but yeah, the 80/20 rule is what you want to follow, right? Most of your gains are going to come from some small niche thing that you want to work on, but yeah, hindsight is 20/20.

So I think if I were to go back, I would probably focus on the right things instead of looking at a lot at the same time, which... Yeah, so that's my advice to my older self.

Sheikh Shuvo: The converse of that question, looking back at your time in school, are there any particular courses that you think were most influential in how you think now?

Atul Dhingra: One of the grad courses on ML that I took, just to kind of set context, like I mentioned, my background in undergrad was not in CS. So one of the ML courses was a PhD level course that I took at Rutgers. So before each lecture, I had to basically read up on a different computer science fundamental course just to understand what's going on.

So I think that was pivotal because that helped me kind of get up to speed and...

Sheikh Shuvo: Oh, that's a kick in the butt.

Awesome. Well, Atul, thank you so much for sharing yourself and your stories with us. Lots of actionable things in there. I'm looking forward to seeing what you do next.

Atul Dhingra: Nice talking to you, sir.