Pioneering AI Fairness and Decolonial Perspectives with Kush Varshney

Sheikh Shuvo: Today's guest is Kush Varshney. Kush is a distinguished research scientist and senior manager at IBM's Watson Research Center. A prolific researcher and writer, Kush is deeply curious about how to build trustworthy machine learning systems, which he recently published a book on. In addition to this, he's a founding member of IBM's Science for Social Good initiative.

In our chat, you'll hear about Kush's intellectual and career journey, advice on how to stand out as a researcher, and insights from his latest work on the colonialism of AI, analysis of the deeply ingrained Western values in AI, and how to make it more inclusive and useful for a global community.

Sheikh Shuvo: Hi, everyone. I'm Sheikh. Welcome back to Humans of AI, where we meet the people that are building the technology that's changing our world.

Kush, thank you so much for joining us.

Kush Varshney: Yeah, no, it's my pleasure, Sheikh. Thanks for having me.

Sheikh Shuvo: Yeah, Kush, the very first question I like to ask my guests is, how would you describe your work to a five-year-old?

Kush Varshney: Yeah, I actually have twin seven-year-olds at home, so I have a lot of experience explaining things in this way. So, I mean, I think, right now, especially with these really powerful language models that we're seeing, I think relating it back to their own growth is a good way to say it, right?

So, when kids are just, maybe like a year old or so, they're starting to repeat words. And so that's where language models start. They're, I mean, kind of completing sentences, completing words and so forth, but they're kind of babbling a little bit. Then, a little bit later, they start following instructions.

So, I mean, tell them, put a cup on the table and maybe like, half the time they'll do that. And then they start chatting. I mean, then they're at a point where, I mean, they really respond. They're talking back and forth. So, I think that's the initial steps. And then at that point is when we start teaching them our culture, our values, morals, and these sorts of things.

And, that's where I get involved. So, kind of looking at what are all the good things, the bad things that we can do with AI, and kind of working on making sure that that happens.

Sheikh Shuvo: Nice. Nice. I'm in the exact same boat as you. I have two girls that are six and eight, but twin seven-year-olds sound a lot more dangerous.

Kush Varshney: Yeah.

Sheikh Shuvo: As they've gotten older, have your thoughts on introducing them to technology and concepts in AI shifted at all?

Kush Varshney: Actually, I mean, I have had them play around with ChatGPT and with Stable Diffusion and these sorts of things because, I think it is valuable. I mean, not alone, sitting with me and doing it, but yeah, because this is going to be the future. They should have some sense of what things are like, and I'd rather have them doing this than other mischief they can get into. So, yeah.

Sheikh Shuvo: My favorite use case so far has been writing custom bedtime stories for them. So they think I'm a lot more creative than I actually am.

Kush Varshney: Nice.

Sheikh Shuvo: Cool. Well, you know, you've done so many cool things over the years. Could you tell us an overview of what your career story is, and maybe what were some of the inflection points along the way that led to your current research?

Kush Varshney: As you did as well, I went to Cornell for my undergrad, was majoring in electrical and computer engineering, and then went straight to grad school at MIT. And, I was drawn to more of the mathematical side of electrical engineering, which included signal processing and related topics.

And, that kind of morphed into doing more machine learning stuff as well. So, they're very intimately related and kind of form a continuum. And then, as I was going through that and doing my Ph.D., I kind of had a sense that I do like doing research a lot, but maybe not so much some of the other aspects of being a professor.

So then I was looking for industrial research labs to join. And, at that time, this was before deep learning and all that stuff. So, there weren't a lot of companies who had really strong machine learning research going on. IBM was one of them and really felt like a great opportunity to be here.

And, yeah, the group that I ended up joining was led by Sashko Moisilevich. And, they were really doing a lot of amazing stuff, that I hadn't even thought of, that, oh, you can use machine learning and apply it to making predictions about people. Like these days, it seems very natural, but back then, it was like, oh, you can make predictions about employees to manage them better.

You can make predictions about the healthcare system to do all sorts of good things. So, yeah, I mean, that was quite unique at that time. And it just kind of let me do this, and it was great. And I've been here ever since, as we were doing a lot of those sorts of projects.

So, balancing kind of fundamental mathematical research with applications, we kind of understood that when you're making these predictions about people, it really has a big, consequential effect on their lives. And that's when the shift started happening. So we got into doing more things like fairness, explainability, robustness, and so forth.

And, at the same time, I also saw an opportunity and started volunteering with this organization named DataKind, which connects practicing data scientists to nonprofits and other social change organizations, and started doing some projects with them. And that experience led to Sashko and me also starting our own social good program at IBM that we ran for several years.

So all of those things then came together. We created a bunch of open-source toolkits. So the need for that, in addition to kind of fundamental research that we were doing, so AI Fairness 360 was the first open-source toolkit that our team created. It was a Python toolkit that a lot of practicing data scientists could include in their workflows.

And we followed that up with other toolkits, pushed stuff into some of the IBM products, as well. So, the first fairness, sort of bias mitigation algorithm in an enterprise-grade commercial product came out of our work and so forth. So all of that was happening. And then, some other inflection points, I guess, with the family, I had a chance to spend 3 months at the IBM research lab in Nairobi, Kenya. That was kind of a change. Let me have a different vantage point on the work and it kind of gave me some space so that when I came back, I decided to write a book on trustworthy machine learning, and that was a different experience and it's led to different opportunities and so forth.

And then, some things, without my active participation, but they've also been an inflection point is, I think, what we mentioned before, ChatGPT. So since that's kind of caught the popular imagination, we already were working on some large language models, sort of things related to trustworthy SNDI, but it kind of really forced us to go all in and kind of do things there.

So that's what we've been doing in the last year.

Sheikh Shuvo: There are so many gems in there to unpack. The first thing to start with then would be, you mentioned your experience in founding IBM's Science for Social Good Initiative. Could you give an example of some of the projects you've undertaken there?

Kush Varshney: Yeah, it's been something we've been doing since around 2015, early 2016, and we've done probably close to 40 projects with different nonprofits and so forth. Let me mention one project that we're still writing the paper for. It's an outgrowth of our research program and working with our business unit as well on it.

So it's a collaboration with this organization called Alabama Appleseed, as well as another organization called the Center for Court Innovation. And what we analyzed was some things called legal financial obligations, which are basically fines and fees that are part of the sentence for different sorts of misdemeanors in the state of Alabama and, in Jefferson County specifically.

There's this practice, which is much more prevalent than in other states and so forth, where people will get these fines, which are much more than what might reasonably be thought of as a reasonable sort of thing. It's already known in some sense that, yes, there's racism and other things happening, systemic racism and so forth.

But what we actually did was look at it and analyze it in a very fine-grained manner, in a more actionable manner. And so, we utilized this set of techniques called multidimensional subset scanning, which I learned about from my Africa Lab colleagues. They're some of the pioneers in this technical sort of piece of work.

And we've been using it, so highlighting, finding anomalous subpopulations, which happen to be ones that are related to, kind of race and, neighborhood characteristics and so forth. But, it kind of illustrates like certain age groups, certain education levels as well, of where fines are more than others and on what kinds of crimes.

And we've been working with the judiciary in Jefferson County, through Alabama Appleseed, and they've been really responsive to it, and that analysis has actually now led to further work that Alabama Appleseed is doing with the judiciary there, and so, they're actually now starting a pilot program. Instead of these large fines, they're doing this sort of thing where they'll just charge $100 for everything, and if you immediately pay it, then you're done as the defendant.

And so, that kind of reduces the debt burden a lot, also just takes away a lot of the stress for the people and so forth. So, that's been a really great piece of work.

Sheikh Shuvo: That's incredible. Yeah. Wow. Now, extrapolating from that experience and looking at an organization like Alabama Appleseed, they were largely able to accomplish this work because they were serendipitously paired with you and the expertise of your team. But there are probably so many different social good organizations out there that would benefit from a similar approach. What would be your advice to one of these organizations that wants to do something like this but doesn't have the technical expertise in-house to start?

Kush Varshney: Actually, the introduction of these large language models and so forth is democratizing access to a lot of the technologies. So, let me just give you a quick example of another group that we had worked with in the past, and now that we have the foundation models, how easy it was for them to do the same analysis in a much quicker fashion.

So, about five years ago or so, we worked with this organization, the International Center for Advocates Against Discrimination. And the task we were doing with them was a natural language processing task where they had certain sorts of documents which relate to human rights violations and so forth.

And they wanted to label different sentences by what are called the United Nations Sustainable Development Goals, which is a set of 17 goals that were ratified by the member nations of the UN in 2015. They relate to hunger, poverty, all sorts of different things. So these human rights sort of things, when they map them to the UNSDGs, then it's useful for further analysis that they do.

So, we spent a couple of months developing this fancy NLP stuff five years ago. And then it worked pretty well. But then, in January of this year, I met up with one of the folks from ICAD, and in just five minutes, we used ChatGPT and were able to do, I mean, exactly the same thing, right, out of the box, and she was doing it herself, I mean, I was just sitting next to her and she did it. So, the point is, skill level, I think, is going down in terms of these interfaces now that we have are quite low code, no code and so forth.

They are empowering different sorts of subject matter experts to do things themselves. And, I think this is going to be the change that we see. So the sort of data science profession isn't going to be required for a lot of analyses going forward.

Sheikh Shuvo: What a difference 5 years makes.

Kush Varshney: Yeah, absolutely.

Sheikh Shuvo: Shifting gears and looking at other aspects of your research, you recently published a fascinating paper called 'Decolonial AI Alignment' that looks at the development of large language models and how it follows colonial values, largely driven by Western companies that impose what you call a monoculture of ideas. What was the inspiration that led to your thesis there?

Kush Varshney: Yeah. So, again, I mean, in this past year, it's been like a sort of a reflective time for me. Like, everything is changing. Everything is different than it was. And so, yeah, I was looking at it, and there are a few companies that, if you look at it through a particular lens, are acting in some kind of metropole sort of fashion.

So, like, in the colonial sort of way, there are a few powerful companies, they are kind of going in and not really allowing people to have their own values reflected in the behaviors of these models, although that might be changing as well. But, the Western values or whatever the values of the actual developers are reflected in the constitutions or the sort behaviors of the choosing of what to guardrail, what not to guardrail, and so forth in there.

So when you look at it, in that sense, in the past, colonialism wasn't just an economic sort of thing. So, part of what happened in many examples of colonialism is that the colonizers' moral philosophies were imposed in such a way that they were made to be seen as kind of like the neutral, secular, rational sort of way of looking at it, whereas the moral philosophies of the colonized people were made to be exotic or just weird in some ways, and kind of discounted in ways as well.

So, that kind of view in my mind is recapitulated when we look at what is happening a little bit, and it doesn't kind of stop just on that value, like imposition, but another thing that is part of this sort of thing is that, in Western philosophy, there's a strong trend towards universalism, that whatever philosophy you choose, you kind of believe it, you go all in on it, and you don't allow for other philosophies to coexist.

Whereas in certain other philosophical traditions, that is the case where it's actually very natural, very normal for even a single person to hold multiple philosophies in their mind, even if they conflict a little bit, at the same time. So, what I think this implies for AI development is especially the alignment processes that we should have approaches that allow for some conflict, some multiple values, especially ones where the end user, the deployer of the models, their affected communities are able to express their own values and, in the behavior, and then based on the context, be able to choose the right ones and so forth.

Kush Varshney: So, yeah, that's something I strongly believe in. I think, how did it come to me? Where did it come from? I mean, my family is originally from a place that was colonized, so maybe there's some of that. Some of my ancestors did actively work toward what was called Swaraj, or independence from the British in India and so forth.

I think spending some time in Africa was maybe something that influenced my thinking. And just being at IBM, I think, also, because we are different than many other tech companies where the idea is for our technology to help other companies do what they need to do, rather than having our own platform and so forth.

So we deal with all sorts of industries, all sorts of sectors, all sorts of geographies, and making sure that the technology is appropriate for all those different settings, whatever the social norms, the laws, the industry standards, the corporate policies, et cetera, might be. So I think that is also part of the thinking.

Sheikh Shuvo: Two of the terms that you use are Dharma and Pluralism. Could you talk a bit more about what those terms are and how that's a preferred approach to ethics?

Kush Varshney: Yeah, sure. So Pluralism, or value pluralism specifically, is the idea I was saying before that there shouldn't be like a single universal sort of ethics that everyone should be abiding by in that sort of sense. So value pluralism is that, yes, there can be many sorts of value systems that are equally valid.

So that's that. Dharma, it's also a technical term if you think about it. It's kind of coming from a lot of South Asian or Indian traditions. And it's the idea of having, I mean, what is the law of what is right or wrong? What is good and bad? And specifically, there are two different categories of Dharma, so one is Sādhāraṇa Dharma, and the other is Viśeṣa Dharma.

So Sādhāraṇa Dharma is sort of general good and bad that a lot of people would agree to, like not stealing and not lying, these sorts of things. And then Viśeṣa Dharma is a little bit more specific. It's things that aren't like the generic ones, but they're particular to your own situation, to your own station in life, to your community, to the situation and context that you're in.

And then, in those cases, what are the right things to do or wrong things to do, and not really even what they are, but how to think about it. So, in a kind of moral sense, like, if there's a prince who's about to enter into a war where the opposite side has a lot of his relatives, then should he participate in that war, kill them, and so forth, but based on whatever his sort of context is, maybe that is the right thing to do. So that would be an example, but in the AI sort of space, we're kind of seeing this all the time as well.

So when we talk to different customers, different clients, and different industries, they'll have these general sorts of things that they want to align according to, which are things like they don't want hallucination, they don't want hate speech, toxicity, leakage of private information, these sorts of general things. But then there happen to be specific things for their own purposes as well, which are very much this Viśeṣa Dharma sort of thing. So, like the grocery store chain we were talking to recently, they don't want their chatbot to ever mention poisonous food items.

Kush Varshney: So, yeah, I mean, it's a good thing for them. It wouldn't apply in general. Or, there's a law in China right now, which says that all generative content must espouse the core socialist values, which is great for language models deployed in China, right? But it might not be needed elsewhere. Or, a bank that we're talking to wants to make sure that their chatbot doesn't refer to other banks or other bank's products, which again is fine.

It makes sense for what they need to do. Or IBM ourselves, we have strict business conduct guidelines. So we want to make sure that our models, for our internal use cases, respect those business conduct guidelines. So we're seeing this alignment process needing to, not just be a generic general sort of thing.

This thing, but the particular specific thing as well.

Sheikh Shuvo: It seems you could make the case that the most important machine learning textbook is the Bhagavad Gita.

Kush Varshney: Yeah, that is the prince who is entering into the battle against his relatives on the enemy side.

Sheikh Shuvo: On the more practical side, given where AI tech is right now, let's say I'm a PM at a big company, and I'm working on really injecting a feature into a product. What are some of the ways I can incorporate these frameworks into my own work? And what would be the right point in the development pipeline to ask these questions?

Kush Varshney: So in my book, 'Trustworthy Machine Learning', I repeatedly kind of say not to take shortcuts as one of the main messages. And, I think that is the biggest piece of advice I would give a project manager, product developer, anyone. And what I mean by that is, throughout a development lifecycle, whether it's traditional machine learning or the kind of foundation model lifecycle that's now emerging, every point of that process, there are things to consider.

So just in the framing of the problem or the specification of the problem, what are the things I need to worry about? And what do I need to consider when I'm doing data stuff? What are the things that I need to look out for kind of later on in modeling, evaluation? I mean, all of the parts have certain things that need to happen.

And, all of that, we're starting to call it as in governance, right? So throughout the entire lifecycle, making sure that all of the potential mitigations of the different checks, all of them are done and that when things fail, there is some sort of warning or some sort of stop that's put in and, um, just controlling the process, going forward according to whatever those values and principles and laws and so forth are.

So, yeah, I don't say that there's any one point. I mean, it's really, it should permeate throughout.

Sheikh Shuvo: You mentioned the book that you wrote, 'Trustworthy Machine Learning', and it was first published in February of 2022. If you were to create a revised edition of that book now, given everything that's happened in the past year and a half, what would you modify or what additional content would you inject in there?

Kush Varshney: Yeah, so I wouldn't modify a whole lot, but I would certainly add more stuff about foundation models. And, yeah, I was actually just in Dublin a few weeks ago, giving a whole day's worth of course on the book, essentially, and the last hour I spent on the additional new content on foundation models.

A few different things come up, right? So one is that this life cycle is a bit different with foundation models versus traditional machine learning. So, the process with traditional models starts with this problem specification, then comes to getting data, preparing it, then modeling, and then evaluating and deploying. Whereas with the foundation models, there's just a lot of data stuff first, training of this pre-trained base model, and then you come to adapting it. So figuring out what the problem is and so forth. So I mean, some alignment or instruction tuning and these sorts of things. And then you come to evaluating, which is not even doable fully in the way that you could for traditional machine learning. So it's more auditing than really testing. So probing into different potential behaviors that might have emerged and so forth. And then adding some guardrails and then integrating into applications and going from there. So just the life cycle being different, I think, is one major thing that I would certainly discuss.

The other is some new risks that are emerging with the language models that we didn't see with the traditional models. So I mentioned them before, but one big category is the hallucination sort of stuff. So lack of faithfulness, lack of factuality in generated output. Also, lack of attributing the sources. Where did the information come from in the prompt and the fine-tuning data and the original training data and so forth? So how do we know where the information came from? So all of that, I think, is one big category.

A second big category is related to the interaction itself. So now that there's generative output, there's much more possibility for bullying, gaslighting sort of behaviors, for hate speech, for other toxicity, and these sorts of things. So that would be another main component and how to work on those.

And the other thing I would maybe add on is also some discussion about safety versus creativity. So the book already talks a lot about safety, defining it, and what it means and so forth. But the creativity aspect, which is what generative AI lets us do, is interesting, right? We can think of safety as a constraint, so that even mathematically, we can show that it limits creativity in compositional generative senses. And so, like, there's a trade-off between safety and creativity, and for different applications, that's actually good, right?

Because if you want the model to help you write poetry or your kids' bedtime stories, then you want creativity. Safety isn't so much of a concern, maybe a little bit, but not so much. But then there's more serious sorts of applications, let's say with contracts or other things, where safety is paramount, and creativity is not really part of the question. So kind of like thinking through the safety-creativity trade-off is an interesting new thing that's also emerging, I think.

Sheikh Shuvo: Without giving away any spoilers of what you have cooking up, what's next for you in terms of research focus and things we can expect out of your lab in the next year?

Kush Varshney: I mean, it's no secret we're working on foundation model stuff. So a lot of our work right now is geared towards a new IBM product called Watson X governance. And, so we're adding a bunch of source attribution sort of capabilities, more transparency and governance sort of capabilities for the foundation model sort of use cases. We're developing different detectors for these new behaviors, so detecting potential hate speech, profanity, implicit hate, gender ambiguity, the model talking about violence, hallucinating, all sorts of different things. So those are all on our plate. And, yeah, I mean, some farther out looking stuff that we're starting to do more of is thinking about how to make models more empathetic, have more of a mutual theory of mind with the user. So, kind of in the sense of, can the model understand, like, have the model itself having a model for the user, like, what's in their mind, what would they expect, and helping the user also better understand a mental model of the LLM, so that there's a more productive and trusted sort of relationship between the two, and that's something that we're trying to even figure out what that means, but it's something of interest.

Sheikh Shuvo: Well, the very last question I have. You, looking through your career story, it seemed the emergent pattern was really just following your own curiosity and being clear about what your values were. I'm wondering, as you look at all the researchers on your team and people you've worked with throughout the years, are there any particular habits that you'd say the best researchers have that they've been able to generate results with, things that set people apart?

Kush Varshney: I would say there are a couple of things in combination that are good characteristics of researchers. So one is having technical depth on things. Not kind of looking at things superficially, but certainly going in and making sure that the fundamentals are strong and across different areas. And that leads to the second point, which is having breadth in the sense of, like, curiosity, like in my case, let's say moral philosophy or how the social sector works or this or that or whatever.

And because the best sort of research, in my opinion, happens at the intersection of different fields. So one piece of work we did on privacy in the healthcare system and these sorts of things actually used a technique from audio signal processing, and it kind of combined these different sorts of fields together to do something unique, which wouldn't have happened otherwise.

So, like, having that open-mindedness, the ability to bring in something from a different field into what you're doing, and having that technical depth to actually do it in a good way.

Sheikh Shuvo: Nice. Well, Kush, this has been a fascinating conversation. Thank you so much for making time to share your story.

Kush Varshney: Yeah, it was my pleasure. And, yeah, thanks again for having me.