Transcription
This slide deck covers Whose Knowledge Counts, Part Two. You may have heard of the algorithm as a black box before. The reading I assigned about the AI-powered HR platform called Workday is about this very thing.
Either you have applied to a job before or you've heard about job search in this new era of automation in the news. Many applicants complain that after they click "apply" on a job posting, they never hear back and they don't know why. And you could send out a hundred applications and not hear back from not a single one of them.
Now, it used to be in the old days that you interacted with people. You were able to get some sort of indirect or, in the best-case scenarios, direct feedback of the strength of your application against other candidates. Now, you don't even know if the thing that's screening out your applications, or your CV, if that is a human being. The first step in the screening process now is that there's a machine that's doing it. There's some common input that we are not aware of that can scan through your resume and throw it out. And the thing is that your employers don't—the place that you're applying for, they don't even know exactly why your application got thrown out.
So, the question that people have is: Was it bias? Is there bias in this process, or is there a black-box score you can't challenge? So, this is the blind spot these authors, Barocas and Selbst, exposed in 2016 in their essay, and this blind spot is still with us today.
So, like I mentioned, I assigned a reading that's related to this. It's called "AI-Powered HR Platform." This is the way it's uploaded into Brightspace. The title is slightly different.
And then we're going to talk about in this slide deck, we're going to talk about this essay by Barocas and Selbst called "Big Data's Disparate Impact." This essay laid bare a huge blind spot. Our laws punish intentional bias but stumble over hidden machine decisions. This was published in 2016. A decade ago. Okay. This was published nearly a decade ago. And though it was published in 2016, little has changed. Algorithms still evade accountability under Title VII. So that's why we are talking about this.
By the end of this slide deck, you will be able to:
* Explain how algorithms and current laws exclude knowledge.
* Call out the difference between "they meant to discriminate" versus "the effect was equal." There are specific terms for that that you will learn.
* Explain why black-box algorithms evade current legal frameworks.
* Articulate what legislative or regulatory changes could force machines to count in court.
* And finally, connect Actor Network Theory to the legal issue of black-box algorithms.
All right, let's start with two legal tools in Title VII. Remember, this is an essay that we're discussing, an essay that's been written in a law review. So, it's very legal-focused. Okay? Okay. So, we're going to talk about these legal terms and how they impact understanding whose knowledge counts. All right.
So, first, we have disparate treatment. This is when someone deliberately discriminates. There's intent. Okay. So, this is where you can point and say, "Yep, they did that on purpose." But with today's black-box algorithms, the issue is that there usually isn't anyone to accuse of doing it on purpose.
And this is where disparate impact comes in. It's a neutral rule, or a neutral rule ends up harming a protected group more. Focus is on effects and not intent. Okay? Where now we're looking at outcome, where it's not about the intent, it's about the outcome. And this is where we ask, even if they didn't mean to do it, did this rule that's being applied, did it hurt one group more than the other? Okay. So, even if they didn't mean to do it, did this rule hurt one group more than the other?
So, imagine a boss who openly says, "I don't hire parents." This is disparate treatment. There's intent, right? Now imagine a resume screener that flags anyone with a career gap. Unintentionally, this sidelines caregivers, and this is disparate impact.
Why algorithms evade treatment tests:
1. There's no human actor. You can't accuse code of intent.
2. There's no single decision. Models bundle thousands of factors into one score.
3. The procedure is invisible. Courts demand clear steps. Algorithms give none in plain English.
So, picture this. In a normal case, you'd ask the boss, "Why didn't you promote Alice?" But with an automated system, there's no boss. There's just a sealed algorithm. That means the "did they intend to discriminate" test falls apart. Okay. So again, it's an automated system. There's no boss. It's a sealed algorithm. And this translates into the test falling apart because they simply—did they intend to discriminate? We can't figure that out.
Courts want to grill the decision-maker, right? They want you to say, "Show me your work." But with AI, there's no one to call up on the stand. There's just a sealed code. You can't walk anyone through the process. People can't poke at your process and see what you did. What was your procedure? There's nothing to show.
Okay, so now we're looking at why algorithms frustrate impact tests. Before, we were looking at treatment tests, right, related to disparate treatment. Now, we're looking at impact tests related to disparate impact.
You have composite scores. So, when you have composite scores compiled from a bunch of details of facts, it's hard to isolate the discriminatory component. And sometimes it's not just located in one hard fact. Sometimes you have a combination of them that create a discriminatory component. So, it's really hard to isolate it.
The business has a necessity defense. Employers argue that the model's efficiency justifies it. They're saying that where they had 25 people applying for a job, now they're receiving 200 applicants for the same job. And this is because everything is automated. People can put together resumes faster. A lot of people are using ChatGPT to format and to organize their resumes and their cover letters and sending it out. And so businesses don't have the means to go through 200 applications. And so it's hard to shoot down this employer's "model is essential" defense.
And then you have the high legal bar. You must identify the specific practice, show the gap, and propose a less biased alternative. It's a lot to ask for. It's like asking a mountain to move. It's a lot to do in order to hold someone accountable.
Human versus machine actor. All right. So, here we're calling the machine an actor. This goes back to the previous slide deck where we talk about the Actor Network Theory framework. And in this case, the machine, according to that, would be an actor.
Law counts humans. This is now the problem: the law counts humans. Courts require someone to explain their decision steps, but the machine is silent. Algorithms produce outputs, but they can't testify. And in our knowledge hierarchy, human testimony outweighs hidden code.
In the current knowledge hierarchy that we have, we don't count the machine as an actor. Right? In Actor Network Theory, what does that say? That says that everything co-produces knowledge together. It co-produces. In this case, according to that Actor Network Theory, the machine would be an actor and it would be a co-producer of knowledge. However, in our current system, we have a knowledge hierarchy. This means that knowledge is not co-produced. Some people or some things are on the higher end of producing knowledge. In this case, it's human testimony, and it outweighs hidden code. Legally, if it can't be voiced in court, it can't—it doesn't count. And that silences the algorithm's shield.
This is a really important slide. This is where you start to kind of put things together. This is where you should be able to tie this back to Actor Network Theory. How Actor Network Theory might help address this issue. Because in our current system, we have a knowledge hierarchy in which human testimony outweighs hidden code. When we talk about whose knowledge isn't counted here, we can think about it being the machine, right? The machine's knowledge is not being counted here.
Another way to look at this is when we talk about whose knowledge is not counted, we say the experience of those who have been harmed, their knowledge, their voice is not—has not been counted here. When we don't have a way to penalize and hold employers accountable for their harmful black-box algorithms, what happens is that we also silence the people who have been harmed, and we don't bring them justice.
What would change the game? What would clear us from this impasse? Imagine a world where your algorithm must explain itself like a human witness. This may be what we need. Until then, all that hidden code remains invisible under Title VII.
Remember, we have to adapt very quickly to technology. Technology makes very fast and sometimes big-scale changes. And so the outcomes of that, whether they're positive or negative, can be huge, and sometimes they can be devastating for people's lives and well-being. And so we do need to make changes to our current legal system because it hasn't adapted yet. And one of the ways would be that we refer back to strong scholarly work by Bruno Latour on his work with Actor Network Theory and we recognize algorithms as actors. We treat software as capable of decision-making like a human. We would have a very strong argument for it because there's a whole scholarly body of work behind that framework.
We can mandate explainability, push for models to expose clear, interpretable rules. By this, we mean, you know, a model is—you have various factors that go into a model. You have various independent variables or inputs that go in to create, to lead you to an output, to cause an output. And so we need to know what all those inputs are. And each of those inputs, there's a classification, there's a category classification where you have created a boundary around what gets included within those meanings and what gets excluded from those meanings. And so what gets included and excluded and classified in a certain way, and these have to be—not just the stale, you know, the reason why a lot of us can't, when we look at a dataset, when we go to those websites and we look at a list of what those data variables mean, it's really hard to make sense of it because they are decontextualized. And so you—it's once you do spend time with it, you study it and read papers on it, you understand more and more. But the issue is that there needs to be a track record of constantly recording what, how something is defined and what's being left out and what that could mean potentially, what the interpretation of that, and what sort of limits and interpretation that might lead to. And so this is something that's not being done. This is something that if a—because these variables come from those who collect the data, and maybe they have it as their personal notes somewhere, that stuff doesn't get transferred when things are made more official and sort of standardized. You have a codebook, and the codebook is giving very concise definitions of what each variable is. And it's hard to read between those lines. You could—it's not impossible to do because you will be doing it in this class. I will be asking you to do that with your dataset. But it would be—when we have these algorithms, it would be easier if we had that information, that transparency, follow from the onset of the data collection all the way to where the algorithm is being used.
Remember, a lot of times this stuff is being used by third parties. It's being imported. Software systems are being bought from other places and being implemented for an employer. So, it's important to have this sort of trail and this transparency.
Okay, so update statuses. We could rewrite laws to include automated decision systems as changeable entities. So, what I want you to do is I want you to reflect on this yourself. I want you to reflect on if you could choose one, whether it's explainability, which is number two, legal status, which is three, or actor recognition, which is item number one, which would you lobby Congress for first? What would you lobby Congress for first? What do you think would help with the challenges that we currently have?
Okay, so the purpose of this slide is for you to get a grasp on the key developments since this essay so that you can understand how much time has passed. Technological change brings changes very quickly and at a large scale, and our social systems can lag behind, and we can't afford for that to happen. There are vulnerable groups that can be further adversely impacted because it might be that our technology is just a replication of the biases that we as a society have. And so, in any case, the issue here is that—well, one question is, how do you stay vigilant? To me, one of the ways is that you stay on top of the news. You have a good sense of bioethics, right? And by "good sense," I mean a basic sense. I exposed three of its core values to you. The fourth value we will talk about more in depth in our next modules. But so, staying on top of the news, knowing what's going on, and advocating for these changes that can help move things along faster.
I have a list here of the key developments that have occurred, but listen, the bottom line is that federal civil rights statutes haven't been amended to cover the black-box harms. And so unless Congress updates Title VII or similar laws to explicitly include automated systems, algorithms will continue to operate largely outside the scope of law. I mean, this is the current state of things. But these are some examples of what's been happening.
So, in 2023, the EEOC has AI disparate impact guidance. The first couple of these items are just guidance and recommendations given to employers. For instance, they're non-binding guidance explaining how employers should apply Title VII's disparate impact framework to AI and automated tools. They reinforce that you must still identify a specific selection procedure, validate its job-relatedness, but nothing new is added to the law itself. The next item, again, it publicizes how emerging technologies can run afoul of civil rights laws, and they produced best practices as recommendations for bias auditing, but again, it does not change statutory language.
Number four—I'm just going to skip over three. Number four is a collection of congressional bills. They're so far stalled that they would have required high-risk automated decision systems to undergo impact assessments and bias audits before deployment. We had the pandemic since then. We had a major government change since then. So, these have stalled, and I'm sure there are a number of other reasons why they're stalled. And lastly, we have these local and state AI fairness laws in New York City and in Hawaii. However, these are local ordinances, and largely what happens is that algorithms will continue to operate largely outside the scope of law.
So, again, understand that it's important for you to be aware of these effects anytime there's some sort of a new technology. Ask yourself, or do some reading up, and understand: What are the bioethics on this technology? How is it impacting everyone? What are the concerns with it? And how is the current legal system adapting to it?
Okay, so this brings us to our last slide. I want you to be able to give yourself a self-check. Can you answer these questions? Whose knowledge is not counted in our black-box algorithm example? I mentioned two actors. Can you identify them and explain why? This is just a self-check, so don't skip over it. I won't be able to check you on it, but do yourself the service and see if you can answer these.
How does ANT offer a solution for current cracks in our legal system? The answers are in this slide. If you don't remember, please go back and just have a look and try to answer them to yourself.
And then next will be what's tied to our hot topic. What's tied to—yes, our next hot topic is going to be this slide and the Part One slide. Remember, the Part One slide talked about sustainability as a concept, right? It talked about how there are two things: one is that SARE is an organization that is progressive. They have this great initiative in the sense that they brought farmers to the table. They moved away from that top-down approach where experts run the research approach, and they said, "We're going to have farmers at the table to help us be co-producers in this knowledge." And so it goes into detail how they do that. And remember, SARE stands for Sustainable Agriculture Research and Education.
And so you have these authors, Tanfa and Bavsar, who appreciate this initiative, and they say, "Okay, so let's see if SARE is doing as intended. Are they achieving their goal by addressing sustainability?" And if you recall, SARE, what they do is they produce knowledge. And so it's a perfect study focus for these sociologists who are discussing knowledge production, and specifically social constructionism. And so, for instance, SARE, what they do is—and I'm just telling you how the mechanics—I did explain this to you in the slide, but I'm going to explain the mechanics of it again. They are a funding organization. They fund projects. Projects that are co-produced by farmer and researcher and educator. They create this knowledge network. So, farmers apply for this grant. In return, they have to produce a report where they talk about the progress and the outcome of the grant and their study. And these authors said, "Okay, this is a good artifact to look at, these reports, to see what sort of projects were chosen for funding. Can we find a pattern there? Were these projects science-driven? Expert-driven? Were they economist-driven? These fields of knowledge meaning, are they more concerned with production? And are they more concerned with the environmental science aspect of it? Or do we see the last leg of sustainability called social sustainability or quality of life metrics? Do we see these also being reflected and being given equal weight?"
Okay, it's not just are they being—okay, so sustainability has included this concept, right? Because as a response to the industrial era of saying, "We can't just be profit-focused. We're not going to sustain as a society this way. We have too many health issues from contaminants and pollution, and from these factory conditions, and the health of factory workers, and it's creating poverty and havoc and all these sort of situations in our society." And so we need to think beyond profit. We need to think about the environment. We need to think about social conditions. And so these are—this is how the concept of sustainability came up, and they defined the three core legs or the core values of sustainability. But still, don't be surprised, still to this day, the last leg of sustainability does not receive equal attention. And there, I discussed the reasons for it. And so, a question is, yes, we include these components to it, but is it fair to say that we should give more weight, more attention, more funding to understand these types of knowledge versus this other type of knowledge? And so that is what slide one is all about: Whose Knowledge Counts?
And again, in that one too, there were two types of actors that we're talking about in that first one. They gave farmers a seat at the table. So, the farmers got included. But then the authors were looking at if something else was being still left out. That's another type of knowledge was being left out.
Okay, so this brings us to discussion hot topic. You will need to have a good sense of these two slide decks in order to complete that assignment.