Invisible Rulers: Information Warfare and Public Trust

Good afternoon everybody. My name is Anatmati. I am uh a finance professor here and a faculty director of the cooperation society initiative which uh I know some familiar faces here. Some of you know well. We've been around since 2018. And it's actually the year in which our wonderful student leader moderating this event, Dylan Cassie, is a a initiative powered by um faculty, staff, and students. Uh we actually have a turnover for new student leaders. So, we're excited about that. Please come in. Um our events are are sometimes moderated by faculty or others but oftentimes and there couldn't be a better moderator for this event than Dylan Akerman who uh spent five years prior to coming to GSB uh in Facebook doing precisely uh issu dealing with issues that we're going to discuss today.

Uh the topic for today is um has to do with governance of speech with uh we called it information what what did we call it information war warfare uh which we now have a lot of free speech in our democracies and the question is you know issues around control and distribution and all of that. All of these come particularly intensely with social media platforms which is where Dylan uh worked. But they obviously the issues themselves have been around forever since any kind of form of media. uh and our guest uh today um worked at Stanford for a while uh in something called the internet u observatory and uh and both of these people uh no longer would have had their jobs because their job no longer exists. So that's one topic we might uh might discuss today. Uh without further ado, I'm going to let Dylan take it from here. Thank you very much for joining us.

Thank you. Thank you everyone for for joining us. Uh we're talking about an issue that is uh near and dear to my heart having spent several years working on it and having known uh Rene's name for for quite some time as a result of that. Um, and I I think that it's an issue that is we were talking about it this morning over coffee particularly pertinent giving what we're seeing uh around the world and you know sort of the the different forms that disinformation and misinformation uh is taking in in digital spaces and sort of both the speed and the content that we're seeing um both could be argued or in some ways unprecedented. So uh really excited for this conversation. Renee, I wanted to give you the opportunity to to go a little bit more into your backgrounds, how long you've been working in this space, in what capacity, what first got you sort of interested in in this misinformation, I know both of us don't love that word, uh, disinformation and sort of platform safety part of social media.

>> Yeah, thanks for having me. Um, so I I got into this entirely accidentally in uh 2013. So, I had just moved uh I moved from New York to uh to San Francisco in 2011, had my first baby, and I was putting him on these preschool waiting lists and um and I started looking at the vaccine rates, the vaccination rates in the schools in the kindergartens. Uh I was kind of horrified and um you know, in New York, we did not have this thing called the personal belief exemption in California at the time it existed. And I started uh I pulled down about 10 years of California Department of Public Health data and I started writing about this phenomenon of um the intersection between personal belief exemptions and the prevalence of the antivaccine movement on social media. Um and I got very involved. The Disneyland measles outbreak happened almost immediately after that. And I got very involved in um in the campaign to eliminate the PBR and to move to just medical and um to medical exemptions. And as I was doing that, I felt like I had this sort of firsthand view into how um how influence and persuasion was going to look in the future. And at the time in 2015, uh you know, you could just run bots on Twitter. This was not this was it was actually not even a thing that people concealed. It was actually completely transparent. Um there was this effort to try to make uh make people's opinions look like they had more support than they did just by using fake accounts to amplify them or to try to get something trending. So I was writing a little bit about the interesting new ways in which social media allowed um the shaping of opinion or the the perception of the majority to be shifted just through the design affordances that the platforms gave people. And this wasn't a normative conversation. This was just like this is how it works now. People should understand this if they want to fight back in this space. Like public health was just completely out of the conversation. They were not engaging online at all. Uh so just trying to explain that this is where this is where the future is going. And around the same time that I was doing this, I wrote I made a couple of like network visualizations, really tried to tell the story how this is, you know, not even the future, this is the now. Uh, ISIS had become quite prominent on social media and I was asked to come down to the state department um to help with this question of why was ISIS so popular online? What what was it that they were doing so well? Um why were their networks growing? What was the appropriate response? How should the US government respond to this? How should the platforms respond to it? And that was sort of what took me into again I just had a startup. I was just in tech and and all of a sudden um my sort of uh you know data science night hobby became um became my job and that was uh that was how it happened and then Stanford Internet Observatory. So I wound up I worked on the ISIS thing. Then I was asked to work on the Senate's Russia investigation um into the data set that Facebook, Twitter and Alphabet provided in the around the 2016 election. And I led one of two outside teams on behalf of the Senate Intelligence Committee doing an analysis into those data sets, writing about what the Internet Research Agency had done, how now we had gone from, you know, activists were doing this, terrorist organizations were doing this, and now state actors were doing this, too. So, how could we talk about that? And um and then I I wrote that report for Um Alex Damos started Stanford Internet Observatory a couple months later and then he reached out to me. Uh and again, that was how it became um my job. So entirely accidental.

>> Yeah. Um and and you've been been working on this >> about 12 years now. Yeah. >> Yeah. I was say uh probably as long as anyone. Um which is >> it wasn't really a field then. It was um a lot of us were just like data scientists in a Slack channel saying hey we're seeing this stuff. Are you seeing this stuff? Uh trying to find ways to communicate with each other. Um because I was in San Francisco I did have a lot of friends who worked at the tech companies. And so I was in an interesting position where I could actually have conversations like hey your recommendation engine is pushing this account of mine that I made QAnon content like why is that happening? Um and so so I I think also some of us having technical backgrounds by virtue of being in tech or you know Alex came right out of Facebook um made us much more focused on things like design and incentives as opposed to this content is bad which was you know a conversation that you don't want to be in because that's where you start to get at um you don't want the problem to be the speech you want it to be a lot of you want to be looking at for example the internet research agency a lot of the speech that they put out a lot of the fake content from the Russian troll accounts was just plagiarized content from real American media. So the problem is not the objection is not that they were saying these specific words. The issue is that these were inauthentic actors trying to artificially inflate and amplify and divide society using the affordances that social media platforms had given them. So we wanted to keep the focus really on more of the actors and behaviors as opposed to what is the content saying. We didn't want to be adjudicating the facts or anything along those lines. And that was how we thought about our role in the space.

Yeah, I think it it might be worth kind of early in our conversation taking a step back and looking broadly at that and saying I think you know for folks who might be newer to how social media works especially on the integrity side or trust and safety side. Um can you walk us through like what are the factors? You touched on some of those like the difference between the actors, the platforms and the content and how those interact and kind of how you view them as well as breaking actors maybe into folks who you know entities that create content as well as entities that spread content can be very different.

>> Yep. So um so there was a there's a woman named uh Camille Francois who had been at a company called Graphica at the time who wrote this very nice rubric called actors behaviors content um trying to again because there was no field at the time what are the ways in which we think about doing these investigations um in a rigorous way and her she had also led one of the teams for the Senate Intelligence Committee. We were blinded, so we didn't know that each other were doing it at the time, but she was the other person who um who led one of these one of these projects. And the so the actor looks at um who is the who is the entity behind the content. Uh sometimes it's quite obvious, right? An influencer maybe makes a makes a piece of content. You know exactly who they are. They're real. They're human. They're quite transparently who you are choosing to engage with. Uh sometimes though, you'll see accounts pretending to be something they're not. I don't know how many of you guys are active on X, but do you remember um maybe a couple, it must have been almost maybe a year ago now. Uh they turned on this thing where all of a sudden they turned on location and all of these accounts that were pretending to be right-wing influencers turned out they were like in Pakistan or in Nigeria or, you know, so these are the sorts of things, right? Like that's inauthenticity right there. That's where somebody is pretending to be speaking as a member of a community but is really not a member of that community. And so the platforms treated that as a distinct category of policy violation in a sense. This is inauthentic behavior. The term came to be coordinated inauthentic behavior. As Meta also is sort of trying to figure this out at the time. Um they come up with the rubrics and some of the vocabulary for this stuff. Um behaviors refers to are there efforts to coordinate, right? Are you uh intentionally using automated accounts, for example, to um or bots in this case to make your content look like it has more engagement and more followers? Are you buying fake engagement? Right? You can do that. Turns out um it's actually quite easy. So, are you buying fake engagement to look more popular than you are? Are you um are you using uh certain types of techniques to you know to link a whole lot of accounts together to create a perception that there is a a media entity with multiple properties at work here? So basically the behaviors part speaks to this notion of um manipulating the affordances the platform gives to people for amplification or for reach uh to try to sort of juice your numbers basically. And then the content refers to what is it these accounts are actually saying. And that becomes salient at certain times like around an election. Um the policies that the platforms put in place which say like you can't tell people vote on Wednesday not on Tuesday. You can't tell them that their polling place is closed when it's not. That's voter suppression. Uh you can't delegitimize an election. You can't prematurely claim victory. So they do have some policies that they put out that are specific to certain types of content in certain types of areas. They also had some for hate speech. I never worked on that but that becomes an area and so the platforms begin to carve out and set rules for uh these sorts of things. Most of what I focused on except for during elections and then subsequently during co uh was the focus on sort of state actors and investigations and more of the intersection with design and um and incentives.

So >> yeah and Christian you can speak to you touched on a little bit I guess how this has evolved and I guess both you witnessed as well as your interaction with the platforms you talked about a decade ago you were able to sort of you know ping people and they give you there's like a little bit more agreeable and then seems like there was an era where uh you know when I had my job that there was some effort to sort of correct some of this right there was a huge investment following the 2016 election I'm curious if you can >> I had a really interesting arc on that one I was um I actually got I got like I got profiled in like the New York Times for being like a godfly of the platforms in 2016. It was a sort of a funny um funny thing to get attention for, but um we were we were those of us in the data science community were calling for the hearings that they had in 2017 saying uh you need to account for what happened in public. You need to tell the public what happened. like we had such adversarial relationships. Even Alex and I originally met when I was like yelling at him on Twitter, but the um the we had very adversarial relationships with the platforms in the early days because it was um I think in their defense and I imagine you had this experience. There would often be these like gotcha kind of stories like oh I found one fake account doing a thing and then media would cover it and like kind of blow it up and it would become a whole story of like the platforms are doing these terrible things, right? So there's that. Um, and then there were other instances where we're saying like actually they had this tool called Crowd Tangle. We we sort of found like a little bit of a um something that you know little kind of back door in the kind of Crowdangle API, this guy at Columbia Journalism School. Um, and even as we were being told early on that the Internet Research Agency hadn't gotten much engagement, just $100,000 worth of Facebook ads, we found the engagement data for the actual pages, which were, you know, millions of engagements. And we're like, okay, you guys are talking about the ads over here, but this is what happened on the organic pages. Like, this is what matters. This is such a redirect when you're constantly redirecting media back to that stupid ads number. Let's talk about what happened over here. And so for me a lot of it was like you know sitting with Senator Warner and others and saying like you've got to have hearings like come on make them make them account for what happened. Um, and then when I you at the same time though you can't do an investigation solely on the outside and that is because I can say I think these accounts are behaving in a coordinated way based on a you know set of I can gather a lot of data. I can look at evidence of coordination. You know, I remember we did this investigation into US Pentagon propaganda accounts where like they literally all tweeted on the zero millisecond of the minute, you know, like these sorts of things you would see where you're like, "Okay, come on." You know, or uh sometimes the content is the red flag. Like we saw these accounts in Libya talking about um Shugeli who was I forget his first name now. Um Maxim Shugle maybe. uh who is this sort of Russian Vagner group uh soldier who had been arrested and all of a sudden all these you know Libyans are talking about Suge like no there's no way that's not authentic at all. So that was kind of a red flag and so I started digging into that network to try to to try to sus out could we say conclusively these are Vagner group linked accounts. These are Russia linked accounts pretending to be Libyans. In that particular case um SIO had started then I reached out to Twitter. I reached out to Facebook and I said this is what we think we have. What do you think this is? And the way that relationship changed was it became much more instead of it being adversarial, it became much more of a collaborative process where we were able to say this is what we see on the outside. What do you see on the inside and you are what they started to do and you can talk about this from your perspective. They built these uh these investigations teams internally that would sometimes share data with researchers that they vetted. You know, we had to sign a bunch of papers saying we weren't going to leak the data. we had protections in place to you know to to deal with it. Um but we could do these investigations jointly and that meant that we had the capability to go and do much more um longitudinal or looking across platforms or you know tracing things back to other forms of media and the platforms would do the depth uh the deep investigations into what happened on their platform. And then we would release these reports where they would put out their statements about what they said happened and we would put out an independent research report that we wrote about our understanding of what was in those data sets. And we pushed back occasionally where I remember once we got a Latin American data set and it had a cluster of accounts. This was from Twitter and we were like we think these are authentic activists. We actually don't think these are fake at all and here's the reason why. Um and so you could actually go back and forth with them to do these investigations in a much more holistic way. And I think personally and I and I still stand by this that um you don't want the platforms in my opinion that like unaccountable private power having the ability to do that takeown and not disclose what happened. I think that the fact that there were academics who were doing those investigations and looking at it and writing those public reports was the ideal was an ideal working relationship where there was at least some collaboration and some information exchange even while both entities behaved independently.

Yeah, I think that last piece on the ability to do the investigation, there's not much in the way of oversight is I think what leads to accusations of censorship and con concerns often well-meaning concerns from users from the public around the amount of power that these platforms have over what all of us all of our information ecosystems and I'm curious you'd said before that you you lean more towards how do we design these platforms more effectively and moving away from kind of that we're going to take down speific content, take down specific actors. I'm curious if you can speak to sort of what you've seen that works and what is what drives you in that direction and say, "No, what the the correct solution here is is better design and better incentives and not we're going to set better rules and better enforcement of those rules."

>> So, I my feeling from watching the antivax stuff very early on was that there's always a backfire effect when you took the accounts down, right? It turned it into forbidden knowledge. It turned it into something where like they don't want you to know, right? And and it makes it more interesting. It's, you know, the the term the Stryand effect, right? Okay. So, um, so I was never a a fan of takedowns except in very limited cases, inauthentic accounts. Yes. Like take down the Russians pretending to be Libyans. Like that's a that's an authenticity vertical and I think that is completely worth doing, right? Um, that was like such a bright line as far as I was concerned. it didn't even again as long as you're putting out the data sets and showing your work on like here is why we believe these are Vagner group accounts um you've kind of like checked your boxes there when you get into things like the policies around elections the policies that they had around COVID um around you know bunch of whether hate speech all these other areas that that I didn't work in um you still at least see you want to have some visibility and transparency into what has happened and often times that didn't happen so there would be takedowns where the person doesn't have a right to appeal. Um they don't really know why their stuff has come down. Some AI has taken it down, right? Because you things gradually begin to move in the way of in the realm of like automated moderation, right? So it's not even human review. It's just the you know the the machine god decides that this is bad and like there it goes. Um so I argue very strongly for transparency and for user agency, right? How do you give users more control over that experience? Um, I also really do believe that labeling is, you know, my my personal theory on this, my personal feeling, um, Asa Raskin and I wrote this article in 2018, freedom of speech, not freedom of reach, which was like, leave it up and then think about what you're curating, that it was actually the curation piece that mattered a lot more than what came down. Um, because every piece of content in a feed is ranked. There is no neutral. Once you internalize that, you realize that the platform has at completely at its own discretion the right to decide how it is going to rank. That in itself is an incredible power. So I wrote a lot of arguments that that power should be devolved to users. That users should have much more control over that ability to do that ranking. But that on a centralized platform, the right to curate is the platform's first amendment right. That is what you were there for. That is what you were signing up for. Um, and so the other thing I wrote a lot about was like we need more platforms, right? And I wrote about we need more platforms because the argument was um if you are moderated or deplatformed or whatever um you should have another place to go and there should be communities that are palatable to the rules and values of certain communities that want to speak in certain ways to each other. And this was actually what you started to see, right? You had Parlor begins to emerge. you know, start to see the rise of first it happens on the right and then it happens on the left. So, you start to see the rise of Parlor, Truth Social, and the very right-wing coded um platforms that emerge with their own very specific moderation rules like on Rumble, you can't share Antifa content, right? That is a very specific terms of service that only Rumble has. And I think it's great, right? That that's what you want to see. You want to see the rise and the proliferation of these platforms so people have places to go. Uh and then you start to see after Elon buys X um the diaspora of left-leaning users moving to Blue Sky and Threads and uh Mastadon for a while there. And so again this question of transparency design and and the my personal belief again that the platform is ranking, it is always ranking. It is always curating. Uh and all of that has to be done transparently and with um ideally a right to appeal. I suppose I I can understand the transparency piece around it's great to have that clear what the rules of engagement are on a site clear. I think the lack of clarity is the root cause of most frustration that I saw like directly from users at Facebook of of why actually happened.

>> On the other hand, you know, examples as sort of those left and right wing coded like at what point does that sort of risk sort of pushing us more and more into echo chambers? Oh yeah, that's >> the trade-off in those two >> 100%. So this has always been um this is the question where I'm not convinced that that is a technical problem at this point. And the the intentional effort to reframe any sort of moderation as censorship began on the right in 2018. Um, and that was for a variety of reasons. But, uh, you started to see Donald Trump get, you know, upset about certain decisions that were made. And then by 2020 when his tweets begin to get labeled during the election, labeled, again, labeled. This is before January 6, there has been no takedown. Labeled. Um, you start to see that um that argument that a label is censorship, right? So we begin to see this expansive definition of that term uh where it becomes you know it sort of in my opinion leaves the realm of reality and moves into this you know anything I don't like is censorship at that point. Um so that that's a thing that happens but the you know that sense of these rules aren't for us is constantly reinforced by the influencers and the community members who are like no no come follow me over here. This is where you start to get at can you help people understand a little bit more what are the incentives of the people who are talking to them um how are they making their money why are they you know why is this happening I think the question of how do you bring people back together how do you depolarize them um that's bigger than tech what we're starting to see if you want to think about it from a design standpoint the best arguments are what's called bridgen-based recommenders I don't know how many people here are familiar with that term Um, but it's the idea that since again, no content ranking is neutral, rather than ranking when you open your feed for something that is going to be like rage bait or incredibly weird that you're going to engage with because you're like, "What the hell is this?" You know, um, or something that is uh that is highly sensational, instead of ranking in those terms, you can actually do what's called bridging based ranking where you're saying, "Here is some content that is liked by divergent Publix." So people on the right and people on the left like this content, that's what we're going to prioritize showing. So this allows you to surface content even about um highly, you know, inflammatory issues, abortion maybe or something like that where it's the idea of like disagreement without being disagreeable. You're still surfacing the issue. You're still getting the content out there, but you're not rewarding the rage entrepreneurs by putting them at the top of the feed every time. So instead, what you're trying to do is rethink what that curation model looks like and you just show people content that is not coming from people who are actively monetizing outreach. That's where I think the the best possible depolarization um option is from a design standpoint. But otherwise, I think a lot of it is um is offline at this point. It is, you know, how do you how do you engage your community? I know there's like things like America in one room and other efforts that people here at Stanford work on um that make people recognize kind of humanity as opposed to like that's my side and that's your side because that is way bigger than just a technical problem.

>> Yeah, certainly don't disagree that it's it's uh often pointed at social media is sort of almost the cause of that when obviously it's happening you know in and out of digital spaces. I'm curious. I love the framing that the ability to rank and the ability to create an algorithm is the first amendment right of these platforms. It is also their biggest responsibility. And I I guess I'm curious from your standpoint. >> Do they do they take that responsibility seriously enough? Because it seems often like they are they would say yeah like our first amendment right to do this but almost like in a very defensive reactionary way of like oh you can't regulate us. This is our first amendment right. But then to your point, like nothing is neutral. Yeah. >> And they don't seem to acknowledge that all of the time.

>> This I mean it's an interesting question. I felt like for a while there um there was a sort of period from maybe 2018 to 2022 or so. Um they were constantly putting out these explanations, right? You could read the Facebook policy blog and you would see this is why we're doing this. Uh Twitter during CO was you can actually still see it. The blog is still up. It's remarkably communicative where they're like constantly putting out updates. This is what we're doing. This is what we're doing. This is what we're doing. This is what we're doing. Here are the stats of our enforcement. We've taken down 8,000 posts. You know, this is what we've done. Here's why we've taken them down. So, you do see them. Um, and interestingly, in those I just wrote a paper on this. I read 10 years of platform policy when I was doing this work. They're making moral arguments in every single one of these posts over that period from 2015, 2016 up till about 2022 where they are saying it is our responsibility to do this. It is our responsibility to make sure that people have accurate information about their health. It is our responsibility to make sure people have accurate information about elections. Here is how we balance that with free expression. And this is the undercurrent that is through every single one of the um of the policy shifts and enforcement uh data sets that they release. So they are making that argument using moral language and then in 2022 um when the investigations begin that is almost that is dismantled incredibly quickly and all of a sudden um you don't hear that moral language so much anymore. So was it all like BS? I don't know. I wasn't internal. I was you know always on the outside. Um, but I think where I engaged with them most directly was really on the inauthentic actor stuff, which as I say is like a very clear bright line. I I think even today um you'd be hardressed to to find many people who object to that. Um the election stuff became very very uh very very polarizing and that was because of the 2020 election and the false allegations that it was rigged and stolen. And that was um that was where you started to see the uh the politicization of that and the arguments that when the platforms labeled that content or um you know they they made some high-profile mistakes, right? There was there is the one that everybody talks about. There's Hunterbine's laptop which we didn't work on at SIO just for the record for all the trolls who will watch this later and clip that out of context. Um we didn't work on that. That was out of scope for us. We were only looking at voting related content and Hunter's laptop had nothing to do with voting. Um, but they did throttle that in a very public way. Um, and it was a bad call and they apologized for it, right? They they did respond to it, but you did see in that moment, I think it became clear to people the the power um that they have. Now, that was stricanded to all hell. The idea that nobody knew about Hunter Biden's laptop is complete BS. It was shared on Meta 400,000 times before it was temporarily throttled, just to be clear. So again, with a lot of these things, you have to like the nuance is really um what you need to get to, which is it was a bad call and also it was not what the media circus turned it into. And unfortunately because of the political nature of what is happening because of the political impacts of what they have the power to do um these stories do come to serve as examples of uh you know but look what they could have done but look what this could have been you wouldn't know if they put their thumbs on the scale and that is the uh that is the issue that I think we continue to face.

Yeah, absolutely true that the what is covered in the media is only sort of a glimpse into what's really happening on on sort of both directions of many of these stories. I think you touched on that there was this era that kind of ended in 2022, maybe 2023 >> um where there was a little bit more openness and sort of movement around design. I'm curious if you can you for folks who aren't in this world day-to-day like explain like what has happened in the last couple of years and the impact that's had on your work. we're talking a little about this morning and kind of the impact that that like the reaction and sort of response from these platforms uh sort of you know secondary to all that push back really stemming from the 2020 election and early 2021.

>> Yeah. So in 2022 what happens is the house flips right. Right. So, November 2022, um the House of Representatives slips and um and Jim Jordan gets the gavvel to something that comes we can call shorthand the weaponization committee, right? And um and this is where for months leading up to that, starting in August of uh of 2022, there had been blog posts about our work. And I say blog because it labeled itself a foundation, but it was never even legally registered. It's like one guy with a blog that, you know, calls himself a foundation. This is how propaganda works, right? Um and uh and so this guy and his fake foundation start writing this these stories about how the research we did in 2020 tracking election rumors was a mass censorship campaign and that we had been directed by the government to do it and that the government had used us right over there as a cutout to demand that the platforms take content down. Um now there was no evidence of this but one of the ways that this works is like the narrative gets laundered through more and more outlets uh and they create this sort of surround sound dynamic around it. Even though the allegations all trace back to one blog post the way this is done is it's like this person says a thing uh he goes on this news station just the news was the first one to do this and then every other outlet says just the news is reporting and then the next one says the daily whatever is reporting and then so on and so forth. So it creates this perception that this is a real thing even though it's just echoes of this one blog post. But Jim Jordan gets his gavvel and we were like, "Okay, we're going to be investigated because this momentum had been building since August as they're saying because literally at the bottom of the blog post it would say um a Republican committee with subpoena power needs to investigate these people." It wasn't like anybody was hiding the ball, right? We knew what was going to happen. Um and uh turned out this person had been an election denier who worked in the White House right before he went off and started his blog. And so we get then the um the Twitter files happens and that was a phenomenal exercise in decontextualization where you just take an email, cut it in half, post it to the internet and pretend that the top said something it didn't. And again it it is covered through these these media cycles. Um, and so we get we get our subpoena in March of 2023. And um, and the subpoena requests, the last 10 years of our emails, keep in mind SIO only existed for five, four, three by that point, maybe last 10 years of our emails with social media platforms and the executive branch of the government because they're investigating the Biden censorship regime. And I'm like, who ran the government in 2020? It wasn't the Biden censorship regime. like we were working with the Trump, you know, we worked with the Trump administration, DHS, when we did talk to DHS, those were Trump appointees we were talking to, but whatever. Sure, the Biden censorship regime. So, it was like transparently political from the very second that that subpoena came down, but the platforms all got those subpoenas, too, right? And there is nothing that lawyers do faster when you get a legal request than say like shut up and say nothing. Um and so that was you really the chilling effect was profound. It was like instantaneous. It was just very much a um you know so we were so Stanford is a private university but they also began foying uh every state university that had done anything remotely related to elections also. So just mass foying um because while they were waiting for the subpoena documents and then they started suing us. So, America First Legal Suit us um in Louisiana and then New Civil Liberties Alliance suit us in Texas. And again, you start to see these dynamics where uh they make it a liability to speak to other people. Um your emails will be turned over. Your your communication will be reframed as a cabal. Even though there's no evidence of this, the insinuation is enough. And that's where you start to see um in my opinion, that was the entire point of the endeavor. It was election deniers mad about what we did during the election pushing apart the entities that had collaborated to try to understand and triage rumors in the election and that was the goal and it was a very very effective thing um because I think all of the different entities retreated to their respective corners and said uh that's it you know it's too much of a liability to even speak to each other anymore.

>> Yeah. And I think that's first of all really troubling and you know awful to to hear about and I think one thing that that struck me that echoed back to my time at Facebook is not that we got everything most things right but when we did get it right we failed to make that case publicly or sort of and there's the adage that the the lie flies around the room faster than the truth can keep up and you know I think that's at the heart of a lot of these issues but it but it does feel like there was a failure to even try to like combat that with the with the other side that this was not a cabal that they these were principal decisions that they were made uh not at the behest of any sort of government official or outside entity. They were they were you know deliberate decisions made by the platforms in line with you know pre-released and public and transparent guidelines. Um and and likewise you know in the the civil society orgs that we worked with there was a failure to kind of defend that work and it was so much easier to sort of back off.

>> Yep. So, um I I'm still like writing about this now. Um because I I've made it sort of like a little bit of a project to go after one particular number, which is the idea that Stanford Internet Observatory censored 22 million tweets, which is a staggering number, and it didn't happen. Um, we did this post-election analysis where we pulled uh all of the tweets related to the top 10 most viral stories in most viral rumors in the 2020 election. So, Sharpiegate, Dominion, all these big moments. And Kate Starboard's team at UDub did this data poll. We like write out the methodology in a report that sat on the internet for two years. It's like academic work. Um, and they take that number that we did after the election in December. We did this data poll. Um, and they say, "No, no, no. this they sent 22 million tweets to the platforms and got them labeled and censored uh during the election and that's how they stole the election. Right? And we keep pointing out that there is no basis for this at all. And it's one thing to say it in 2023, but it's another after you have turned over all of your documents under subpoena and there is no evidence of it in the turned over documents. So I have actually started asking them when they when they you know when crazy media outlets write this stuff now I send a request for a correction just to see how they handle it. I'm sorry, but did you know all of my emails were turned over? All of our work product was turned over. You can go look at it. Jim Jordan doesn't even defend this number. Why is it in your publication? Right? And you and you like watch the watch the magic happen and the the ways that they try to move the goalposts about what they did or didn't say regarding that number. But it is such an outrageous, demonstrably false claim that I find it funny actually to to like litigate it with them to like make them explain to me how they are justifying keeping that in their publication. Um, and it's just, you know, at this point I think it's just tribal, right? It's just like we we want to believe and so we're so we're keeping this here. But what we didn't do a good job of was when those when that um when that number was being laid down originally again through that laundering process of like this literally it came from the same guy like this blog says this thing and then everybody repeats it. Um we never requested a correction. Stanford Comm did not believe that requesting a correction was worthwhile. And so it let all of these articles that were written speculatively at the time, we did not request corrections. And that I think was um just an absolutely devastatingly bad decision because it meant that even now I don't know how many times you guys like argue with LLMs about things, but but this is another thing that I write a lot about, right, which is like how do you synthesize reality? because it is uh it is incredible. You know, I I one thing I will say is the LLMs realize that the 22 million number is BS now too. And I'll send that back to them also. Like ask Grock, ask Grock if this is true, you know. Um because even Grock knows it's you know. But the uh but as I as I go through this process, it's like um they work with what they have. And we were sitting there telling election officials, don't let rumors oify, right? You've got to push back. you know, you've got to you've got to say like, "No, the Sharpie markers actually didn't bleed through and blah blah blah and all the details related to the specific election rumor. You can't just let this stand because this is going to be reality, right? If it's viral, it's reality." Uh, and then when it happened to us, we didn't do anything. And it just just it was like wild to watch that. So, these articles um from that time frame just all have this stuff in there and it's like trying to dig out of a of a hole. Um, so I write a lot now about institutional comms and uh and what not to do. So

>> yeah, there's definitely not that it's funny, but almost funny that exactly and you said this earlier, this ended up being a case study in exact social dynamics fight against for years um just with kind of the subject that this time the content was the sort of the referees themselves as it were.

>> Yes. Yes. and and you I think the other reason why you have to engage um I I I really do understand look it is very hard when you're being sued um and I am still being sued right those court cases are still going on um that you don't want to do anything that hurts your defense right um and yet on the flip side there's the matter of where is the court of public opinion when it takes again It's been three years now that those court cases have been going on, right? The the uh Texas one, they just appealed again on um I think April 16th was the deadline. They got it in three days before the deadline. So now we're going to the fifth circuit court of appeals again, you know, and so it'll last another seven months uh at least. Um but as you work through these things and you realize it's going to be three to four years and one of the reasons to file those lawsuits is to shut you up, right? It is to put you in that position where you can't talk. And so then I think that we have to be thinking about this, I've started getting to know um as I've been writing about this from like a what is it what should you do, right? What is the answer to that question?

As I've started talking to uh law firms and PR firms, as I've gone through this kind of investigations process, like you know, postmortem this now, like what, what would you have done differently here? Um, the answer is like new firms or like firms that are much more versed in how the internet works actually do mount um defenses in the court of public opinion. Also, they do find ways to uh to push back. Again, it is like a balancing act, but the idea that you're just going to go silent for four years while some litigation winds its way through the courts um, it, it is like it's so destructive and um, and yet that is that is how it happened.

You saw. So Jim Jordan actually released um, so as the weaponization committee came to a close, as that session of Congress ended, he released 17,000 pages of investigation documents and you can read his four-chapter report, you know, on the Biden censorship regime and then you go into the 12,000-page appendix, which I have read. You throw it in notebook LLM, that's how you want to deal with this, but I actually read it um, 12,000-page transcripts of all the interviews with the tech company executives. So, so he's got 12,000 pages of interviews with the tech company executives saying, "None of this ever happened. We weren't pressured by the government. This is how we made our independent decisions. This is what our policy was. This is how we executed on our policy." None of the 12,000 pages, like they don't reflect the reality of like, or I should say the reality that is in the 12,000 pages is not what you read in the reports. But interestingly, again, speaking of the LLMs, they're like, "Well, this is a report from congress.gov, so we're going to cite it." You know, and and you're and you're like, kind of the Democrats didn't write a counter report, so this is this is reality here, right? And and uh and you're trying to tell um the LLM, but did you look at the 12,000-page appendix that wasn't even indexed? And it's, you know, it's like, no.

So, you're really um, you know, trying to to think through again from an infrastructure and design perspective, what synthesizes reality at this point and how do you, how do you push back?

Yeah. And and my my last question before we'll we'll take some questions for the audience for uh a good amount of time at the end here is, you know, you talked about the purpose of that aimed at you and other journalists and other civil society actors was to to sort of silence. And I'm curious, you know, for those folks who don't necessarily have the inside look at like maybe if what was different, can you explain just the impact that had on the platforms themselves? Like I think there's a cynical take that's, well, the platforms were never doing anything to begin with, so this, so this lawsuit doesn't necessarily shift anything. But I think speaking to whether it's through your relationship or just what you've seen, has that led to a market shift in how the platforms view this question?

This is a really interesting question. So, the platforms got sued also. They were not, we're not codependents in any of the cases, just to be clear. We were sued as de facto agents of the United States government. So, I'm mostly co-defendants with government people, even though I was not paid by a government, nor am I an agent of the government. That's a whole other thing. The platforms were sued um as uh our f, you know, by by people who said our First Amendment rights were violated when you moderated us or took us down. Those cases are tossed constantly, right? They are just not because again, the platform has the First Amendment rights, not the user. This is just American law. This is something where you can you can reframe it and yell about it on social media, but when a court looks at it, the court says, "No, this did not happen. No, this isn't real." The most prominent case was the Murthy v. Missouri case where again the platforms were defendants or that was government. But you did see uh the Supreme Court Amy Coney Barrett, Justice Barrett say um, "There's no evidence here connecting a platform decision to anything the government asked for in relation to any of these defendants. In fact, there's no evidence that the government requested anything related to these defendants. So, we're going to toss this for standing, right? We're going to kick this back down for standing."

So, what happens with the platform court cases? um, they too are constantly getting tossed because the platform has the right if they wanted to say like, "We're not letting you on here if you type the word cat," that would be covered, right? That is their right. They can do that. It's crazy, but they could do it if they wanted to, right? It's just the sort of court of public opinion that would that would prevent that from happening. But so what you start to see is this um, they're winning in the court cases, but they are not willing to stand up and say um, "We made this choice. Here is why we did it. This was the ethics underlying it. Here's where we might have done something differently. Here's where we wouldn't have done something differently." Instead, you just see um, you just see a retreat, I think. And that's the in response to, ironically, political pressure. Um, you know, they argue that there was a lot of political pressure from the left during COVID. Then you see political pressure on the right afterwards. And that is the uh, that's the dynamic there.

Absolutely. Um, I'd love to open for the last 15 minutes to open it to questions from the audience. Folks raise your hand. We'll call on you.

Sure. Yeah.

Hello.

So good to see you.

You too. Congratulations. Amazing book. What do you think is the future of um, the kind of work you were doing of uh, you know, monitoring for u election-related disinformation? The future of um, platform content moderation, which seems to be going radically in the reverse direction. And if I can just add on one more angle to this, we thought we were seeing a massive and virtually unstoppable train wreck coming in terms of um, deep fakes.

Yeah.

And it, it doesn't seem yet to have had the impact on elections >> that we were expecting. Is that, do you think that's just because the technology isn't perfect enough yet, or do you have any thoughts on why it hasn't had a more profound and, you know, determinative effect on election outcomes?

So, I think on the first question, the, it's the collaboration piece that was uh dismantled. So, you do not see researchers talking to state and local election officials anymore. Right? So, the way that our election work worked, um, if a state or local election official saw something, we had an open tip email. They could just send us a, "Hey, can you look at this? Hey, what do you think this is?" Again, that same question because we were like, "Their job is to run elections. It's not to be monitoring like the latest, you know, random account that they think is fake." Sometimes they would think it was Russian. That was like, you know, um, but we would, you know, we would try to look at it and understand it and send them something back saying what it might be or whether it was like a big deal or not. And then every now and then we would um, we would tag in a platform if it did seem to violate policy. Um, so it's the, it's that connection that's not happening. You'll still see like Kate Starboard's team at UDub still does rumor triage, right? They're still talking about this is going viral. Here's how it's going viral. Here's why it's not true. They do pre-bunking work. They try to explain things in terms of tropes and storytelling. Around this phase in the election, you are likely to see X, Y, and Z. Um, but again, it was uh, it is the ability to have multistakeholder conversations that is done. Even by 2022, we actually, our finding in 2020 was that the platforms ignored 65% of the things that we tagged. So we were like, well, okay, tagging is not really the best use of our time. So we're just, you know, so by 2022, we weren't really doing that. Uh, but that was the um, that was the the sort of firsthand, you know, the effort in 2020 was was different. We were kind of trying it out for the first time. Um, I think you have seen people retreat from the space, right? The idea that um, that it is too much of a liability to do election work and um, you know, like when we were still under subpoena, Stanford sent a letter saying that like SIO wouldn't do this in 2024, wouldn't do real-time election rumor research in 2024. So you do see, I think universities getting spooked um when it comes to seem like a massive liability, like if you do this, you're going to get sued. It's going to be a vexatious nuisance lawsuit, but you're going to fight it for millions of dollars over four years, and is the juice worth the squeeze? So that chilling effect is real. And then to your point about AI, um, the technology gets better and better. And so my feeling on this was always, again, in that nuance space, two things are true. One, authentic content or decontextualized real videos are highly persuasive. They they move, you know, they're um, anytime there's like fires in the Brazilian rainforest, someone will grab a forest fire image from India and throw the picture into X and it'll go viral, right? So decontextualized real images are a thing. Um, the question is, how do you help people understand what is real and what is true? And these are not the same thing. And that and generative AI creates a significant degree of distrust. It makes people really doubt what they're seeing. They doubt what's real and they doubt, you know, so so you can actually use it to diminish confidence in the real. Uh, and at the same time, you can use it to create the fake. I think that I see it as an extension. It's just um, another way to create propaganda, another way to you know, to to get content out there. I think we have seen in some elections um, I want to say maybe it was Slovakia, fake leaked audio dropped, you know, 48 hours before the vote that um, there has been some speculation that actually did have some impact. You know, leaked audio of a politician on a hot mic. This is obviously fake. This is a fake audio um, saying, "Oh, I'm, you know, when I get into power, this is what I'm really going to do." That kind of thing. And as people try to figure out if it's real, it really comes down to just what do you trust? Do you trust this speaker? Do you trust this outlet? Uh, and so you, you see that that problem, I think, is um, it is going to continue. I don't think it's going anywhere. But again, it's not um, it's not it's not massive. It's it's not like the biggest thing, but it is also a thing. I will say the the AI question is the question I got most from friends and family while I worked at Meta was, "How bad are first deep fakes and then generative AI? Like, how much will that shift the ecosystem?"

It's a real bipartisan concern. Um, I, I joined um, I joined Bill O'Reilly actually to talk to, he has nothing right to talk to his audience about it. And because because he had a friend who was taken in by a video and he wanted to talk about this and he wanted to talk about this for his audience. And I said, sure, you know, I'd be happy to come on and talk about it. And I got such nice emails from the audience afterwards um talking about how it had impacted them, right? Like my mom was taken in by this scammer, my, you know, a lot of it touches people in their personal lives in the context of spam and scams. But it's really not a partisan concern. Actually, it is quite a bipartisan concern that people are going to be duped. They're not going to know what's real. They're going to be constantly taken in by this stuff. And then you're going to have to try to persuade them that something is true or not true. And how can we find a way maybe to bring people back together around this idea of um, it is a common value, I think, across the political spectrum to want to believe, to to want to have some confidence in what you see in front of you. And I think we should be leaning into that more and trying to emphasize that this is not a partisan issue.

Two more. Um,

Yeah.

I wonder if you looked at all at Wikipedia.

Yes.

And the effect that Wikipedia articles have uh in establishing >> reality >> reality and and also shutting down the communication. I mean, and specifically if Steve Jobs at West would have written a piece on Wikipedia, it would have been banned because he was talking about the company he founded. That is a policy that they have. So you now have shut out a historical voice, right?

So I have written about Wikipedia most recently in the Atlantic when Graedia was launching. Um, uh, this might sound strange, but Graedia does some things very well, right? And uh, and what it does very well is that it crawls and ingests massive quantities of information. And um, I was one of the fortunate few. The the first 800,000 entries included a bio for me, which was like two-thirds normal and then it went into like InfoWars down in the bottom third. And I was like, there we go. Um, but what was very interesting about it is you see the war for reality happening because Wikipedia is such a big piece of training data for AI and it and it really heavily influences the um, uh, Google in particular, their automated summaries, right? So if it's in Wikipedia, it's it's treated as being legitimate. At the same time, it's a lot of entries in Wikipedia are like stubs because who has the time? Like I'm not that interesting, right, for Wikipedia to go write a bio about. Grock did this massive crawl and pull and it was like finding random interviews I'd given where I talked about like, "My dad taught me how to code," you know, and and like that stuff's in there. And I was like, nobody on Wikipedia would ever find that, right? Or if they're like, they should be writing articles about more interesting people. And so the watching like this is what the AI can do. And I actually was kind of fortunate enough. Jimmy Wales had just written a book at the same time, so I got to interview Jimmy Wales about this um, because Wikipedia also doesn't want AI-generated entries, which is an interesting um challenge because one of the things that was fascinating about the Graedia experience is like, you have to plead your case to a robot. You have to try to convince the robot that the InfoWars article about you is crazy, right? And that is a that is a wild exchange, you know. I was like, well, of course Desta would say that, you know, um, whereas on Wikipedia, you can go and try to appeal your case to a human, right? So, so you have these like these different strengths and weaknesses. Um, but for Wikipedia to you know, to be um, competitive going forward, the concern is that the Graedia model is going to come to feel so much more rich and robust whereas relying on human moderators and volunteers to do this stuff because as you say, there's all these rules, right? I can't go edit my page to say this sentence about me is wrong. I have to leave a comment for somebody and hope that the person does it. So it's just very different ways of treating it. And what you, the last piece I'll add to this is that members of Congress then send letters to Wikip Wikipdia Foundation um saying, you know, "We think there's been evidence of suspect editing rings colluding to shut out political perspectives. How do you respond to this? Or your approved source list doesn't include ONN and Newsmax. Why is that? You're biased, right?" You know, and so this is where you start to see those war for reality is happening because that is where that synthesis is happening. And it's very interesting to watch this um to watch this play out. So yeah, it is it is very much at the forefront of this because of that uh that importance in in the synthesis space.

Take one more question.

Yeah. So, focusing on the public trust part of your title, um, you know, autocrats love the lack of public trust in in media. And this, you know, this final sentence of 1984, if I remember right, >> it strikes me that, you know, average citizens are left a wash in this, you know, massive amount of misinformation and they need tools, first of all, to be able to navigate their way through it and also kind education to teach them to be skeptical, you know, you know, to follow the Carl Sagan's uh adjunct of, you know, extraordinary claims require extraordinary uh proof.

So, what tools is an average person going to have access to in order to try to fret their way through the system?

Really not very many at this point, candidly. And that's one of the big problems I think um even that a lot of this you're seeing happen on Reddit right now. People are trying to crowdsource uh answers like the um is this AI sub, right? Uh, where people are asking other humans, "What do you think this is? How can I tell? How can I have access?" They don't have access to models that do detection or uh tools that expose watermarks or anything like that. And there, you know, every now and then though, the the detection tools get it wrong and then it becomes a whole um whole news cycle where, well, the detection tool said that this was or was not AI and it, you know, gets it wrong. And then it becomes the people then have like, well, I saw it, I searched for it. I did that myself. I asked that tool. What do you mean it's not right? And so we've just hit this really bad state, I think, where you might start to see is content provenance come into play a little bit more. This is where I think um, there was an article in the Financial Times that made this point that social media kind of rewarded extremes and AI might be more of a um, reward um, accurate information and a um, it was it was framing it more around the center. I don't think it's necessarily the center. I think it's more of a widespread agreement in, you know, certain things being real or true. And that's because when you have like provenance tools, you'll eventually have a New York Times reporter with a camera that says, "I am at this place. I took this photo and here is every piece of um editing that happened to this photo in the time since it was taken." And on the New York Times, we will display that content credential, and anybody who wants to see it can see it. So it's almost like credentialing in the good stuff as opposed to trying to um constantly detect and correct the bad stuff, if that makes sense. So the opposite side of the coin. Okay. And I think that the result of that will be certain outlets that have that capacity to do that and to say, "We are going to do this credentialing," um will potentially enjoy a higher trust from readers, higher engagement with readers as they become a voice that sort of shows their work through this more um, you know, leverages the technology appropriately. It's going to be a weird intermediate period though because you're going to have parts of the world that don't have access to those tools, don't aren't using the latest iPhone, don't have the latest Canon camera with this credentialing uh, you know, protocol built in. And then you're going to have that challenge of like, well, this person in this remote area of this conflict zone says they took that photo and just because they're not a New York Times photographer, can we really discount it? So it's it's going to be, I think, uh, for a while, a very uncomfortable intermediate period as people are going to make their decisions in part based on what are their communities of trust tell them that they can believe. I wish I had a better answer for you, but

I think that's more realistic.

Unfortunate to leave it there. We do have to to join me in thanking Renee for coming in.