Transcription
[NICOLAS] Hello and welcome to this presentation of our paper on Ethical Mining for the 17th International Conference on Mining Software Repositories. My name is Nicolas Gold.
[JENS] And I am Jens Krinke, and we work together at CREST in the Computer Science Department at University College London.
[NICOLAS] In our talk, we're going to present our recent work looking at the kind of ethical issues that may come up when studying software repositories. We will consider these through the lens of a fairly recent ethics framework for ICT research, namely the Menlo Report, and look at the kinds of data that are often used in MSR research and how the two interact. We also make some suggestions for raising the profile of ethics in MSR research.
Given the nature of the subject here and its interaction with legal matters, we feel we should start with a disclaimer, simply to say that neither of us are trained or qualified in law and that nothing we say or present here should be construed as or used as legal advice. Our talk and paper are based on our best understanding of the legal and ethical environments at the time they were prepared and you should consult your own legal advisors and ethics oversight bodies for the current position if and when this becomes relevant to you.
[JENS] Ethics issues have become a topic of interest in the MSR community. At last year's MSR in Montreal we had a session on ethical MSR and this year we had a tutorial on privacy. There are two reasons for this raised interest: First, there is the recent data protection legislation like the GDPR or the California Consumer Privacy Act of 2018. Second, there have been a few issues identified in MSR work. For example, email addresses have been removed from GHTorrent in 2016 and age information has been removed from Stack Overflow's public data as part of an audit for the GDPR. Moreover, Baltes and Diehl report on GitHub developers getting annoyed by the number of survey invitations and questionnaires they are receiving from researchers.
[NICOLAS] At the outset we want to establish why we think it is important to have this conversation as a community, and to set out the position from which Jens and I have approached what we're doing here. In particular we want to make clear that our intention is not to critique researchers or research that has already happened. We have worked on the basis that all the necessary research ethics issues for previous work were considered at the time, according the community's ethical standards as they were then, and as they were understood at that time. We also worked on the basis that all appropriate permissions were obtained according to whatever oversight frameworks applied to the context in which the research was done.
In looking back at the previous MSR literature, we did so simply to verify our hypothesis that ethics is not often talked about in publications in this area, and to identify systematically the kinds of data that have been used, to inform and structure our subsequent discussion. Research ethics is not a static thing. It reflects the changing priorities and values of the societies that the research serves, and changing views on the burdens of research placed on individual members of those societies who participate in that research. The ethical standards that apply to a piece of research are at least partly a function of the time at which that research was done. The procedures that apply to oversight of that research express those standards operationally and this is itself a function of context as procedures vary between institutions and nations.
Ethical standards emerge through rich ongoing implicit and explicit conversations between and among researchers, research communities, oversight bodies, and wider society. Therefore, we believe it's important that research communities are self-reflective, responsive, transparent and aware of current concerns, so that they can participate in that conversation and evaluate and modify their current practices if necessary.
Over the years there has been discussion of ethical empirical software engineering research and its associated frameworks. For example, Vinson and Singer's practical guide to ethical research involving humans in software engineering. The Menlo Report is a recent research ethics framework that explains various properties of ICT research that make it unlike many other fields. Among other things it identifies the speed, scale, coupling, de-centralisation, and opacity of ICT as creating challenges for ICT research. ICT researchers now regularly engage with sociological and cultural research questions, often in the contexts of data mining, data science, and AI. As such they need to address research ethics issues that do not traditionally arise in the development of new algorithms and other technologies.
The Menlo Report sets out four principles for the consideration of ethics: Respect for Persons, Beneficence, Justice, Respect for Law and Public Interest. Respect for Persons involves factors such as ensuring voluntary participation following informed consent, respecting individuals as autonomous and able to determine their own best interests, respecting individuals impacted by but not the targets of, research, and protecting those with limited autonomy. Beneficence means ensuring that the research maximises benefit, minimises harm, and that this is systematically assessed. Justice involves considering each individual equally, fairly distributing the benefits of research, considering need, effort, societal contribution and merit, fairly selecting subjects and allocating the burdens of research equitably. Finally, respect for law and public interest involves undertaking due diligence, having transparency in methods and results, and having accountability for the research.
[JENS] As we stated at the outset, our intention is not to revisit previous work from an ethics perspective. But it's important to ground our discussion in the context of the kinds of data used in mining software repositories. Trying to identify everything ever used everywhere would be a near impossible task so we simply looked at the data used in the MSR mining challenges of the past ten years. This indicated six types: Version Control Data, Issue Tracker Data, Mail Archives, Stack Overflow data, Build Logs and IDE Events. Our paper looks at all six types, but in this presentation, we don't have enough time to cover them all, so we will focus only on two: version control data and Stack Overflow data.
[NICOLAS] VCS data is a good example of data gathered for one purpose but used for another. It's gathered for the purpose of software development, but may be mined for a variety of reasons from technical quality through to aspects of developers themselves. A range of ethics issues can be raised by this kind of secondary data analysis but perhaps the most common is that of informed consent. When we talk about informed consent here, we are referring to ethical consent rather than consent as a lawful basis for data processing under the GDPR. There are differences between these two types of consent and their implications. Consent as a lawful basis for data processing is not always required as there are other bases that can be used in some circumstances, but addressing ethical consent would normally require consideration in all circumstances, even if only to justify why it will not be sought.
Informed consent is typically interpreted to mean an active consent: in other words, a participant is given information about the proposed research, what will happen to them and their data during and after the study, any foreseeable risks or benefits, and their options to withdraw. They then make a decision to participate or not. Ethics committees and institutional review boards usually examine the quality of the information to be provided to ensure that participants have the full opportunity to assess their potential participation.
Secondary data analysis in datasets like software repositories poses challenges to this well-understood consent process. The data exists but its ethical provenance may be unclear, the original participants are probably unreachable, and we may not know what permissions they gave for future research if any. In resolving this, it is perhaps helpful to look at the principles underlying informed consent. The key ethical principle of concern is autonomy: respecting the fact that those whose data we process as researchers are autonomous moral agents and should be treated in such a way that their agency over their data is retained. That agency extends not just to the disposition of the data in terms of whether it can be accessed, but also what is done with it.
A common approach to this in secondary data analysis is to rely on the terms under which data is made available, and the terms under which it was gathered, as evidence that those contributing data were fully aware of what would happen to their data. There are circumstances where that may be the case but such arguments should be treated with care. There is research showing that users' perceptions of private and public spaces and activity online are unclear, that users do not read or understand the terms and conditions when signing up to a site, and therefore that their expectations in relation to the use of their data may be varied. We are not arguing that terms and conditions can never be used as part of an ethical defence, simply that such arguments need to be treated with care and consideration needs to be given to the likely expectations of users, drawing on various evidences to argue for setting aside direct informed consent.
In terms of ethics issues in data analysis itself, it's harder to identify common issues in general because one cannot predict future research. In the VCS context there may be potential risk to participants in terms of privacy or to personal or professional reputation. The open nature of software repositories makes it hard to fully anonymise studies because code itself can be resolved to particular commits and developers. Other risks may arise from aggregating data sets across platforms, raising potential ethical risks if this allows profile building of individuals.
[JENS] Stack Overflow is the go-to Q&A website for programmers and as the data contained is very rich, it has been used for all kinds of research. This has been made easy as Stack Exchange, the organisation behind, even provides official dumps of the Stack Overflow data. Using the Stack Overflow data in research raises similar ethics considerations to research in other areas using secondary data from websites. When users register with Stack Overflow, they are referred to the Terms of Service, which explicitly states that by registering one agrees to make all content available under Creative Commons license terms, including regular dumps of the content. The clear and explicit licence gives strong support to an ethical defence for data use in general.
Although consent to the creation and sharing of the dataset has been given, this does not imply that informed consent has been given to any and all research using the data. There may be particular risks that require explicit consent. The creators of the Stack Overflow dump ensured to not accidentally release personally identifying information, but there are risks to the users posting content: Many users include personal data like their real name, their website, their location, or their GitHub username in their public profile which makes them identifiable. One example of research that needs to take this into consideration is research that aims at observing and analysing user behaviour. Such research needs to protect users from the risks of revealing behaviour that could negatively affect them, personally or professionally. Stack Overflow does not reveal data about gender and race but does contain location data which should not be used to target specific persons or groups. Until 2018 the data dump contained the age of the user, but as part of an audit for GDPR it was removed from public data as it is considered sensitive information. But as a consequence, it is not possible to exclude minors from participation and having minors in the group of Stack Overflow users usually requires additional ethics procedures.
We have only looked at two examples of data that have been used in the MSR mining challenges, but already seen a large number of ethics issues. One particular issue is that all the challenges asked researchers to use third-party provided data. This leads to a dilemma: Should they trust that the data has been collected for the purpose of their intended use in an ethical way, or do they need to try and verify this for themselves? Even when assuming that the data has been collected in an ethical way, the intended research using the provided data needs to also consider and balance ethics issues again. It is important to distinguish between the data collection and the data usage, as most ethical issues will arise at the usage level. For example, I created a collection of code review data and released it publicly. The creation of the dataset did not require full review through our institution's Research Ethics Committee, but if we would use it for profiling reviewers' productivity, we would certainly require full review. We are also more and more encouraged to provide replication packages or other research artefacts. However, we have to be careful to not expose identifiable data in them. For example, a paper may carefully not use commit ids in the discussion to protect developers, the artefact may still contain the ids, which could be an ethics issue.
There are widespread ethics issues in MSR research and one has to assume that review through Research Ethics Committees or Boards will often be necessary. Preparing such an application for review and getting approval takes time and can be complex. For example, I had to abandon the idea of submitting to this year's mining challenges as it was impossible to gain approval in time. We have raised many issues in ethical mining of software repositories. Our takeaway message should not be that these are issues making mining software repositories unethical as we strongly believe that our community acts ethically and rises to the challenges. However, we conclude with a few suggestions to the MSR community that can help future researchers and our suggestions focus on raising awareness and openly discussing the ethics considerations of our datasets and our research. While we often discuss threats to validity in detail in papers, we should also discuss ethics issues and risks. For example, the Empirical Software Engineering journal already has a policy that authors should include a section on "Compliance with Ethical Standard". We have to incentify authors to discuss ethics considerations, for example by allowing such a discussion outside the page limit. Moreover, future authors of dataset papers could help future users of those datasets by providing a detailed discussion of ethics considerations in the collection of data and its potential applications in research.
[NICOLAS] Thank you for watching this presentation. We hope that we have discussed these issues in a helpful way. Of course, ours is only one perspective and we are interested to understand the community's views and to return this information in the form of further publications. So we would like to invite you to participate in a short anonymous survey about data and ethics in MSR. The link and QR code link to the survey is on the slide and you will find a full information sheet there with all the details. We look forward to discussing these matters with you in the Q+A session and subsequently.