Recorded on Tuesday, May 12, 2025
TRANSCRIPT
Daniela Kucz ’14 All right. Welcome everyone. My name is Daniela Kucz. I'm class of 2014 and I am a member of the Alumni Council, which organizes the SwatTalks initiative to spotlight alumni excelling in their fields. For this evening's SwatTalk, I'm very excited to welcome Rachel Thomas, Swarthmore class of 05, for her talk titled “Artificial Intelligence and Medicine: Promise and Peril”.
Rachel Thomas is a pioneering AI researcher. Through her work co-founding fast AI in 2016, she helped create the longest running deep learning course in the world and was recognized by Forbes as one of 20 Incredible Women in AI. She was also the founding director of the center for Applied Data Ethics at the University of San Francisco, was an early data scientist at Uber, and received a PhD in mathematics from Duke. Now Rachel is back in school pursuing a PhD in microbiology with the goal of applying her machine learning and data ethics expertise to the fields. Today, Rachel is joining us to discuss the history of AI, the exciting opportunities for AI in medicine, the risks of what can go wrong and the positives that occur when people from unlikely backgrounds get involved. With that, I will now turn it over to you, Rachel.
Rachel Thomas ’05 Sorry, I was in the midst of sharing my screen. Thank you. Thank you, Daniela, for that introduction. All right. Okay, here we go. Yeah, “AI and Medicine: Promise and Peril”. And I think this is an area where there is a lot of really exciting work happening, as well as some, kind of very serious, risk that we need to consider. Daniela already covered my background that have had a kind of winding career journey, from mathematics through working in the tech industry, to really focusing on deep learning, which is a family of AI algorithms and all kind of defined more what those are in a moment, as well as data ethics of written three chapter book chapters on data ethics. And then in the past few years, getting really fascinated with microbiology and immunology. If the topics of today are of interest to you, I blog about all of these, on my blog Rachel.fast.ai and that includes links to lots of references and other helpful resources on these topics.
All right. So I want to share some of the discoveries that I think are exciting and areas of interest in AI and medicine to keep an eye on. And one area is drug discovery in drug design. And so deep learning is being used to identify kind of promising drug candidates, whether that is from repurposed drugs, screening them or designing new drugs in these images from photo using or from a paper that was using deep learning to look at repurposed drug candidates. And this is something where there's just such a large chemical landscape of possible drugs that it's very helpful to have automated tools for, kind of screening, which are the most, most promising ones to look into further. And this is a kind of a big issue, particularly with antibiotics, with the rise of anti antimicrobial resistance and the need for, for new classes of antibiotics.
Another huge area in AI in medicine is detecting diseases from medical images. This is a paper that was in nature two years ago on what can we tell from a retinal image? And so here, the researchers have trained an algorithm that could recognize stroke, Parkinson's disease, heart attack from these images of the back of the eye. And so this is a really fascinating area as well. And then another, another huge area is, 3D protein structure prediction. So on the left this is hemoglobin written as a sequence of amino acids. And amino acids are the building blocks of proteins. And in many cases it's more, more informative to know what that looks like as a 3D structure. And so the information on the right is the, the same as the left, only with the, the 3D structure of what these amino acids look like when the protein is folded. And in the past few years, the computer program AlphaFold has really made international headlines for smashing previous records of how accurate this can be done computationally. And in fact, the creators of AlphaFold won a Nobel Prize in chemistry last year. And so this is quite unusual for the creators of an AI algorithm to win a Nobel Prize. But just because of the kind of incredible accuracy of AlphaFold. And I should also give partial credit, or credit to - this was possible because of a really rich data source, the protein databank, keeping track of well known 3D structures for proteins. But I'll talk more about the relationship with data in a moment.
And in terms of medical applications of this problem of protein folding, one that I find particularly interesting is trying to think about what a T cell will bind to. And T cells are a really crucial part of the adaptive immune system. And so kind of this question of what proteins they're likely to bind to has a lot of therapeutic implications, including, trying to design better cancer vaccines. So this is a bit about why you might be excited about AI in medicine.
And now I kind of want to explain a little bit more about what AI and what machine learning are. So the term and I should say AI is a pretty, pretty wide umbrella. I'll be focusing on a particular family of machine learning algorithms called deep learning. But the term machine learning goes back to 1959. An IBM researcher named Arthur Samuel defined it in a paper. And Arthur Samuel had this idea that he did not want to teach a computer how to play checkers. He wanted to teach a computer how to learn to play checkers. And so this is a kind of a big distinction. So non machine learning rules based approaches are about handcrafting relevant rules and so that could look like coding - you know these are the rules of checkers, you know your piece can jump diagonally over another piece. For protein folding that would be you know, here's a score about hydrogen bonds, you know, since we know this from the biochemistry. Whereas machine learning is all about learning from previous data and so that would be a bunch of old checkers games or a bunch of previous, protein structures that are known. And Arthur Samuel wrote in his, in his paper on this that rather than specifying methods in quote “minute and exact detail that computers could instead learn from data”. And so this is a really powerful idea that's been around for, for quite a while.
And it's really important to understand machine learning algorithms are all about matching inputs to outputs. And so when you hear about an advance in machine learning, or if somebody is trying to sell you an AI based product and there's a lot of snake oil out there. And so it's really important to ask what are the inputs and the outputs that this was trained with. The quality and nature of the input data really matters. And so in many cases, people are making claims that aren't possible, based on the data that they are working with. And, this is kind of a key first step to trying to understand any claims or products about AI.
So for deep learning, there's a famous example from the 1980s, and Professor Yann LeCune developed an algorithm together with a team to automatically identify zip codes for the U.S. Postal Service. And so on the left, we have some examples of how people hand write different digits. And notice there are a lot of different ways to write each digit. Lots of people have messy handwriting, and there are various variation lines. On the right, this is just a single digit eight that's been written kind of in a crooked or slanted way. But it is, you can see how that can just be encoded as this, this matrix of numbers. And Yann LeCun used a multilayered neural network. And so a multilayer neural network is, those are what are now called deep learning. And this was implemented by the U.S. Postal Service and was useful for them to kind of have machine sort, sort the letters by zip code.
And so even though the idea of neural networks has been around for decades, we've really kind of only seen this explosion and popularization of it, particularly since 2012, so kind of in the last decade or a bit more. And so kind of why is that? Why now? And there are three key ingredients that have come together. One is the existence of these large data sets and so ImageNet was a collection and a competition of images that really spurred on computer vision research to have these available, kind of as I mentioned, the protein Data Bank, that there was also competition dating back to the 90s about protein structure prediction and that's what spurred on AlphaFold, in 2020. So kind of having the existence of these data sets. Second key ingredient is the existence of the type of computers that we need to do these matrix calculations. And because it's so multi-layer neural networks that involve a lot of linear algebra, a lot of matrix calculations. And that's actually the same type of computation that's needed to render video games. And so the video gaming industry has done a lot in the development of graphics processing units and that's kind of the same, same technology needed for deep learning. And so this is a picture of an NVIDIA GPU, but that's been really crucial. And then the third ingredient is that there have been some algorithmic breakthroughs that make these easier and more feasible to train. This is a picture from AlexNet, which won the 2012 ImageNet competition and that was a real kind of key moment in terms of popularizing deep learning. But these algorithmic breakthroughs have been significant as well. And so that's, that's kind of why we're, we're seeing such, kind of such a decade of all these, advances in deep learning and all, most of most of the claims that you hear now about AI or most news stories about AI breakthroughs are based on deep learning.
So I saw that this was exciting and really interesting. And I personally found it, as I was living in San Francisco in 2012 and found it very difficult to get practical information about how to code deep learning. At the time, it was really you kind of had to have done your PhD with just this small handful of advisors, and then kind of only the theoretical math was being published, but not the practical information you needed to code. And so in 2016, together with Jeremy Howard, we founded Fast AI. And our goal was to make deep learning more accessible and easier to use. And we were particularly interested in people with little datasets and limited resources, unusual domains and nonstandard backgrounds. And so we really kind of wanted to get this to a broader and more diverse audience. And we did this through, we have a very long running course, practical deep learning for coders, that's available for free online and we also taught in person versions, as well as our software library, the fast.ai library. But really, really taking a different perspective than the major tech companies were, which were kind of just focused on, I don't know, almost assuming that people have infinite resources. And we really felt the opposite of, you know, most people have limited resources and how can we do this and also that the value is going to be and applying deep learning to these different domains. And to do that you really need a deep domain expertise. And that it's important to, to take domain expertise, seriously and that we wanted experts from other domains to be able to learn this technology. When I referred to our students and most people who took the course were, working professionals, kind of doing this part time.
And what we were doing with state of the art. So a team of our students beat Google and Intel in a competition in 2018 that was hosted by Stanford. And, a key component of this is that constraints can actually drive innovation. And I think that the temptation at the big companies is often to just kind of throw, throw more money, more computers at a problem, as opposed to always thinking creatively of how, how can you do this with limited resources? And that can actually actually be a strength, to think about, having to get creative. And so I'm really, really proud of our students for this. And this is something that they did through kind of doing some things more creative algorithmically and not just being like, oh, let's just get even more GPUs and even more GPUs. And so, I think that shows that limited resources can still, still produce state of the art results.
And then I'll just mention one of our students in detail to, to give an example, but pictured here at the bottom right is Corey Spencer, and he is a Canadian dairy farmer, who raises goats and he used deep learning to identify goat udder infections earlier. And this was an issue because by the time udder infection is visible to the human eye, it may have already caused irreparable damage to the goat udder and so, using heat sensors, together with deep learning, he was able to identify these infections sooner. And I just love this application because it's something I never, ever would have thought of and it really illustrates to me that kind of everybody, you know, you all know about problems no one else knows about and it really is so valuable to get people from other, you know, a variety of domains and unexpected domains who can understand this technology for themselves.
So a question that I have received in various forms over the years and still often here is, well, you know, isn't it dangerous to make AI accessible to more people or, you know, don't you worry about bad actors or this falling into the wrong hands. And I think it's, it's usually not intentional, but there's often there's kind of an implicit assumption here that, oh, you know, as long as it's just like billion dollar tech companies and, you know, kind of, very elite people that have access to this, they're going to be more ethical. And that's, that's not the case and that'll actually show up in some of the ethical risk and harms that I'll, I'll speak about in a moment. But I really think having a homogeneous group create tech that impacts the entire world is even more dangerous. And that, having this be limited to kind of elite billion dollar corporations is not, does not make a safer world for anyone.
But that said, there is a lot that can go wrong and that has gone wrong and people are already being harmed by algorithmic systems. And as I said before, to me it's not directly linked to access because in many cases and examples, I'll show, the people creating these systems actually were quite powerful and, and elite.
So I spent, I spent years kind of studying how things are going wrong, how are people being harmed? And this is something I did a lot of researching and writing about and a kind of a broad pattern that emerges about what I see as many of the most immediate harms are that machine learning can have the effect of centralizing power and it does this for several reasons. And I'm going to give some concrete examples of this in a moment. But one is it can be used at a massive scale cheaply. The second is that it's often implemented with no system for recourse and no way to identify mistakes. And that can be related to the first and, you know, if you're doing something as a cost cutting measure, you know, you may not be interested in inviting mistakes because providing avenues for actionable, actionable recourse can seem, you know, more expensive in the short term. It can be used to evade responsibility; it's making more complex systems. Danah Boyd has described this as, machine learning extends bureaucracy and kind of gives yet another node for people to point fingers to and to, to not take responsibility for the outcomes of the systems that impact us. It can create feedback loops. And I'll go into more detail on that in a moment. And it can amplify not just encode bias. And so I've had people ask me, you know, like, oh, humans are biased, so why is it a big deal if machine learning also produces biased results? But there are multiple papers showing that machine learning systems don't just encode existing bias, but in some cases can also amplify it.
So an example that I return to a lot because I think it illustrates possibly all of these properties. It comes from an algorithm that's used to determine home health care benefits in many U.S. states. And when it was implemented in, in one particular state, there was a bug in the code that incorrectly cut care for patients with cerebral palsy. And so this is from a Verge article. This photo is of a woman named Tammy Dobbs who has cerebral palsy and she received a drastic cut in her care due to this, the software bug that really impacted her quality of life. And she couldn't even get an explanation for why this had happened. But she didn't just need an explanation she needed actionable recourse. Like she needed her care reinstated. And this is a very common system where algorithmic systems are implemented with no way to identify and address mistakes. And that can be made even worse by biases that cause people to think that computers don't make errors or to assume, you know, a computer must be more accurate or impartial than a human. And this eventually was discovered through a lengthy court case, but that's not an ideal way at all. And, you know, and it turns out everybody with cerebral palsy in the system had been going without care that they should have received until the court case revealed the error. And this was also a case where nobody took responsibility. So the creator of the algorithm who's, you know, it's a private company, he's earning royalties from it, and then he contracts with state governments, you know, he blamed the state policymakers, the policymakers are able to implement the software engineers that implemented it. And there's no sense of responsibility for the outcomes. And this, this is continuing. So, that was an article from 2018, in 2023, Stat News did an investigation on how Medicare Advantage is using AI to, in many cases, incorrectly cut care for seniors in need. And this is a huge problem because the people that are losing care under this, in some cases, by the time they go through a month-long appeal process, their health has deteriorated further and they may have even died. And the Stat News where these journalists were nominated for a Pulitzer Prize for this investigation. But this is a serious, serious issue.
And this is not unique to the US. Where I live in Australia, we have had a significant scandal with Robodebt. And so Robodebt was a computer system used by the government to automatically issue debts to people that it claimed had been overpaid welfare benefits. And so these people were poor to begin with and then they’re issued debts, which were later shown to be unlawful, they were based on a narronious calculation. But people did not have a straightforward way to contest these debts. They were asked for a very unreasonable amount of historical documentation. And this ruined lives and this led to suicides. And it was something that the implementation of this, really, illustrates this principle of scale. Prior to when the computer program was implemented, the government used to go through this process, but they issued 20,000 deaths a year. And then they used this computer program for a 50 x scale up to start issuing 50 or, sorry, 20,000 debts per week. And so it was this kind of scaling, this terrible and narronious process.
And this is, sadly, there were lots of similar examples. I won't get into this here, but if you're interested, Human Rights Watch did a really good report on how the EU is dealing with these AI issues and some very disturbing cases, from the EU as well. So this is kind of a common problem, these types of systems.
And so now I want to dive into feedback loops a bit more. So this is a paper by another Swarthmore alun Sorelle Friedler about feedback loops and predictive policing. And so I've written about this a little bit too, with the reliance on metrics being kind of both this strength but also this real challenge in AI. But the idea is that algorithms don't just make predictions, they determine outcomes. And I think people often forget this and kind of focus on this, this predictive aspect. And so the way this works or can work in predictive policing is, police departments, in some cases, are using software to tell them, you know, we think more crimes will happen in this neighborhood and fewer crimes and Y neighborhood, so you should send more officers to X neighborhood. And then the issue with that is if there are a lot more police officers in a particular neighborhood, they're going to make more arrests there than, a neighborhood that they haven't gone to. That data gets fed back into the model and so then the model might further amplify its predictions and say, wow, you're making a lot of arrests in that neighborhood you should send even more police officers there. And so this is very, very concerning how you can create these feedback loops. And this is the same, the same mechanism that is at work in recommendation systems. So when you log on to YouTube or really any social media or media platform and it says, ‘hey, you know, we think you would enjoy watching this video’, the YouTube recommender algorithm has learned that, hey, if I show people conspiracy theory and sorry, I shouldn't, anthropomorphize it like this, but when people are shown conspiracy theory videos, then that might, if they get hooked on conspiracy theories, it makes them want to watch even more conspiracy theory videos. And so it's not just predicting, hey, this person will like a conspiracy theory video, it can create the outcome of now I've exposed people to all these conspiracy theories, and they have this demand to see even more conspiracy theories. And so this is a real issue with, with AI systems, this issue of feedback loops.
So just to kind of return to those points in this pattern of how machine learning can centralize power and it can be used at, sorry, I just saw the time, we need to speed up a bit. I did practice this, in my timing, but it can be used at a massive scale cheaply, it can be implemented with no system for recourse and no way to identify mistakes, be used to evade responsibility, to create feedback loops and to amplify not just encode bias. And many of the examples I shared illustrated multiple points from there.
And so moving back to medicine - this really concerns me, because these risks that are seen, not just risks but existing harms that are a pattern in machine learning systems, mirror existing shortcomings in the medical system. Disempowerment is already a problem in medicine. And I could make it worse. And so I just want to talk about the medical system briefly.
So this is a case, there is a woman, neurologist, doctor Eileen. She's a MD, PhD, and she developed a brain tumor that was seven centimeters large, and she was experiencing disturbing neurological symptoms. This also happens to be within her area of expertise, she's a neurologist. And yet, when she was going to the doctor, she went to multiple doctors, she said she was told that she knew too much. She was working too hard. She was stressed out. She was anxious. In other words, she was having her symptoms dismissed. She was being incorrectly, diagnosed or misdiagnosed and dismissed as kind of, oh, this is just anxiety. And unfortunately, her or her experience is not unique at all. The article I've shown here covers a number of such experiences, but there are many articles like this, as well as a lot of research studies that back this up. 1 in 3 patients with brain tumors had to visit doctors at least five times before receiving an accurate diagnosis. And so this is something to always keep in mind with AI breakthroughs. The idea of, you know, AI that reads an MRI more accurately is not going to help, patient whose doctor won't order an MRI in the first place and, you know, in her case, the brain tumor was actually large and obvious that it was just she couldn't couldn't actually get an MRI. And this doesn't just apply to brain tumors. There is a host of research on racial and gender biases in medicine. And this includes diagnostic delays for cancer, disparities in how much pain pain medication is given to people of color, including children of color, gender biases and, again, diagnostic delays in inflammatory bowel disease and a variety of other issues. And so keep in mind that this is all in the data set and this is the data that is used to train algorithms including these misdiagnoses and kind of incorrect, incorrect and biased information.
And while bias is a huge problem, I think that the root of the issue goes much deeper. And really, I think that it fundamentally comes down to kind of notions of patient expertise and medical authority and the way that, often it's kind of seen as, you know, the, the medical provider is the authority, they are the arbiter, kind of determining what is, what is factual and not and ignores kind of all the expertise that patients have and that expertise takes, kind of many, many different forms. But I think it's a fundamental way that the medical system is often kind of blocked from knowledge. And it really mirrors, kind of a lot of the problems we see in machine learning systems when they break down is kind of whose expertise wasn't included. you know, going back to this case, with the algorithm that incorrectly cut care for people with cerebral palsy, those are the people that saw the problem first, the people that were most impacted. But their expertise was not included or listened to in any way and there's, I can give some references later. There's a lot of research looking at kind of when, when machine learning systems fail, why is that? And this is, this is a key reason.
So just kind of to wrap up before our Q&A, which I'm looking forward to. A few things that make me feel hopeful, one is so just coming from the machine learning side, one of the, you know, probably best event I've attended in the past five years was the Participatory Approaches to Machine Learning workshop at ICML, which is a major, major machine learning conference and the organizers at this workshop called for more democratic, cooperative and participatory systems and recognize that even a lot of notions out there of kind of, fair machine learning still kind of give all the power to the developer of the system and really thinking like, what does that look like for it to be more democratic or collaborative. And all the, I should say, all the talks and papers for that are available online, if that's something of interest to you.
A great article, kind of on a similar theme, was “Don't ask if AI is good or fair, ask how it shifts power”, and I think that's really important too, of thinking, you know, there's been, a focus sometimes on, you know, AI for good or, you know, fair machine learning. And while those have their place, I think that those can, in some cases, kind of wash over this kind of central issue of who has more or less power, with the implementation of an AI system. And so I really think this, this reframing is very helpful.
So that's something, yes, it is helpful, helpful for me in the area of machine learning. And then kind of what's helpful for me in medicine is, looking at examples of patient-led research and so this is a paper that came out late last year that I thought was really interesting. And this is written by the lead author, has long Covid and also has a doctorate in pharmacology and so is very knowledgeable about medications and supplements. And she conducted a survey of, I believe, 4000 patients, who had tried over 150 different types of medication supplements and other treatments. She clustered them kind of in terms of subtypes, clustering of symptoms and the effectiveness of the treatments they've tried and it was really fascinating and it was something that, seemed like it was really informed from her experiences as a patient and as someone very active in patient communities of even knowing kind of the what are the different things that seem promising and that are that are being tried. And so I think that that type of patient led research is really, really interesting.
In addition to that, there's a patient-led research collaborative, which again, I think is putting out some interesting work. And so those are those avenues that I am curious and hopeful about, to see more of, of how we can get to a kind of more, participatory and more patient-led approach to AI in medicine. Just to address these, kind of these existing risks. And from there, we can head into the Q&A.
Daniela Kucz ’14 Fantastic. Thank you so much, Rachel. That was fascinating. As a reminder, if you have a question, please use the Q&A feature to ask it and we will be selecting folks' questions from there. The first one, is on the topic of the patient experience. How can we use the patient experience to better train AI systems in healthcare, given that these incidents of misdiagnosis or systemic biases are not insignificant? Are there, you know, historical scenarios that can inform medical expertise that can be then included in training sets? Are there other means beyond the workshops you described? Would love to hear more about your perspective on how institutions can better incorporate them?
Rachel Thomas ’05 This is such an important question. This is something I think we're still kind of very much at the early stages of, yes, how do we do this? Because, someone told me the key issue in machine learning is really just gathering data is such a challenge. And gathering kind of the quality of data you want and like, a big area where things go wrong is when an algorithm is trained on one data set and then deployed on a different data set with different properties. And so I'm not saying it's not out there, I haven't heard examples where I'm particularly like, oh wow, that's what everyone should be doing around this, collecting patients, the like the patient perspective and the patient expertise. But it is definitely something I plan to spend the next decade thinking about at least. And it's also an area where we need more people working on this, I think.
Daniela Kucz ’14 A slightly different topic, but, and maybe you could describe briefly what some of these concepts are for those of us who aren't familiar, but, one of our participants asked about your perspective about the arguments for neuro symbolic AI, brute force AI, and gen AI. Is there one better in terms of ethics or, you know, sort of protections for those that might be exploited or forgotten by AI? Or generally, you know, what are your thoughts about this?
Rachel Thomas ’05 Yeah, so I should say my background is almost entirely in kind of a machine learning approach of learning specifically from data. And so this is, it's getting outside my area, my understanding is neuro symbolic is more kind of like, focused on the underlying, kind of like conceptual representations and that hasn't, hasn't been my area at all. I should also say I'm not as interested in, like, notions of human consciousness or how does a machine replicate human consciousness, I really find the kind of the practical kind of like what has the data shown in the past, and how can we use that to kind of accurately classify things or accurately make predictions is what interests me. I did work with, like, some more kind of traditional modeling systems earlier in my career. And I found those to often be, kind of brittle when you're you know, modeling kind of like, well, I know these pathways and so I went to the computer model. Whereas the machine learning I've worked with, like the last decade has been much more kind of these are, these are the inputs and these are the outputs. And you know, you take into account some of those characteristics, but yeah, so I would say that I am less qualified to kind of speak on the, the other areas.
Daniela Kucz ’14 Thank you. So going back to some of the healthcare pieces specifically, I'm curious if you are, speaking with, say, a healthcare institution or someone in the healthcare space, given that AI is more and more prevalent in different ways, whether that's, you know, in the back office function or with clinical notes, if you're a provider, how should individuals in the system, especially you know, both providers, but also maybe decision makers best think about how to select AI tools, if at all. Like are there ones that are perhaps better suited in particular for medicine? And beyond that, are there sort of general ways to evaluate what is being pitched, given that so much also gets over-exaggerated?
Rachel Thomas ’05 Yeah. No, that's that's a great point. I think so. One is yeah. Like ideally like keeping in mind that, ideally any tool you use, you want it to make the lives of, you know, both the providers and the patients kind of better or easier. And so keeping peace, keeping that kind of data, which, you know, it sounds obvious, but I think often the approach of people creating machine learning or AI systems is kind of more like, ‘Oh, it'd be exciting to automate this process’. And then they don't fully automate it and then you have to kind of put a person back in the loop in this way that doesn't take advantage of human strengths or interest, at all. And so really, I mean, the designers of these systems should ideally be thinking holistically of like, you know, how do we take advantage of human, human strengths and also make something that's solving a real problem for people? And so, I think a kind of a starting point for someone, though, on, in terms of considering purchasing a product is, you know, is this even an actual pain point that my providers or patients are experiencing? And then I think to continue checking on it regularly as well. I know there was an example, and there's been reporting on this, of like a sepsis algorithm, from Epic that I think initially had higher accuracy and then the accuracy really, really decreased and that's something where you do want to have systems in place for your you're checking regularly if kind of like, okay, is this maintaining the accuracy and is this, you know, is this user friendly and not causing kind of more pain to, to the people using it?
I should also give a plug and maybe I can put a link in the chat - the Markkula Center at the University of Santa Clara has a really neat, tech ethics toolkit and it's different practices that your company can implement for, kind of checking for tech ethics and they have different kind of like, red teaming and, different ways of kind of making sure you're regularly kind of checking for potential problems and hopefully identifying that before they cause issues.
Daniela Kucz ’14 Thank you. That would be great. All right. In terms of physicians, and sort of individual providers, do you believe that systems and employers are effectively advising them about the risks of relying on AI or perhaps over relying, whether to use it as a supplement in diagnosis or in sort of screenings?
Rachel Thomas ’05 So I'm not close enough to that space to know, but I would be concerned just because I know in general, I think there are a lot of misconceptions often about how these tools work and that I think often companies and even kind of very well respected major tech companies put out marketing materials that I think border on misinformation sometimes in kind of overhyping what their products can do.
Daniela Kucz ’14 Absolutely. So as a next step, maybe we can talk about how to better contain AI and sort of mitigate some of the risks, in particular in healthcare. But I think probably more broadly as well. What do you see the role of regulation to be? How do we make sure that sort of these larger problems and errors that are caused by algorithms, are stopped before they impact real lives?
Rachel Thomas ’05 So I think one aspect is to keep in mind that many of these issues come back to human rights issues, and I didn't talk about all of these as much, but, I mean, definitely access to health as a human rights issue, but also, you know, AI systems are being used to determine kind of who, who gets selected for, like, an apartment, for housing, for loans, it's being used to make educational decisions. And a lot of these things touch on human rights and so I think there is some kind of existing existing framework of kind of, well, how much are we going to protect human rights, which is the great concerning concerning point for in many ways, but that not everything is going to need to completely be reinvented and that there are questions of kind of, you know, also the use within the criminal justice system is another huge one where it's like, okay, this is definitely a human rights issue and so kind of recognizing that those rights are protected.
And then there's a set of principles that are there specific to, to social media, called the Santa Clara Principles. I believe that they call for, I think it's timely and actionable. No, it’s not actionable, timely recourse. But they have this notion of the idea of kind of when you file a complaint or raise an issue that you can kind of, get a response in a way that is timely and fair and that's something that's really lacking in a lot of systems where you know, even if, you know, so like the, the Medicare Advantage example, you know, it's taking people like months and months, if not years to contest these like medical decisions where it's like, no, they they needed that care like five months ago, it doesn't help to, you know, prove that the the algorithm made a mistake. So, I think giving people rights around kind of like the, being able to contest things in a timely way and get a meaningful response.
Daniela Kucz ’14 And I think a relevant question to that as well. So in terms of responsibility for some of these outcomes, in particular in healthcare, but I think more broadly, you gave some predictive policing examples as well. Where should the responsibility fall in terms of ensuring that some of these AI systems are diverse and fair for patients, for individuals in society? Is it the developers and researchers building the tools? The organizations selling them? The government? Is it a shared responsibility? And I think there is a second part to this as well, where, it seems that, you know, at the organizational level, some of these companies, either publishing models or building tools on top, currently don't have a lot of ownership of the outcomes. And how, how do we enable that power dynamic to shift?
Rachel Thomas ’05 Yeah. And that is an important question, but it's a hard question. And yeah, I don't have an easy answer to it. Like because I think that gets to a lot of governance issues where, yeah, kind of the examples you gave, it's like, yes, the creators and, you know, the government deploying it should both have some responsibility, but I know that, that has to be carefully thought about, because often shared responsibility kind of becomes just nobody's responsibility of, you know, this is this is somebody else.
Daniela Kucz ’14 Tough question. I think in particular in the healthcare space, what do you think the role of the FDA, for example, should be? Obviously, the FDA is making some moves on the front of regulating software as a device and making sure that they're sort of keeping a pulse on AI. But, I guess at what level of granularity do you think the FDA should be examining some of these algorithms when they're being used in the healthcare space?
Rachel Thomas ’05 Yeah, this is really hard. And I know this is the frustration of a lot of founders and companies that, you know, for the most part, it really does seem like the FDA has it hasn't kept up and it is really hard from a regulatory side often to test or approve these. Yeah. I'm not I'm not sure. I know that I think a more flexible and knowledgeable framework is needed, but I don't know what that would look like and I recognize that that's a tough kind of governance question.
Daniela Kucz ’14 Absolutely. Have you seen, I guess, examples of better incentive structures between, I would say that insurer to insure provider and the medical outcomes in terms of AI? Does that help in terms of improving the accuracy of some of these models?
Rachel Thomas ’05 I think that could, I think a tough thing right now is that I think a lot of these models are being kind of, implicitly or explicitly implemented as cost cutting measures. And so often I think there is almost like a starting point of like, well, we're not going to give everybody the health care they need or the housing they need and starting with this kind of very, overly scarce resources and then like, well, let's use an algorithm to, you know, like, automate how we how we divide this, like not having enough resources for people. And I think, I guess recognizing and aiming that is important of like, AI isn't going to magically fix, like if you decide like, as a society, we're not putting enough resources into these areas, there's a limit to how much you can get out from efficiency when it's like sometimes the final point is know people, just people need more care and you can't just kind of like, efficiency AI that away.
Daniela Kucz ’14 And I think we have maybe one more question on healthcare specifically. I'm curious, are there any AI tools or applications that you would recommend, or are you skeptical about them as a whole?
Rachel Thomas ’05 Oh, it's not that there is none that I would recommend. I think I don't feel necessarily close enough to the applications that I am trying to like, I'm really trying to immerse myself in microbiology now because it was like, I need to have a very deep domain expertise if I'm going to do this, kind of as accurately as I want and because I find it fascinating. So it's not not necessarily that there, there aren't, aren't products that are good, but I think I'm not close enough to a particular, particular domain that's using one to, to say,
Daniela Kucz ’14 Makes sense.
Rachel Thomas ’05 And then there's also an additional, there was an interesting post, recently, by Abhishek Maharan on kind of what happened to AI pathology companies and he kind of goes through a lot of the, issues of, kind of even even when AI is doing a decent job at reading pathology slides of why, why there are a lot of other reasons that might not make for a successful company or even a product that hospitals will want to buy and, you know, that, kind of there are some upstream issues of, like, a lot of hospitals don't have the machines to scan whole slide images or, kind of other issues too.
Yeah, absolutely. It's a very complex landscape in terms of interest in acquiring and deploying tools, so.
Daniela Kucz ’14 Alright, a slightly different question related back to your talk and the RoboCop example that you gave. How were the errors actually eventually discovered?
Rachel Thomas ’05 So this is something where people were raising court cases and then actually the government was like, kind of solving them at or like would resolve the court case at a level before it went to kind of a higher level where it would get more attention. So it was kind of a bit, it was shady in terms of like, they weren't wanting this to come to light. And I don't know what the tipping point moment was, but I think they were just growing, so many people were impacted that there was kind of this growing like people were bringing cases and it was getting media attention, until it became something that they kind of couldn't ignore anymore.
Daniela Kucz ’14 Excellent. Next, there's a sort of question about conceptually framing artificial intelligence. Do you think that taking machine learning and using the terminology artificial intelligence and sort of trying to or implying that we're trying to replicate human intelligence, might be causing some of these ethical challenges and negative impacts?
Rachel Thomas ’05 Yeah. Yes, yes I do. Yeah. This is, yeah. I don't like the term artificial intelligence, personally. And like, I think it's led to a lot of confusion, and I kind of feel like I have to use it, though, because a lot of this stuff that I'm doing does get categorized under that. But yeah, I would, I would prefer machine learning. I guess I also find the term deep learning really confusing, and it's unfortunate that that's what they went with for multilayered neural networks. I mean, like, I understand wanting something concise, but yeah, no, I think I think it contributes to a lot of a lot of confusion and you've got people that are like picturing, like, humanoid robots when they hear artificial intelligence. And then it's also hard because there are, you know, several different types of work that are going on underneath this umbrella and they are, you know, some in some cases overlapping. But yeah, I think I think artificial intelligence as a term can be quite confusing, particularly because really, I mean, a lot of the stuff that is making the news really is more classic machine learning about matching inputs to outputs.
Daniela Kucz ’14 I think we have time for a couple more questions, so if you haven't asked them yet, please do now. The next question is also specific to one of your slides, which is, about the use of AI in connection with retinal scans. Can you talk a bit more about the use of AI there to improve individual health and devolved public health protocols?
Rachel Thomas ’05 Yeah. So, for individual health and public health protocols. So it's hard because there's also and this is a separate area, but it's very relevant. A lot of times the things that would have the highest impact don't necessarily involve machine learning or have kind of like, exciting, make for exciting headlines and that's a challenge as well. Also, a lot of times, things that would be, I think, quite interesting and valuable we lack the data for because we're not collecting them and this comes back to kind of, one of the first questions about getting more data from patients on their experience, which I think in many cases we’re lacking. And this is, kind of a find a fundamental limitation of recognizing, you know, okay, like machine learns from data and so you, you can't really learn from the data you don't have. And so I think that, there are many very interesting public health projects where we either don't have the data or you don't need machine learning for them. And that's also something we have to be open to when you ask a question about a domain of like, okay, maybe the best solution, does it involve machine learning? And this isn't a machine learning problem. So, those are kind of some broader things to keep in mind. With retinal images in particular, I would have to look back at the details, but I know there was a case maybe like five or 5 or 6 years ago where an algorithm was deployed and then it turns out like there were issues with the cameras not being clean, kind of some use issues in the clinic and so it didn't see the clinical accuracy that had been seen in the, when it was being tested and developed. And that's always an issue as well of kind of when you, when you have this good performance and testing and then there kind of like these practical things of like, okay, do you have the technology and is it is it being run and just yet even, you know, is your camera lens clean? And so, that was the case from several years ago.
Daniela Kucz ’14 Thank you. All right. I think we have time for one more question. I'm sure a lot of the folks who joined want to know how to better get involved in AI more broadly. I think you talked a little bit about that, including some of the free courses that fast.ai offers, but what are some other ways that individuals who want to make sure AI is being effectively and ethically used, for their problem sets in their sort of field of work? What can we do as a next step?
Rachel Thomas ’05 Yeah. So I mean, I just think that learning to code is really useful, even if it's not something you ever want to do professionally or just knowing some coding is helpful. And I think also recognizing that, AI is not magic and it's not, I feel like there is a very kind of harmful narrative of like, oh, you have to be this super genius to work on this stuff. And that's not the case. And to recognize that, like, this really, can be for everyone and to not assume that, you know, you don't have the background or the ability that you would need to understand this. And yeah, I think this is an important question because I do think increasingly, kind of data literacy and AI literacy are things that everyone in society will need, even just to kind of be able to understand the products they don't want to use or why, kind of, why these are ethical issues.
Daniela Kucz ’14 Absolutely. Thank you so much, Rachel. This has been absolutely fascinating and very relevant to a lot of our lives and work, even if we're not necessarily in health care, I think we are all patients in one hospital system or another. So I really appreciate all of the work that you're doing, on our behalf. And thank you all for coming as well. We will be sending out some resources that Rachel mentioned, including her blog, if you'd like to learn more. And as I mentioned we will also be posting a copy of this recording on the SwatTalks website in 2 to 3 weeks. So thanks everyone, and I hope you have a great rest of your day.
Rachel Thomas ’05 Thank you
