James Kirby is an NIH-funded Principal Investigator in the Experimental Pathology Division of Beth Israel Deaconess Medical Center; Director of the Clinical Microbiology Laboratory at BIDMC; Program Director of the Medical Microbiology Fellowships at BIDMC; and an Associate Professor of Pathology at Harvard Medical School. One of the goals of his lab is to shorten the antimicrobial testing gap through the use of novel technologies. That can range from exploring the use of digital dispensing technology to developing a Superfast Microscopy-Based Antimicrobial Susceptibility Testing (MAST) system. Recently, Dr. Kirby turned his sights to AI and artificial neural networks.
SCIENTIFIC INQUIRER: Can you provide some background? Why did you decide to test how AI-based diagnostics perform?
JAMES KIRBY: We have long been interested in improving our ability to diagnose infection and identify appropriate treatments more quickly. Infections are caused by microbes, very tiny living beings. Despite their small size, we can see them under a microscope, discriminate among them based on size and shape, and observe their behavior when treated with antibiotics. So infectious disease diagnostics is a “visual” or “image-based” activity, and artificial intelligence excels at image discrimination (and can, in some cases, exceed human capacity for image recognition). We are beginning to use such AI extensively in familiar technologies, for example, AI that can identify family members in pictures on an iPhone based on facial features. That utility and promise gave us the idea of testing use of AI in infectious disease diagnostics.
SCINQ: What is a convolutional neural network and how does it work?
JK: A convolutional neural network is a type of artificial intelligence. It is modeled on the human optical cortex. This area of the brain processes the images that we see and converts them into something that the brain can understand. The optical cortex has multiple layers, each of which is used to extract information from an image that can be used in deeper layers to determine whether that something is a cat or car or beach ball. For example, information may include color, outline, shape, reflectance, movement. If one were to probe the deeper layers the extracted information would look nothing like a beach ball, but would be really useful in distinguishing a beach ball from other objects.
The convolutional neural network works similarly. Like the human brain, CNN’s must be trained to recognize objects. To do this, the CNN is presented with large numbers (tens of thousands) of examples of objects of interest and “background”, in other words examples that are not the object of interest. During training, the CNN iteratively modifies its programming in each layer to maximize its ability to perform the assigned task, for example, in our case, identifying a particularly type of bacteria correctly.
SCINQ: How long did it take to essentially train the neural network to recognize the three categories of bacteria?
JK: The majority of the time was spent acquiring and classifying images of different types of bacteria to train the neural network. We were fortunate to have access to an automated microscope system developed by a company called MetaSystems that allowed us to acquire a large number of images for training the CNN. With 100,000 images in hand, training of the network took about a week of computer time. When we performed our initial published study, we performed the training on a very underpowered computer, using its central processing unit or CPU. Since then we have acquired a powerful graphic processing unit (GPU, also the mainstay of computer gaming processing power) which are standardly used in the field to perform computationally intensive CNN training and implementation. Training could have been accomplished more quickly with a GPU. Since we were originally limited in computer power, we also started out with a pre-trained CNN that had already been trained to recognize 1000 different objects, and limited ourselves to trained its final layer to recognize bacterial types. This is referred to in the field as “transfer learning” and has been shown to be a very efficient way to train complex CNNs that would otherwise require extensive computational infrastructure. In essence we started with a CNN that already had a primary and secondary school education, and we put it through an associate’s degree program to develop a new skill set.
SCINQ: When you allowed the algorithm to work on its own, it performed extremely well – roughly 93 percent accuracy. What about the 7 percent it got wrong? How and why did it struggle?
JK: We used real world slides prepared in from patient blood culture specimens. These slides were prepared manually and therefore had unequal distribution of blood culture sample. Our initial imaging protocol used a microscope to automatically obtain images from invariant, predetermined positions on each slide. Sometimes there was specimen in the imaged positions, sometimes there was not. Therefore, our most frequent error was non-detection of bacteria because of adequate sampling.
Future iterations of this technology will use dynamic image sampling, where images will only be acquired from positions on each slide where there is actual blood culture sample. This could be accomplished by telling the microscope to only obtain images from areas with a certain contrast threshold (no sample = low contrast; patient sample present = higher contrast). Furthermore, we could also take advantage of automatic slide preparation instruments, which are fairly common in clinical labs, and which would ensure greater uniformity of slide preparations upfront.
Lastly, CNN’s show better performance with increased training. In our study, our goal was to show that fully automated Gram stain analysis was possible. In the future, we will train with a larger number of images to enhance accuracy further.
SCINQ: Overall, did the AI-enhanced automated microscope perform as you expected? Or did it outperform or underperform?
JK: The automated microscope performed well beyond our expectations. Typically, Gram stains are interpreted using a 100X objective which requires addition of oil and has a very small field of view. Using the MetaSystems microscope we were able to collect images using a 40X objective (no oil) with a much larger field of view, allowing us to efficiently collect a large amount of training data.
The AI itself performed as well as expected based upon or level of training. Using our previous analogy, we have begun the first year of post-secondary school education. We did not expect the proficiency of a college graduate. We would like to move the program through college and perhaps even graduate school to get it ready for use in different stages of clinical diagnostics.
SCINQ: What else needs to happen before this system can be considered reliable enough to use in actual clinical settings?
JK: There are two ways the system could be used in a clinical setting. The first is something that we call Technologist Assist or TA. Here the AI microscope system would supplement medical technologist (MT) and medical laboratory technician (MLT) capabilities. It would pre-scan slides and present select images containing bacteria to a MT or MLT. The TA program itself would offer a provisional diagnosis but the MT or MLT would be responsible for rendering a final diagnosis based on their interpretation of the images. Importantly, the automated system would save technologists significant amounts of time, as they would not have to evaluate large numbers of fields under the microscope to find examples of bacteria sufficient for a diagnosis. Importantly, too, the AI provisional diagnosis would help improve overall accuracy of the final diagnosis, by helping technologists consider the full range of diagnostic possibilities.
The system would also allow remote slide analysis and images could be sent to a centralized location for MLT/MT final interpretation. This would help address staffing shortages during certain shifts in hospital networks and would provide a way for expertise to be shared with remote locations. In this system, reliability would not have to be much greater than at present, either when used on site or remotely, because of MLT/MT backup.
To replace technologists, reliability would have to be significantly higher and would require an extensive degree of convolutional network training and validation to ensure that the CNN performs with a very high degree of accuracy and also knows when it does not know an answer so it can ask for help. Also, before use in a clinical setting, the AI would need to recognize the range of possible pathogens that could be seen in blood cultures. To that end, we envision performing additional training to include additional types of bacteria since our published proof-of-principle study, we limited ourselves to the three most common types of bacteria morphologies seen.
SCINQ: What are the possible advantages that arise from using AI in the laboratory? Do you foresee AI eventually replacing diagnostic professionals in the future?
JK: There is a national shortage of medical technologists. Diagnostic smear interpretation is one of the most labor-intensive activities in the clinical microbiology laboratory. We perform all sorts of such interpretations from the diagnosis of tuberculosis, to the identification of intestinal parasites, to Gram stain interpretation to determine what types of organisms are in the blood or a tissue infection. We look at molds, isolated from patients on chemotherapy or with organ or bone marrow transplants, under the microscope to determine their species and suggest treatment options. All of these microscopic diagnoses require a lot of time spent finding organisms and a high degree of skill to make correct interpretations. TA could facilitate interpretation in all of these areas, shorten the time needed for diagnosis, and improve overall diagnostic accuracy by providing an expert first or second opinion. I do foresee such technology eventually allowing diagnostics with fewer personnel. Personnel would be freed to focus on the most challenging cases and thereby improve overall diagnostic care.
SCINQ: On a personal/professional level, you graduated Yale College with a BS in Biochemistry and Biophysics, but after a two year stint at the NIH, you developed a interest in bacteria. What brought this about?
JK: I was always interested in bacteria and biology more generally. Molecular biophysics and biochemistry provided a broad background that could be applied to many areas of scientific investigation. But in terms of my career studying bacteria, as with many things in life, there was also serendipity. When I interrupted my medical school training to pursue research at the National Institutes of Health, I wanted to explore scientific investigation as a potential future career combined with also becoming a physician. This is called a physician-scientist track. As part of the unique program at NIH, I could work pretty much in any lab that would take me. I spoke with a lot of scientists there about potential opportunities, and I was referred to a scientist named Susan Gottesman who works on gene regulation in bacteria. The point was made that I could explore a lot of scientific questions in a short time in a bacterial system, since bacteria because grow so quickly. That way I could make an informed decision about whether I enjoyed science. I met with Susan, and I was hooked. The scientific questions were fascinating; her mentoring and commitment exceptional; and the scientific environment inspirational. I also read through two influential books, all 2000 pages of Escherichia coli and Salmonella edited by Frederick Neidhardt, a compendium of how a “simple” life form works, and also Lambda II, a book about a virus that infects bacteria. I have now forgotten almost all of the text, but the total experience of my two years at NIH left its mark. When I returned to medical school my interest in bacteria remained. I found infectious diseases in medical school fascinating. All of this led to a specialty in clinical microbiology (where I could help diagnose infectious disease), a postdoctoral research fellowship under the mentorship of Ralph Isberg learning how bacterial pathogens exploit human cells, and a career as a physician-scientist. I should note that the Foundation for the NIH (FNIH) has started a new program to introduce medical, dental and veterinary students to biomedical research through a one year period performing research on the NIH campus. This is the descendant of the program that I attended almost three decades ago. I am very excited that others can once again take advantage of this unique experience of living and performing research on the NIH campus and become part of the next generation of biomedical scientists that are also trained in a clinical specialty.
SCINQ: What fascinates you the most about bacteria?
JK: Bacteria are very creative. They use so many different strategies to persist in hostile environments. I am particularly fascinated by bacterial pathogens. They have found ways to survive and cause infection despite our very robust immune defenses. They have figured out multiple ways to thwart all of the antibiotics that we have developed. I am fascinated by the challenge that they offer: a challenge to understand how they cause disease; a challenge to find new vulnerabilities that can be exploited to develop new therapies; and a challenge to tame them whenever they cross the boundary between harmless colonizer and pathogen.
SCINQ: What is your stance on the future of antibiotics? Are we already in a post-antibiotic world, as some people assert? Considering the rapidity at which bacteria develop resistance to antibiotic classes already in existence, is a whole new strategy (whatever that may be) the only chance we have to prevent a return to pre-antibiotic days?
JK: I think we are straddling on the knife-edge of a post-antibiotic world. We have pathogens that are completely resistant or effectively resistant to all antibiotic because we cannot use any available antibiotics that still retain activity in some patients because of unacceptable side effects. Most large pharmaceutical companies have dismantled their antibiotic discovery efforts. It is far more profitable to produce a therapeutic that patients need to take for the rest of their life than an antibiotic that has to be taken for a week, and at best, an antibiotic that perhaps should be reserved for special cases to prevent development of resistance.
On the positive side, there has been recent recognition that we have to do something. There have been new incentives both in the United States and Europe to de-risk investment in antimicrobial research and several new antimicrobials are working through the development pipeline and clinical trials. Most of these antibiotics are variations on what is already in use, and so rapid development of cross-resistance to these new therapies is definitely of concern. There needs to be additional and steady investment in future antimicrobial development and a commitment to discovery of new classes of agents. Also, we need to understand resistance mechanisms at a deep level. This will potentially allow us to engineer derivatives of highly effective agents through chemical modification in a way that avoids existing resistance mechanisms. I am excited about new advances in medicinal chemistry and the ability to build antibiotics from bottom-up to retain and enhance activity, and at the same time avoid known resistance mechanisms. So I am cautiously optimistic that a post-antibiotic world can be avoided. However, success in this endeavor will require substantial investment at a societal level to create new medicines that are intrinsically not particularly attractive to pharmaceutical companies.
It is easy to lose site of the fact that many of the advances in modern medicine — new cures for cancer and immunological disease — and longevity itself make patients more susceptible to infection. A post-antibiotic world would have devastating effects on our health. It is a calamity that can be avoided with commitment and resolve.
For more information about James Kirby and his research, visit his lab page.