Google’s DeepMind team has just published a paper “International evaluation of an AI system for breast cancer screening” showing that their research on artificial intelligence applied to medical data is making amazing progress. Their system makes breast cancer predictions by assessing mammogram images, and seems to work better than radiologists at performing this task. Comparing to the status quo, the system improved both in terms of making fewer mistaken cancer diagnoses, and missed fewer cases where breast cancer was evident.
Demis Hassabis, chief executive officer and co-founder of DeepMind Technologies Ltd., speaks during ... [+]
Patient data for the DeepMind project was from the UK and the US. Six human radiologists were compared to the computer-based approach for assessing mammograms. The DeepMind team has taken the approach to use artificial intelligence as an assistant, rather than as a replacement for a radiologist, for a few very good reasons. Primarily, it’s hard to get approval to replace doctors with computers. As with Elon Musk’s Neuralink, the road to clinical availability will be a long one.
In order to bring this technology to market, it will likely need to get regulatory approval as a diagnostic medical device and then obtain CE Marking registration. Why a medical device? You may ask yourself why the AI system would be so difficult to gain approval, because, after all, it’s just some software that looks at pictures. Well, breast cancer screening tools, even assistive tools, are highly regulated, as they can lead to very invasive procedures in the case of a false positive, or disease progression in the case of a false negative result. In Europe, the safety and quality standards for medical devices are high, and this means the need for clinical trials, and quality control over many aspects of the project, including the design, development, and even the distribution and installation. It also means broad internal and external audits, and a very tough risk management assessment by outside auditors. Unlike a “me-too product” that can cite a similar device as a predicate, an AI-based system is a one-of-a-kind system than needs to be validated from scratch.
Google is only one of several companies targeting the healthcare market with AI. IBM tried to tackle medical diagnosis head-on with Watson, and although many promising individual results and studies are out there, the medical research community came to feel that the technology overpromised in the demos but under-delivered in everyday clinical application. There are also many students and amateurs working on medical data AI applications. It’s a hot field not limited to Google. And yet, the vast majority of the work in the field remains limited to academic work, because of the high cost and effort of bringing medical AI to market.
A further challenge to bringing DeepMind’s breast cancer screening technology to market is the real risk of fooling an autonomous AI-based medical screening system. In a stunning article filled with fascinating images, a team from Harvard Medical School and MIT presented approaches for fooling this type of AI-based diagnosis system, in their paper “Adversarial Attacks Against Medical Deep Learning Systems”. They found that attempts to fool the system succeeded even when the internal details of the artificial intelligence system was not revealed to the system that fools it. The ability to so completely fool an AI-based system is probably a major factor in the decision to keep humans in the loop, for now. We have seen the same trend in self-driving cars, where the initial promise was full autonomy, but the reality is an increase in driving automation like lane-following and parking assist systems. It’s hard to get people out of the loop in these life-critical applications.
The two challenges with getting to market that talked about so far are regulatory and technological, but there are many more. The reimbursement mechanism is not at all clear. In a single-payer system, the doctors don’t have an incentive to pay a company, and so the company needs to sell governments on the use of their technology, which can be slow. In addition to the reimbursement challenge, there are country-specific regulatory issues such as data privacy that come into the equation. This article started off talking about a promising proof-of-concept AI system for improving breast cancer diagnosis, and we have not even scratched the surface of getting clinical trials done and FDA approval in addition to ISO certification and CE Marking. All in all, the patients in our society don’t experience cutting edge technologies like this until they have been very carefully tested and validated. The progress DeepMind is making is not only crucial, it is amazing and exciting. But we need to temper that excitement with the reality that these techniques are still in their infancy. There is so much work still to do.
