Karen Simonyan, Abstract Co-Author: Nothing to Disclose

Andrew Zisserman MA, PhD, Abstract Co-Author: Nothing to Disclose

Antonio Criminisi, Presenter: Employee, Microsoft Corporation, Cambridge, United Kingdom

This paper presents an algorithm for content based medical image search, which allows querying by a specific region of interest. The user provides a bounding box for the region in a query image. The algorithm then retrieves images similar in content to the query with the region of interest detected and localized in each of them.

The search process is performed in two stages. At first, images of the same class as the query (e.g. lung or hand X-rays) are retrieved from a large repository. Image-level classification is based on a supervised multiple kernel support vector machine with gradient-based visual features. Then, the query region of interest is detected and localized within the retrieved images. The procedure is cast into a structured output regression framework which utilizes adaptive spatial context of the query region. The context model is learnt in a semi-supervised setting (no ground truth annotation required).

The image classification stage was evaluated on a repository of 675 X-Ray images of 5 image classes, taken from the public IRMA database. The repository was split into training (350 images) and test sets. The mean classification accuracy on the test set exceeded 99%. The region of interest detection was assessed on a subset of 44 hand X-ray images. Ground truth bounding box annotation was provided for 3 types of bones present in the images. The mean value of the area under the precision-recall curve was measured to be 92% (across 3 types of structure queries). Both stages demonstrate a high level of performance, allowing for accurate and scalable image search.

A novel technique for anatomy-driven image search has been presented. We plan to extend the proposed methods to medical images of other modalities, such as CT and fMRI, as well as more disease-specific search (e.g. finding similar liver lesions).

Anatomy-driven image search allows radiologists to retrieve images from repositories and automatically localize regions of interest in them based on their content and the specified region of interest.

Simonyan, K, Zisserman, A, Criminisi, A, Anatomy-driven Medical Image Search.  Radiological Society of North America 2010 Scientific Assembly and Annual Meeting, November 28 - December 3, 2010 ,Chicago IL. http://archive.rsna.org/2010/9008273.html