Invented by Synho Do, Jung Hwan Cho, General Hospital Corp
The General Hospital Corp invention works as followsBoth supervised and unsupervised learning methods can be used to label medical images automatically for deep learning applications. A small initial training dataset can be used to generate large labeled data sets using an iterative sampling scheme. This paper also provides a machine-learning powered automatic organ classification for imaging datasets such as CT datasets with a deep Convolutional Neural Network (CNN), followed by an instrument dose calculation. This technique can also be used to estimate the organ dose for each individual patient, since the sizes and locations of the organs can be calculated separately.
Background for The system and method of automated labeling and annotation for unstructured medical data sets
Diagnostic medical imaging is now a central part of the practice and the volume of diagnostic examinations has grown over the last decade. As systems became more advanced and had higher resolutions, the number images within a study also increased. Increased demand for diagnostic imaging poses a risk of human error, which can delay diagnosis. Computer-aided detection (CADe), and diagnosis (CADx), systems can reduce these problems but they are still limited by their dependence on hand-crafted features. Deep-learning techniques avoid this problem because they extract these features themselves. Recent advances in deep-learning technology have enabled data driven learning of nonlinear filters and classifiers. This has improved detection and segmentation for multiple medical applications, including brain infarcts and automated bone age analysis. “Despite these advancements, large-scale training datasets with well-labeled labels are necessary for deep learning networks to learn hierarchical and representative abstractions.
This labeling requirement can be difficult to meet, as medical expertise costs a lot, the labeling process is time-consuming and tedious, and certain diseases may only occur rarely. Several automated annotation techniques have been tried on brain CT, MR and other biomedical images with different feature representation, classification and clustering algorithms. These approaches are limited, because they only extract low-level features, such as colors, edges, or color layouts. Even when higher-level features are extracted from MR voxels using hierarchical learning with two-layer random forest, segmentation performance does not always outperform that of deep convolutional networks. All current methods for annotating medical pictures still require large labeled datasets to train the model.
Axial Image Location Classification is a fundamental process in the initial classification of an image during a volumetric CT exam. The difficulty in determining the exact location of an image is that the body’s details can differ dramatically from patient to patient. For example, the brain gyral pattern, cervical vertebral structure, pulmonary vessels and bowel distribution may vary significantly. Degenerative changes may also cause bony anatomy to be distorted enough to confuse the neural network. In order to ensure that algorithms are accurate enough, they often require large sets of training data.
Body-part identification is important for automatic medical image analysis, as it is the prerequisite step to organ segmentation and anatomy identification. By reducing the range of search for a particular organ, accurate body-part classification can facilitate organ detection and segmentation. To classify up to 10 organs, multiple techniques were developed using decision forests and multi-class random regressions. These classifiers are able to discriminate between similar structures, such as the heart and aortic arches. These previous works, however, focus on a classification of anatomical parts in general.
The use of high-quality data for training neural networks is essential to unlocking the potential that neural networks have to improve clinical image analysis. But creating high-quality datasets can be expensive and time consuming.
The present disclosure addresses these drawbacks with a system and a method that uses supervised and non-supervised learning schemes for automatically labeling medical images to be used in deep learning applications. The system is able to generate a large, labeled dataset using an iterative sampling scheme. This paper also provides a machine-learning-powered, automatic organ classification system for imaging datasets such as CT datasets. It uses a deep convolutional network (CNN), followed by an instrument dose calculation. This technique is useful for estimating organ doses that are specific to each patient, as the sizes and locations of organs can be calculated separately for each patient.
In one configuration, the method comprises automatically processing unstructured data from medical imaging to produce classified images. The method involves acquiring medical images of a person and submitting the medical images of that person to a neural net to produce classified image data. The method can also include comparing classified image to a confidence level test and, if the classified data fails the test, submitting the classified data to variational autoencoders (VAEs) that implement a snowball sample algorithm to refine classified data by converting features in the classified data into latent spaces with a Gaussian distributed. This may be repeated in some configurations until the classified data passes the test. The classified image data can then be used to generate annotated images.
In one configuration, the method provides automatic annotation and labeling for unstructured medical data sets with snowball sampling. The method comprises acquiring images of an area of a subject, and labeling them to create a training dataset. The method includes also training a neural network such as a Convolutional Neural Network with the training data and classifying images that are not labeled using the trained network. The method can also include determining whether a threshold performance is exceeded by the classified images. If the threshold is not reached, the dataset can be refined by using a variational encoder to label unlabeled pictures to create labeled photos and updating the dataset to include the labeled photos.
The system includes a computer system configured to: i) acquire images of a region of a subject and label the images in order to generate corresponding training datasets; ii) train cnn with the training dataset; eiii) classify unlabeled images using trained network; v), refine the dataset if threshold is not exceeded by using var autoencoders to label unlabeled image data. The system comprises a computer configured to: (i) acquire images from a region on a subject, and (ii), train a convolutional network using the training dataset. (iii), classify unlabeled pictures using the trained network.
In one configuration, an organ classification method for unstructured medical data is provided. The method comprises acquiring images for a particular region on a subject, and labeling them to create a training dataset using the images. The method can also include training an artificial neural network such as convolutional network with the training data. The trained network can be used to classify a region of the images. Convolutional neural networks can be used to segment the classified images, generating segmented images which distinguish at least two organs within the classified regions. The calculated dose of radiation for one or more of the classified organs can be reported.
The following description will reveal the above and other aspects of and advantages of this disclosure. The description makes reference to the drawings which are a part of the disclosure and illustrate a preferred embodiment. The embodiment shown here does not necessarily reflect the entire scope of the present invention. For that reason, the scope is interpreted by referring to the claims.
The present disclosure describes systems and methods for supervised and non-supervised learning schemes which can be used to label medical images automatically for deep learning applications. A small initial training dataset can be used to generate large labeled data sets using an iterative sampling scheme. This paper also provides a machine-learning powered automatic organ classification for imaging datasets such as CT datasets with a deep Convolutional Neural Network (CNN), followed by an instrument dose calculation. This technique can also be used to estimate the organ dose for each individual patient, since the sizes and locations of organs can be calculated separately.
In one configuration, the desired classification accuracy can be achieved by using a minimal labeling procedure. A large medical image dataset can be automatically annotated with a smaller subset of training images using an iterative approach called snowball sampling. The automatic labeling may use a variational encoder (VAE), for feature representation. Gaussian mixtures models (GMMs), for clustering, refinement of mislabeled categories, and deep convolutional networks (DCNNs), for classification. The system and method are also able to quickly and accurately identify an organ with a greater accuracy than the current text-based information found in DICOM headers. In one configuration, it selects candidates and classifies them using the DCNN. It then refines the results by learning the features of a VAE, and clustering them according to GMM.
Referring to FIG. In Figure 1, an example of a labeling system 100 for images is shown. The image data used to label the images corresponds with certain aspects of the disclosed material. As shown in FIG. As shown in FIG. In certain configurations, the computing devices 110 can run at least a part of an automatic labelling system 104 in order to determine automatically whether a particular feature is present on images of a specific subject.
In some configurations, a computing device 110 may communicate image data from an image source 102 over a network 108 to a server, who can then execute a portion or all of the automatic labelling system 104 in order to determine automatically whether a particular feature is present on images of a given subject. In these configurations, the computer 110 (or any other computing device suitable) can receive information from the server 120 indicating the output of the automatic labelling system 104 in order to determine if a feature is absent or present.
In some configurations the computing device 120 and/or server 110 can be any computing device or combination, such as desktop computers, laptop computers, smartphones, tablet computers, wearable computers, server computing devices, virtual machines being executed by physical computing devices, etc. In certain configurations, automatic image labelling systems 104 can extract (e.g. labeled) features from labeled image data, using a CNN that has been trained as a general classifier. They can also perform a correlation to calculate correlations between features corresponding to image data and database. In some embodiments the labeled image data can be used as input to a classification model such as a Support Vector Machine (SVM) to classify certain features as indicative or disease, condition or normal. In certain configurations, automatic image labelling systems 104 can provide unlabeled data features to the trained classification models.
In certain configurations, an image source 102 could be any source of image information, including MRI, CT scan, ultrasound, PET scan, SPECT scan, x-ray or another computing device, like a server that stores image data. In some configurations the image source can be located locally to the computing device. The image source 102, for example, can be integrated with the computing devices 110 (e.g. the computing devices 110 can be configured to be part of a device that captures and/or stores images). Another example is that the image source can be directly connected to the computing system 110 via a wireless or cable connection. In some configurations, image sources 102 may be located both locally and remotely from computing devices 110. They can then communicate image data via a network (e.g. the communication network 108) to the computing devices 110 and server 120.
In some configurations the communication network 108 may be any communication network, or combination of networks. The communication network 108, for example, can be a Wi-Fi (which includes one or multiple wireless routers and switches). A peer-to-peer (e.g. a Bluetooth network) or cellular network can be used. A wired network is another option. In certain configurations, communication network 108 may be a local network, wide network, public network (e.g. the Internet), private or semi-private (e.g. a corporate intranet or university intranet), or any other type of suitable network. Communication links shown in FIG. The communications links shown in FIG.
FIG. “FIG. As shown in FIG. In some configurations the computing device 110 may include a CPU 202, display 204, inputs 206 and/or communication systems 208. In certain configurations, the processing device 110 can include any processor or combination thereof, including a central processor unit (CPU), graphics processing unit unit (GPU), and so on. In some configurations the display 204 may include any suitable display device, such as a monitor, touchscreen, television, etc. In some configurations the inputs 206 may include any suitable input device and/or sensor that can be used to accept user input. Examples of such devices are a keyboard or mouse, touchscreen, microphone, etc.
In some configurations, communications systems 208 may include various hardware, software, or firmware for transmitting information over communication networks 108, or any other suitable networks. The communications systems 208 may include, for example, one or several transceivers or communication chips or chip sets. The communications systems 208, for example, can include hardware and/or firmware that can be used to create a WiFi connection, Bluetooth connection, cell phone connection, Ethernet connection, etc.
In some configurations, memory 210 may include any storage device that is suitable for storing instructions, values, or other data that could be used by the processor 202, for instance, to display content on the display 204 and communicate with the server via the communication system(s) (208). Memory 210 may include a combination of volatile memory, nonvolatile storage, and/or any combination of these. The memory 210 may include, for example, RAM, ROM and EEPROM. It can also include one or several flash drives, hard disks, solid state drives or optical drives. In certain configurations, a computer programme can be encoded on the memory 210 to control the operation of the computing devices 110. In these configurations, the CPU 202 can run at least a part of the computer programme to display content (e.g. MRI images or user interfaces with graphics and tables), receive content from server 120 and transmit information to server 120.Click here to view the patent on Google Patents.