Method and apparatus for indexing of topics using foils
Application no:09593207 -
Filed date:2000-06-14 -
A method and structure for indexing multi-media data comprising deriving keywords from a first media type (slides), matching the keywords to a second media type (video), identifying an appearance of the first media type in the second media type, and calculating a co-occurrence of the keywords and the appearance of the first media type in the second media type. The invention produces an index of the second media type based on the co-occurrence.
No:707 - Data processing: database and file management or data structures / No:102 - Generating database or data structure (e.g., via user interface)
No:707 - Data processing: database and file management or data structures / No:E17.028 - subclass
What is claimed is:
1. A computer implemented method of indexing multi-media data comprising: deriving keywords from a first media type; matching said keywords to a second media type; identifying an appearance of said first media type in said second media type; and calculating a co-occurrence of said keywords and said appearance of said first media type in said second media type.
2. The method in claim 1, further comprising producing an index of said second media type based on said co-occurrence.
3. The method in claim 1, wherein said identifying comprises: generating keyframes from said second media type; extracting geometric keyframe features from said keyframes and geometric slide features from said first media type; and matching said geometric slide features and said geometric keyframe features.
4. The method in claim 3, further comprising identifying background matching regions in said keyframes having colors matching colors of backgrounds in said second media type.
5. The method in claim 4, wherein said extracting of said geometric keyframe features is performed only in said background matching regions.
6. The method in claim 3, wherein said matching identifies which portion of said first media type has a highest number of geometric slide features matching geometric keyframe features in a keyframe of said keyframes.
7. The method in claim 3, wherein said extracting includes: identifying changes in image intensity as edges; forming curves connecting said edges; identifying corners where said curves change direction; grouping said curves into curve-groups; and designating a sequence of three consecutive features in each of said curve-groups as basis triples.
8. The method in claim 7, wherein said matching comprises: computing coordinates of said basis triples; identifying which portion of said first media type has a highest number of basis triples matching basis triples in a keyframe of said keyframes.
9. The method in claim 3, wherein said generating of said keyframes comprises: dividing said second media type into portions based upon scene changes; and selecting one frame from each portion of said second media type as a keyframe.
10. The method in claim 1, wherein said calculating comprises: processing said multi-media data to extract relevance scores and time reference points of matches to individual media modes; identifying overlapping time periods of matching keywords and said appearance of said first media type in said second media type; and ranking a relevance of said overlapping time periods.
11. The method in claim 10, wherein said ranking includes finding an overlapping time period having a highest relevance score.
12. The method in claim 10, wherein said ranking includes segmenting said overlapping time period to identify beginning and ending events.
13. The method in claim 10, wherein said ranking includes finding a largest number of different modes of overlap.
14. The method in claim 10, wherein said modes comprise two or more of audio, video, text, and graphic display.
15. A computer implemented method of indexing multi-media data comprising: deriving keywords from slides; matching said keywords to an audio portion of video data; identifying an appearance of said slides in a video portion of said video data; and calculating a co-occurrence of said keywords and said appearance of said slides in said video data.
16. The method in claim 15, further comprising producing an index of said video data based on said co-occurrence.
17. The method in claim 15, wherein said identifying comprises: generating keyframes from said video portion; extracting geometric keyframe features from said keyframes and geometric slide features from said slides; and matching said geometric slide features and said geometric keyframe features.
18. The method in claim 17, further comprising identifying background matching regions in said keyframes having colors matching colors of backgrounds in said slides.
19. The method in claim 18, wherein said extracting of said geometric keyframe features is performed only in said background matching regions.
20. The method in claim 17, wherein said matching identifies which slide of said slides has a highest number of geometric slide features matching geometric keyframe features in a keyframe of said keyframes.
21. The method in claim 17, wherein said extracting includes: identifying changes in image intensity as edges; forming curves connecting said edges; identifying corners where said curves change direction; grouping said curves into curve-groups; and designating a sequence of three consecutive features in each of said curve-groups as basis triples.
22. The method in claim 21, wherein said matching comprises: computing coordinates of said basis triples; identifying which slide of said slides has a highest number of basis triples matching basis triples in a key frame of said keyframes.
23. The method in claim 17, wherein said generating of said keyframes comprises: dividing said video data into portions based upon scene changes; and selecting one frame from each portion of said video as a keyframe.
24. The method in claim 15, wherein said calculating comprises: processing said video data to extract relevance scores and time reference points of matching keywords and said appearance of said slide in said video portion; identifying overlapping time periods of matching keywords and said appearance of said slides in said video portion; and ranking a relevance of said overlapping time periods.
25. The method in claim 24, wherein said ranking includes finding an overlapping time period having a highest relevance score.
26. The method in claim 24, wherein said ranking includes segmenting said overlapping time period to identify beginning and ending events.
27. The method in claim 24, wherein said ranking includes finding a largest number of different modes of overlap.
28. The method in claim 24, wherein said modes comprise two or more of audio, video, text, and graphic display.
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention generally relates to systems that search and index different modes of media (audio, text, video, etc.) based upon a query and more particularly to an improved system that indexes one form of media using keywords from another form of media using a relevance-based time overlap between the matches in the different media modes.
2. Description of the Related Art
Information can be stored in many different forms (modes). Before the advent of audio recordings, the only form to record information was the written word, written symbols, and/or numbers. Subsequently, audio and video recordings were used to supplement or replace written information. Regardless of the mode in which information is recorded, there is always a need to search and index the information so that only relevant portions need to be reviewed when the user has a question (query) on a very specific topic.
Conventional searches primarily involve keyword queries of previously created text or indexes. Thus, it is common to perform a simple Boolean combination such as AND/OR, or perform a search based on individual relevance scores of the textural data. However, with the increasing use of different media modes to record information, there is a need to logically search video, audio, and graphics, as well as textual information.
Detecting and recognizing slides or transparencies used by a speaker in a video is important for detecting teaching-related events in video. This is particularly important in the domain of distributed or distance learning whose long-standing goal has been to provide a quality of learning comparable to the face-to-face environment of a traditional classroom for teaching or training. Effective preparation of online multimedia courses or training material for users is currently ridden with problems including the high cost of manual indexing, slow turnaround, and inconsistencies from human interpretation. Automatic methods of cross-linking and indexing multimedia information are very desirable in such applications, as they can provide an ability to respond to higher level semantic queries, such as for the retrieval of learning material relating to a topic of discussion.
Automatic cross-linking multimedia information, however, is a non-trivial problem as it requires the detection and identification of events whose common threads appear in multiple information modalities. An example of such events include points in a video where a topic was discussed. From a survey of the distance learning community, it has been found that the single most useful query found by students is the querying of topic of interest in a long recorded video of a course lecture. Such classroom lectures and talks are often accompanied by foils (also called slides), some of which convey the topic being discussed at that point in time. When such lectures are video taped, at least one of the cameras can be used to capture the displayed slide, so that the visual appearance of a slide in video can be a good indication of the beginning of a discussion relating to a topic.
The recognition of slides in the video, however, is a challenging problem for several reasons. First, the imaging scenario in which a lecture or talk was taped could be of a variety of forms. There could be a single camera looking at the speaker, the screen showing the slide, and the audience. Also, the camera often zooms or pans. Alternately, a single camera could be dedicated to looking at the screen, or the overhead projector where the transparency is being displayed, while another camera looks at the speaker. The final video in this case, may have been edited to merge the two video streams.
Thus, the slide appearing in a video stream could consume an entire video frame or be a region in a video frame. Secondly, depending on the distance between the camera and the projected slide, the scale at which the slide image appears in the video could be reduced, making it difficult to read the text or slide and/or make it hard to detect text using conventional text recognition methods. Additionally, the color on the slides undergoes transformation due to projection geometry, lighting conditions of the scene, noise in the video capture and MPEG conversion. Finally, the slide image often appears occluded and/or skewed. For example, the slide image may be only partially visible because another object (e.g., person giving the talk) is covering or blocking the slide image.
Previous approaches of slide detection have worked on the premise that the camera is focused on the slides, so that a simple change detection through frame subtraction can be used to detect changes in slides. There has been some work done in the multimedia authoring community to address this problem from the point of synchronization of foils with video. The predominant approach has been to do on-line synchronization using a structured note-taking environment to record the times of change of slide electronically and synchronize with the video stream.
Current presentation environments such as Lotus Freelance or Powerpoint have features that can also record the change time of slides when performed in rehearse modes. In distributed learning, however, there is often a need for off-line synchronization of slides since they are often provided by the teacher after a video recording of the lecture has been made.
The detection of foils in a video stream under these settings can be challenging. A solution to this problem has been proposed for a two-camera geometry, in which one of the cameras was fixed on the screen depicting the slide. Since this was a more-or-less calibrated setting, the boundary of the slide was visible so that the task of selecting a slide-containing region in a video frame was made easy. Further, corners of the visible quadrilateral structure could be used to solve for the âposeâ of the slide under the general projective transform.
Therefore, there is a need for a system which can recognize slides, regardless of whether the slides are distorted, blurred, or partially blocked. A system and process for ârecognizingâ slides and indexing the video using such information has not been explored before. The inventive approach to foil detection is meant to consider more general imaging situations involving one or more cameras, and greater variations in scale, pose and occlusions.
Other related art includes the principle of geometric hashing and its variants[Lamdan & Wolfson, Proc. Int. Conf. Computer Vision,(ICCV) 1988, Rigoustous and Wolfson, Spl. Issue on geometric hashing, IEEE Computational Science and Engg., 1997, Wolfson, âOn curve matchingâ in IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 12, pp.483-489, 1990, incorporated herein by reference], that has been applied earlier to the problem of model indexing in computer vision, and the related technique of location hashing that has also been disclosed earlier [Syeda-Mahmood, Proc. Intl. Conf. Computer Vision and Pattern Recognition, CVPR 1999, incorporated herein by reference].
SUMMARY OF THE INVENTION
It is, therefore, an object of the present invention to provide a structure and method for indexing multi-media data comprising deriving keywords from a first media type (slides), matching the keywords to a second media type (video), identifying an appearance of the first media type in the second media type, and calculating a co-occurrence of the keywords and the appearance of the first media type in the second media type. The invention produces an index of the second media type based on the co-occurrence.
The identifying of the appearence of the first media type in the second media type includes generating keyframes from the second media type, extracting geometric keyframe features from the keyframes and geometric slide features from the first media type and matching the geometric slide features and the geometric keyframe features.
The invention further identifies background matching regions in the keyframes having colors matching colors of backgrounds in the second media type. The extracting of the geometric keyframe features is performed only in the background matching regions. The matching identifies which portion of the first media type has a highest number of geometric slide features matching geometric keyframe features in a keyframe.
The extracting process includes identifying changes in image intensity as edges, forming curves connecting the edges, identifying comers where the curves change direction, grouping the curves into curve-groups, and designating a sequence of three consecutive features in each of the curve-groups as basis triples. The matching process comprises computing coordinates of the basis triples, and identifying which portion of the first media type has a highest number of basis triples matching basis triples in a keyframe. Generating the keyframes comprises dividing the second media type into portions based upon scene changes and selecting one frame from each portion of the second media type as a keyframe. The calculating process comprises processing the multi-media data to extract relevance scores and time reference points of matches to individual media modes, identifying overlapping time periods of matching keywords and the appearance of the first media type in the second media type, and ranking the relevance of the overlapping time periods. The ranking can include finding an overlapping time period having a highest relevance score, segmenting the overlapping time period to identify beginning and ending events, and finding the largest number of different modes of overlap. Such modes can comprise audio, video, text, and graphic display.
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which:
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION
The invention is directed toward combining searches using multi-modal queries on multi-media data to index the various media sources. More specifically, the invention automatically indexes the information contained within one media source, based upon keywords from a different media source. For example, in one simplified embodiment, the invention determines keywords from captions on foils and produces an index for a related video based upon the co-occurrence of the foils in the video and keywords in the audio portion of the video. Since, the invention automatically finds the keywords from the foils and automatically produces the index of the video, without any manual contributions, the invention is extremely efficient and more consistent than manual indexing.
As discussed above, current methods of combining searches involve a simple Boolean combination such as AND/OR or operate based on the relevance scores of the individual modal matches. The invention goes well beyond conventional methods and takes into account an important aspect of multimedia content, namely, a relevance-based time co-occurrence of events.
Multimedia data such as learning material associated with an educational course can have multiple forms of data associated with it. Such data can be in the form of text derived from multiple sources, such as text books, scripts (for scripted presentations e.g., CEO communiques), text content within slides or overhead used during presentation, or speech converted to text from an analysis of the audio tract. Multimedia data also consists of audio information derived from the audio tract as well as visual information in the form of associated images. Finally, a video taping of the presentation/course can also be provided.
Referring now to
Each of the searches based on individual cues can return multiple locations in the media as matches which could possibly overlap in time. Combining these matches to develop a coherent response to a user's high-level semantic query or to automatically index any of the media sources, is an important and challenging problem. Performing multiple-mode queries is a very valuable tool for searching and browsing videos. Previous work in this area has concentrated on combining matches generated from textual sources, with current search engines exploiting one or more of the combining methods for text search such as Boolean AND/OR etc.
The invention provides a generic combining process for multimedia data that incorporates both relevance scores and co-occurrence of time intervals to automatically index the different media sources. Such a combination (e.g., relevance scores and co-occurrence) point to relevant semantic events in a lecture or course to identifying times when an index topic was discussed. For example, it can be reasonably concluded that places in the video in which a speaker put up a slide representing the index topic (seen, say, as a match for the slide pattern in the video tract) and also talks about it (heard, say, using words relating to the topic in the audio tract). Similarly, such co-occurrance events are more relevant to a topic or a common thread of discussion than video segments, in which an index topic was mentioned but not reinforced with a visual slide, or segments in which a slide was put up but never talked about.
During the presentation illustrated in
In the illustration shown in
Toward the end of the discussion of topic A (
As mentioned above, the invention automatically indexes the information contained within one media source, based upon keywords from a different media source. Therefore, using the previous example of indexing a video recording using keywords from the presentation foils, the invention searches the audio portion of the video to locate matching keywords and also searches the video portion for occurrances of when the foils appeared in the video.
However, the invention is not limited to simply indexing a video based upon foils which appear in that video. Instead, with the invention, any media source can be used by the invention to index any other media source. Therefore, index keywords could be derived from the text database
The ability to develop keywords from text is a well-developed art field. Therefore, there are a number of systems available to those of ordinary skill in the art that can be used to automatically identify the keywords upon which the index will be based. A detailed description of such systems is omitted here so as not to unnecessarily obscure the invention.
In the example discussed above, one such conventional system could base the keyword selection upon the frequency with which the words occur in the foil captions. In a very simplified system, a predetermined number (e.g., 10) of the most frequent words could comprise the keywords. However, as mentioned above, much more elaborate keyword detection systems are available and could readily be used with the invention.
The invention not only searches the audio portion of the video for index keyword matches, but also searches for occurrences when different foils appear in the video. The recognition of the slides within videos has conventionally been a difficult and error-prone process. The invention performs such matching by, first, generating a representation of the video stream as a set of keyframes. Next, in one embodiment, the invention identifies likely matches of electronic slide images and keyframes by color-matching the electronic slide and keyframe images. The invention extracts the geometric features of slide images and keyframe images. Lastly, the invention matches the keyframe images and slide images.
Thus, the invention matches slides supplied in electronic or other form, to their appearance in a video stream. This invention allows indexing for such events through natural language queries. The invention makes no assumptions on camera geometry. As a result, the invention is successful even when the slides appearing in video stream are partially occluded; color variations exist; and/or the content of slides is very similar. The invention also does not make assumption of presentation order in slides and can recognize them in any arbitrary order including recognizing their multiple occurrences in the video stream.
The invention takes, as input, a set of slides that were created in electronic form (using any conventional presentation program such as Powerpoint, Freelance Graphics, etc.) or color or black and white images of slides; and a video stream, and outputs the set of frames in the video stream along with the position in the frame where one of the electronic slides or images on the electronic slides are depicted.
As a preliminary step, the invention prepares the slides and video stream for matching. The invention generates slide images from the electronic slides. This can be done by extracting images from any conventional graphic presentation program using a method such as the one described by Niblack [Niblack, âSlide finder . . . â, in Proc. IEEE Workshop on Content-based Access of Image and Video Libraries, 1999, incorporated herein by reference]. The invention generates a set of keyframes from the video stream by processing the video stream to detect scene changes, cuts and different camera shots. The invention then separates the video stream into keyframes.
In other words, each time the scene changes in the video stream, the invention creates a still image of that portion of the video stream. In doing so, the invention divides the video stream into individual still images. This method affectively summarizes the video stream with a limited set of keyframes.
Typically, a video stream will concentrate upon a specific feature for a period of time, after which a different feature becomes the subject of concentration. Thus, the invention creates a keyframe (or still image) of each feature which was the primary subject at some point during the video stream. As discussed below, the invention reviews each keyframe to determine whether any of the slide images appear within any portion of the key frame.
Next, the invention performs a first comparison of the slides and keyframes. In one embodiment, the invention compares the background color of a given slide image to the keyframes and identifies regions of the keyframe in which the slide image background color was detected. In this embodiment, and the following processes (such as matching geometric features) are only performed on regions of the keyframes which have a background color corresponding to the background color of the slide image. This feature of the invention substantially increases the accuracy and speed of the inventive process by limiting the regions of the keyframes which are analyzed. However, in other embodiments, the invention can omit the process of matching background color if, for example, the user decides that a more thorough review of all aspects of the keyframes is more important than increasing the processing speed, or if it is determined that there is a poor color match between the video stream and the slides.
The invention also identifies geometric features of each slide image and the images contained in the identified regions of the keyframes. More specifically, the invention concentrates upon the intensity components of the images (e.g., by converting the images into gray images and retaining only the intensity component of the images). Then, the invention extracts edges from the intensity components. Edges generally occur where there is a sharp change in intensity.
The invention can then create line patterns (or âcurvesâ) by connecting the edges. Also, the invention identifies corner features, by locating places on the curves that have sharp changes in curvature. The corners serve as the basis features from which the geometric features for recognition are derived. A set of three such features forms a basis triple or an object-centered coordinate system in which positions of all other features can be expressed as affine coordinates as shown in FIG.
The affine coordinates serve as the basis geometric features to characterize slides as well as keyframe images. The advantage of using affine coordinates as geometric features is that they are affine-invariant. That is, even when the slide appears in the keyframe in a skewed, rotated or scaled form, its geometric features with respect to corresponding basis triples will remain the same, leading to robust recognition. To restrict the number of basis triples and affine coordinates to a reasonable number, groups of curves are formed from the initial edge map using a suitable grouping constraint such as intersection, parallelism, etc. Then triplets of consecutive corners on single curves in curve-groups are chosen as basis triples. This ensures that there is only a linear number of basis triples as against the cubic number of basis triples that would exist otherwise. Next, the invention computes the affine coordinates of all corners on all curves in curve-groups with respect to the basis triples. The curve-group number of each comer is associated with the affine coordinate to serve as a geometric region identity for the affine coordinate feature which will be useful during the matching of slides to their appearance in keyframes. The choice of features to form basis triples and affine coordinates is illustrated in FIG.
Next, the invention arranges the slide image affine coordinates in a manner that permits a comparison between the slide image affine coordinates and the keyframe image affine coordinates. The invention places the slide image affine coordinates in a balanced binary search tree (called a location hash tree) where each node in the tree includes the background color and an index of the affine coordinates e.g., (I
The invention indexes the location hash tree using the keyframe image basis triples. As a result of the index, the invention creates a histogram of slide image basis triples hit, and a histogram of the corresponding slide images hit. Each keyframe is paired with the slide image with the most hits. The slide which contains the most hits for a given keyframe is identified as corresponding to the given keyframe.
In addition, not every keyframe may contain a slide. Therefore, the invention also includes a feature which monitors the minimum threshold of basis triple matches before a keyframe is considered to correspond to any of the slides. This minimum threshold can be varied by the user depending upon the accuracy required and the quality of the video image.
The use of a limited number of basis triples and affine coordinates together with the manner of computing affine coordinates across curve-groups greatly reduces the time required to perform the search and the amount of data that is generated and analyzed. The invention also optionally reduces the time required time to perform the search and the amount of data analyzed by first comparing the background colors of the keyframes and the slide image to detect potential slide image-containing regions in the keyframes. This allows the invention to isolate the potential slide image containing regions on the keyframes and analyze only the images in these regions rather than the entire key frame.
The use of affine coordinates allows definition of the images in relation to the image itself, rather than a set of Cartesian coordinates or points on the slide or to keyframe. As a result, the invention permits searching for and recognizing an image at any location on the slide or key frame. The invention also reduces the potential for missing a partially occluded image on a keyframe. The feature extraction portion of the invention includes a built-in redundancy in the location hash tree.
As discussed above, the invention automatically generates keywords form foils and searches text (e.g., the audio portion of the video, or the handout material) for the occurrence of keywords. In addition, the invention automatically identifies points in the video where the foils appear. Next, the invention combines this information to automatically produce an index. Combining results of matches to multiple index keyword queries to generate an overall index is a challenging problem. Simple solutions such as ranking matches based on their individual cue scores are not sufficient as they cannot take into account the co-occurrence of matches, i.e., the fact that two or more of these matches occur in the same time interval.
Not all ways of taking the match co-occurrence into account are helpful either. For example, assigning a fixed score to the fact that there is an overlap is not necessarily helpful since this would weigh heavily overlapping segments the same as segments with small time overlap. Assigning a score to the extent of overlap and combining this with the individual scores is also not enough since the duration over which such overlaps occur could potentially span the entire video returning the entire video as a match and causing a loss in precision.
The invention combines multi-modal searches of multimedia data by exploiting the time overlap or co-occurrence of event matches in addition to their relevance scores in a new way. Specifically, the invention proceeds by dividing the overlapping time intervals into groups and uses the relevance scores of individual matches within the groups and their extent of overlap to produce an overall score. Each match from an individual cue can be denoted by âL
The invention forms a function of time given by
This function is 0 at times where there are no matches due to any cue. This forms a cumulative distribution of the individual match scores along the time axis.
The invention then takes the derivative of the above functionâT(t)/ât. Then, the invention notes the time period or the cycle of the derivative by noting three consecutive zero crossings of the derivative curve (e.g., the beginning of the time period when the derivative bell-shaped curve goes from negative to positive, the maximum of the derivative curve where the curve changes from positive to negative, and the end of the time period when the derivative curve again changes from negative to positive). For the special cases where the profile of the cumulative distribution has less than three consecutive zero crossings, the time period between two consecutive zero crossings can be taken.
Depending on the visual query, a number of different schemes of object detection and recognition in the video are admitted under this embodiment (item
The matching time durations and locations are as marked by the dark rectangles with the lighter larger rectangle representing the entire time line of the data (e.g., total video duration). The cumulative distribution obtained by combining the relevance scores is shown in row
The invention forms time interval groups between these triplets of adjacent zero crossings. The above process divides the time interval of matches into groups forming clusters that represent places where evidence for matches to one or more query types was found. The three consecutive zero crossings are based on the rationale that overlapping time intervals go through a pattern of overlapping cycles in which the overlap of interval grows and gradually shrinks. It can be seen in
Lastly, the above scores are normalized to lie between 0 and 1 using the maximum value of F
Thus, the invention groups the time intervals between three consecutive zero crossings, and combines individual scores as a linear combination of the individual scores with their overlaps to produce superior results when compared to conventional query programs. Finally, the notion of time co-occurrence of events combined with multiple match results produces unique advantages for the invention.
Existing methods of combining search results have been primarily targeted towards text searches of textual documents with Boolean and other relevance-scoping searches combinations now possible. The search of multimedia data based on visual, text and audio query types and their combination of results that take into account their time co-occurrence have not been considered conventionally.
Therefore, as shown above, the invention utilizes more than a search of the text within the database
The invention can be embodied in any number of different types of systems and executed in any number of different ways, as would be known by one ordinarily skilled in the art, given this disclosure. For example, as illustrated in
By taking into account the time co-occurrence of matches in addition to their individual relevance scores, this methods provides a higher degree of robustness to errors in producing an index. As is conventionally believed, manual visual and audio searches are notoriously lacking in precision. In such cases, the combining of evidence can tremendously enhance the robustness of the index produced.
Thus, the invention automatically develops an index for any type of media. The invention automatically produces topical index search items. In the above example, the topical search items were the most frequent terms occurring in the captions of foils. Then, the invention searches different media types (e.g., video and audio) to produce matches. The matches are then observed all along a time line to determine the co-occurrence of matches. The co-occurrence information is combined with relevance information to accurately index the media type in question. Since this process is fully automated, it is very efficient and does not require subjective judgment of an operator.
Further, the accuracy of the index produced with the invention is a very high for a number of reasons. In particular, by looking at multiple media mode searches (e.g., audio and video), the likelihood of correctly identifying the occurrence of the index search item dramatically increases. Conventional efforts which observe only a single media type produce false positive and false negative results much more easily. To the contrary, the different media-type searches utilized by the invention provide a cross-check to eliminate false positive and false negative results. In addition, the invention goes beyond a simple identification of a co-occurrence of a search term by including relevance information to more precisely evaluate which co-occurrence has the highest probability of correctly identifying material relating to the index search item.
While the invention has been described in terms of preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.
Field of search:
6021412 - Method and system for automatically adding graphics to a document to illustrate concepts referred to therein
6353450 - Placing and monitoring transparent user interface elements in a live video stream as a method for user input
Browse by classes
Automotives and Transportation
Business and Commerce
Fashion and Accessories
Hardware and Tools
Health and Medicine
Materials and Material Science
Paper and Office Materials