Chapter 5: Pattern Matching – Practical Guide to Machine Vision Software: An Introduction with LabVIEW

5
Pattern Matching

Pattern matching is a method for finding regions in a grayscale image that match a reference image pattern. If the initial image source is a color image, the image needs to be converted to grayscale first in order to use pattern matching. The pattern matching VI uses a reference or template image to find like images within a new image regardless of location, rotation, or scaling of the template.

Pattern matching is often used to locate the positions of a fiducial mark, or unique characteristic features, of an object in an image. You can use the positions to compute length, angles, and other measurements. As a result, pattern matching has been widely used in various applications such as alignment, gauging, and inspection. Pattern matching has an advantage over particle analysis or edge detection because the pattern search does not rely on distinct brightness of the imaged object compared with the image background.

Figure 5.1 shows an example VI provided in LabVIEW, which can be found in the following folder:

Figure 5.1 Pattern matching example VI.

As seen in Figure 5.1, pattern matching requires several steps.

Run the example VI above to gain an understanding of LabVIEW pattern matching concept:

  1. Click on Image File Path to select and read in an image file (Figure 5.1 ).
  2. Select Create Template (Figure 5.1 ) and a pop-up ROI Constructor window will appear. Using the Rectangle ROI tool, select the part of the image to be used as the reference or template (right mouse button on the image to drag out a rectangle). Then, select OK to return to main VI. The selected image portion will appear as the Template, as seen in Figure 5.1 and will be used for pattern matching in a target image.
  3. Select Match (Figure 5.1 ) to search for the matched pattern in the image. The matching results include locations, number of matches, and boundary rectangle information, as seen in Figure 5.1 . The found patterns will then be shown on Image display via overlay, as seen in Figure 5.1 .

Figure 5.2 Image for pattern matching example.

5.1 Pattern Matching Using Vision Assistant

To find patterns that match with a reference pattern, Vision Assistant will be used for this section. Perform the following steps:

  1. Acquire or read in an image for image processing by using Vision Assistant.
  2. Color images will need to be converted to grayscale via the Color Plane Extraction function (Processing Function: Color» Color Plane Extraction). This process has been discussed in Chapter 3.
  3. From Vision Assistant, select the pattern matching function from Processing Functions: Machine Vision»Pattern Matching.
  4. Select New Template from Pattern Matching Setup, as seen in Figure 5.3 .
  5. You will be presented with a pop-up window to select a region of the image as the reference pattern for the pattern matching operation using the mouse, as seen in Figure 5.4. After the image template is selected, click on Next to proceed.
  6. From the NI Vision Template Editor as shown in Figure 5.5, you can select the regions in the template image to ignore by using the drawing tool () to draw a region around the object. Here, the ignored area is indicated in Figure 5.5. This area will not be used for pattern matching and more accurate matching results can be obtained by excluding unnecessary parts.
  7. Select Next to adjust the desired X and Y coordinates within the matched object, as seen in Figure 5.6. This Match Offset point will identify the object's point of location when it is found in the searched image. Now, select Finish to save the Template image for pattern matching.
  8. The resulting template image and template image path can now be seen in the Template tab of the Pattern Matching Setup, as seen in Figure 5.7.
  9. By selecting the Settings tab from Pattern Matching Setup in Figure 5.7, you can set parameters values, as seen in Figure 5.8 and . Table 5.1 summarizes the parameter settings for pattern matching.
  10. Select a ROI rectangle in the image to define the area (Figure 5.8 ) to perform pattern matching.
  11. Observe and confirm the pattern matching results, as seen in Figure 5.8 . The results include X and Y coordinate locations, angle of rotation, and score. Three patterns are found in this example. Note that matched pattern 1 has the score value of 1000 since this pattern was used as the reference template image. The scores of other matched patterns are 983.9 and 727.5. From the score values, the level of similarity of matched patterns can be evaluated. If you set the minimum score to 900, the pattern with score value of 727.5 will not be included in the set of found objects. Note that the score can be affected by differences in lighting conditions as well as object similarity. The rotation angle of matched patterns with respect to reference pattern can be obtained, as seen in Figure 5.8 . If you had set allowable angle range to 30°, the pattern with a 329.9° rotation (result 3) would not have been reported in the search results.
  12. If the parameters are acceptable, select OK from the Pattern Matching Setup to finish pattern matching using Vision Assistant. You are now ready to create a VI to build your own program for pattern matching.

Figure 5.3 Create New Template for pattern matching.

Figure 5.4 Selection of template region from the image.

Figure 5.5 Regions to Ignore (Define Pattern Matching Mask).

Figure 5.6 Matching offset adjustment.

Figure 5.7 Pattern matching setup.

Figure 5.8 Pattern matching setup and the results.

Table 5.1 Parameters for pattern matching.

Number of matches to find (Figure 5.8 ) Minimum score (Figure 5.8 ) Search for rotated pattern (Figure 5.8 )
In this example, the value of 3 is used to find three ovals that match with template pattern. Only objects with higher matching score than the minimum score are considered valid in the search. The score value ranges from 0 to 1000. If matching score of an object is close to 1000, the matched object is likely to be perfectly matched with the reference template. However, if the value of minimum score is set too high, some of matched patterns with low matching score may not be found. Rotated pattern can be found by using Search for Rotated Patterns. Note that allowable angle of 180° ranging from −90 to 90 is selected since the oval is symmetrical in shape.

5.2 LabVIEW Code Creation and Modification

Now we can use Vision Assistant to create a LabVIEW VI (Tools»Create LabVIEW VI). The created VI needs to be modified according to your needs. LabVIEW programming skills are required for this purpose. You may want to skip Sections 5.2 and 5.3 if you prefer an easier approach with the use of Vision Express, which will be presented in Section 5.4.

Within the LabVIEW VI Creation Wizard, you may select Image File as the image source (step 3 of 4). Note that the VI code that specifies the image source may not be appropriate for the intended application and may need to be modified. To build the SubVI for your applications efficiently, select the indicators and controls for the created VI (step 4 of 4), as seen in Figure 5.9.

Figure 5.9 Selection of controls and indicators for pattern matching VI.

The created VI from Vision Assistant will need to be modified in order to be used for real-time analysis. In this section, modification techniques are discussed for real-time pattern matching. However, before any modifications are performed, save the created LabVIEW code under the name pattern_maching_SUB.vi. The resulting LabVIEW SubVI has inputs and outputs, as seen in Figure 5.10.

Figure 5.10 SubVI for pattern matching.

Figure 5.11 shows the SubVI that has been slightly modified from the created VI to add controls and indicators, as shown in Figure 5.10. Note that most of the indicators and controls were created automatically by the VI Creation Wizard (step 4 of 4). The parts of the VI to be discussed are indicated as circled numbers in Figure 5.11.

  1. shows an Image control that is placed on the front panel (Vision»IMAQ Image ctl) so that the image can be used as an input of the SubVI. In this way, the SubVI can receive an acquired image from the main or calling VI.
  2. shows error controls to handle possible errors for debugging purpose and used as an input of the SubVI.
  3. shows the control for a ROI Descriptor, which was automatically generated during the VI creation. The ROI is often used to define image processing area. Initially, the ROI Descriptor control was given default values, which were defined in Vision Assistant prior to the SubVI code creation. However, the ROI information needs to be changeable either interactively or programmatically according to the location of object. The ROI information will be defined and updated from the main VI.
  4. shows the control for the Template File Path, which is the file path for the reference template image. The File Path control was created in the control selection from VI Creation Wizard. Note that the reference image was defined and saved using the Vision Assistant, as seen in Figure 5.7.
  5. shows the control for the pattern matching parameters. From the front panel, the number of matches requested, minimum match score, and angle range can be modified. Note that the default values for this control were generated from Vision Assistant. By using this control as an input for the SubVI, the parameters can be modified and passed from the main VI.
  6. shows the pattern matching part, which searches for the matched pattern.
  7. shows the indicator, Matches, which contains the results from pattern matching. The results include position, angle, scale, and bounding box. Matches is an array of clusters and will have the same size as the number of found patterns. The position and bounding box information will be used to overlay the searched patterns on image.
  8. Number of Matches receives the number of matched (or searched) objects.
  9. shows a Vision Image.ctl indicator that is placed on the front panel so that the image can be used as an output of the SubVI.

Figure 5.11 Block diagram of pattern matching SubVI.

5.3 Main VI for Pattern Matching

Figure 5.12 shows the main VI's block diagram for pattern matching. In this main program, there are two SubVIs: one is to find matched patterns as seen in and the other is to overlay the matched pattern on image display as seen in .

Figure 5.12 Main VI for pattern matching.

The method to make the SubVI for pattern matching (pattern_matching_SUB.vi) was discussed in previous section. In the main program, all the inputs of the SubVI in are provided to perform the pattern matching. The inputs of the SubVI include acquired images from a camera, ROI information (property node of Image ROI) defined by mouse selection, and the template file path. The property node of the Image ROI (Figure 5.12 ) can be generated by using the right mouse click on the Image indicator (Figure 5.12 ).

To indicate the matched results on the Image display, an overlay function is used (Figure 5.13). For this purpose, you may reference the example provided in LabVIEW from C:\Program Files\National Instruments\LabVIEW 2013\Examples\Vision\Pattern Matching. The example VI, Overlay Pattern Matching Results.vi effectively overlays the results of pattern matching on the image display. Here, we slightly modify this code as seen in Figure 5.14.

Figure 5.13 Overlay SubVI from NI provided example program.

Figure 5.14 Overlay for Matches.

Matches, which is one of the inputs to the SubVI is a cluster array, has the same number of elements as matched objects. To overlay the matched results, autoindexing of the Matches array in the FOR loop is used to overlay each of the resulting matched objects. There are two sets of location information for each matched object: position and boundary box. Both locations are used for overlaying the matched results.

In Figure 5.14 , the Overlay Oval function (Vision and Motion»Vision Utilities»Overlay»Overlay Oval) is used to draw a circle around the position indicator of the matched objects. Additionally, two overlay line functions (Figure 5.14 and ) are used to draw crosses at the indicator positions. The bounding box information, which defines the boundary of each object, can be used to draw a boundary line by using the Overlay Multiple Lines function (Figure 5.14 ).

Figure 5.15 shows the final result of pattern matching when the main VI is run. As seen in Figure 5.15, template image file path as saved in Vision Assistant is selected in (refer to Figure 5.5). In the process of running the VI, define the ROI area (Figure 5.15 ) on image display after selecting the rectangle tool (Figure 5.15 ) from ROI tool menu. You can adjust the pattern matching parameters (Figure 5.15 ) in order to provide tuned real-time pattern matching results, as seen in Figure 5.15 . As a result of pattern matching, the position, angle, size, and score of the objects are determined. Note the angle indicates the amount of rotated angle of matched pattern with respect to reference template image. Matched results (Matches) are in the form of an array, of which the size is the same as the number of located matched patterns.

Figure 5.15 Pattern matching results.

Note that the acquired color image was converted to a grayscale image because the pattern matching function requires a grayscale image. If you want to show original color image, Image Copy can be used to keep original image. The IMAQ Image Create function in Figure 5.16 is used to allocate memory for the original image acquired by camera. The Overlay SubVI in Figure 5.16 can then be configured to overlay on the original image. To keep the original image, the copied image should be used for image processing. The image processing includes image conversion of the color image to a grayscale image. For this purpose, additional image memory is allocated, as seen in Figure 5.16 , and is connected to the destination (Dst) of IMAQ Copy (Vision and Motion»Image Management»IMAQ Copy) in Figure 5.16 in order to make a copy of the acquired image. The copied image is connected to the pattern matching SubVI (pattern_matching_SUB.vi) in Figure 5.16 . In this way the original color image can be used to display the result, while the copy is used for conversion to grayscale and image processing.

Figure 5.16 Modified VI to keep original image.

5.4 Vision Assistant Express

Creating a pattern matching VI by using Vision Express is discussed. Prior to creating pattern matching using Vision Express, a VI for the continuous image acquisition using the Grab function (Vision and Motion»NI-IMAQdx»Grab) can be used, as shown in Figure 5.17, to capture and save an image for analysis using Vision Assistant Express. The image can be saved by right mouse clicking on the front panel image display and selecting Save image from the pop-up menu.

Figure 5.17 Vision Assistant Express.

Then, the Vision Assistant Express function is dragged down onto block diagram, as seen in Figure 5.17.

As a result of the drag the function onto the block diagram, the Vision Express wizard will appear. As a first step, open the saved image to use for pattern matching using Vision Assistant. Then, the same process (described in from Figures 5.35.8) can be used for pattern matching. Since it is the same Vision Assistant process for pattern matching, a detailed discussion will be skipped. As a final step, you can select controls and indicators, as seen in Figure 5.18, by clicking on Select Controls. In this way, the inputs and outputs can be easily accessed by the LabVIEW main VI.

Figure 5.18 Selection of controls and indicators from Vision Assistant Express.

As a final step of Vision Assistant Express, select Finish to return to LabVIEW. The created Vision Assistant Express VI for pattern matching will have controls and indicators, as seen in Figure 5.19 . The image source and destination controls are automatically selected and generated so that the acquired image (related to create memory in Figure 5.20 ) is connected to Image Src (source) and the additional created memory (Figure 5.19 ) is connected to Image Dst (destination) for copying and processing the image. In this way, the original acquired image is left unchanged, while the Image Dst Out can be converted to a grayscale image for pattern matching. Finally, the overlay SubVI described in Figure 5.13 is used for overlaying the matched results on the original image, as seen in Figure 5.19 .

Figure 5.19 Final code for pattern matching using Vision Assistant Express.

Figure 5.20 Acceptable pattern search based on pattern matching.

By using the Vision Assistant Express, image processing subroutines can be easily created and directly inserted into LabVIEW VIs. In Chapter 6, Vision Assistant Express will be mainly discussed as a means to create VIs for your application.

Exercise 5.1

Nano-imprinted patterns are inspected to find any defects. The image has 100 imprinted patterns. Use the pattern matching to obtain the correct number of valid patterns. As matched results, display the number of acceptable patterns and overlay the pattern with boundary rectangle around acceptable patterns.

Exercise 5.2

Find the image C:\Program Files\National Instruments\Vision\Example\Images\Particle 01.png (Figure 5.21). From the image, find the screws by using pattern matching and overlay the boundary box at position of matched screws. Incomplete screw images should be ignored.

Figure 5.21 Finding screws using pattern matching.