Joseph Tarango
Department of Computer Science and Engineering
Bourns College of Engineering
University of California, Riverside

Home
CS122A | Syllabus
Intro to FPGAs | Basics of VHDL | Intro to Xilinx | Turnin
Lab 1 | Lab 2 | Lab 3 | Lab 4 | Lab 5 | Project

Project : Accelerating Software Computation with Custom Hardware

Introduction

Image processing is a specific field of study in computer vision. Image processing itself is any form of signal processing for which the input is an image, such as a photograph or video frame; the output of image processing may be either an image or, a set of characteristics or parameters related to the image. Most image-processing techniques involve treating the image as a two-dimensional signal and applying standard signal-processing techniques to it. In the project for this class, we are going to perform edge detection using software, then using a combination using software and a coprocessor.

Edge detection refers to the process of identifying and locating sharp discontinuities in an image. The discontinuities are abrupt changes in pixel intensity, which characterize boundaries of objects in a scene. Classical methods of edge detection involve convolution on the image with an operator (a 2-D filter), which is constructed to be sensitive to large gradients in the image while returning values of zero in uniform regions. There are an extremely large number of edge detection operators available, each designed to be sensitive to certain types of edges. Variables involved in the selection of an edge detection operator include:
  • Edge orientation: The geometry of the operator determines a characteristic direction in which it is most sensitive to edges. Operators can be optimized to look for horizontal, vertical, or diagonal edges.

  • Noise environment: Edge detection is difficult in noisy images, since both the noise and the edges contain high-frequency content. Attempts to reduce the noise result in blurred and distorted edges. Operators used on noisy images are typically larger in scope, so they can average enough data to discount localized noisy pixels. This results in less accurate localization of the detected edges.

  • Edge structure: Not all edges involve a step change in intensity. Effects such as refraction or poor focus can result in objects with boundaries defined by a gradual change in intensity. The operator needs to be chosen to be responsive to such a gradual change in those cases. Newer wavelet-based techniques actually characterize the nature of the transition for each edge in order to distinguish, for example, edges associated with hair from edges associated with a face.
There are many ways to perform edge detection. However, the majority of different methods may be grouped into two categories Gradient and Laplacian; in this project, we will only work with gradient. The gradient method detects the edges by looking for the maximum and minimum in the first derivative of the image.

Sobel Operator (Gradient)

The Sobel operator is used in image processing, particularly within edge detection algorithms. Technically, it is a discrete differentiation operator, computing an approximation of the gradient of the image intensity function. At each point in the image, the result of the Sobel operator is either the corresponding gradient vector or the norm of this vector. The Sobel operator is based on convolving the image with a small, separable, and integer valued filter in horizontal and vertical direction and is therefore relatively inexpensive in terms of computations. On the other hand, the gradient approximation, which it produces, is relatively crude, in particular for high frequency variations in the image.

The operator consists of a pair of 3x3 convolution kernels; a kernel a constant matrix. One kernel is simply the other rotated by 90 degrees. For the calculation we use kernel’s: Kx = { {-1, 0, 1}, {-2, 0, 2}, {-1, 0, 1} }, Ky = { {1, 2, 1}, {0, 0, 0}, {-1, -2, -1} }. The calculation uses the kernels with the picture P, where Gx=Kx*P and Gy=Ky*P.

These kernels are designed to respond maximally to edges running vertically and horizontally relative to the pixel grid, one kernel for each of the two perpendicular orientations. The kernels can be applied separately to the input image, to produce separate measurements of the gradient component in each orientation (call these Gx and Gy). These can then be combined together to find the absolute magnitude of the gradient at each point and the orientation of that gradient. The gradient magnitude is given by: |G|=sqr(Gx2+Gy2). Typically, an approximate magnitude is computed using: |G|=|Gx|+|Gy|, which is much faster to compute. The angle of orientation of the edge (relative to the pixel grid) giving rise to the spatial gradient is given by: θ=arctan(Gy/Gx).

The result of the Sobel operator is a 2-dimensional map of the gradient at each point. It can be processed and viewed as though it is itself an image, with the areas of high gradient (the likely edges) visible as white lines. The following images illustrate this, by showing the computation of the Sobel operator on a simple image.

Sample code for Sobel operation:

/*Function performs the partial Sobel convolution on a single pixel value. Parameters: x,y correspond to the pixel location on the image. The return value is the convoluted pixel at x,y.*/

unsigned int SobelOperator(const unsigned int& x, const unsigned int& y, const unsigned int& picture[][]) {

      //constants

      const int MATRIX_MAX = 3;

      const unsigned int PIXEL_MAX = 255;

     

      //value for the new cell calculation

      unsigned int GXpixel, GYpixel, Gpixel = 0;

 

      //matrix for the Kx convolution

      const int SobelKxfilterMatrix[3][3] =

      { {-1, 0, 1}, {-2, 0, 2}, {-1, 0, 1} };

 

      //matrix for the Ky convolution

      const int SobelKyfilterMatrix[3][3] =

      { {-1,-2, -1}, {0, 0, 0}, {1, 2, 1} };

     

      //for addressing into filter array and into picture itself

      int iFilter, jFilter;

      int iPic, jPic;

 

      //Loop to iterate over picture and filter window to perform Sobel operation

      //The loop is not optimized and does not check for boundary exceptions.

      for(iFilter = 0 , iPic =-1;

            iFilter < MATRIX_MAX && iPic >= 1;

            iFilter++, iPic++) {

            for(jFilter = 0 , jPic =-1;

                  jFilter < MATRIX_MAX && jPic >= 1;

                  jFilter++, jPic++) {

                 

                  GXpixel += (unsigned int)(picture[x+iPic][y+jPic] *

                  SobelKxfilterMatrix[iFilter][jFilter]);

                 

                  GYpixel += (unsigned int)(picture[x+iPic][y+jPic] *

                  SobelKyfilterMatrix[iFilter][jFilter]);

            }

      }

      //check for pixel saturation

    if (GXPixel>PIXEL_MAX) {GXpixel=PIXEL_MAX;}

    if (GYPixel>PIXEL_MAX) {GYpixel=PIXEL_MAX;}

     

      //normalize pixel

    Gpixel = (unsigned int)sqrt(pow((double)GXpixel,2)+pow((double)GYpixel,2));

     

      //check for pixel saturation

    if (Gpixel>PIXEL_MAX) {Gpixel=PIXEL_MAX;}

 

      return Gpixel;

}

Prewitt Operator (Gradient)

Prewitt operator is similar to the Sobel operator and is used for detecting vertical and horizontal edges in images. However, the kennels differ as follows: Kx = { {1, 1, 1}, {0, 0, 0}, {-1, -1, -1} }, Ky = { {-1, 0, 1}, {-1, 0, 1}, {-1, 0, 1} }.

Software

Part 1: Software Implementation

Perform the Prewitt and Sobel filter on the given image in software using C/C++ on the lab machines, then verify the operation visually.

You may use your own images, just remember your space is limited on the FPGA. For all others, the pictures.zip has several images included.

Recommended Steps:
1. EasyBMP to Data structure
Read the images color channels using the EasyBMP library and put values into a data structure. The libraries, documentation, and a example for EasyBMP is provided below.
EasyBMP_1.06.zip
EasyBMP_Documentation_1.06.00.zip
BMP_Image_Parce_to_RGB.cpp

2. Prewitt and Sobel Operator Functions
Develop functions for the Prewitt and Sobel operator.

3. Perform the Edge Detection
Create a program to use the operator functions and output to a new BMP file.

4. Verify Visually
Visually compare the original image to the new filtered images.

MicroBlaze

Part 2: MicroBlaze Software

Create a self-contained MicroBlaze software implementation of the Prewitt and Sobel filter. Hard -code the image into the software, perform the filters, output the data in a standard format to the UART, and convert the data to a BMP image.

Recommended Steps:
1. BMP to RGB
Use Easy BMP to convert a 100 x 100 image to a RGB Map or  B/W (Black/White) Map.

2. (RGB or B/W) to C Matrix
Modify your program to take the (RGB or B/W) Map data and generate a file with a C matrix.

3. Sobel Filter Software
From the source code above, develop a Sobel filter function.

4. Create a MicroBlaze program
From the previous parts, create a program that reads a C matrix, performs the Sobel operation (Part 3), writes to another C matrix, and outputs the filtered image to the UART. Note: The processed data can be read from the Cutecom log file.

5. UART Data to BMP
Taking the UART data, make a C/C++ program to convert UART data to a BMP image.

Profile MicroBlaze Software

Part 3: Course Gained Evaluation of C Instructions on the MicroBlaze

From the cycle counter in the previous lab, profile the number of clock cycles to perform the Prewitt and Sobel function on the MicroBlaze. Run the profile several times and calculate the average from a set of at least ten runs.

Coprocessor Hardware

Part 4: Sobel Hardware Module

Taking what you have learned from the Prewitt and Sobel operator, make hardware modules to calculate a single pixel. The hardware module inputs will be the 9 pixel values and the output be the calculated pixel value. Replace the software implementation with the hardware module and output the filtered image to the UART and recombine the data into a BMP image for visual confirmation.

Profile Coprocessor Hardware

Part 5: Course Gained Evaluation of C Instructions

Using the cycle counter profile the number of clock cycles to do the Prewitt and Sobel function on the Coprocessor. Run the profile several times and calculate the average from a set of at least ten runs.

Direct Display

Part 6: Course Gained Evaluation of C Instructions

Using what you have already created, create a new peripheral to use a BRAM in combination with your VGA driver to display the image on the monitor Screen.

Application Specific Hardware

Part 7: Application Specific Processor

Create a RTL design of the Prewitt and Sobel Filter and display it to the VGA monitor. The design will read your image from a BRAM, run the Prewitt and Sobel Filter on the data from the BRAM, write the processed image to a BRAM, and display it to the VGA monitor using your previously made VGA Driver. Use a switch to reset the computation and change from Prewitt to Sobel.

Recommended Steps:
1. Draw HLFSM
Create a high level finite state machine of the overall operation and verify correctness.

2. Create Course-Grained Datapath Design
Evaluate your HLFSM and draw data path with black box operators.

3. Create Fine-Grained Component Design
From your previous source code, develop BRAM initialization, Prewitt operator, Sobel operator, and other necessary operators.

4. Connect Components on Data-Path
Combining the designs from previous steps, incrementally connect components and test the data flow. The step should involve several testbenches to verify integrity of the RTL components and HLFSM functionality. IE: Input image BRAM to operators, operators to output BRAM, BRAM to VGA.

5. Display to VGA
Visually verify the result is comparable to the software implementation.

Profile Application Specific Hardware

Part 8: Course Gained Evaluation

Using cycle counter profile the number of clock cycles to do the Prewitt and Sobel function on the application specific hardware. Run the profile several times and calculate the average from a set of at least ten runs. The cycle counter should start when the calculation beings and stops when the calculation ends. The cycle calculation can be output on the LCD, UART, or VGA over the image.

Project Report

Each group must submit one report. The report can significantly affect your grade based on how many parts of the project each group completed. Each page must be single spaced with at least 10-point font in Arial or Garamond or Times New Roman at 1-inch margins on all sides. The report will include a Title Page, Description of Project, User Guide, Technical Design Specifications, Results and Performance Evaluation, and Conclusion.

The Title page will contain the project title, course name, university, instructor and teaching assistant names, names of the group members, the members' emails, and a statement of original work. The statement will be as follows "We, [member 1] and [member 2], attest that this document and the work described herein is entirely our work".

The Description of Project will describe the constructed project. The description should include a brief description of the project in the members own words. The reader of this section should be able to understand the purpose of the project, the theory related to project content, and high level implementation.

The User Guide section will describe how to employ your project. Is should not make reference to unnecessary technical details and should be easy clear enough so any engineer can reproduce results. You can assume the user will have all source files in your hierarchy.

The Technical Design Specifications section will describe the fine details of your project. Is should not make reference to unnecessary technical details and should be easy clear enough so any other student in the class can reproduce results. It is encouraged to include block diagrams, data paths, and other related figures.

The Results Performance Evaluation section will analytically show the performance of MicroBlaze software to coprocessor and application specific processor implementations. Show these results by a  brief description, before and after images, equations used in calculation, and a graph comparison.

The Conclusion section will state a summary, problems encountered and solved, suggestion for better approaches, division of labor (itemized), contact methods, and knowledge learned.

Here are templates: projectreport.docx  projectreport.pdf

Note: The project turnin will follow the specification from the turnin page,  except include all Xilinx files in the extra folder and the report in DOCX and PDF format.

References

http://zone.ni.com/devzone/conceptd.nsf/webmain/37BF3B76F646EF5286256802007BA00F

H.S. Neoh, and A. Hazanchuk.2004. Adaptive edge detection for real-time video processing using FPGAs. GSPx 2004 confernce.
URL: http://www.altera.com/literature/cp/gspx/edge-detection.pdf

N. Anastasiadis, I. Sideris, and K.l Pekmestzi. 2010. A fast multiplier-less edge detection accelerator for FPGAs. In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC '10). ACM, New York, NY, USA, 510-515. DOI=10.1145/1774088.1774193
URL: http://doi.acm.org/10.1145/1774088.1774193

M.D. Heath, S. Sarkar, T. Sanocki, K.W. Bowyer. "A robust visual method for assessing the relative performance of edge-detection algorithms," Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol.19, no.12, pp.1338-1359, Dec 1997
DOI = 10.1109/34.643893
URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=643893&isnumber=14015