Curso de Visão Computacional Marcelo Gattass 2009.2 Processamento de imagens capturadas para algoritmos de Visão Computacional Curso de Visão Computacional Marcelo Gattass 2009.2

Resumo Motivação (aplicações) Requisitos Deteção de pontos e arestas Analise da variação local Deteção de arestas Deteção de pontos Haris SHIFT Viola Jones

object instance recognition (matching) Motivação object instance recognition (matching) David Lowe

Sample results using the Viola-Jones Detector Motivação Sample results using the Viola-Jones Detector Notice detection at multiple scales

Small set of 111 Training Images Motivação Small set of 111 Training Images

How do we build panorama? Motivação How do we build panorama? We need to match (align) images Darya Frolova, Denis Simakov, The Weizmann Institute of Science, 2004

Matching with Features Motivação Matching with Features Detect feature points in both images Darya Frolova, Denis Simakov, The Weizmann Institute of Science, 2004

Matching with Features Motivação Matching with Features Detect feature points in both images Find corresponding pairs Darya Frolova, Denis Simakov, The Weizmann Institute of Science, 2004

Matching with Features Motivação Matching with Features Detect feature points in both images Find corresponding pairs Use these pairs to align images

Matching with Features Motivação Matching with Features Problem 1: Detect the same point independently in both images no chance to match! We need a repeatable detector Darya Frolova, Denis Simakov, The Weizmann Institute of Science, 2004

Matching with Features Motivação Matching with Features Problem 2: For each point correctly recognize the corresponding one ? We need a reliable and distinctive descriptor Darya Frolova, Denis Simakov, The Weizmann Institute of Science, 2004

Example: Build a Panorama Motivação Example: Build a Panorama M. Brown and D. G. Lowe. Recognising Panoramas. ICCV 2003

Motivação Photosynth

Busca de padrões geométricos Motivação Mauricio Ferreira, Diogo Carneiro e Carlos Eduardo Lara Visão 2005 http://www.tecgraf.puc-rio.br/~mgattass/ra/trb05/T1-VirtualOnReal/MauricioFerreira_DiogoCarneiro_CarlosEduardoLara/

Requisitos Uma caracteristica deve ser robusta o suficiente para continuar se destacando mesmo quando a cena é capturada em diferentes condições

Requisitos Tom Duerig Types of invariance Illumination

Requisitos Tom Duerig Types of invariance Illumination Scale

Tom Duerig Types of invariance Illumination Scale Rotation

Types of invariance Illumination Scale Rotation Affine Tom Duerig Requisitos Tom Duerig Types of invariance Illumination Scale Rotation Affine

Types of invariance Illumination Scale Rotation Affine Requisitos Tom Duerig Types of invariance Illumination Scale Rotation Affine Full Perspective

Tipos de características de uma imagem Globais: histograma, conteúdo de freqüências, etc... Locais: regiões com determinada propriedade, arestas, cantos, curvas, etc... 21

Modelo Matemático: Função Tipos Modelo Matemático: Função 0% 20% 40% 60% 80% 100% Níveis de cinza Posição ao longo da linha x u v L L(u,v) Função

Derivadas direcionais Requisitos Derivadas direcionais f(x,y) y x 23

Tipos Norma da derivada

Tipos Mínimo de formas quadráticas de matrizes simétricas positivas definidas

Autovetores e autovalores de matrizes simétricas positivas definidas Tipos Autovetores e autovalores de matrizes simétricas positivas definidas

Minimização como um problema de autovalores y p x' y' x mínimo

Teorema Espectral (Teorema dos Eixos Principais) Toda matriz simétrica S (S = ST ) pode ser fatorada em:  - matriz diagonal real Q – matriz ortogonal, formada pelos autovetores de S MMachado

Estimando Orientação Local em Imagens Usando o Teorema Espectral no problema de orientação: Mudança de base por rotação J é máximo se y só tem componente na direção do autovetor de maior autovalor MMachado

Identificando Estruturas Lineares com PCA Problema. Dados os vetores v1,...,vk, em N dimensões, estimar a orientação média quando o sinal de vi é ignorado. Solução. A orientação média é dada pelo eixo principal da matriz MMachado

Matriz de Variância-Covariância N variáveis M observações Variância Covariância Matriz de Variância-Covariância MMachado

PCA X2 X1 Maior Componente Principal maximiza a variância minimiza a variância Menor Componente Principal MarcoMachado X1

Interpretação geométrica (max)-1/2 (min)-1/2 1, 2 – autovalores of S 33

PCA Variância total = soma das variâncias Variância total = traço de S Eixos principais também representam a variância total do conjunto de dados. Primeiro eixo: 1/traço(S) Segundo eixo: 2/traço(S) MarcoMachado

Estimando Orientação Local em Imagens Interpretação dos Autovalores 1=0, 2=0 Intensidade constante, sem estrutura 1>0, 2=0 Estrutura linear (invariante por deslocamento em uma única direção) 1>0, 2>0 A estrutura desvia do modelo de estrutura linear Ruído Curvatura Múltiplas orientações 1=2 Estrutura isotrópica

Comportamento local: Classificação 2 “Edge” 2 >> 1 “Corner” 1 e 2 são grandes, 1 ~ 2; aumenta em todas as direções 1 e 2 são pequenos; Quase constante em “Edge” 1 >> 2 “Flat” 1 36

Detecção de arestas 1 > 2

Operadores clássicos Prewitt’s Diferencia Suavisa 38

Openadores clássicos Sobel’s Diferencia Suavisa 39

Detector de arestas 40

Quality of an Edge Detector Robustness to Noise Localization Too Many/Too less Responses True Edge Poor robustness to noise Poor localization Too many responses Khurram Hassan-Shafique 41

Canny Edge Detector Criterion 1: Good Detection: The optimal detector must minimize the probability of false positives as well as false negatives. Criterion 2: Good Localization: The edges detected must be as close as possible to the true edges. Single Response Constraint: The detector must return one point only for each edge point. Khurram Hassan-Shafique 42

Hai Tao 43

The result General form of the filter (N.B. the filter is odd so h(x) = -h(-x) the following expression is for x < 0 only) Camillo J. Taylor 44

Approximation Canny’s filter can be approximated by the derivative of a Gaussian Canny Derivative of Gaussian Camillo J. Taylor 45

Canny Edge Detector Convolution with derivative of Gaussian Non-maximum Suppression Hysteresis Thresholding Khurram Hassan-Shafique 46

Algorithm Canny_Enhancer Smooth by Gaussian Compute x and y derivatives Compute gradient magnitude and orientation Khurram Hassan-Shafique 47

Canny Edge Operator Khurram Hassan-Shafique 48

Canny Edge Detector Khurram Hassan-Shafique 49

Canny Edge Detector Khurram Hassan-Shafique 50

Algorithm Non-Maximum Suppression We wish to mark points along the curve where the magnitude is biggest. We can do this by looking for a maximum along a slice normal to the curve (non-maximum suppression). These points should form a curve. There are then two algorithmic issues: at which point is the maximum, and where is the next one? Khurram Hassan-Shafique 51

Non-Maximum Suppression Suppress the pixels in ‘Gradient Magnitude Image’ which are not local maximum Khurram Hassan-Shafique 52

Non-Maximum Suppression Khurram Hassan-Shafique 53

Non-Maximum Suppression Khurram Hassan-Shafique 54

Hysteresis Thresholding Khurram Hassan-Shafique 55

Hysteresis Thresholding If the gradient at a pixel is above ‘High’, declare it an ‘edge pixel’ If the gradient at a pixel is below ‘Low’, declare it a ‘non-edge-pixel’ If the gradient at a pixel is between ‘Low’ and ‘High’ then declare it an ‘edge pixel’ if and only if it is connected to an ‘edge pixel’ directly or via pixels between ‘Low’ and ‘ High’ Khurram Hassan-Shafique 56

Hysteresis Thresholding Khurram Hassan-Shafique 57

Resultado de algoritmo de histerese 58

Subpixel Localization One can try to further localize the position of the edge within a pixel by analyzing the response to the edge enhancement filter One common approach is to fit a quadratic polynomial to the filter response in the region of a maxima and compute the true maximum. -1 1 59

Detector de cantos

Comportamento local: Classificação 2 “Edge” 2 >> 1 “Corner” 1 e 2 são grandes, 1 ~ 2; aumenta em todas as direções 1 e 2 são pequenos; Quase constante em “Edge” 1 >> 2 “Flat” 1 61

Harris Detector: Mathematics Measure of corner response: (k – empirical constant, k = 0.04-0.06) 62

Harris Detector: Mathematics 1 “Corner” “Edge” “Flat” R > 0 R < 0 |R| small 2 R depends only on eigenvalues of M R is large for a corner R is negative with large magnitude for an edge |R| is small for a flat region 63

Algoritmo Comparação dos gráficos 2 “Edge” “Corner” R < 0 R > 0 “Flat” “Edge” |R| small R < 0 1 64

Algoritmo Comparação dos gráficos 2 “Edge” “Corner” R < 0 R > 0 “Flat” “Edge” |R| small R < 0 1 65

Algoritmo Comparação dos gráficos 2 “Edge” “Corner” R < 0 R > 0 “Flat” “Edge” |R| small R < 0 1 66

Harris Detector The Algorithm: Find points with large corner response function R (R > threshold) Take the points of local maxima of R 67

Harris Detector: Workflow 68

Harris Detector: Workflow Compute corner response R 69

Harris Detector: Workflow Find points with large corner response: R>threshold 70

Harris Detector: Workflow Take only the points of local maxima of R 71

Harris Detector: Workflow 72

Example: Gradient Covariances Corners are where both eigenvalues are big from Forsyth & Ponce Detail of image with gradient covar- iance ellipses for 3 x 3 windows Full image 73

Example: Corner Detection (for camera calibration) courtesy of B. Wilburn 74

Example: Corner Detection courtesy of S. Smith SUSAN corners 75

Harris Detector: Summary Average intensity change in direction [u,v] can be expressed as a bilinear form: Describe a point in terms of eigenvalues of M: measure of corner response A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive 76

Harris Detector: Some Properties Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation

Harris Detector: Some Properties Partial invariance to affine intensity change Only derivatives are used => invariance to intensity shift I  I + b Intensity scale: I  a I R x (image coordinate) threshold

Harris Detector: Some Properties But: non-invariant to image scale! All points will be classified as edges Corner !

Harris Detector: Some Properties Quality of Harris detector for different scale changes Repeatability rate: # correspondences # possible correspondences C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

SIFT (Scale Invariant Feature Transform)

A 500x500 image gives about 2000 features SIFT stages: Scale-space extrema detection Keypoint localization Orientation assignment Keypoint descriptor detector descriptor matching ( ) local descriptor A 500x500 image gives about 2000 features

1. Detection of scale-space extrema For scale invariance, search for stable features across all possible scales using a continuous function of scale, scale space. SIFT uses DoG filter for scale space because it is efficient and as stable as scale-normalized Laplacian of Gaussian.

Scale space  doubles for the next octave K=2(1/s)

Detection of scale-space extrema

Keypoint localization X is selected if it is larger or smaller than all 26 neighbors

2. Accurate keypoint localization Reject points with low contrast and poorly localized along an edge Fit a 3D quadratic function for sub-pixel maxima

Accurate keypoint localization Change sample point if offset is larger than 0.5 Throw out low contrast (<0.03)

Eliminating edge responses Let Keep the points with r=10

Maxima in D

Remove low contrast and edges

3. Orientation assignment By assigning a consistent orientation, the keypoint descriptor can be orientation invariant. For a keypoint, L is the image with the closest scale, orientation histogram

Orientation assignment

Orientation assignment

Orientation assignment

Orientation assignment

SIFT descriptor

4. Local image descriptor Thresholded image gradients are sampled over 16x16 array of locations in scale space Create array of orientation histograms (w.r.t. key orientation) 8 orientations x 4x4 histogram array = 128 dimensions Normalized, clip values larger than 0.2, renormalize σ=0.5*width

SIFT extensions

PCA

PCA-SIFT Only change step 4 Pre-compute an eigen-space for local gradient patches of size 41x41 2x39x39=3042 elements Only keep 20 components A more compact descriptor

GLOH (Gradient location-orientation histogram) SIFT 17 location bins 16 orientation bins Analyze the 17x16=272-d eigen-space, keep 128 components

Multi-Scale Oriented Patches Simpler than SIFT. Designed for image matching. [Brown, Szeliski, Winder, CVPR’2005] Feature detector Multi-scale Harris corners Orientation from blurred gradient Geometrically invariant to rotation Feature descriptor Bias/gain normalized sampling of local patch (8x8) Photometrically invariant to affine changes in intensity

Multi-Scale Harris corner detector Image stitching is mostly concerned with matching images that have the same scale, so sub-octave pyramid might not be necessary.

Multi-Scale Harris corner detector gradient of smoother version Corner detection function: Pick local maxima of 3x3 and larger than 10

Orientation assignment Orientation = blurred gradient

MOPS descriptor vector Scale-space position (x, y, s) + orientation () 8x8 oriented patch sampled at 5 x scale. See the Technical Report for more details. Bias/gain normalisation: I’ = (I – )/ 40 pixels 8 pixels

Detections at multiple scales

Feature matching Exhaustive search Hashing Nearest neighbor techniques for each feature in one image, look at all the other features in the other image(s) Hashing compute a short descriptor from each feature vector, or hash longer descriptors (randomly) Nearest neighbor techniques k-trees and their variants (Best Bin First)

Wavelet-based hashing Compute a short (3-vector) descriptor from an 8x8 patch using a Haar “wavelet” Quantize each value into 10 (overlapping) bins (103 total entries) [Brown, Szeliski, Winder, CVPR’2005]

Nearest neighbor techniques k-D tree and Best Bin First (BBF) Indexing Without Invariants in 3D Object Recognition, Beis and Lowe, PAMI’99

Reference Chris Harris, Mike Stephens, A Combined Corner and Edge Detector, 4th Alvey Vision Conference, 1988, pp147-151. David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, 60(2), 2004, pp91-110. Yan Ke, Rahul Sukthankar, PCA-SIFT: A More Distinctive Representation for Local Image Descriptors, CVPR 2004. Krystian Mikolajczyk, Cordelia Schmid, A performance evaluation of local descriptors, Submitted to PAMI, 2004. SIFT Keypoint Detector, David Lowe. Matlab SIFT Tutorial, University of Toronto. Matthew Brown, Richard Szeliski, Simon Winder, Multi-Scale Oriented Patches, MSR-TR-2004-133, 2004.

FIM

Arestas e cantos Locais de mudanças significativas na intensidade da imagem 114

Edgedels = edge elements 115