# Support Vector Machine

A general note. In SVM, the class is labed using +1/-1. More precisely, $$y_i \in \{+1, -1\}$$.

#### Margin

• Functional margin: $$\hat{\gamma}^{(i)} = y^{(i)}(w^T x + b)$$
• Geometric margin: $$\gamma^{(i)} = y^{(i)}(\frac{w^T x}{||w||} + \frac{b}{||w||})$$

#### Optimization Problem

In the lecture note, three forms of the optimization problem are presented. Note that each formulation consists of two parts

• objective function
• constraints

The first formulation is based on geometric margin. The objective function and constraints are both defined using geometric margin $$\gamma$$.

The second formulation is a mixed form. The objective function is still based on geometric margin (recall that we have $$\gamma = \frac{\hat{\gamma}} {||w||}$$) while the constraints are defined using functional margin. This transformation is valid because in the first formulation, we have constraint $$||w|| = 1$$ therefore we can replace the geometric margin constraints with functional margin constraints.

The third formulation is the following. The idea is that we can scale $$w$$ and $$b$$ such that $$\hat{\gamma}$$ equals to 1.

Finally, for non-separable case, we have

#### Hinge Loss

Hinge loss is defined as the following

$$Loss_{hinge} = max(0, 1 - y(\omega^T x + b))$$

This is related to the constraints in the non-separable case. In fact, the loss is a reformulation of the constraints.

----- END -----

Welcome to join reddit self-learning community.

Want some fun stuff?