• Support Vector Machine History
  • by HolyPython

Contributions of a prolific mind

From steppes of Central Asia to Murray Hill, New Jersey

Vapnik showing off his ERM framework with humor. Claiming superiority to Bayesian Statistics (with ERM formula at the top)

Statistical progress continues

Support Vector Machine algorithm is yet another flag race in the history.

There are chain of events that lead to the invention of support vector machines generally dating back to the middle of 20th century:

In 1950 Aronszajn publishes the “Theory of Reproducing Kernels”. In 1957 Frank Rosenblatt took this idea and invented perceptron, a simple linear classifier.

6 years later Vapnik and Lerner comes and announces “Generalized Portrait Algorithm” (1963). This was the true inspiration for Boser, Guyon and Vapnik’s 1992 paper at the COLT conference introducing Support Vector Machines.

The original paper, “A training algorithm for optimal margin classifiers” (1992), can be found here (ISBN 978-0897914970).

Another big leap between Generalized Portrait Algorithm (1963) and SVM (1992) paper was Vapnik and Chervonenkis’ statistical learning theory in 1974 and then Vapnik’s advancement on it in 1979. 

Vapnik’s work were originally in Russian then and have been translated to English and German in the following years.

Vladimir Vapnik

Vladimir Vapnik, born in 1936, studied Mathematics at Uzbek State University (then in Soviet Union) and moved to the US in 1990 to work with AT&T Bell Labs (now Nokia Bell Labs) shortly before publishing his major work in SVM.

His book Statistical Learning Theory (1998) has been outstandingly cited more than 60.000 times. Book here.

Since supervised machine learning techniques cannot be used with unlabeled data, Vapnik with Hava Siegelmann also developed SVC (Support Vector Clustering) an  unsupervised extension of Support Vector Machines in November 2001. You can see the research paper here.

Vapnik also founded VC theory with Chervonenkis (Vapnik–Chervonenkis theory ), a theory in computational learning.

Bell Labs hosted 9 Nobel prize winners and many other prolific minds in its 40+ years history (Photo cred: Alex Napoliello | NJ Advance Media for NJ.com)

Further Advancements

Vanpik has been making countless amounts of outstanding contributions to the statistics field. Talk about a prolific life! In 1995, he found soft margin classifier along with Cortes and extended it to a regression application in the same year, paper here:
The Nature of Statistical Learning Theory (1995)

 

In 1998 Shawe, Taylor et al. made a significant contribution to the generalization of hard margin Support Vector Machines. Shawe, Taylor, Cristianini then gave statistical bounds to the generalization of soft margin Support Vector Machines in 2000.

Finishing Thoughts

I think when it comes to Support Vector Machines, Vapnik stands out as a true legend that contributed to the statistical learning for decades, fathered SVMs along with Boser and Guyon, continued to advance the models and then founded SVC (Support Vector Clustering) with Siegelmann.

Vapnik is still alive and quite active. He resides in New Jersey, USA.