Welcome to this third video on ensemble techniques where we look generally at 'Boosting', and more specifically at a method called 'AdaBoost' (short for Adaptive Boosting).
Hit '>Play' above to learn more about Boosting generally & AdaBoost specifically.
This is a modern AI technique that has evolved primarily from Random Forests. It is normally described and demonstrated to work with decision trees and forests as we shall see. But the basic boosting technique is agnostic to the choice of AI algorithm. It can be applied to support vector machines, K-nearest neighbours, neural networks, indeed any Machine Learning or AI technique.
The key idea behind boosting is not to introduce a completely randomised dataset, but to produce a dataset that is richer in samples that the classifier struggles to correctly identify.
A good analogy here is a sports coach. They will spend less time on coaching where their client is already world-class. Instead they will focus on elements where the athlete is weaker, with a greater improvement in overall performance.
There are three fundamental concepts behind AdaBoost that make it different from other machine learning algorithms.
They are:
1. Create a large amount of weak learners
2. Give models a weighted vote, based on their strength of classification
3. Allow the algorithm to be an adaptive learner; develop learners in order, with the errors in prior models influencing the building of the next learner.
The individual models (called 'stumps' in AdaBoost) are built and optimised separately. This makes this technique much less prone to overfitting than other machine learning models.
Please see the following videos to refresh your knowledge of Gini impurity, decision trees and Random Forests...
Gini impurity & Decision Trees:
- https://www.lucidate.co.uk/forum/machine-learning/ml-2-decision-trees
- https://youtu.be/5swHVbJNWpw
Random Forests:
- https://www.lucidate.co.uk/forum/ensemble-techniques/ensembles-1-random-forests-pt-1
- https://www.lucidate.co.uk/forum/ensemble-techniques/ensembles-2-random-forests-pt-2 - https://youtu.be/jz1ZBTrIx5Y
- https://youtu.be/4SjkF13Sl_0
