Machine Learning explained
The development and use of artificial intelligence (AI) is currently keeping the digital world on its toes. One aspect of AI is machine learning. Here, IT systems learn to recognize patterns on the basis of existing data and thus automate the creation of analytical models, such as forecasts. But how does this system work? Here is a short summary.
Machine Learning can be used e.g. for speech and text recognition and stock market analysis. In the following, we will concentrate on the "validation set method", a part of supervised machine learning, which aims to predict future developments from existing data. In order to guarantee a useful prediction, the deviation between generated data and actual data must be kept as small as possible. Therefore, the existing data is divided into validation and training data. The system is trained with the help of the training data. The accuracy of the data is then checked using the validation data.
The higher the model complexity (e.g. higher polynomials), the lower the error rate of the training data, i.e. the deviations between actual data and generated data. This is due to the fact that a more complex function can reproduce the data more accurately.
The error rate of the validation data initially decreases more slowly than the training data. At a certain point, the error rate increases again because the data is overfitting. Individual fluctuations are dealt with in too much detail (“noise”) instead of the function. The sweet spot you want to hit is the point, where the validation error is at its minimum.