300 times faster than Amazon, Google and Microsoft: Brainome fundamentally changes the game for Auto ML
Auto ML (AML) enables no-code predictive analysis for datasets of all sizes. Existing AML approaches available from Amazon, Microsoft and Google use brute-force (lots of compute resources) to try as many combinations of models + hyperparameters as the user is willing to pay for. This works, but it takes too much time, even for small datasets, and gets expensive if you have to generate lots of models.
At Brainome, we have an entirely different approach to the AML task. We measure the learnability of datasets and use these measurements to drive a reproducible, zero-code solution that (1) accelerates model generation up to 40 times, (2) predicts up to 300 times faster and (3) creates extremely compact models while achieving equivalent test accuracy and F1 when compared to Amazon, Microsoft and Google.
While our approach works well for datasets of all sizes, it is a revelation for customers who have very large datasets and / or need to generate many models. Think genomic analysis, disease discovery and / or financial markets. Analysis that previously took too long or was too expensive can now be done very cost efficiently in seconds or minutes.
This article describes the methodology used for comparing Brainome against Amazon SageMaker, Microsoft Azure Machine Learning and Google Cloud AutoML. We report the performance results and outline the differences between the various platforms.
(“Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools“). This paper attempted to compare the various AutoML platforms available at the time.
Each dataset was split into a training dataset (70%) used exclusively for training of the model and a held back test dataset (30%), used to compute the accuracy of the model.
Our benchmark recorded five key performance metrics: test accuracy, F1 score, training speed, prediction (or inference) speed and model size.
The four AutoML systems benchmarked are:
There are no “version” numbers for GCML, SageMaker and AzureML but all tests were conducted between December 2021 and January 2022.
For consistency, all tests were automated via scripting using the respective API of each system.
We can provide the scripts if you’d like to repeat the experiment – email us at email@example.com
Brainome, SageMaker and AzureML were run on similar hardware platforms equivalent to an EC2 m5.2xlarge general purpose instance (the suggested default for SageMaker). We did not have control of GCML’s hardware.
The following table is a summary of the five key metrics used for comparison:
Average Test Accuracy
Average training time (seconds)
Aggregate training time (seconds)
Average prediction time (seconds)
Aggregate prediction time (seconds)
Average predictor size (KB)
* Google could not produce results for 3 of the datasets
These two graphs show the accuracy and F1 score for each dataset for each system:
As we can see from the graphs, all four systems create models that perform very similarly in terms of accuracy and F1 score.
However there is a significant difference in training time and prediction time. Brainome builds and trains models 22 to 40 times faster than its competitors. Additionally, the very compact models produced by Brainome predict (infer) 12 to 300 times faster.
Observations and Limitations
Dataset SelectionGCML requires at least 1,000 data points in any dataset used for training, while SageMaker requires at least 500 data points.
Because our methodology used 70% of each dataset for training and 30% as the test set, this meant that GCML could not produce results for 3 datasets which had less than 1,428 total data points.
There were also limitations on the number of features (columns) a dataset could have:
- GCML rejects any dataset that has more than 1,000 features
- SageMaker rejects any dataset that has more than 100,000 characters in a single line. This error message was displayed when trying to upload a data set with 72K features: “ClientError: One or more lines in the data exceeded the max allowed line length of 100000. Please reduce the length of the input and retry.”
PreprocessingWe used Brainome to clean the raw data files downloaded from OpenML to convert all non-numeric data into numbers and fill in any missing values before submitting to all 4 AutoML systems. This was necessary to ensure that all platforms were using the same exact starting point for training.
Prediction SpeedBrainome has the fastest inference times (by orders of magnitude) thanks to its very compact, bespoke models. Both SageMaker and GCML are very slow to infer. This appears to be the result of combining model deployment with the actual inference. We were not able to separate these two steps for SageMaker and GCML. AzureML appears to keep the deployed model alive and the measured time is clearly just the inference time.
Model Size and TypesIt was not possible to extract accurate model sizes and model types from GCML, SageMaker or AzureML. Brainome’s average model size is 172 Kilobytes. The smallest model was 25 Kilobytes and the largest one was 1,561 Kilobytes. The models (aka “predictors”) are entirely self-contained within a single Python file (a.py). It is worth noting that the a.py file contains a fair amount of code beyond just the inference model, including:
- comments to make the a.py human readable
- all data pre-processing (so raw data can be passed directly to the a.py for inference)
- code that enables multi-core inference (if multi-core is available)
- code to generate statistics, confusion matrices and other utilities
UsabilityIt took ~6 engineering work days to set up the automation pipeline for each of the GCML, SageMaker and AzureML systems. Brainome’s Python PIP installation only takes a few minutes to set up locally or in a cloud environment and is ready for interactive use immediately. Integration with an automation pipeline takes ~2 hours.
ReproducibilityBrainome’s measurements driven approach ensures 100% reproducible model building, test accuracy and F1 scores. AzureML explicitly states in their documentation that one should not expect reproducible results when using their platform. Neither SageMaker nor GCML say anything about reproducibility but running the same dataset multiple times on each platform (with the same exact parameters) yielded different test accuracy results each time.
These results demonstrate the advantages of Brainome’s unique measurements-based approach compared to standard AutoML systems.
To try Brainome with your own data, please follow our installation instructions. Additional documentation is found here. Note that when using Brainome your data will always stay private and never leave your computer. For large datasets, you can request a demo license key by emailing firstname.lastname@example.org
A PDF with the full results for all 21 data sets is available for email distribution: