Documentation

Access all resources and documentation or contact us for additional assistance.

Measure and evaluate

Unique to Brainome is the ability to measure the learnability of your data. If you are not interested in learning about this right now and would rather skip ahead and let Brainome build a model first, please jump to: Build and Run.

Why measure up front?

Every scientific endeavor starts with measurements, and so should machine learning. (Or, if you prefer, consider that, just as in remodeling a kitchen or sewing a dress, being successful requires that you always measure before you cut.)

If you do the up-front work of measuring and “right-sizing” the training process before you build your machine learning model, you will save time and money by avoiding mistakes.

If you don’t measure up front:

  • You have no way of knowing whether the predictor you build will actually do what you want it to do when it sees new data that it wasn’t trained on
  • You will probably build a model that is much larger than it needs to be
  • Your training times will probably be much longer than they need to be
  • Your run times for the deployed predictor could be much longer than they need to be
  • You can end up in a situation where you just don’t know whether you have the right amount or right type of training data, even after extensive training and testing

Brainome’s measurements give you the insight you need up front!

  • Learnability (using capacity progression and risk of overfit)
  • Generalization ratio
  • Noise resilience
  • Attribute ranking
  • And, of course, all the standard accuracy figures

(For more information on terminology in bold, check out the Brainome Glossary.)

Steps

      1.  Run the measuring process in your experiment directory

pwd
/Users/user1/Documents/Brainome/Experiment

ls -l Experiment_data
total 255560
-rw-r–r–@ 1 user1  staff  22115 Jul 26 14:07 experiment_test.csv
-rw-r–r–@ 1 user1  staff  22191 Jul 26 14:07 experiment_train.csv
brainome Experiment_data/experiment_train.csv -measureonly > experiment.measurements

      2.  Look at the results

more experiment.measurements
Brainome Table Compiler v1.002-fractor-alpha
Copyright (c) 2019-2021 Brainome, Inc. All Rights Reserved.
Licensed to:                 Brainome User 1  (Enterprise)
Expiration Date:             2021-12-12   209 days left
Maximum File Size:           30 GB
Maximum Instances:           unlimited
Maximum Attributes:          unlimited
Maximum Classes:             unlimited
Connected to:                daimensions.brainome.ai  (local execution)

Command:
    brainome experiment_train.csv -measureonly

Start Time:                 05/17/2021, 14:14 PDT

Cleaning…done.
Splitting into training and validation…done.
Pre-training measurements…done.

Pre-training Measurements
Data:
    Input:                      experiment_train.csv
    Target Column:              Survived
    Number of instances:        800
    Number of attributes:        11 out of 11
    Number of classes:            2

Class Balance:                
                            died: 61.50% 
                        survived: 38.50%

Learnability:
    Best guess accuracy:          61.50%
    Data Sufficiency:             Maybe enough data to generalize. [yellow]

Capacity Progression:             at [ 5%, 10%, 20%, 40%, 80%, 100% ]
    Ideal Machine Learner:              6,   7,   8,   8,   9,   9

Estimated Capacity Utilized:
    Trained Neural Network:       15 bits

Expected Generalization:
    Decision Tree:                 1.99 bits/bit
    Neural Network:                6.52 bits/bit
    Random Forest:                10.13 bits/bit

Expected Accuracy:              Training            Validation
    Decision Tree:               100.00%                51.62%
    Neural Network:               54.00%                50.50%
    Random Forest:               100.00%                80.25%

Recommendations:
    Warning: Data has high information density. Using effort 5 and larger ( -e 5 ) can improve results.
    We recommend using Random Forest -f RF.
    If predictor accuracy is insufficient, try using the option -rank to automatically select the important attributes.
    If predictor accuracy is insufficient, try using the effort option -e with a value of 5 or more to increase training time. 

Time to Build Estimates:
    Decision Tree:                a few seconds
    Neural Network:               less than a minute
Measurement Runtime Duration:     8s

      3. Decide what to do next

To decide what to do next, you need to look at the measurements obtained from your training data. If the Data Sufficiency “Recommendation” is green and your Accuracy and Generalization numbers are where you’d like them to be, then you can move ahead to the next step, where you’ll build your predictor.

However, in the example above, the Data Sufficiency “Recommendation” is yellow, not green. We’re also seeing a warning that we may need to ask Brainome to do some extra work (using the “-e” option, see below) to get the most out of what appears to be very information-dense (complex) data. So in this case, when we move on to build the predictor, we will already have some ideas about what we can do to get the best results.

Sometimes at this measurement-only stage Brainome will give specific advice about which type of predictor to build (decision tree, neural network or random forest). In this particular case this did not happen — there’s nothing in the “-measureonly” output that gives specific feedback on this point. So our first approach will be to let Brainome decide automatically what kind of predictor to build, and only consider  one type of model vs. another later on if needed.

Here are some common situations that you may encounter after running in “measure only” mode, together with suggested actions to take:

Measurements

Options/actions

Data Sufficiency “Recommendation” is red.

This means you don’t have enough data or your data is not clean enough and Brainome is having a hard time extracting a general predictor. 

You have the following options:

  1. Use attribute selection to prioritize the most useful columns. Identifying the most important attributes first can help increase learnability by not using columns that are irrelevant. You can either ignore columns by using the -ignorecolumns option or use Brainome’s automated attribute ranking algorithm via the -rank option.
  2. If you’re doing multi-class predictions, reduce the number of output classes. Sometimes ignoring classes can help learnability. Once you understand the problem better, it may be easier to ignore irrelevant columns and add classes back in.
  3. Re-check the quality and consistency of your training data annotation. What is the annotator agreement? Are the classes balanced? Higher accuracy in annotation combined with balanced classes will give better results.
  4. If you are using perceptual data such as images, sound or videos, you may need to do some pre-processing. You may need to use a feature extractor or a convolution algorithm as a first step before using Brainome.
  5. If you started with very few columns, add more attributes (columns) and try again.
  6. If nothing else helps, try collecting more data of the same sort that you started with, then run Brainome again to obtain new measurements.

Data Sufficiency “Recommendation” is yellow.

Choose the predictor (decision tree, neural network or random forest) with the highest generalization first. If that doesn’t work well, increase effort with the -e training option. Try incrementally increasing effort levels to see if you can get better results. If you max out on available effort options, then review and try the options (in order!) listed above for “Data Sufficiency is red.”

Data Sufficiency “Recommendation” is green.

Choose the predictor with the highest estimated generalization. The automatic choices that Brainome makes should give you good results.

It is rare that a data set will produce a “measureonly” Data Sufficiency “Recommendation” result of “green” right out of the box. Because you’re unlikely to see this “green” outcome when working with your own data, we’re including a clear-cut example here for comparison. This is a simple dataset of 200 numbers in which all numbers <100 are class 0 and all numbers >= 100 are class 1:

brainome Experiment_data/simple_example_train.csv -measureonly > simple_example.measurements

ls
simple_example.measurements

more simple_example.measurements
Brainome Table Compiler v1.002
Copyright (c) 2019-2021 Brainome, Inc. All Rights Reserved.
Licensed to:                 Brainome User 1  (Enterprise)
Expiration Date:             2021-12-12   209 days left
Maximum File Size:           30 GB
Maximum Instances:           unlimited
Maximum Attributes:          unlimited
Maximum Classes:             unlimited
Connected to:                daimensions.brainome.ai  (local execution)

Command:
    brainome simple.example.measurements.csv -measureonly

Start Time:                 05/17/2021, 14:53 PDT

Cleaning…done.
Splitting into training and validation…done.
Pre-training measurements…done.

Pre-training Measurements
Data:
    Input:                      simple.example.measurements.csv
    Target Column:              Class
    Number of instances:        200
    Number of attributes:         1 out of 1
    Number of classes:            2

Class Balance:                
                               0: 50.00%
                               1: 50.00%

Learnability:
    Best guess accuracy:          50.00%
    Data Sufficiency:             Enough data to generalize. [green]

Capacity Progression:             at [ 5%, 10%, 20%, 40%, 80%, 100% ]
    Ideal Machine Learner:              1,   1,   1,   1,   1,   1

Estimated Capacity Utilized:
    Trained Neural Network:        1 bits

Expected Generalization:
    Decision Tree:               200.00 bits/bit
    Neural Network:              100.00 bits/bit
    Random Forest:               100.00 bits/bit

Expected Accuracy:              Training            Validation
    Decision Tree:               100.00%                99.50%
    Neural Network:              100.00%                99.50%
    Random Forest:               100.00%                99.00%

Recommendations:
    We recommend using Random Forest -f RF.
    If predictor accuracy is insufficient, try using the option -rank to automatically select the important attributes.
    If predictor accuracy is insufficient, try using the effort option -e with a value of 5 or more to increase training time. 

Time to Build Estimates:
    Decision Tree:                a few seconds
    Neural Network:               less than a minute
Measurement Runtime Duration:     6s

It’s not just simple data sets that can yield a “green” value for data sufficiency, however.  Sometimes very large data sets can also yield a “green” result. Most of the work you’ll do with Brainome will be with data sets that yield a “yellow” result. This is normal and it is by design. Use the measurement capabilities of Brainome to understand your data more thoroughly so that you can build a useful and reliable predictor while knowing exactly how much time, effort, and money to put into data collection. You don’t need to dramatically over-collect data in an effort to get to a “green” outcome! Instead, it’s much better to use all of the measurements provided by Brainome together in concert so that you can determine the right amount and type of data you need.

     4. Rank and select wgucg attributes to use

Another unique feature of Brainome is the ability to select which data attributes (columns) are the most important for your classification BEFORE building a predictor. To do this, use the -rank option in combination with -measureonly.

brainome titanic.csv -e 3 -f NN -rank -measureonly > titaniceNNrank.output

When we use -rank on our experiment data, we see that Brainome automatically decides to use only 4 of the 11 attributes (columns) in the training data:

more titaniceNNrank.output

Attribute Ranking:
    Columns selected:           Sex, Sibling_Spouse, Parent_Children, Cabin_Class
    Risk of coincidental column correlation:    0.0%    

    Test Accuracy Progression:
                                Sex :   78.75%
                     Sibling_Spouse :   79.62% change   +0.88%
                    Parent_Children :   80.25% change   +0.62%
                        Cabin_Class :   80.88% change   +0.63%
.
.
.

Using the -rank option often gives valuable insight into the data itself, while saving considerable effort and cost before you move forward with model building. Over time, as you work with different datasets, you will develop an intuition for what the measurements mean. Often the measurements work best when used as a relative comparison between two choices. For example:

  • If you have the choice between two training datasets, the one that achieves higher Generalization over all possible machine learning algorithms (decision tree, neural network, random forest) is the one to use.