This summer, I worked on FastAI.jl as a part of Google Summer of Code'21 under The Julia Language. My work involved adding tabular support to the package.
If you are unfamiliar with FastAI.jl, it is a package inspired by fastai[1] and is a repository of best practices for deep learning in julia. It offers a layered API that lets you use the higher-level functionalities to perform various learning tasks in around 5 lines of code, and the middle and lower-level APIs can be mixed and matched to create new approaches.
The GSoC project page can be found here.
We'll go through an end-to-end task to understand the added functionalities. Here's we will be working with the adult dataset and try to perform tabular classification on the salary column to predict if its <50k or >=50k
The first thing I worked on was adding an index-based data container suitable for tabular data, which follows the interface defined by MLDataPattern.jl. TableDataset accepts any type satisfying the Tables.jl interface and allows querying for any row using getobs and the total number of observations using nobs.
On top of this, among some of the other things the container lets you do are performing arbitrary functions on the observations lazily, splitting containers, and creating a DataLoader suitable for training.
using FastAI
data = TableDataset(joinpath(datasetpath("adult_sample"), "adult.csv"));
getobs(data, 1), nobs(data)(DataFrameRow
Row │ age workclass fnlwgt education education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country salary
│ Int64 String Int64 String Float64? String String? String String String Int64 Int64 Int64 String String
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 49 Private 101320 Assoc-acdm 12.0 Married-civ-spouse missing Wife White Female 0 1902 40 United-States >=50k, 32561)We'll now split off our target column from the data container using mapobs.
splitdata = mapobs(row -> (row, row[:salary]), data)Once we have our container, to make it usable for training we'll pre-process it using the tabular transformations added to DataAugmentation.jl, which are -
Normalization - Normalizes a row of data using column mean and standard deviation.
Fill Missing - Fills any missing values in the row.
Categorify - Label encodes the categorical columns
As these transformations are applied on individual rows (or TabularItem to be precise), we collect all the needed dataset statistics beforehand. This is done by creating an indexable collection (such as Dict) with the column as keys and required statistics as the value. For example, to create the normalization dictionary, we'll need the values to be tuples of the mean and standard deviations of the columns.
There are various helper methods defined which make this process very easy if there is a TableDataset created already, but it is still possible to prepare everything manually and pass in the required statistics for maximum flexibility.
using DataAugmentation
catcols, contcols = FastAI.getcoltypes(data)
normdict = FastAI.gettransformdict(data, DataAugmentation.NormalizeRow, contcols)Dict{Any, Any} with 6 entries:
:fnlwgt => (1.89778e5, 105550.0)
:age => (38.5816, 13.6404)
Symbol("education-num") => (10.0798, 2.573)
Symbol("capital-loss") => (87.3038, 402.96)
Symbol("capital-gain") => (1077.65, 7385.29)
Symbol("hours-per-week") => (40.4375, 12.3474)Now we can create the transform by passing in the constructed collections.
normalize = DataAugmentation.NormalizeRow(normdict, contcols)DataAugmentation.NormalizeRow{Dict{Any, Any}, NTuple{6, Symbol}}(Dict{Any, Any}(:fnlwgt => (189778.36651208502, 105549.97769702229), :age => (38.58164675532078, 13.640432553581315), Symbol("education-num") => (10.079815426825466, 2.572999154730833), Symbol("capital-loss") => (87.303829734959, 402.9602186489992), Symbol("capital-gain") => (1077.6488437087312, 7385.292084840338), Symbol("hours-per-week") => (40.437455852092995, 12.34742868173185)), (:age, :fnlwgt, Symbol("education-num"), Symbol("capital-gain"), Symbol("capital-loss"), Symbol("hours-per-week")))To now use the transformation, we'll have to create a TabularItem to apply it on, and then call apply on it.
using Tables
item = TabularItem(getobs(data, 1), Tables.columnnames(data.table))DataAugmentation.TabularItem{DataFrames.DataFrameRow{DataFrames.DataFrame, DataFrames.Index}}(DataFrameRow
Row │ age workclass fnlwgt education education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country salary
│ Int64 String Int64 String Float64? String String? String String String Int64 Int64 Int64 String String
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 49 Private 101320 Assoc-acdm 12.0 Married-civ-spouse missing Wife White Female 0 1902 40 United-States >=50k, [:age, :workclass, :fnlwgt, :education, Symbol("education-num"), Symbol("marital-status"), :occupation, :relationship, :race, :sex, Symbol("capital-gain"), Symbol("capital-loss"), Symbol("hours-per-week"), Symbol("native-country"), :salary])apply(normalize, item).data(age = 0.7637846676602542, workclass = " Private", fnlwgt = -0.8380709161872286, education = " Assoc-acdm", education-num = 0.7462826288318035, marital-status = " Married-civ-spouse", occupation = missing, relationship = " Wife", race = " White", sex = " Female", capital-gain = -0.14591824281680102, capital-loss = 4.5034127099423245, hours-per-week = -0.035428902921319616, native-country = " United-States", salary = ">=50k")The other transforms work similarly.
Link to Model PR
Link to Embedding PR
We created a deep learning model suitable for tabular data by re-engineering the model present in fastai[2]. The backbone of the model is structured with a categorical and a continuous component. It accepts a 2-tuple of categorical (label or one-hot encoded) and continuous values, where each backbone applies to each element of the tuple. The output from these backbones is then concatenated and passed through a series of linear-batchnorm-dropout layers before a finalclassifier block, whose size could be task dependant.
The categorical backbone consists of an embedding layer for each categorical column and makes use of the concept of entity embeddings[3]. Instead of just one-hot encoding the categorical columns, representing these values in an "embedding space" makes it so that similar values could be closer to each other. This can reveal the intrinsic properties of the categorical variables and even reduce memory usage.
We have mainly two methods for creating a TabularModel.
by passing required metadata
by passing custom backbones
The simplest method of constructing a TabularModel is through the first method, which involves passing the number of continuous columns, output size and a collection of cardinalities (number of unique classes) for the categorical columns.
num_cont = length(contcols)
outsize = 2
catdict = FastAI.gettransformdict(data, DataAugmentation.Categorify, catcols)
cardinalities = [length(catdict[col]) for col in catcols]
FastAI.TabularModel(num_cont, outsize; cardinalities=cardinalities)Chain(
Parallel(
vcat,
Chain(
FastAI.Models.var"#43#45"(),
Parallel(
vcat,
Embedding(10, 6), # 60 parameters
Embedding(17, 8), # 136 parameters
Embedding(8, 5), # 40 parameters
Embedding(17, 8), # 136 parameters
Embedding(7, 5), # 35 parameters
Embedding(6, 4), # 24 parameters
Embedding(3, 3), # 9 parameters
Embedding(43, 13), # 559 parameters
Embedding(3, 3), # 9 parameters
),
identity,
),
BatchNorm(6), # 12 parameters, plus 12
),
Chain(
Dense(61, 200, relu; bias=false), # 12_200 parameters
BatchNorm(200), # 400 parameters, plus 400
identity,
),
Chain(
Dense(200, 100, relu; bias=false), # 20_000 parameters
BatchNorm(100), # 200 parameters, plus 200
identity,
),
Dense(100, 2), # 202 parameters
) # Total: 19 arrays, 34_022 parameters, 132.523 KiB.While using the second method for creating the model, we'll have to define our categorical backbone, continuous backbone, and an optional finalclassifier layer.
To create a categorical backbone with entity embeddings, we'll have to decide on the embedding dimensions. By default, these can be calculated using fastai's rule of thumb (see emb_sz_rule function) but can be overridden easily using size_overrides. Or we can even pass in an arbitrary vector of embedding sizes if you prefer not to use get_emb_sz.
Check out the docs for more information about different methods available for get_emb_sz.
embedszs = FastAI.Models.get_emb_sz(cardinalities, catcols, Dict(:workclass => 20))9-element Vector{Tuple{Int64, Int64}}:
(10, 20)
(17, 8)
(8, 5)
(17, 8)
(7, 5)
(6, 4)
(3, 3)
(43, 13)
(3, 3)contback = FastAI.Models.tabular_continuous_backbone(6)
catback = FastAI.Models.tabular_embedding_backbone(embedszs, 0.2)Chain(
FastAI.Models.var"#43#45"(),
Parallel(
vcat,
Embedding(10, 20), # 200 parameters
Embedding(17, 8), # 136 parameters
Embedding(8, 5), # 40 parameters
Embedding(17, 8), # 136 parameters
Embedding(7, 5), # 35 parameters
Embedding(6, 4), # 24 parameters
Embedding(3, 3), # 9 parameters
Embedding(43, 13), # 559 parameters
Embedding(3, 3), # 9 parameters
),
Dropout(0.2),
) # Total: 9 arrays, 1_148 parameters, 168 bytes.TabularModel, along with a bunch of other optional keyword args to get our model.
FastAI.TabularModel(
catback,
contback,
Chain(Dense(100, 2), x->FastAI.Models.sigmoidrange(x, 2, 5)),
layersizes=(200, 100, 100),
dropout_rates = [0.1, 0.2, 0.1],
activation=Flux.sigmoid
)Chain(
Parallel(
vcat,
Chain(
FastAI.Models.var"#43#45"(),
Parallel(
vcat,
Embedding(10, 20), # 200 parameters
Embedding(17, 8), # 136 parameters
Embedding(8, 5), # 40 parameters
Embedding(17, 8), # 136 parameters
Embedding(7, 5), # 35 parameters
Embedding(6, 4), # 24 parameters
Embedding(3, 3), # 9 parameters
Embedding(43, 13), # 559 parameters
Embedding(3, 3), # 9 parameters
),
Dropout(0.2),
),
BatchNorm(6), # 12 parameters, plus 12
),
Chain(
Dense(75, 200, σ; bias=false), # 15_000 parameters
BatchNorm(200), # 400 parameters, plus 400
Dropout(0.1),
),
Chain(
Dense(200, 100, σ; bias=false), # 20_000 parameters
BatchNorm(100), # 200 parameters, plus 200
Dropout(0.2),
),
Chain(
Dense(100, 100, σ; bias=false), # 10_000 parameters
BatchNorm(100), # 200 parameters, plus 200
Dropout(0.1),
),
Chain(
Dense(100, 2), # 202 parameters
var"#5#6"(),
),
) # Total: 22 arrays, 47_162 parameters, 184.641 KiB.layersizes and size_overrides keyword args are also available in the first method if needed.
A Learning Method can be thought of as a concrete approach for solving a "learning task" (eg. tabular classification, tabular regression etc.)[4]
To implement a Learning Method in FastAI.jl, we have 2 main options.
Implement a Learning Method from scratch satisfying the DLPipelines.jl interface.
Use the DataBlock API to create a BlockMethod, or use an implemented high-level wrapper.
In most cases, using the DataBlock API should be sufficient for most learning tasks.
The DataBlock API lets us put together a learning method using "data blocks" which represent the type of data. For example, a TableRow data block carries all the information about the row, like which columns are categorical and continuous, and what classes a particular categorical column can have. In addition to TableRow, a Continuous block was also added which can help in regression tasks.
One important thing to keep in mind is that these data blocks don't actually carry the data, but just contain meta-data about the underlying data.
In addition to specifying the data blocks, we can also choose to apply any processing step which might be required to make the data suitable for training. For example TabularPreprocessing allows us to use the transformations specified previously to pre-process the rows.
Since our target column is a categorical value and we want to perform classification, we'll use the Label block to represent the target.
method = BlockMethod(
(
TableRow(catcols, contcols, catdict),
Label(unique(data.table[:, :salary]))
),
((FastAI.TabularPreprocessing(data)), FastAI.OneHot())
)BlockMethod(FastAI.TableRow{9, 6, Dict{Any, Any}} -> FastAI.Label{String})The DataBlock API is very flexible and we can put together an arbitrary number of blocks for any kind of learning task.
Creating a learning method for regression tasks is as easy as substituting the target block with a Continuous block where the size would represent the number of target columns.
method2 = BlockMethod(
(
TableRow(catcols, contcols, catdict),
Continuous(3)
),
((FastAI.TabularPreprocessing(data),))
)BlockMethod(FastAI.TableRow{9, 6, Dict{Any, Any}} -> FastAI.Continuous)If our learning task is either tabular classification or regression, we can even use the high-level wrappers defined for these tasks to get the method easily.
method = TabularClassificationSingle(
catcols,
contcols,
unique(data.table[:, :salary]);
data)BlockMethod(FastAI.TableRow{9, 6, Dict{Any, Any}} -> FastAI.Label{String})With this method, it is possible for us to encode our data, get a model and loss function suitable for the data, and even directly create a Learner which can be used for training.
To get a quick summary of the steps which will be performed while encoding, we can use the describemethod function.
describemethod(method) LearningMethod summary
------------------------
• Task: FastAI.TableRow{9, 6, Dict{Any, Any}} ->
FastAI.Label{String}
• Model blocks: FastAI.EncodedTableRow{9, 6, Dict{Any, Any}} ->
FastAI.OneHotTensor{0, String}
Encoding a sample (encode(method, context, sample))
Encoding Name method.blocks[1] method.blocks[2]
–––––––––––––––––––– ––––––––––––––– –––––––––––––––––––––––––––––––––––––––––––– ––––––––––––––––––––––––––––––
(input, target) FastAI.TableRow{9, 6, Dict{Any, Any}} FastAI.Label{String}
TabularPreprocessing FastAI.EncodedTableRow{9, 6, Dict{Any, Any}} FastAI.Label{String}
OneHot (x, y) FastAI.EncodedTableRow{9, 6, Dict{Any, Any}} FastAI.OneHotTensor{0, String}
Decoding a model output (decode(method, context, ŷ))
Decoding Name method.outputblock
–––––––––––––––––––– ––––––––––– ––––––––––––––––––––––––––––––
ŷ FastAI.OneHotTensor{0, String}
OneHot FastAI.Label{String}
TabularPreprocessing target_pred FastAI.Label{String}Let's use this method to encode a row of data.
The input here is a tuple of all row values and the target value.
row = getobs(splitdata, 1000)(DataFrameRow
Row │ age workclass fnlwgt education education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country salary
│ Int64 String Int64 String Float64? String String? String String String Int64 Int64 Int64 String String
──────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1000 │ 61 State-gov 162678 5th-6th 3.0 Married-civ-spouse Transport-moving Husband White Male 0 0 40 United-States <50k, "<50k")encode(method, Training(), row)(([5, 16, 2, 10, 5, 2, 3, 2, 3], [1.6435221651965317, -0.2567538819371021, -2.751580937680526, -0.14591824281680102, -0.21665620002803673, -0.035428902921319616]), Float32[0.0, 1.0])To get a model suitable for this learning method, we can use the methodmodel function.
methodmodel(method, NamedTuple())Chain(
Parallel(
vcat,
Chain(
FastAI.Models.var"#43#45"(),
Parallel(
vcat,
Embedding(10, 6), # 60 parameters
Embedding(17, 8), # 136 parameters
Embedding(8, 5), # 40 parameters
Embedding(17, 8), # 136 parameters
Embedding(7, 5), # 35 parameters
Embedding(6, 4), # 24 parameters
Embedding(3, 3), # 9 parameters
Embedding(43, 13), # 559 parameters
Embedding(3, 3), # 9 parameters
),
identity,
),
BatchNorm(6), # 12 parameters, plus 12
),
Chain(
Dense(61, 200, relu; bias=false), # 12_200 parameters
BatchNorm(200), # 400 parameters, plus 400
identity,
),
Chain(
Dense(200, 100, relu; bias=false), # 20_000 parameters
BatchNorm(100), # 200 parameters, plus 200
identity,
),
Dense(100, 2), # 202 parameters
) # Total: 19 arrays, 34_022 parameters, 132.523 KiB.Here the second parameter is a NamedTuple of backbones. The keys can be
:categorical
:continuous
:finalclassifier
corresponding to the specific backbones, and we can choose to specify any combination of these to customize our model.
methodmodel(method, (categorical=catback,))Chain(
Parallel(
vcat,
Chain(
FastAI.Models.var"#43#45"(),
Parallel(
vcat,
Embedding(10, 20), # 200 parameters
Embedding(17, 8), # 136 parameters
Embedding(8, 5), # 40 parameters
Embedding(17, 8), # 136 parameters
Embedding(7, 5), # 35 parameters
Embedding(6, 4), # 24 parameters
Embedding(3, 3), # 9 parameters
Embedding(43, 13), # 559 parameters
Embedding(3, 3), # 9 parameters
),
Dropout(0.2),
),
BatchNorm(6), # 12 parameters, plus 12
),
Chain(
Dense(75, 200, relu; bias=false), # 15_000 parameters
BatchNorm(200), # 400 parameters, plus 400
identity,
),
Chain(
Dense(200, 100, relu; bias=false), # 20_000 parameters
BatchNorm(100), # 200 parameters, plus 200
identity,
),
Dense(100, 2), # 202 parameters
) # Total: 19 arrays, 36_962 parameters, 143.523 KiB.To get an iterable for our data, we can use the methoddataloader function
traindl, valdl = methoddataloaders(splitdata, method, 128; pctgval = 0.2, shuffle = true, buffered=false)(DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{DLPipelines.MethodDataset{FastAI.BlockMethod{Tuple{FastAI.TableRow{9, 6, Dict{Any, Any}}, FastAI.Label{String}}, Tuple{FastAI.TabularPreprocessing{DataAugmentation.Sequence{Tuple{DataAugmentation.FillMissing{Dict{Any, Any}, NTuple{6, Symbol}}, DataAugmentation.NormalizeRow{Dict{Any, Any}, NTuple{6, Symbol}}, DataAugmentation.Categorify{Dict{Any, Any}, NTuple{9, Symbol}}}}}, FastAI.OneHot{DataType}}, FastAI.OneHotTensor{0, String}}}}}(batchviewcollated() with 204 batches of size 128, true), DataLoaders.GetObsParallel{DataLoaders.BatchViewCollated{DLPipelines.MethodDataset{FastAI.BlockMethod{Tuple{FastAI.TableRow{9, 6, Dict{Any, Any}}, FastAI.Label{String}}, Tuple{FastAI.TabularPreprocessing{DataAugmentation.Sequence{Tuple{DataAugmentation.FillMissing{Dict{Any, Any}, NTuple{6, Symbol}}, DataAugmentation.NormalizeRow{Dict{Any, Any}, NTuple{6, Symbol}}, DataAugmentation.Categorify{Dict{Any, Any}, NTuple{9, Symbol}}}}}, FastAI.OneHot{DataType}}, FastAI.OneHotTensor{0, String}}}}}(batchviewcollated() with 26 batches of size 256, true))and for getting a suitable loss function, methodlossfn comes in handy.
methodlossfn(method)logitcrossentropy (generic function with 1 method)All of these steps can be customized according to requirements and can be put together to create a Learner for training.
The methodlearner function abstracts these functions to directly get a Learner.
learner = methodlearner(method, splitdata, (categorical=catback,), Metrics(accuracy), batchsize=128, dlkwargs=NamedTuple(zip([:buffered], [false])))Learner()Now, we can use this for training our model by calling the fit or fitonecycle function on it provided by FluxTraining.
fit!(learner, 1)┌───────────────────────────────────┬───────┬────────┬──────────┐
│ Phase │ Epoch │ Loss │ Accuracy │
├───────────────────────────────────┼───────┼────────┼──────────┤
│ FluxTraining.Phases.TrainingPhase │ 1.0 │ 0.1368 │ 0.94719 │
└───────────────────────────────────┴───────┴────────┴──────────┘
┌─────────────────────────────────────┬───────┬─────────┬──────────┐
│ Phase │ Epoch │ Loss │ Accuracy │
├─────────────────────────────────────┼───────┼─────────┼──────────┤
│ FluxTraining.Phases.ValidationPhase │ 1.0 │ 0.00205 │ 1.0 │
└─────────────────────────────────────┴───────┴─────────┴──────────┘
| Link to PR/commit | Description | Status |
|---|---|---|
| FastAI.jl #26 | Adds TableContainer | Merged |
| DataAugmentation.jl #45 | Adds table transforms and item | Merged |
| FastAI.jl #124 | Adds table model | Merged |
| Flux.jl #1656 | Updates Embedding layer, adds Flux.outputsize support for Embedding layer, and doc updates | Waiting for a decision on the best way to handle this case for Flux.outputsize |
| FastAI.jl #141 | Adds tabular blocks and encodings, fixes some bugs, adds hash for adult_sample dataset. | Merged |
| fluxml.github.io #94 | Blog post showing some of the implemented functionalities. | Will be moved to the new FastAI.jl website (under construction) |
More comprehensive documentation and notebooks can be added to demonstrate the various features. Out of the box support for additional learning methods like multiple column classification or a combination of regression and classification tasks would also be nice to have. Implementation for complex tabular models like SAINT[5] will further improve the capabilities of the package.
All of this wouldn't have been possible at all without the continuous support and teachings of my mentors Kyle Daruwalla, Brian Chen and Lorenz Ohly. From knowing nothing about the Julia Language to completing this project really shows the amount of effort they have put in for helping me through every step of this project.
The whole community has been really helpful as well, with their constant support and suggestions. From the informative references Ari provided about what's happening in the ecosystem, the various critiques and suggestions from Dhairya, the helpful reviews from Michael, Logan and the rest of the community, it has really been a very fun and great learning experience.
I would also like to thank Google and the whole Summer of Code team for creating a really wonderful program and providing us with this opportunity.
| [1] | fastai: A Layered API for Deep Learning |
| [2] | fastai's TabularModel |
| [3] | Entity Embeddings of Categorical Variables |
| [4] | From the DLPipelines.jl docs |
| [5] | SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training |