ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (2024)

Jintai Chen^∗jtchen721@gmail.comUniv. of Illinois Urbana-ChampaignUrbana, IL, USA,Jiahuan Yan^∗jyansir@zju.edu.cnZhejiang UniversityHangZhou, China,Qiyuan Chenchenqiyuan1012@gmail.comZhejiang UniversityHangZhou, China,Danny Z. Chendchen@nd.eduUniversity of Notre DameUrbana, IL, USA,Jian Wuwujian2000@zju.edu.cnZhejiang UniversityHangZhou, ChinaandJimeng Sunjimeng@illinois.eduUniv. of Illinois Urbana-ChampaignUrbana, IL, USA

(2024; 8 Feb 2024; 17 May 2024)

Abstract.

Data organized in tabular format is ubiquitous in real-world applications, and users often craft tables with biased feature definitions and flexibly set prediction targets of their interests. Thus, a rapid development of a robust, effective, dataset-versatile, user-friendly tabular prediction approach is highly desired. While Gradient Boosting Decision Trees (GBDTs) and existing deep neural networks (DNNs) have been extensively utilized by professional users, they present several challenges for casual users, particularly: (i) the dilemma of model selection due to their different dataset preferences, and (ii) the need for heavy hyperparameter searching, failing which their performances are deemed inadequate.In this paper, we delve into this question: Can we develop a deep learning model that serves as a sure bet solution for a wide range of tabular prediction tasks, while also being user-friendly for casual users? We delve into three key drawbacks of deep tabular models, encompassing: (P1) lack of rotational variance property, (P2) large data demand, and (P3) over-smooth solution. We propose ExcelFormer, addressing these challenges through a semi-permeable attention module that effectively constrains the influence of less informative features to break the DNNs’ rotational invariance property (for P1), data augmentation approaches tailored for tabular data (for P2), and attentive feedforward network to boost the model fitting capability (for P3). These designs collectively make ExcelFormer a sure bet solution for diverse tabular datasets. Extensive and stratified experiments conducted on real-world datasets demonstrate that our model outperforms previous approaches across diverse tabular data prediction tasks, and this framework can be friendly to casual users, offering ease of use without the heavy hyperparameter tuning. The codes are available at https://github.com/whatashot/excelformer.

Tabular data prediction, Mixup

*: Co-first Authors.

^†^†copyright: acmlicensed^†^†journalyear: 2024^†^†doi: XXXXXXX.XXXXXXX^†^†conference: SIGKDD Conference on Knowledge Discovery and Data Mining; Aug 25–29, 2024; Barcelona, Spain^†^†isbn: 978-1-4503-XXXX-X/18/06^†^†ccs: Computing methodologiesArtificial intelligence

1. Introduction

ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (1)

Tabular data is ubiquitous and plays a critical role in real-world applications, spanning diverse domains, such as medical prediction(Wu etal., 2022; Yan etal., 2024), market prediction(Wang etal., 2017), and financial risk forecasting(Kim etal., 2020b).However, unlike fields such as image and natural language processing, where data from disparate datasets frequently exhibit similar spatial or sequential feature relations and aligned semantics, tabular data often lack such “common” and “standard” data structures.Tables are typically created by casual users for diverse purposes. The features and targets can be defined subjectively, and table columns (features) are added or removed arbitrarily, even sometimes resulting in missing information or adding noise.Therefore, while some bespoke frameworks following specific inductive biases have thrived in the domains of image and textual data, achieving comparable success on tabular data is notably challenging.

Therefore, users are compelled to undergo computationally intensive hyperparameter searching in model development for specific tabular datasets, and there is currently no universally recognized method for selecting a model and determining a set of hyperparameters without comprehensive testing on the target datasets. In this paper, we endeavor to design a DNN framework that serves as a sure bet solution for diverse tabular datasets, by solving key drawbacks of existing DNNs. Inspired by (Grinsztajn etal., 2022), we summarize three key drawbacks of current deep tabular models, including:

(P1) lack of rotational variance property.As each table column holds distinct semantics and tabular data lack rotational invariance(Ng, 2004), rotational variant algorithms like decision trees are more efficient on tabular datasets. However, DNNs are a kind of rotationally invariant algorithms that has a worst-case sample complexity growing at least linearly in the number of uninformative features. As mentioned above, tables are created by casual users who frequently include uninformative features, underscoring the importance of rotational variance property.

(P2) large data demand. DNNs typically possess larger hypothesis spaces, necessitating more training data to obtain robust performances. Thus, it is widely noted that DNNs frequently exhibit competitive, and at times, superior performance compared to GBDTs on large-scale tabular datasets. Yet, their performances tend to be subpar on smaller datasets.

(P3) over-smooth solution. Observations suggest that DNNs tend to produce overly smooth solutions, a factor pinpointed by(Grinsztajn etal., 2022) as a contributor to suboptimal performance on datasets featured by irregular decision boundaries (as illustrated in Fig.3). In contrast, decision trees partition the feature space based on thresholds along each axis, resulting in sharp boundaries that are demonstrated to be more suitable for a majority of datasets.

ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (2)

Among these, the large data demand (P2) should be the primary obstacle to current deep tabular models: Intuitively, rotationally-invariant DNNs are data-inefficient(Grinsztajn etal., 2022), but this drawback can be mitigated if there are sufficient training data(Ng, 2004). Moreover, modern DNNs have been demonstrated to be able to fit any functions(Goodfellow etal., 2016). Thus, if a plethora of discrete data points adequately fulfill the feature space to accurately represent the underlying feature-target functions, DNNs can definitely fit such functions rather than obtain overly smooth solutions. We empirically observed that even though DNNs outperform GBDTs on large tabular datasets, their performance significantly declines and falls short of GBDTs when fewer training samples are used. Three examples are presented in Fig.1. Thus, boosting the effectiveness of DNNs on small datasets is the key to achieve a sure bet model.

In this paper, we delve into addressing the limitations of existing DNNs and present a robust model, ExcelFormer, for diverse tabular data prediction tasks. To address (P1), we design a novel attention module named the semi-permeable attention (SPA), which selectively permits more informative features to gather information from less informative ones. This results in a noticeably reduced influence of less informative features. A special interaction-attenuation initialization approach is devised to boost this module. This initialization approach sets SPA’s parameters with minimal values, effectively attenuating tabular data feature interactions during the initial stages of training. Consequently, SPA undertakes more cautious feature interactions at the beginning, learning key interactions and aiding ExcelFormer to break the rotational invariance.

To address (P2), we introduce two interpolation-based data augmentation approaches for tabular data: Feat-Mix and Hid-Mix. Interpolation-based data augmentation approaches, such as Mixup(Zhang etal., 2018) and its variants(Verma etal., 2019; Uddin etal., 2020), have demonstrated their effectiveness in computer vision tasks. However, as the feature-target functions are often irregular(Grinsztajn etal., 2022), simple interpolation methods, like Mixup, tend to regularize DNNs to behave linearly in-between training examples(Zhang etal., 2018), which potentially conflicts with the irregular functions. Therefore, they often fall short of improving and even degrading the performance of DNNs. We propose two data augmentation approaches, Feat-Mix and Hid-Mix, to avoid such conflicts and respectively encourage DNNs to learn independent feature transformations and to conduct sparse feature interactions.

To address (P3), we employ an attentive feedforward network to replace the vanilla multilayer perceptron-based feedforward network in the Transformer. Both consisting of two fully connected layers, this substitution integrates an attentive module to enhance the model’s ability to effectively fit irregular boundaries. Following previous work, we utilize Gated Linear Units (GLU)(Shazeer, 2020) and do not introduce any new modules. Experimental results confirm that this simple substitution effectively improves model performances.

Notably, replacing the vanilla self-attention and feedforward network with the SPA and the attentive feedforward network does not incur any additional computational burden, and the data augmentation approaches come at negligible cost. That means, the proposed ExcelFormer maintains a comparable size to that of cutting-edge tabular Transformers (e.g., FT-Transformer(Rubachev etal., 2022)). Comprehensive and stratified experiments have demonstrated the superiority of our designed ExcelFormer:

(1)
Our model outperforms existing GBDTs and DNNs not only on small tabular datasets where existing DNNs typically underperform GBDTs, but also on large-scale datasets where traditional DNNs have shown preference (Sec.6.2).
(2)
Across a spectrum of stratified experiments in terms of feature quantity, dataset scale, and task types, our ExcelFormer consistently outperforms both GBDTs and DNNs. Additionally, we observe that besides our ExcelFormer, existing approaches excel in handling different types of datasets.This underscores its status as a reliable solution for non-expert users, mitigating the need for intricate model selection (Sec.6.3 and Sec.6.4).
(3)
Notably, while most existing approaches necessitate intensive hyperparameter tuning through repeated model runs (typically 50-100 times), our model achieves superior performance with pre-fixed parameters, offering a time-efficient and user-friendly solution (AppendixD).
See Also
Warm Audio WA-19 Dynamic Microphone Nickel Used Volkswagen Golf Cars for Sale | Second Hand Volkswagen Golf | Big Motoring World The odds were never in my favour - Chapter 118 - Antony444 - Harry Potter Polish Media System in a Comparative Perspective

2. Related Work

2.1. Supervised Tabular Data Prediction

While deep neural networks (DNNs) have proven to be effective in computer vision(Khan etal., 2022) and natural language processing(Vaswani etal., 2017), GBDT approaches like XGBoost continue to be the preferred choice for tabular data prediction tasks(Katzir etal., 2020; Grinsztajn etal., 2022), particularly on smaller-scale datasets, due to their consistently superior performance.To enhance the performance of DNNs, recent studies have focused on developing sophisticated neural modules for (i) handling heterogeneous feature interactions(Gorishniy etal., 2021; Chen etal., 2022; Yan etal., 2023; Chen etal., 2023), (ii) seeking for decision paths by emulating decision-tree-like approaches(Katzir etal., 2020; Popov etal., 2019; Arik and Pfister, 2021), or (iii) resorting to conventional approaches(Cheng etal., 2016; Guo etal., 2017) and regularizations(Jeffares etal., 2023). In addition to model designs, various feature representation approaches, such as feature embedding(Gorishniy etal., 2022; Chen etal., 2023), discretization of continuous features(Guo etal., 2021; Wang etal., 2020), and Boolean algebra based methods(Wang etal., 2021b), were well explored.All these efforts suggested the potentials of DNNs, but they have not yet surpassed GBDTs in performance, especially on small-scale datasets.Moreover, there were several attempts(Wang and Sun, 2022; Arik and Pfister, 2021; Yoon etal., 2020; Zhu etal., 2023) to apply self-supervision learning on tabular datasets. However, many of these approaches are dataset- or domain-specific, and transferring these models to distant domains remains challenging due to the heterogeneity across tabular datasets. While pretrained on a substantial dataset corpus, XTab(Zhu etal., 2023) offered only a modest performance improvement due to the limited shared knowledge across datasets.TapPFN(Hollmann etal., 2022) concentrated on solving classification problems for small-scale tabular datasets and achieved commendable results.However, its efficiency waned when applied to larger datasets and regression tasks. In summary, compared to decision tree-based GBDTs, DNNs still fall short on tabular data, especially on small-scale ones, which remains an open challenge.

2.2. Mixup and other Data Augmentations

The vanilla Mixup(Zhang etal., 2018) generates a new data through convex interpolations of two existing data, which was proved beneficial on computer vision tasks(Tajbakhsh etal., 2020; Touvron etal., 2021a). However, we have observed that vanilla Mixup may conflict with irregular target patterns (please refer to Fig.3) and typically achieves inferior performance. For instance, in the context of predicting therapy feasibility, a 70-year-old man (elderly individual) and a 10-year-old boy (young individual) may not meet the criteria for a particular therapy, but an individual with an interpolated feature value (aged 40) would benefit from it. Namely, the vanilla Mixup can lead to over smooth solution, which is considered to be unsuitable(Grinsztajn etal., 2022). ManifoldMix(Verma etal., 2019) applied similar interpolations in the hidden states, which did not fundamentally alter the data synthesis approach of Mixup and exhibited similar characteristics to the vanilla Mixup. The follow-up variants CutMix(Yun etal., 2019), AttentiveMix(Walawalkar etal., 2020), SaliencyMix(Uddin etal., 2020), ResizeMix(Qin etal., 2020), and PuzzleMix(Kim etal., 2020a) spliced image pieces spatially, preserving local image patterns but being not directly applicable to tabular data. CutMix is used in SAINT(Somepalli etal., 2021) for tabular data prediction, but it is highly impacted by uninformative features, as shown in Table6.Kadra etal. (2021) investigated various data augmentation techniques aimed at enhancing the performance of MLPs on tabular data. However, these methods were found to be effective only on a limited number of tabular datasets, requiring time-consuming enumeration and testing of these options. In contrast, this paper introduced two novel data augmentation approaches for tabular data, Hid-Mix and Feat-Mix, which avoid the conflicts encountered with Mixup and contribute to ExcelFormer achieving superior performance.

3. ExcelFormer

The workflow of ExcelFormer is illustrated in Fig.2. ExcelFormer processes data by the following components: 1) After pre-processing likeGorishniy etal. (2021), the embedding layer featurizes and embeds tabular features to token-level embeddings; 2) token-level embeddings were alternately processed by the newly proposed semi-permeable attention module (SPA) and gated linear units (GLUs). 3) Finally, a prediction head yields the final target. In the following, we will introduce the novel semi-permeable attention with the interaction attenuated initialization and the GLU based attentive feedforward network first and then the rest parts of ExcelFormer.

3.1. Solving (P1) with Semi-Permeable Attention

ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (3)

As stated in (Ng, 2004), less informative features make minor contributions on target prediction but still necessitate at least a linear increase in the requirement for training samples to learn how to “ignore” them. DNNs are rotationally invariant algorithms, which are data-inefficient with a worst-case sample complexity increasing at least linearly with the number of uninformative features (Grinsztajn etal., 2022).

Our idea is to incorporate an inductive bias into the self-attention mechanism, which selectively restricts the impacts of a feature to only those that are less informative, thereby reducing the overall impact of uninformative features on prediction outcomes. We propose a semi-permeable attention module (SPA), as:

(1)

z^{\prime}=\text{softmax}(\frac{(zW_{q})(zW_{k})^{T}\oplus M}{\sqrt{d}})(zW_{v%}),

where $z\in\mathbb{R}^{f\times d}$ is the input embeddings and $z^{\prime}$ the output embeddings, $W_{q},W_{k}$ , $W_{v}\in\mathbb{R}^{d\times d}$ are all learnable matrices, and $\oplus$ is element-wise addition. $M\in\mathbb{R}^{f\times f}$ is an unoptimizable mask, where the element at the $i$ -th row and $j$ -th column is defined by:

(2)

M[i,j]=\begin{cases}-\infty&I(\mathbf{f}_{i})>I(\mathbf{f}_{j})\\0&I(\mathbf{f}_{i})\leq I(\mathbf{f}_{j})\end{cases}

The function $I(\cdot)$ represents a measure of feature importance, and we use the “mutual information” metric in this paper (see AppendixE for details). If a feature $\mathbf{f}_{i}$ is more informative compared to $\mathbf{f}_{j}$ , $M[i,j]$ is set $-\infty$ (we use $-10^{5}$ in implementation) and thus the $(i,j)$ grid on the attention map is masked. It prevents the transfer of the embedding of the feature $\mathbf{f}_{j}$ to the $\mathbf{f}_{i}$ ’s embedding.

In this way, only more informative features are permitted to propagate information to the less informative ones, and the reverse is not allowed. By doing so, SPA still maintains interaction pathways between any two features while constraining the impacts of less informative ones. Intuitively, when training samples are insufficient, some feature interactions conducted by the model may be sub-optimal, as vanilla self-attention was proved data-inefficient(Touvron etal., 2021a). When using SPA, it can avoid the excessive impacts of a noisy feature on prediction outcomes in case some associated interaction pathways are ill-suited.Furthermore, the SPA inhibits certain feature transfer pathways, thereby obviating the need for the ExcelFormer to learn partial rotational directions. The rotational properties of the ExcelFormer lie between those of the rotational invariant counterpart with vanilla self-attention (e.g., FT-Transformer) and a fully rotational variant model (e.g., feedforward network conducting no feature interactions). Namely, our SPA partially disrupts DNNs’ rotational invariance property.In practice, SPA is extended to a multi-head self-attention version, with 32 heads by default.

ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (4)

Interaction Attenuated Initialization.

Similar to how the SPA disrupts rotational invariance by diminishing feature interactions,we present a specialized initialization approach for SPA to ensure that ExcelFormer starts as a largely non-rotationally invariant model. Notably, removing all self-attention operations from a Transformer model, features are processed individually, which makes the Transformer model nearly non-rotationally invariant (if we set aside the full connection layers that fuse features for target prediction). Concurrently, prior researches have evidenced the indispensable role of feature interactions (e.g., through self-attention) in Transformer-based models on tabular data(Gorishniy etal., 2021; Yan etal., 2023). By integrating these insights, our proposed interaction attenuated initialization scheme initially dampens the impact of SPA during the early stages of training, allowing essential feature interactions progressively grow under the driving force of the data.

Our interaction attenuated initialization scheme is built upon the commonly used He’s initialization(He etal., 2015) or Xavier initialization(Glorot and Bengio, 2010), by rescaling the variance of an initialized weight $w$ with $\gamma$ ( $\gamma\rightarrow 0^{+}$ ) while keeping the expectation at 0:

(3)

\text{Var}(w)=\gamma\text{Var}_{\text{prev}}(w),

where $\text{Var}_{\text{prev}}(w)$ denotes the weight variance used in the He’s initialization and Xavier initialization. In this work, we set $\gamma=10^{-4}$ . To reduce the impacts of SPA,we apply Eq.(3) to all the parameters in the SPA module. Thus, ExcelFormer works like a non-rotationally invariant model initially.

Actually, for a module with an additive identity shortcut like $y=\mathcal{F}(x)+x$ , our initialization approach attenuates the sub-network $\mathcal{F}(x)$ and satisfies the property of dynamical isometry(Saxe etal., 2014) for better trainability. Some previous work(Bachlechner etal., 2021; Touvron etal., 2021b) suggested to rescale the $\mathcal{F}(x)$ path as $y=\eta\mathcal{F}(x)+x$ , where $\eta$ is a learnable scalar initialized as 0 or a learnable diagonal matrix whose elements are of very small values. Different from these methods, our attenuated initialization approach directly assigns minuscule values to the weights during initialization. Our approach is better suited for the flexible learning of whether each feature interaction pathway should be activated or not, thereby achieving sparse attention.

3.2. Solving (P3) with GLU layer

The irregularities (Fig.3 shows an example) present in the tabular feature-target relationship make it particularly advantageous for decision trees utilizing multiple thresholds to split the feature space.On the contrary, existing Transformer employs a two-layer MLP as its feedforward network (FFN), which possesses a lesser degree of non-linear fitting capability. Therefore, we replace the vanilla FFN by a Gated Linear Unit (GLU) layer. Diverging from the standard GLU architecture, we employ the “tanh” activation in lieu of the “sigmoid” activation for better optimization properties(LeCun etal., 2002), as:

(4)

z^{\prime}=\tanh{(\texttt{Linear}_{1}(z))}\odot\texttt{Linear}_{2}(z),

where $\texttt{Linear}_{1}$ and $\texttt{Linear}_{2}$ are applied onto the embedding dimension $d$ of $z$ , $\odot$ denotes element-wise product. Please note that both the vanilla FFN and GLU employ two fully connection layers (FFN is defined by $z^{\prime}=\texttt{Linear}_{1}(\text{ReLU}(\texttt{Linear}_{2}(z)))$ ), resulting in similar computational costs. The SPA and GLU modules are alternately stacked to form the core structure of the ExcelFormer model, as shown in Fig.2.

Existing tabular Transformers(Yan etal., 2023; Gorishniy etal., 2021) use linear embedding layer to independently deal with each feature $\mathbf{f}_{i}\in\mathbb{R}$ into an embedding $\mathbf{z}_{i}\in\mathbb{R}^{d}$ , by $z_{i}=\mathbf{f}_{i}W_{i,1}+b_{i,1}$ . Here we also use GLU to replace it by $\mathbf{z}_{i}=\tanh{(\mathbf{f}_{i}W_{i,1}+b_{i,1})}\odot(\mathbf{f}_{i}W_{i,%2}+b_{i,2})$ , where $W_{i,1},W_{i,2}\in\mathbb{R}^{1\times d}$ and $b_{i,1},b_{i,2}\in\mathbb{R}^{d}$ are learnable parameters. Then, the initial feature embedding $z^{(0)}$ are obtained by stacking all $\mathbf{z}_{i}(i=1,2,\ldots,f)$ , as $z^{(0)}=[\mathbf{z}_{1},\mathbf{z}_{2},\mathbf{z}_{3},\ldots,\mathbf{z}_{f}]^{%T}\in\mathbb{R}^{f\times d}$ like previous work.

3.3. The rest part of ExcelFormer

Feature Pre-processing.

Feature values are pre-processed before feeding into ExcelFormer. The numerical features are normalized by quantile transformation and the categorical features are converted into numerical ones using the CatBoost Encoder implemented with the Sklearn Python package¹¹1https://contrib.scikit-learn.org/category_encoders/catboost.html. This step performs similar to previous works (e.g., FT-Transformer (Gorishniy etal., 2021)).

Prediction Head.

The prediction head is directly applied to the output of the topmost transformer block, which contains two fully connection layers to separately compress the information along the token embeddings and fuse the information from features, by:

4. Solving (P2) with Data Augmentation

A straightforward approach to tackle data insufficiency is to create training data. While Mixup(Zhang etal., 2018) regularizes DNNs to favor linear behaviors between samples and stands as one of the most effective data augmentation methods in computer vision, empirical evidence suggests that it does not perform optimally on tabular datasets (e.g., see Table5).This discrepancy may be due to the conflict between the model’s linear behavior and the irregularity of target functions, as intuitively illustrated in Fig.3.To address this challenge, we introduce two Mixup variants, Hid-Mix and Feat-Mix, which mitigate the conflicts in creating samples.

Hid-Mix. Our Hid-Mix is applied to the token-level embeddings after the input samples have been processed by the embedding layer, along with their corresponding labels. It randomly exchanges some embedding “dimensions” between two samples (please refer to Fig.4(a)). Let $z_{1}^{(0)},z_{2}^{(0)}\in\mathbb{R}^{f\times d}$ be the token-level embeddings of two randomly selected samples, with $y_{1}$ and $y_{2}$ denoting their respective labels. A new sample represented as a token-label pair $(z_{\text{m}}^{(0)},y_{\text{m}})$ is synthesized by:

(6)

\left\{\begin{aligned} &z_{\text{m}}^{(0)}=S_{H}\odot z_{1}^{(0)}+(\mathbbm{1}%_{H}-S_{H})\odot z_{2}^{(0)},\\&y_{\text{m}}=\lambda_{H}y_{1}+(1-\lambda_{H})y_{2},\end{aligned}\right.

where the matrix $S_{H}$ is of size $f\times d$ and is formed by stacking $f$ identical $d$ -dimensional binary vectors denoted as $s_{h}$ : $S_{H}=[s_{h},s_{h},\ldots,s_{h}]^{T}$ . $s_{h}$ consists of $\lfloor\lambda_{H}\times d\rfloor$ randomly selected elements set to 1 and the rest elements set to 0. The scalar coefficient $\lambda_{H}$ for labels is sampled from the $\mathcal{B}eta(\alpha_{H},\alpha_{H})$ distribution, where $\alpha_{H}$ is a hyper-parameter. $\mathbbm{1}_{H}$ is an all-one matrix with dimensions $f\times d$ . In practice, $\lambda_{H}$ is first sampled from given $\mathcal{B}eta(\alpha_{H},\alpha_{H})$ distribution. Subsequently, we randomly select $\lfloor\lambda_{H}\times d\rfloor$ elements to construct the vector $s_{h}$ and the matrix $S_{H}$ .

Since the embedding “dimensions” from different samples may be randomly combined in training, ExcelFormer is encouraged to independently and equally handle various embedding dimensions. Considering each embedding dimension as a distinct “profile” version of input data (as each embedding element is projected from a scalar feature value), Hid-Mix regularizes ExcelFormer to behave like a bagging predictor(Breiman, 1996). Therefore, Hid-Mix may also help mitigate the effects of data noise and perturbations, in addition to increasing the amount of training data.

Feat-Mix. Our idea of Feat-Mix is visualized as in Fig.4. Unlike Hid-Mix that operates on the embedding dimension, our Feat-Mix synthesizes new sample $(x_{m},y_{m})$ by swapping parts of features between two randomly selected samples $x_{1},x_{2}\in\mathbb{R}^{f}$ , and blending their labels $y_{1}$ and $y_{2}$ guided by feature importance, by:

(7)

\left\{\begin{aligned} &x_{\text{m}}=\mathbf{s}_{F}\odot x_{1}+(\mathbbm{1}_{F%}-\mathbf{s}_{F})\odot x_{2},\\&y_{\text{m}}=\Lambda y_{1}+(1-\Lambda)y_{2},\end{aligned}\right.

wherethe vector $\mathbf{s}_{F}$ andthe all-one vector $\mathbbm{1}_{F}$ are of size $f$ , $\mathbf{s}_{F}$ contains $\lfloor\lambda_{F}\times f\rfloor$ randomly chosen elements set to 1 and the remaining elements set to 0. $\lambda_{F}\sim\mathcal{B}eta(\alpha_{F},\alpha_{F})$ .The coefficient value, $\Lambda$ , is determined based on the contribution of $x_{1}$ and $x_{2}$ , taking into account feature importance, by:

(8)

\Lambda=\frac{\sum_{\mathbf{s}_{F}^{(i)}=1}I(\mathbf{f}_{i})}{\sum^{f}_{i=1}I(%\mathbf{f}_{i})},

where $\mathbf{s}_{F}^{(i)}$ represents the $i$ -th element of $\mathbf{s}_{F}$ , and $I(\cdot)$ returns the feature importance using mutual information. When disregarding feature importance, $\Lambda=\lambda_{F}$ (assuming $\lfloor\lambda_{F}\times f\rfloor=\lambda_{F}\times f$ ), making Feat-Mix degenerate into a form similar to cutmix(Yun etal., 2019). However, due to the presence of uninformative features in tabular datasets, Feat-Mix emerges as a more robust scheme.

As features from two distinct samples are randomly combined to create new samples, Feat-Mix promotes a solution with fewer feature interaction. This aligns with the functionality similar to our Interaction Attenuated Initialization (see Sec.3.1). We argue that Feat-Mix not only supplements the training dataset as a data augmentation method, but also encourages ExcelFormer to predominantly exhibit like a non-rotationally invariant algorithm.

5. Training and Loss Functions

ExcelFormer can handle both classification and regression tasks on tabular datasets in supervised learning. In training, our two proposed data augmentation schemes can be applied successively by $\textsc{Hid-Mix}(\text{Embedding Layer}(\textsc{Feat-Mix}(x,y)))$ or used independently. But, our tests suggest that the effect of ExcelFormer on a certain dataset could be better by using only Feat-Mix or Hid-Mix. Thus, we use only one scheme in dealing with certain tabular datasets. The cross-entropy loss is used for classification tasks, and the mean square error loss is for regression tasks.

ExcelFormer setting:	No DA (t)	Feat-Mix (d)	Hid-Mix (d)	Mix Tuned (t)	Fully Tuned (t)
XGboost (t)	4.20 $\pm$ 2.76	4.21 $\pm$ 2.70	4.29 $\pm$ 2.73	4.34 $\pm$ 2.73	4.28 $\pm$ 2.77
Catboost (t)	4.61 $\pm$ 2.73	4.57 $\pm$ 2.69	4.63 $\pm$ 2.68	4.66 $\pm$ 2.61	4.64 $\pm$ 2.68
FTT (t)	4.32 $\pm$ 2.36	4.35 $\pm$ 2.35	4.41 $\pm$ 2.25	4.44 $\pm$ 2.32	4.39 $\pm$ 2.37
MLP (t)	5.23 $\pm$ 2.31	5.27 $\pm$ 2.34	5.26 $\pm$ 2.32	5.30 $\pm$ 2.37	5.32 $\pm$ 2.33
DCN v2 (t)	6.01 $\pm$ 2.78	5.96 $\pm$ 2.75	5.99 $\pm$ 2.27	6.03 $\pm$ 2.74	6.02 $\pm$ 2.73
AutoInt (t)	5.70 $\pm$ 2.61	5.78 $\pm$ 2.51	5.77 $\pm$ 2.56	5.88 $\pm$ 2.53	5.80 $\pm$ 2.55
SAINT (t)	5.48 $\pm$ 2.59	5.48 $\pm$ 2.55	5.56 $\pm$ 2.56	5.61 $\pm$ 2.55	5.56 $\pm$ 2.58
TransTab (d)	6.78 $\pm$ 2.52	6.80 $\pm$ 2.59	6.82 $\pm$ 2.57	6.86 $\pm$ 2.59	6.87 $\pm$ 2.55
XTab (d)	8.56 $\pm$ 2.20	8.68 $\pm$ 2.19	8.67 $\pm$ 2.19	8.67 $\pm$ 2.19	8.71 $\pm$ 2.14
ExcelFormer (ours)	4.11 $\pm$ 2.68	3.91 $\pm$ 2.60	3.62 $\pm$ 2.59	3.20 $\pm$ 2.10	3.41 $\pm$ 2.12

6. Experiments

6.1. Experimental Setups

Implementation Details. We configure the number of SPA and GRU modules as $L=3$ , set the feature embedding size to $d=256$ , and apply a dropout rate of 0.3 to the attention map. AdamW optimizer(Loshchilov and Hutter, 2018) is used with default settings. The learning rate is set to $10^{-4}$ without weight decay, and $\alpha_{H}$ and $\alpha_{F}$ for $\mathcal{B}eta$ distributions are both set to 0.5. These settings are the default hyperparameters for our ExcelFormer.In the hyperparameter fine-tuning process, we utilized the Optuna library(Akiba etal., 2019) for all approaches. Consistent with(Gorishniy etal., 2021), we randomly select 80% of the data as training samples and the remaining 20% as test samples. During training, we reserve 20% training samples for validation. To fine-tune our ExcelFormer, we designate two tuning configurations: “Mix Tuned” and “Fully Tuned”. “Mix Tuned” refers to only fine-tune hyperparameters of data augmentation (for Feat-Mix and Hid-Mix), while “Fully Tuned” optimizes all hyperparameters, including those related to data augmentation and model architecture. A comprehensive description of all settings can be found in AppendixC. We applied early stopping with a patience of 32 for ExcelFormer.

Datasets. A total of 96 small datasets sourced from the Taptap dataset benchmark²²2https://huggingface.co/datasets/ztphs980/taptap_datasets were utilized. The criterion for classifying datasets as small is based on having a sample size of less than 10,000.Besides, 21 larger public tabular datasets, ranging in scale from over 10,000 to 581,835 samples were also used. The detailed dataset descriptions are provided in AppendixF.

Compared Models. We compare our new ExcelFormer with two prominent GBDT approaches XGboost(Chen and Guestrin, 2016) and Catboost(Prokhorenkova etal., 2018) and several representative DNNs: FT-Transformer (FTT)(Gorishniy etal., 2021), SAINT(Somepalli etal., 2021), Multilayer Perceptron (MLP), DCN v2(Wang etal., 2021a), AutoInt(Song etal., 2019), and TapPFN (Hollmann etal., 2022). We also include two pre-trained DNNs: TransTab(Wang and Sun, 2022) and XTab(Zhu etal., 2023) for reference. The implementations of XGboost and Catboost mainly follow(Gorishniy etal., 2021). Since we aim to extensively tune XGboost and Catboost for their best performances, we increase the maximum number of estimators/ iterations (i.e., the number of decision trees) from 2000 to 4096 and the number of tuning iterations from 100 to 500, which give a more stringent setting and better performances. The settings for XGboost and Catboost are given in AppendixC. We use the default hyperparameters of pretrained models, TransTab and XTab, and fine-tune them on each dataset. They are not hyperparameter tuned, since their hyperparameter tuning spaces are not given. For large-scale datasets, FT-Transformer, SAINT, and TapFPN were fine-tuned based on the hyperparameters outlined in their respective papers. The architectures and hyperparameter tuning settings of the remaining DNNs follows the paper(Gorishniy etal., 2021). On small datasets, we tuned 50 iterations for each datasets.

Evaluation metrics. We use the area under the ROC Curve (AUC) and accuracy (ACC) for binary classification tasks and multi-class classification tasks. In regression tasks, we employ the negative root mean square error (nRMSE), where the negative sign is introduced to RMSE, aligning its direction with AUC and ACC, such that higher values across all these metrics indicate superior performance. Due to the high diversity among tabular datasets, performance ranks are used as a comprehensive metric, and the detailed results are given in AppendixG.

Setting	Model	Rank (mean $\pm$ std)
default	XGboost	8.52 $\pm$ 1.86
hyperparameters	Catboost	7.52 $\pm$ 2.44
	FTT	6.71 $\pm$ 1.74
	Excel w/ Feat-Mix	6.62 $\pm$ 2.44
	Excel w/ Hid-Mix	4.76 $\pm$ 1.95
hyperparameter	XGboost	4.29 $\pm$ 2.59
hyperparameter	Catboost	6.24 $\pm$ 2.39
fine-tuned	FT-T	5.19 $\pm$ 2.60
fine-tuned	Excel (Mix Tuned)	2.38 $\pm$ 1.53
	Excel (Fully Tuned)	2.05 $\pm$ 1.40

Model	ExcelFormer	FTT (t)	XGb (t)	Cat (t)	MLP (t)	DCNv2 (t)	AutoInt (t)	SAINT (t)	TransTab (d)	XTab (d)	TabPFN (t)
Characteristics: Task Type	Classification
Proportion	51%
Setting: Hid-Mix (d)	3.88	4.88	5.97	5.77	6.61	6.38	6.63	6.07	6.31	9.50	4.01
Setting: Mix Tuned	3.78	4.91	5.95	5.79	6.60	6.39	6.71	6.10	6.37	9.46	3.95
Characteristics: Task Type	Regression
Proportion	49%
Setting: Hid-Mix (d)	3.81	4.45	3.43	4.26	4.64	6.26	5.53	5.64	8.21	8.79	/
Setting: Mix Tuned	3.17	4.49	3.53	4.28	4.74	6.32	5.68	5.72	8.23	8.83	/
Characteristics: #. Sample	$\geq$ 500
Proportion	43%
Setting: Hid-Mix (d)	3.85	4.50	4.38	5.17	5.57	5.91	5.59	5.24	6.44	8.34	/
Setting: Mix Tuned	3.52	4.50	4.39	5.15	5.60	5.99	5.71	5.33	6.48	8.34	/
Characteristics: #. Sample	¡ 500
Proportion	57%
Setting: Hid-Mix (d)	3.45	4.34	4.22	4.23	5.02	6.05	5.90	5.79	7.10	8.92	/
Setting: Mix Tuned	3.18	4.38	4.28	4.29	5.05	6.05	5.97	5.79	7.13	8.88	/
Characteristics: #. Feature	#. Feature ¡ 8
Proportion	32%
Setting: Hid-Mix (d)	3.45	3.84	3.98	5.08	4.23	6.32	6.16	5.32	7.52	9.10	/
Setting: Mix Tuned	3.27	3.84	4.03	5.06	4.34	6.26	6.21	5.35	7.50	9.13	/
Characteristics: #. Feature	8 $\leq$ #. Feature ¡ 16
Proportion	38%
Setting: Hid-Mix (d)	3.76	4.26	4.44	4.61	6.31	5.75	5.39	5.69	6.61	8.17	/
Setting: Mix Tuned	3.17	4.33	4.49	4.74	6.33	5.81	5.58	5.78	6.64	8.14	/
Characteristics: #. Feature	#. Feature $\geq$ 16
Proportion	30%
Setting: Hid-Mix (d)	3.62	5.19	4.41	4.17	5.05	5.93	5.81	5.64	6.33	8.84	/
Setting: Mix Tuned	3.17	5.22	4.48	4.14	5.05	6.05	5.90	5.69	6.45	8.84	/

6.2. Model Performances and Discussions

Performances on Small Datasets.

DNNs are typically data-inefficient, thus we initially investigate whether the proposed ExcelFormer can effectively perform on small datasets. See Table1, our ExcelFormer consistently outperforms other models that undergo dataset-adaptive hyperparameter tuning, regardless of whether the hyperparameters of the ExcelFormer are tuned or not, which underscores the superiority of our proposed ExcelFormer.We observe that ExcelFormer with Hid-Mix slightly outperforms that with Feat-Mix; and if we tune hyperparameters of ExcelFormer, its performance achieves further improvement. Notably, hyperparameter fine-tuning reduces the standard deviations of performance ranks, indicating that applying hyperparameter fine-tuning onto ExcelFormer can yield more consistently superior results. Interestingly, while fine-tuning all the hyperparameters (“Fully Tuned”) should result in better performance ideally, it shows that, under the same fine-tuning iterations, “Mix Tuned” configuration performs better. This might be attributed to the higher efficiency of finely tuning data augmentation setting. To assess the effectiveness of our ExcelFormer’s architecture, we conducted experiments by excluding all data augmentation (Feat-Mix and Hid-Mix) and compare it with existing models. The results show that even without the use of Feat-Mix and Hid-Mix, ExcelFormer still outperforms previous approaches, underscoring the superiority of our architecture.

Performances on Larger Datasets.

We further conduct a comparison between our model and three previous state-of-the-art models: XGboost, Catboost, and FTT. We excluded other models from the comparison due to their relatively inferior performances and the significant computational load when dealing with large datasets. Each model undergoes evaluation with two settings: using default hyperparameters and dataset-adaptive fine-tuning hyperparameters. As depicted in Table2, our model also outperforms the previous models under both settings. Additionally, it is worth noting that our ExcelFormer with Hid-Mix still achieves comparable performance to prior models that undergo hyperparameter tuning, consistent with the findings on small datasets. Different from the conclusion on small datasets, the Fully Tuned ExcelFormer outperforms the Mix Tuned version on large datasets.

Takeaway. We discovered that (1) our model performs well on GBDT-favored smaller datasets and DNN-favored larger ones. This suggests that our design addresses the existing drawbacks of DNNs in tabular prediction. (2) Even with default hyperparameters, ExcelFormer consistently outperforms hyperparameter-tuned competitors on small datasets and performs competitive with them on larger ones. This implies that for users who are not experts in hyperparameter tuning, using our model can still obtain a strong solution. Moreover, even for professional users, our model also stands out as a top choice since the hyperparameter-tuned ExcelFormer performs excellent on various tabular prediction tasks.

6.3. Can ExcelFormer be a sure bet solution across various types of datasets?

We still have to further rigorously examine whether our model performs poorly on certain tabular dataset types to ensure that we have achieved our goal of building a sure bet solution.We divide datasets into various subgroups according to the task type, dataset size, and the number of features, and examine ExcelFormer performance within each subgroup. We adopt two configurations, Hid-Mix (default) and Mix Tuned, for ExcelFormer, while all of the existing models undergo hyperparameter fine-tuning. As shown in Table3, ExcelFormer with Hid-Mix (default) exhibits the best performance in all subgroups except for regression tasks, where it slightly lags behind hyperparameter-tuned XGboost. The Mix Tuned ExcelFormer outperforms other models in all subgroups, indicating that ExcelFormer does not exhibit overt dataset type preferences.

Takeaway. What changes does ExcelFormer bring to the field of tabular data prediction?Refer to Table3, besides our model, runner-up positions are held by TapFPN, FTT, XGBoost, and CatBoost in different subgroups. Notably, TapFPN is solely applicable to classification tasks, CatBoost performs well on datasets with numerous features, and FTT excels on datasets with fewer than 16 features. However, our proposed model demonstrates strong performance across all dataset types, which further proves its status as a sure be solution for tabular datasets.

6.4. Ablation Analysis

baseline	SPA	IAI	GLU	rank ( $\pm$ std)
✓				4.31 $\pm$ 0.94
✓		✓		3.87 $\pm$ 1.58
✓	✓			3.73 $\pm$ 2.04
✓	✓	✓		2.45 $\pm$ 1.60
✓			✓	3.71 $\pm$ 1.52
✓	✓	✓	✓	2.31 $\pm$ 1.46

Backbone	Data Augmentation	rank ( $\pm$ std)
FT-Transformer	N/A	3.28 $\pm$ 1.66
	Mixup	3.80 $\pm$ 1.39
	CutMix	2.91 $\pm$ 1.37
	Feat-Mix	2.50 $\pm$ 1.03
	Hid-Mix	2.24 $\pm$ 1.00
ExcelFormer	N/A	3.68 $\pm$ 1.43
	Mixup	3.46 $\pm$ 1.63
	CutMix	2.88 $\pm$ 1.21
	Feat-Mix	2.38 $\pm$ 1.25
	Hid-Mix	2.36 $\pm$ 1.03

Here we investigate the effects of architectural design (SPA, GLU) and data augmentation approaches (Hid-Mix and Feat-Mix), with the results presented in Table4 and Table5, respectively.

In Table4, the baseline model employs an ExcelFormer with the vanilla self-attention module initialized using typical Kaiming initialization(He etal., 2015), along with a vanilla MLP-based FFN. Subsequently, we evaluate how the designed approaches enhance this baseline. It is observed that SPA and IAI individually improve baseline performances, and their joint usage achieves even better results. Additionally, GLU can also significantly enhances the baseline. These findings suggest that our architectural designs, SPA with IAI and GLU, are all well-suited for tabular data predictions. In the last row, where all these components are utilized (ExcelFormer), we demonstrate that their combined utilization leads to the best results. The rotational invariance property brought by SPA and IAI are carefully demonstrated in AppendixB.

See Table5, where we report the comparison results among various data augmentation techniques on both the FTT backbone(Rubachev etal., 2022) and our ExcelFormer. It is crucial to recognize that the performance rankings are computed independently for different backbones, making direct comparisons of ranks unfeasible. It is evident that Mixup demonstrates minimal to no effect and sometimes even exhibits a detrimental impact. This could be attributed to Mixup’s interpolation potentially introducing “error” cases or steering the model towards an overly smooth solution. In contrast, CutMix consistently outperforms Mixup, approaching the performance level of Feat-Mix, albeit with slight inferiority. As discussed in Sec.4, without considering feature importance, Feat-Mix may regress to CutMix; however, feature importance computation is crucial to mitigate the impacts of uninformative features, a common occurrence in tabular datasets. Further experiments are detailed in AppendixA. It is evident that our proposed Feat-Mix and Hid-Mix consistently enhance DNN model performance and prove more effective compared to other Mixup variants.

7. Conclusions

This paper introduces a novel approach aimed at addressing three key limitations of DNNs when applied to tabular data prediction. We present a novel semi-permeable attention module incorporated with an interaction attenuated initialization approach, a GLU based FFN, as well as two data augmentation approaches: Hid-Mix and Feat-Mix. Through the integration of these designs, we present ExcelFormer, a model that maintains the same size as previous tabular Transformers but significantly outperforms existing GBDTs and DNNs in terms of performance, without hyperparameter tuning. Extensive and stratified experiments demonstrate that the ExcelFormer stands out as a sure bet solution for tabular prediction. We believe the proposed framework is highly accessible and user-friendly for even novices working with tabular data.

References

(1)
Akiba etal. (2019)Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.Optuna: A next-generation hyperparameter optimization framework. In The ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Arik and Pfister (2021)SercanÖ Arik and Tomas Pfister. 2021.TabNet: Attentive interpretable tabular learning. In The AAAI Conference on Artificial Intelligence.
Bachlechner etal. (2021)Thomas Bachlechner, BodhisattwaPrasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. 2021.Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence.
Breiman (1996)Leo Breiman. 1996.Bagging predictors.Machine Learning (1996).
Chen etal. (2023)Jintai Chen, Kuanlun Liao, Yanwen Fang, DannyZ Chen, and Jian Wu. 2023.TabCaps: A Capsule Neural Network for Tabular Data Classification with BoW Routing. In International Conference on Learning Representations.
Chen etal. (2022)Jintai Chen, Kuanlun Liao, Yao Wan, DannyZ Chen, and Jian Wu. 2022.DANets: Deep abstract networks for tabular data classification and regression. In The AAAI Conference on Artificial Intelligence.
Chen and Guestrin (2016)Tianqi Chen and Carlos Guestrin. 2016.XGBoost: A scalable tree boosting system. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Cheng etal. (2016)Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, etal. 2016.Wide & deep learning for recommender systems. In Workshop on Deep Learning for Recommender Systems.
Glorot and Bengio (2010)Xavier Glorot and Yoshua Bengio. 2010.Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics.
Goodfellow etal. (2016)Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016.Deep Learning.MIT Press.
Gorishniy etal. (2022)Yura Gorishniy, Ivan Rubachev, and Artem Babenko. 2022.On Embeddings for Numerical Features in Tabular Deep Learning. In Advances in Neural Information Processing Systems.
Gorishniy etal. (2021)Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021.Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems.
Grinsztajn etal. (2022)Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. 2022.Why do tree-based models still outperform deep learning on typical tabular data?. In Advances in Neural Information Processing Systems.
Guo etal. (2021)Huifeng Guo, Bo Chen, Ruiming Tang, Weinan Zhang, Zhenguo Li, and Xiuqiang He. 2021.An embedding learning framework for numerical features in CTR prediction. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
Guo etal. (2017)Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.DeepFM: A factorization-machine based neural network for CTR prediction. In International Joint Conference on Artificial Intelligence.
He etal. (2015)Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015.Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In International Conference on Computer Vision.
Hollmann etal. (2022)Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. 2022.TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. In International Conference on Learning Representations.
Jeffares etal. (2023)Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela vander Schaar. 2023.TANGOS: Regularizing tabular neural networks through gradient orthogonalization and specialization.arXiv preprint arXiv:2303.05506 (2023).
Kadra etal. (2021)Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. 2021.Well tuned simple nets excel on tabular datasets.Advances in Neural Information Processing Systems (2021).
Katzir etal. (2020)Liran Katzir, Gal Elidan, and Ran El-Yaniv. 2020.Net-DNF: Effective deep modeling of tabular data. In International Conference on Learning Representations.
Khan etal. (2022)Salman Khan, Muzammal Naseer, Munawar Hayat, SyedWaqas Zamir, FahadShahbaz Khan, and Mubarak Shah. 2022.Transformers in vision: A survey.Comput. Surveys (2022).
Kim etal. (2020b)Alisa Kim, Y Yang, Stefan Lessmann, Tiejun Ma, M-C Sung, and JohnnieEV Johnson. 2020b.Can deep learning predict risky retail investors? A case study in financial risk behavior forecasting.European Journal of Operational Research (2020).
Kim etal. (2020a)Jang-Hyun Kim, Wonho Choo, and HyunOh Song. 2020a.Puzzle Mix: Exploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning.
LeCun etal. (2002)Yann LeCun, Léon Bottou, GenevieveB Orr, and Klaus-Robert Müller. 2002.Efficient backprop.Neural networks: Tricks of the Trade (2002).
Loshchilov and Hutter (2018)Ilya Loshchilov and Frank Hutter. 2018.Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
Ng (2004)AndrewY Ng. 2004.Feature selection, L1 vs. L2 regularization, and rotational invariance. In International Conference on Machine Learning.
Popov etal. (2019)Sergei Popov, Stanislav Morozov, and Artem Babenko. 2019.Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data. In International Conference on Learning Representations.
Prokhorenkova etal. (2018)Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, AnnaVeronika Dorogush, and Andrey Gulin. 2018.CatBoost: Unbiased boosting with categorical features.Advances in Neural Information Processing Systems (2018).
Qin etal. (2020)Jie Qin, Jiemin Fang, Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang. 2020.ResizeMix: Mixing data with preserved object information and true labels.arXiv preprint arXiv:2012.11101 (2020).
Rubachev etal. (2022)Ivan Rubachev, Artem Alekberov, Yury Gorishniy, and Artem Babenko. 2022.Revisiting pretraining objectives for tabular deep learning.arXiv preprint arXiv:2207.03208 (2022).
Saxe etal. (2014)AndrewM Saxe, JamesL McClelland, and Surya Ganguli. 2014.Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations.
Shazeer (2020)Noam Shazeer. 2020.GLU variants improve Transformer.arXiv preprint arXiv:2002.05202 (2020).
Somepalli etal. (2021)Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, CBayan Bruss, and Tom Goldstein. 2021.SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training.arXiv preprint arXiv:2106.01342 (2021).
Song etal. (2019)Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019.AutoInt: Automatic feature interaction learning via self-attentive neural networks. In ACM International Conference on Information and Knowledge Management.
Tajbakhsh etal. (2020)Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, JeffreyN Chiang, Zhihao Wu, and Xiaowei Ding. 2020.Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation.Medical Image Analysis (2020).
Touvron etal. (2021a)Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021a.Training data-efficient image Transformers & distillation through attention. In International Conference on Machine Learning.
Touvron etal. (2021b)Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. 2021b.Going deeper with image Transformers. In IEEE/CVF International Conference on Computer Vision.
Uddin etal. (2020)AFMShahab Uddin, MstSirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. 2020.SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization. In International Conference on Learning Representations.
Vaswani etal. (2017)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, AidanN Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.Attention is all you need.Advances in Neural Information Processing Systems (2017).
Verma etal. (2019)Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019.Manifold Mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning.
Walawalkar etal. (2020)Devesh Walawalkar, Zhiqiang Shen, Zechun Liu, and Marios Savvides. 2020.Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification. In International Conference on Acoustics, Speech and Signal Processing.
Wang etal. (2017)Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017.Deep & cross network for ad click predictions. In ADKDD.
Wang etal. (2021a)Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021a.DCN v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In The ACM Web Conference.
Wang and Sun (2022)Zifeng Wang and Jimeng Sun. 2022.TransTab: Learning Transferable Tabular Transformers Across Tables. In Advances in Neural Information Processing Systems.
Wang etal. (2021b)Zhuo Wang, Wei Zhang, Ning Liu, and Jianyong Wang. 2021b.Scalable rule-based representation learning for interpretable classification.Advances in Neural Information Processing Systems (2021).
Wang etal. (2020)Zhuo Wang, Wei Zhang, LIU Ning, and Jianyong Wang. 2020.Transparent classification with multilayer logical perceptrons and random binarization. In The AAAI Conference on Artificial Intelligence.
Wistuba etal. (2015)Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme. 2015.Learning hyperparameter optimization initializations. In IEEE international conference on data science and advanced analytics.
Wu etal. (2022)Kevin Wu, Eric Wu, Michael DAndrea, Nandini Chitale, Melody Lim, Marek Dabrowski, Klaudia Kantor, Hanoor Rangi, Ruishan Liu, Marius Garmhausen, etal. 2022.Machine learning prediction of clinical trial operational efficiency.The AAPS Journal (2022).
Yan etal. (2023)Jiahuan Yan, Jintai Chen, Yixuan Wu, DannyZ Chen, and Jian Wu. 2023.T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction.The AAAI Conference on Artificial Intelligence (2023).
Yan etal. (2024)Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. 2024.Making Pre-trained Language Models Great on Tabular Prediction. In ICLR.
Yoon etal. (2020)Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela vander Schaar. 2020.VIME: Extending the success of self-and semi-supervised learning to tabular domain.Advances in Neural Information Processing Systems (2020).
Yun etal. (2019)Sangdoo Yun, Dongyoon Han, SeongJoon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019.CutMix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision.
Zhang etal. (2018)Hongyi Zhang, Moustapha Cisse, YannN Dauphin, and David Lopez-Paz. 2018.Mixup: Beyond Empirical Risk Minimization. In International Conference On Learning Representations.
Zhu etal. (2023)Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. 2023.XTab: Cross-table Pretraining for Tabular Transformers. In ICML.

Appendix A Why is Feat-Mix superior to CutMix?

The primary distinction between the Feat-Mix and CutMix approaches(Yun etal., 2019) lies in whether the feature importance is considered when synthesizing new samples. To explore this difference, we conducted experiments on several datasets using the architecture of the ExcelFormer as backbone. Our observations were made on both the original tables and the tables augmented with additional columns containing Gaussian noise. See Table6, generally, Feat-Mix outperforms CutMix or performs on par with CutMix on these datasets. However, in tables with noisy columns, we only observed a slight decline in the effectiveness of Feat-Mix (even with an improvement on the cpu dataset), while CutMix exhibited a more significant performance drop under the influence of noisy columns. Given the prevalence of uninformative features in tabular data(Ng, 2004; Grinsztajn etal., 2022), the comparison of their performance and performance drops with noisy data emphasizes the importance of considering feature importance during interpolation. We find that Feat-Mix stands out as a more resilient choice for tabular datasets. By leveraging Feat-Mix instead of CutMix, ExcelFormer can be deemed a more dependable approach for casual users, aligning with our initial intent in this research.

	Breast	Diabetes	Campus	cpu	fruitfly	yacht
CutMix	0.702	0.822	0.972	-102.06	-16.19	-3.59
Feat-Mix	0.713	0.837	0.980	-79.10	-15.86	-0.83
CutMix (with noise)	0.688	0.809	0.938	-115.10	-17.09	-4.40
Feat-Mix (with noise)	0.700	0.834	0.969	-74.56	-16.60	-0.89
$\Delta$ CutMix ( $\downarrow$ )	0.014	0.013	0.034	13.04	0.90	0.81
$\Delta$ Feat-Mix ( $\downarrow$ )	0.013	0.003	0.011	-4.54	0.74	0.06

Appendix B Rotational Variance Property Evaluation

In (P1) of main paper, we highlight the DNNs’ lack of rotational variance, which makes them less efficient for tabular prediction tasks. This limitation serves as the motivation behind our proposal of the semi-permeable attention (SPA) and the interaction attenuated initialization (IAI) approach.Here we would like to inspect if the ExcelFormer is more non-rotationally invariant with the proposed SPA and IAI.We assess the test performance of ExcelFormer (without using data augmentation) when randomly rotating the datasets. We utilize all binary classification datasets consisting of numerical features and containing fewer than 300 data samples. Additionally, we introduce $f$ uninformative features into each dataset (assuming that the original table comprises $f$ features), which are generated using Gaussian noises. As depicted in Fig.5 (a), it is evident that after randomly rotating the datasets, XGBoost and CatBoost exhibit the most significant decline in performances. This observation suggests that they are algorithms with a higher degree of non-rotational invariance, aligning with the findings of (Ng, 2004). While the decline in performance of ExcelFormer and FTT are not as substantial as those of decision tree-based models, it is still noticeable that ExcelFormer’s performance decreases by a larger extent after random rotations, compared to FTT. This observation indicates that our ExcelFormer exhibits a higher degree of non-rotational variance compared to counterpart FTT.

Inversely, we further conducted additive studies, utilizing FTT as the backbone and incorporating our proposed SPA and IAI on FTT. See Fig.5(b), we find that: (i) both SPA and IAI contribute positively to the performances of FTT. (ii) In the presence of random dataset rotations, FTT with IAI and SPA demonstrated a more pronounced performance drop, thereby showcasing the efficacy of SPA and IAI in enhancing the non-rotational invariance property of FTT. Additionally, see Fig.5(c), ablation studies on the ExcelFormer backbone (where neither Feat-Mix nor Hid-Mix was applied) also highlighted the value of SPA and IAI in mitigating the rotational invariance property of DNN models.

ExcelFormer: Can a DNN be a Sure Bet for Tabular Prediction? (5)

Model	eye	california	house	jannis	higgs-small	average time (s)
XGboost (t)	699.92	325.48	238.95	2260.57	425.67	790.12
Catboost (t)	1241.43	589.10	387.77	4100.39	714.53	1406.64
ExcelFormer (d)	39.29	51.80	59.41	166.57	160.48	95.51

Hyper-parameter	Distribution
#. Layers $L$	UniformInt[2, 5]
Representation size $d$	{64, 128, 256}
#. Heads	{4, 8, 16, 32}
Residual dropout rate	Uniform[0, 0.5]
Learning rate	LogUniform[ $3\times 10^{-5}$ , $10^{-3}$ ]
Weight decay	{0.0, LogUniform[ $10^{-6}$ , $10^{-3}$ ]}
(*) Mixup type	{Feat-Mix, Hid-Mix, neither}
(*) $\alpha$ of $\mathcal{B}eta$ distribution	Uniform[0.1, 3.0]

Hyper-parameter	Distribution
Booster	“gbtree”
N-estimators	Const(4096)
Early-stopping-rounds	Const(50)
Max depth	UniformInt[3, 10]
Min child weight	LogUniform[ $10^{-8}$ , $10^{5}$ ]
Subsample	Uniform[0.5, 1.0]
Learning rate	LogUniform[ $10^{-5},1$ ]
Col sample by level	Uniform[0.5, 1]
Col sample by tree	Uniform[0.5, 1]
Gamma	{0, LogUniform[ $10^{-8},10^{2}$ ]}
Lambda	{0, LogUniform[ $10^{-8},10^{2}$ ]}
Alpha	{0, LogUniform[ $10^{-8},10^{2}$ ]}
#. Tuning iterations	500

Hyper-parameter	Distribution
Iterations (number of trees)	Const(4096)
Od pval	Const(0.001)
Early-stopping-rounds	Const(50)
Max depth	UniformInt[3, 10]
Learning rate	LogUniform[ $10^{-5},1$ ]
Bagging temperature	Uniform[0, 1]
L2 leaf reg	LogUniform[1, 10]
Leaf estimation iterations	UniformInt[1, 10]
#. Tuning iterations	500

Appendix C Details of Hyper-Parameter Fine-Tuning Settings

For XGboost and Catboost, we follow the implementations and settings in(Gorishniy etal., 2021), while increasing the number of estimators/iterations(i.e., decision trees) and the number of tuning iterations, so as to attain better performance. For our ExcelFormer, we apply the Optuna based tuning(Akiba etal., 2019). The hyper-parameter search spaces of ExcelFormer, XGboost, and Catboost are reported in Table8, Table9, and Table10, respectively. For ExcelFormer, we tune just 50 iterations on the configurations with regard to the data augmentation (it is marked as “Mix Tuned”). For “Fully Tuned” version, we finely tune 50 interations on all the hyper-parameters.

Appendix D Development Time Cost

In the main paper, we have demonstrated on a large array of datasets that, with default hyperparameters, our proposed model outperforms existing models that require time-consuming heavy hyperparameter tuning (typically requiring 50-100 iterations). Among these, XGBoost and CatBoost stand out as two of the most efficient approaches. It is apparent in comparison to FTT that our model demands only approximately 1/100 to 1/50 of the development time. This is due to its close architecture of ExcelFormer to FTT while being competitive or even superior without the need for hyperparameter tuning.

We conducted an analysis of the total time invested in model development for XGBoost, CatBoost, and our proposed ExcelFormer. As shown in Table7, we observed that our model can achieve significantly higher efficiency than both, even with just 50 iterations of hyperparameter tuning applied to XGBoost and CatBoost. Developing an XGBoost or CatBoost model requires 8-15 times more development time than our model, indicating that our approach is user-friendly and environmentally friendly.

Appendix E Implementation Details of Metrics Used in this Work

Feature Importance.

In this study, we employ Normalized Mutual Information (NMI) to assess the importance of various features, as mutual information can capture dependencies between features and targets. We implement NMI using the sklearn Python package. Specifically, for classification tasks, we utilize the ”feature_selection.mutual_info_classif” function, and for regression tasks, we utilize the ”feature_selection.mutual_info_regression” function.

Average Normalized AUC across Datasets.

To aggregate the model performances across datasets, we calculate the average normalized scores(Wistuba etal., 2015) for AUC to comprehensively evaluate the model performances. Specifically, we first normalize the scores among the compared models for given datasets, and then average them across datasets. Formally, among $D$ involved datasets, the average normalized score $s_{m}$ for the model $m$ is computed by:

(9)

s^{\prime}_{m,d}=\frac{s_{m,d}-\text{min}_{i\in M_{0}}(s_{i,d})}{\text{max}_{i%\in M_{0}}(s_{i,d})-\text{min}_{i\in M_{0}}(s_{i,d})},s_{m}=\frac{\sum^{D}_{d=%1}s^{\prime}_{m,d}}{D}

where $M_{0}$ encompasses all the models compared. The $s_{m}$ denotes $\text{AUC}_{m}$ . We only use binary classification datasets in Fig.5 since the average normalized AUC, ACC, and nRMSE should be separately computed.

Performance Rank.

We performed 5 runs with different random seeds and calculated the average results for each dataset. Additionally, we computed the overall rank across datasets for comparison. Average rank is given to tied values.

Appendix F Detailed Description of Datasets

The details of the 96 used small-scale tabular datasets are summarized in Table11 and Table12. The details of the 21 large-scale datasets are summarized in Table13. We use the same train-valid-test split for all the approaches.

Dataset	#. Sample	#. Feature	#. Num	#.Cat	Task Type
Analytics Vidhya Loan Prediction	614	11	5	6	classification
Audit Data	776	24	21	3	classification
Automobiles	201	25	13	12	classification
Bigg Boss India	567	21	6	15	classification
Breast Cancer Dataset	569	30	30	0	classification
Campus Recruitment	215	13	6	7	classification
chronic kidney disease	400	13	9	4	classification
House Price	506	17	14	3	classification
Compositions of Glass	214	9	9	0	classification
Credit Card Approval	590	15	6	9	classification
Customer Classification	1000	11	5	6	classification
Development Index	225	6	6	0	classification
fitbit dataset	457	13	12	1	classification
Horse Colic Dataset	299	27	9	18	classification
Penguins Classified	344	6	4	2	classification
Pima-Indians_Diabetes	768	8	8	0	classification
Real Estate DataSet	511	13	11	2	classification
Startup Success Prediction	923	45	9	36	classification
Store Data Performance	135	16	7	9	classification
The Estonia Disaster Passenger List	989	6	1	5	classification
AAPL_stock_price_2021_2022	346	5	5	0	regression
AAPL_stock_price_2021_2022_1	347	5	5	0	regression
AAPL_stock_price_2021_2022_2	348	5	5	0	regression
analcatdata_creditscore	100	6	3	3	classification
analcatdata_homerun	162	26	12	14	regression
analcatdata_lawsuit	264	4	3	1	classification
analcatdata_vineyard	468	3	1	2	regression
auto_price	159	15	13	2	regression
autoPrice	159	15	14	1	regression
bodyfat	252	14	14	0	regression
boston	506	13	11	2	regression
boston_corrected	506	19	15	4	regression
Boston-house-price-data	506	13	11	2	regression
cholesterol	303	13	7	6	regression
cleveland	303	13	7	6	regression
cloud	108	5	3	2	regression
cps_85_wages	534	10	3	7	regression
cpu	209	7	5	2	regression
DEE	365	6	6	0	regression
Diabetes-Data-Set	768	8	8	0	classification
DiabeticMellitus	281	97	6	91	classification
disclosure_x_bias	662	3	3	0	regression
disclosure_x_noise	662	3	3	0	regression
disclosure_x_tampered	662	3	3	0	regression
disclosure_z	662	3	3	0	regression
echom*onths	130	9	7	2	regression
EgyptianSkulls	150	4	3	1	regression
ELE-1	495	2	2	0	regression
fishcatch	158	7	5	2	regression
Fish-market	159	6	5	1	regression

Dataset	#. Sample	#. Feature	#. Num	#.Cat	Task Type
forest_fires	517	12	8	4	regression
Forest-Fire-Area	517	12	8	4	regression
fruitfly	125	4	2	2	regression
HappinessRank_2015	158	9	8	1	regression
Heart_disease_classification	296	13	7	6	classification
hungarian	294	13	11	2	classification
Indian-Liver-Patient-Patient	583	11	9	2	classification
Intersectional-Bias-Assessment	1000	18	14	4	classification
liver-disorders	345	5	5	0	regression
lowbwt	189	9	2	7	regression
lungcancer_shedden	442	23	20	3	regression
machine_cpu	209	6	6	0	regression
meta	528	21	16	5	regression
nki70.arff	144	76	72	4	classification
no2	500	7	7	0	regression
pharynx	195	10	3	7	regression
Pima-Indians-Diabetes	768	8	8	0	classification
pm10	500	7	7	0	regression
Pokmon-Legendary-Data	801	12	9	3	classification
Reading_Hydro	1000	26	11	15	regression
residential_building	372	108	100	8	regression
rmftsa_ladata	508	10	10	0	regression
strikes	625	6	6	0	regression
student-grade-pass-or-fail-prediction	395	29	4	25	classification
Swiss-banknote-conterfeit-detection	200	6	6	0	classification
The-Estonia-Disaster-Passenger-List	989	6	1	5	classification
The-Office-Dataset	188	10	2	8	regression
tokyo1	959	44	42	2	classification
visualizing_environmental	111	3	3	0	regression
weather_ankara	321	9	9	0	regression
wisconsin	194	32	32	0	regression
yacht_hydrodynamics	308	6	6	0	regression
Absenteeism at work	740	20	7	13	classification
Audit Data	776	24	21	3	classification
Breast Cancer Coimbra	116	9	9	0	classification
Cervical cancer (Risk Factors)	858	30	25	5	classification
Climate Model Simulation Crashes	540	19	18	1	classification
Early stage diabetes risk prediction	520	16	1	15	classification
extention of Z-Alizadeh sani dataset	303	57	20	37	classification
HCV data	615	12	11	1	classification
Heart failure clinical records	299	12	7	5	classification
Parkinson Dataset	240	46	44	2	classification
QSAR Bioconcentration classes	779	11	7	4	classification
Quality Assessment of DC	97	62	62	0	classification
User Knowledge Modeling	258	5	5	0	classification
Z-Alizadeh Sani	303	54	20	34	classification

Dataset	Abbr.	Task Type	#. Features	#. Num	#. Cat	#. Sample	Link
sulfur	SU	regression	6	6	0	10,081	https://www.openml.org/d/44145
bank-marketing	BA	classification	7	7	0	10,578	https://www.openml.org/d/44126
Brazilian_houses	BR	regression	8	8	0	10,692	https://www.openml.org/d/44141
eye	EY	multiclass	26	26	0	10,936	http://www.cis.hut.fi/eyechallenge2005
MagicTelescope	MA	classification	10	10	0	13,376	https://www.openml.org/d/44125
Ailerons	AI	regression	33	33	0	13,750	https://www.openml.org/d/44137
pol	PO	regression	26	26	0	15,000	https://www.openml.org/d/722
binarized-pol	BP	classification	48	48	0	15,000	https://www.openml.org/d/722
credit	CR	classification	10	10	0	16,714	https://www.openml.org/d/44089
california	CA	regression	8	8	0	20,640	https://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.html
house_sales	HS	regression	15	15	0	21,613	https://www.openml.org/d/44144
house	HO	regression	16	16	0	22,784	https://www.openml.org/d/574
diamonds	DI	regression	6	6	0	53,940	https://www.openml.org/d/44140
helena	HE	multiclass	27	27	0	65,196	https://www.openml.org/d/41169
jannis	JA	multiclass	54	54	0	83,733	https://www.openml.org/d/41168
higgs-small	HI	classification	28	28	0	98,049	https://www.openml.org/d/23512
road-safety	RO	classification	32	29	3	111,762	https://www.openml.org/d/44161
medicalcharges	ME	regression	3	3	0	163,065	https://www.openml.org/d/44146
SGEMM_GPU_kernel_performance	SG	regression	9	3	6	241,600	https://www.openml.org/d/44069
covtype	CO	multiclass	54	54	0	581,012	https://www.openml.org/d/1596
nyc-taxi-green-dec-2016	NY	regression	9	9	0	581,835	https://www.openml.org/d/44143

Appendix G Detailed Results on Small and Large Datasets

We present the average results (five runs averaged) of all the models for each dataset. The results for the 96 small-scale datasets can be found in Table14, and the performance on the 21 large-scale datasets is provided in Table15.

Datasets	O.+F (d)	O.+H (d)	O. (MT)	O.(FT)	FTT (t)	XGb (t)	Cat (t)	MLP (t)	DCNv2 (t)	AutoInt(t)	SAINT (t)	TT (d)	XT(d)	TapP(t)
Analytics Vidhya Loan Prediction	0.7449	0.7421	0.7421	0.7421	0.7285	0.7486	0.7045	0.7359	0.7464	0.7350	0.7505	0.7291	0.7240	0.7331
Audit Data	0.9984	0.9905	0.9991	0.9941	0.9993	0.9995	1.0000	0.9936	0.9990	0.9983	0.9995	0.9983	0.9822	0.9998
Automobiles	0.9774	0.9798	0.9869	0.9750	0.9583	0.9679	0.9726	0.9512	0.9226	0.9726	0.9536	0.9679	0.9545	0.9845
Bigg Boss India	1.0000	1.0000	0.9861	0.9861	0.9799	1.0000	0.9861	0.9954	0.9985	0.9059	0.9753	1.0000	0.9769	1.0000
Breast Cancer Dataset	0.9970	0.9937	0.9921	0.9944	0.9841	0.9914	0.9851	0.9917	0.9828	0.9795	0.9828	0.9970	0.9927	0.9916
Campus Recruitment	0.9795	0.9846	0.9795	0.9487	0.9487	0.9795	0.9667	0.9256	0.9103	0.9744	0.9590	0.9436	0.9232	0.9821
chronic kidney disease	0.9993	1.0000	0.9967	0.9960	0.9960	0.9900	0.9940	0.9907	0.9767	0.9893	0.9853	0.9947	0.9973	0.9993
House Price	0.8904	0.9015	0.9015	0.8983	0.9011	0.8818	0.8977	0.9026	0.8924	0.9013	0.9054	0.8971	0.8863	0.9007
Compositions of Glass	0.8976	0.8595	0.8976	0.9238	0.8595	0.8655	0.9024	0.8167	0.9000	0.6929	0.7929	0.7929	0.7976	0.8905
Credit Card Approval	0.9583	0.9719	0.9680	0.9701	0.9607	0.9595	0.9680	0.9447	0.9478	0.9550	0.9550	0.9662	0.9376	0.9553
Customer Classification	0.5792	0.5719	0.5280	0.6014	0.5951	0.4727	0.5802	0.5376	0.6417	0.5952	0.6348	0.6125	0.6134	0.6229
Development Index	1.0000	0.9671	1.0000	1.0000	0.9815	0.9259	0.9805	0.9588	0.9362	0.9633	1.0000	0.9856	0.9210	0.9856
fitbit dataset	0.7991	0.8216	0.8092	0.8025	0.8154	0.8138	0.7975	0.8073	0.8101	0.7628	0.7939	0.8164	0.7847	0.8914
Horse Colic Dataset	0.7523	0.7477	0.7373	0.7407	0.6921	0.6921	0.7269	0.7025	0.6169	0.6038	0.7083	0.7199	0.6905	0.7346
Penguins Classified	0.9991	0.9991	0.9991	1.0000	1.0000	0.9966	0.9944	0.9983	0.9983	0.9954	1.0000	0.9940	0.9872	0.9983
Pima-Indians_Diabetes	0.8367	0.8330	0.8154	0.8148	0.8048	0.7887	0.7667	0.8222	0.8069	0.7987	0.8096	0.8135	0.7546	0.8181
Real Estate DataSet	0.9176	0.8894	0.9041	0.9176	0.9045	0.9010	0.9167	0.8621	0.8760	0.9014	0.8945	0.8883	0.8471	0.9029
Startup Success Prediction	0.9938	0.9914	0.8397	0.8445	0.8464	0.9937	0.7277	0.8373	0.7701	0.8456	0.8363	0.8204	0.8371	0.8428
Store Data Performance	0.7632	0.7632	0.6908	0.7566	0.6382	0.6316	0.7829	0.6447	0.6711	0.7627	0.6184	0.6645	0.6513	0.8487
The Estonia Disaster Passenger List	0.7572	0.7559	0.7708	0.7713	0.7546	0.7499	0.7379	0.7464	0.7401	0.7379	0.7436	0.7505	0.7248	0.7518
analcatdata_creditscore	1.0000	1.0000	1.0000	1.0000	1.0000	0.9400	0.9667	1.0000	0.9200	0.9667	0.8533	1.0000	0.8467	1.0000
analcatdata_lawsuit	1.0000	1.0000	1.0000	1.0000	1.0000	0.9949	1.0000	1.0000	0.9898	0.9541	0.9847	0.9796	0.9395	1.0000
Diabetes-Data-Set	0.8356	0.8337	0.8248	0.8294	0.8241	0.7852	0.7754	0.8215	0.7954	0.8311	0.8257	0.8120	0.7903	0.8152
DiabeticMellitus	0.9878	0.9865	0.9865	0.9865	0.9784	0.9743	0.9493	0.7405	0.8242	0.8932	0.9239	0.9041	0.7346	0.9405
Heart_disease_classification	0.8984	0.9096	0.9152	0.9241	0.9342	0.8990	0.9018	0.8984	0.9342	0.9241	0.9174	0.9107	0.9152	0.9252
hungarian	0.8446	0.8596	0.8596	0.8484	0.8822	0.8578	0.8910	0.9273	0.9060	0.8534	0.9261	0.8985	0.9273	0.8722
Indian-Liver-Patient-Patient	0.7133	0.6984	0.7197	0.7232	0.7551	0.7399	0.7206	0.7116	0.7310	0.7133	0.7254	0.7332	0.6935	0.7381
Intersectional-Bias-Assessment	0.9953	0.9958	0.9960	0.9979	0.9916	0.9925	0.9977	0.9956	0.9944	0.9962	0.9982	0.9942	0.9736	0.9965
nki70.arff	0.8158	0.9263	0.8867	0.8737	0.6263	0.8526	0.8526	0.6842	0.6684	0.8105	0.8353	0.8263	0.8263	0.9216
Pima-Indians-Diabetes	0.8356	0.8337	0.8109	0.8052	0.8494	0.8140	0.7528	0.7974	0.7931	0.7638	0.7806	0.8141	0.7903	0.8152
Pokmon-Legendary-Data	0.9767	0.9888	0.9981	0.9869	0.9679	0.9504	0.9840	0.9203	0.9412	0.9917	0.9772	0.9810	0.9679	0.9801
student-grade-pass-or-fail-prediction	0.9898	0.9884	1.0000	1.0000	0.9710	1.0000	1.0000	0.9771	0.8211	0.9993	0.9608	0.9898	0.8494	0.9601
Swiss-banknote-conterfeit-detection	0.9925	0.9950	1.0000	1.0000	0.9975	1.0000	1.0000	1.0000	0.9975	1.0000	1.0000	1.0000	0.9450	1.0000
The-Estonia-Disaster-Passenger-List	0.7572	0.7559	0.7555	0.7585	0.7719	0.7723	0.7366	0.7485	0.7530	0.7390	0.7433	0.7464	0.6948	0.7518
tokyo1	0.9715	0.9740	0.9696	0.9702	0.9692	0.9706	0.9536	0.9716	0.9727	0.9665	0.9653	0.9595	0.9596	0.9705
Absenteeism at work	0.8579	0.8665	0.8669	0.8921	0.8281	0.8402	0.8145	0.8278	0.7973	0.8299	0.8119	0.8352	0.7713	0.8339
Audit Data	0.9984	0.9905	0.9995	0.9995	1.0000	1.0000	1.0000	0.9934	0.9967	0.9991	0.9995	0.9967	0.9822	0.9998
Breast Cancer Coimbra	0.7133	0.7622	0.9276	0.7483	0.7483	0.7762	0.7692	0.6993	0.7168	0.6503	0.6434	0.6923	0.6224	0.5923
Cervical cancer (Risk Factors)	0.7137	0.6431	0.6431	0.5415	0.6194	0.6039	0.7165	0.5680	0.5867	0.5720	0.4839	0.5268	0.5353	0.9798
Climate Model Simulation Crashes	0.9574	0.9776	0.9742	0.9776	0.9192	0.9529	0.9832	0.9439	0.9439	0.8563	0.9733	0.7957	0.7677	0.9973
Early stage diabetes risk prediction	0.9906	0.9684	0.9746	0.9820	0.9793	0.9969	0.9906	0.9957	0.9973	0.9984	0.9980	0.9922	0.9109	0.9629
extention of Z-Alizadeh sani dataset	0.9638	0.9651	0.9651	0.9121	0.9638	0.9606	0.9509	0.9522	0.9651	0.9457	0.9574	0.9134	0.8416	1.0000
HCV data	0.9982	0.9942	0.9965	0.9988	0.9982	0.9924	0.9930	0.9714	1.0000	0.9982	1.0000	0.9977	0.9523	0.8511
Heart failure clinical records	0.8652	0.8883	0.8806	0.8922	0.9127	0.8633	0.8177	0.8370	0.8395	0.8691	0.8203	0.8588	0.7918	0.9214
Parkinson Dataset	0.9167	0.9253	0.9392	0.9253	0.9306	0.8559	0.9071	0.9201	0.9253	0.9253	0.9201	0.9132	0.8941	0.9121
QSAR Bioconcentration classes	0.7721	0.8331	0.8314	0.8454	0.8796	0.8308	0.8363	0.8419	0.8640	0.8242	0.8162	0.8293	0.8079	0.8613
Quality Assessment of DC	0.5490	0.5294	0.5294	0.8235	0.3922	0.6078	0.9412	0.7451	0.7451	0.2941	0.1961	0.4314	0.3922	0.3137
User Knowledge Modeling	0.9771	0.9771	0.9771	0.9559	0.9902	0.9330	0.9673	0.9641	0.9559	0.9673	0.9739	0.8713	0.8547	0.9886
Z-Alizadeh Sani	0.8385	0.8863	0.8863	0.8618	0.8773	0.8424	0.8863	0.8618	0.8760	0.8450	0.8734	0.8592	0.8041	0.8527
AAPL_stock_price_2021_2022	-2.4201	-0.8485	-1.1187	-0.6742	-1.0613	-0.4714	-1.2488	-0.3812	-2.9911	-1.1469	-1.2134	-3.7808	-3.2483	n/a
AAPL_stock_price_2021_2022_1	-1.3351	-0.7599	-0.3584	-0.3108	-0.6781	-0.7711	-1.4369	-0.7036	-1.2721	-1.0302	-2.4377	-2.7802	-2.5602	n/a
AAPL_stock_price_2021_2022_2	-1.4472	-0.5832	-0.3367	-0.3141	-0.4005	-0.7059	-0.9954	-0.2768	-1.2930	-0.9359	-2.2318	-2.8244	-2.4182	n/a
analcatdata_homerun	-0.7584	-0.9188	-0.7432	-0.7514	-0.7456	-0.8075	-0.7366	-0.7425	-0.7731	-0.7706	-0.7452	-0.7574	-0.8417	n/a
analcatdata_vineyard	-2.9582	-2.7122	-3.0034	-2.6116	-2.4820	-2.1594	-2.3821	-2.4549	-2.4602	-2.4509	-2.4403	-3.5946	-3.7126	n/a
auto_price	-1751.0	-2103.5	-1830.7	-2463.1	-2244.8	-1720.8	-1935.9	-2702.4	-2503.7	-2020.4	-2905.1	-3100.4	-3031.4	n/a
autoPrice	-1751.0	-2103.5	-1676.9	-1831.3	-2341.8	-1659.6	-1977.4	-2623.2	-4026.9	-1840.2	-3104.0	-3093.8	-2969.5	n/a
bodyfat	-1.0621	-0.7597	-0.8297	-0.5658	-0.5431	-0.8420	-1.1247	-1.6816	-3.9424	-1.7095	-1.4271	-4.0332	-4.3167	n/a
boston	-2.7724	-2.9132	-3.0481	-3.2503	-4.2412	-3.0366	-3.5042	-3.6890	-4.0360	-4.2976	-4.5132	-4.9288	-4.6731	n/a
boston_corrected	-3.0906	-3.6336	-3.3379	-3.3726	-3.3748	-3.2464	-3.6352	-3.4099	-3.4520	-3.2328	-3.8338	-5.5182	-5.7764	n/a
Boston-house-price-data	-3.1074	-2.9132	-3.0481	-3.9133	-3.4856	-3.1320	-3.5397	-4.4732	-5.6720	-4.1353	-3.9253	-4.7881	-4.6171	n/a
cholesterol	-63.898	-63.607	-62.204	-61.527	-61.702	-60.718	-61.791	-62.238	-61.145	-62.760	-62.621	-61.434	-64.213	n/a
cleveland	-0.8839	-0.8765	-0.8686	-0.8853	-0.9944	-0.8863	-0.8918	-0.9546	-0.9936	-0.9455	-0.8704	-1.1198	-1.2134	n/a
cloud	-0.5701	-0.6858	-0.4539	-0.6851	-0.4608	-0.2720	-0.3458	-0.6079	-0.6326	-0.8258	-0.7637	-0.9437	-1.0297	n/a
cps_85_wages	-4.3197	-4.4261	-4.2873	-4.3968	-4.2237	-4.6278	-4.6009	-4.4570	-4.4097	-4.2979	-4.4403	-4.7683	-4.8651	n/a
cpu	-79.104	-76.943	-76.402	-91.466	-95.975	-62.504	-104.760	-74.299	-68.783	-122.468	-123.213	-137.357	-131.2979	n/a
DEE	-0.4023	-0.4255	-0.4294	-0.4278	-0.3863	-0.4051	-0.4239	-0.3814	-0.8244	-0.4174	-0.4296	-0.6657	-0.4780	n/a
disclosure_x_bias	-21921	-21743	-21919	-21876	-21807	-22587	-21853	-22022	-21878	-21912	-22159	-22481	-23453	n/a
disclosure_x_noise	-26993	-27266	-26843	-26919	-27196	-26943	-27438	-26992	-27944	-27232	-27010	-27412	-27078	n/a
disclosure_x_tampered	-27168	-27275	-27824	-27062	-27245	-27318	-27647	-27180	-27227	-27984	-27114	-27347	-27018	n/a
disclosure_z	-21506	-21374	-21496	-21477	-21791	-21911	-21815	-21753	-30624	-21764	-21804	-22272	-23509	n/a
echom*onths	-8.8668	-9.4200	-10.1428	-9.9251	-11.5086	-12.6059	-10.4651	-11.1439	-12.4521	-10.1922	-9.5287	-13.5465	-14.0546	n/a
EgyptianSkulls	-1425.98	-1393.52	-1403.98	-1403.98	-1360.73	-1460.25	-1487.83	-1519.55	-1243.86	-1480.12	-1603.05	-1575.59	-1669.02	n/a
ELE-1	-736.04	-736.25	-758.72	-749.40	-749.72	-770.96	-782.94	-761.12	-739.36	-779.54	-737.94	-838.68	-816.96	n/a
fishcatch	-50.863	-46.911	-86.628	-76.929	-66.500	-79.260	-102.405	-126.658	-89.525	-155.355	-149.660	-180.285	-205.606	n/a
Fish-market	-72.073	-70.419	-76.484	-80.037	-88.557	-63.291	-70.877	-112.847	-64.847	-128.376	-69.060	-153.163	-172.934	n/a
forest_fires	-109.375	-109.339	-108.763	-107.853	-109.139	-108.803	-107.700	-108.925	-108.707	-108.573	-109.064	-108.921	-108.578	n/a
Forest-Fire-Area	-109.375	-109.339	-108.988	-107.538	-109.026	-108.803	-106.091	-108.945	-109.292	-109.428	-109.349	-109.015	-108.578	n/a
fruitfly	-15.856	-15.752	-15.858	-15.732	-16.438	-20.724	-16.224	-16.251	-18.620	-17.561	-16.023	-15.829	-21.850	n/a
HappinessRank_2015	-0.1402	-0.0753	-0.0640	-0.0753	-0.0800	-0.0856	-0.1765	-0.2066	-0.1220	-0.3450	-0.2596	-0.9244	-1.2549	n/a
liver-disorders	-3.1001	-2.9445	-2.9602	-3.0291	-3.2481	-3.0613	-2.9170	-2.9184	-3.5018	-3.1161	-3.0200	-2.8904	-3.2965	n/a
lowbwt	-419.24	-421.87	-417.13	-419.41	-451.40	-422.76	-443.85	-447.74	-486.07	-406.45	-420.31	-501.78	-580.11	n/a
lungcancer_shedden	-2.8148	-2.5672	-2.6234	-2.6200	-2.7049	-2.7345	-2.5833	-2.5232	-2.5298	-2.8645	-2.5481	-2.8069	-3.4472	n/a
machine_cpu	-71.259	-85.958	-90.238	-82.287	-92.617	-78.152	-107.735	-73.420	-89.281	-125.633	-129.315	-187.953	-177.951	n/a
meta	-153.09	-162.95	-142.67	-128.98	-164.11	-147.92	-236.52	-141.32	-142.79	-237.33	-146.03	-192.56	-273.34	n/a
no2	-0.5015	-0.4864	-0.4972	-0.4967	-0.4948	-0.4912	-0.5082	-0.5289	-0.5212	-0.5127	-0.4985	-0.6629	-0.7692	n/a
pharynx	-286.57	-281.51	-277.71	-273.68	-310.59	-279.05	-277.78	-337.05	-492.43	-270.79	-282.70	-328.23	-391.29	n/a
pm10	-0.7670	-0.7650	-0.7267	-0.8022	-0.8010	-0.7487	-0.7331	-0.8141	-0.9794	-0.7942	-0.8064	-0.8130	-0.9376	n/a
Reading_Hydro	-0.0039	-0.0037	-0.0040	-0.0039	-0.0038	-0.0036	-0.0037	-0.0039	-0.0041	-0.0042	-0.0047	-0.0188	-0.0081	n/a
residential_building	-351.46	-210.01	-168.25	-196.44	-237.14	-200.64	-306.75	-533.02	-723.89	-533.37	-526.71	-584.36	-643.16	n/a
rmftsa_ladata	-2.0287	-1.8550	-1.8238	-2.0475	-2.0305	-2.0150	-1.8144	-2.0999	-2.4473	-2.3571	-2.0216	-2.5843	-2.5447	n/a
strikes	-586.03	-592.41	-587.24	-604.66	-604.16	-592.34	-588.58	-620.24	-660.56	-589.25	-599.57	-615.12	-637.11	n/a
The-Office-Dataset	-0.3876	-0.4189	-0.3654	-0.4127	-0.4152	-0.3350	-0.3730	-0.4256	-0.4068	-0.4148	-0.4843	-0.5396	-0.5697	n/a
visualizing_environmental	-2.5584	-2.8069	-3.0343	-3.4924	-2.8716	-2.3128	-2.8110	-2.5520	-2.9296	-3.1022	-2.7731	-3.4982	-3.8523	n/a
weather_ankara	-1.7222	-1.3999	-1.5609	-1.4824	-1.5756	-1.8900	-1.5884	-2.0655	-2.6293	-1.9707	-2.3743	-3.3254	-2.9359	n/a
wisconsin	-36.915	-37.548	-38.315	-38.603	-36.128	-35.500	-35.720	-34.429	-75.613	-34.541	-37.677	-37.229	-51.180	n/a
yacht_hydrodynamics	-0.8310	-0.9270	-0.7151	-1.0738	-1.0881	-0.7432	-1.0243	-1.2074	-1.2786	-2.3096	-4.4713	-5.1386	-6.5417	n/a

Datasets	XGb (d)	Cat (d)	FTT (d)	Excel + F (d)	Excel + H (d)	XGb (t)	Cat (t)	FTT (t)	Excel (M)	Excel (F)
SU	-0.02025	-0.01994	-0.01825	-0.01840	-0.01740	-0.01770	-0.02200	-0.01920	-0.01730	-0.01610
BA	80.25	89.20	88.26	89.00	88.65	88.97	89.16	88.64	89.21	89.16
BR	-0.07667	-0.07655	-0.07390	-0.11230	-0.06960	-0.07690	-0.09310	-0.07940	-0.06270	-0.06410
EY	69.97	69.85	71.06	71.44	72.09	72.88	72.41	71.73	74.14	78.94
MA	86.21	93.83	93.66	93.38	93.66	93.69	93.66	93.69	94.04	94.11
AI	-0.0001669	-0.0001652	-0.0001637	-0.0001689	-0.0001627	-0.0001605	-0.0001616	-0.0001641	-0.0001615	-0.0001612
PO	-5.342	-6.495	-4.675	-5.694	-2.862	-4.331	-4.622	-2.705	-2.629	-2.636
BP	99.13	99.95	99.13	99.94	99.95	99.96	99.95	99.97	99.93	99.96
CR	76.55	85.15	85.22	85.23	85.22	85.11	85.12	85.19	85.26	85.36
CA	-0.4707	-0.4573	-0.4657	-0.4331	-0.4587	-0.4359	-0.4359	-0.4679	-0.4316	-0.4336
HS	-0.1815	-0.1790	-0.1740	-0.1835	-0.1773	-0.1707	-0.1746	-0.1734	-0.1726	-0.1727
HO	-3.368	-3.258	-3.208	-3.305	-3.147	-3.139	-3.279	-3.142	-3.159	-3.214
DI	-0.2372	-0.2395	-0.2378	-0.2368	-0.2387	-0.2353	-0.2362	-0.2389	-0.2359	-0.2358
HE	35.02	37.77	37.38	37.22	38.20	37.39	37.81	38.86	38.65	38.61
JA	71.62	71.92	72.67	72.51	72.79	72.45	71.97	73.15	73.15	73.55
HI	71.59	80.31	80.65	80.60	80.75	80.28	80.22	80.71	80.88	81.22
RO	80.42	87.98	88.51	88.65	88.15	90.48	89.55	89.29	89.33	89.27
ME	-0.0819	-0.0835	-0.0845	-0.0821	-0.0808	-0.0820	-0.0829	-0.0811	-0.0809	-0.0808
SG	-0.01658	-0.03377	-0.01866	-0.01587	-0.01531	-0.01635	-0.02038	-0.01644	-0.01465	-0.01454
CO	96.42	92.13	96.71	97.38	97.17	96.92	96.25	97.00	97.43	97.43
NY	-0.3805	-0.4459	-0.4135	-0.3887	-0.3930	-0.3683	-0.3808	-0.4248	-0.3710	-0.3625