Data Science and Machine Learning

data-science-ml-ai999x

Evaluation metrics, loss functions and gradients, classical machine-learning algorithms, dimensionality reduction, cross-validation, information theory and neural-network arithmetic.

Data Science & Machine Learning Question Answering Token Classification Text Classification Tabular Classification Tabular Regression Sentence Similarity Feature Extraction

Hugging Face ↗GitHub ↗See it on the capability orb

107capabilities

14topics

107formulas typeset

8worked examples below

1.0.1version

About this model

What Data Science and Machine Learning does

Data Science and Machine Learning (data-science-ml-ai999x) is an offline Python package for data science and machine learning. It covers evaluation metrics, loss functions and gradients, classical machine-learning algorithms, dimensionality reduction, cross-validation, information theory and neural-network arithmetic.

It has 107 capabilities — 107 computational procedures — in 14 topics, the largest being Classification metrics, Classical algorithms, Loss functions and activations, Neural networks, Clustering metrics and Information theory. Ask a question in words with ask(), or call a capability by name with typed inputs with solve(). Every answer carries its value and unit, the steps that produced it and the formula in LaTeX.

When a question is not determined — a value is missing, an input is out of range, or the request is outside the package — it says so and names what it needs, instead of guessing. It runs on an ordinary CPU without a network connection (Python >=3.10), and the same question always gets the same answer.

Install

pip install data-science-ml-ai999x
hf download AI999X/data-science-ml-ai999x --local-dir data-science-ml-ai999x

What it covers

Classical algorithms13Agglomerative hierarchical clustering merge heights · Bernoulli naive Bayes prediction (binary features) · Best decision-tree split (classification, Gini or entropy) …Classification metrics20Average precision and precision-recall AUC · Averaged precision, recall and F1 (multi-class) · Balanced accuracy (multi-class) …Clustering metrics7Adjusted mutual information of two clusterings · Calinski-Harabasz index · Davies-Bouldin index …Dimensionality reduction1Principal component analysis (explained variance and first component)Gradients3Logistic regression loss and gradient (L2-regularised) · Mean squared error loss and gradient of a linear model · One gradient-descent step for linear regression Information theory7Gini impurity decrease of a split · Information gain and gain ratio of a split · Joint, marginal and conditional entropy and mutual information …Linear models6Lasso regression coefficients (stated alpha) · Linear SVM decision values, classes and margin · Logistic regression fit (L2, stated C) …Loss functions and activations13Binary cross-entropy with logits and its gradient · Binary focal loss and its gradient · Cross-entropy loss from logits and its gradient …Model selection5Bias-variance decomposition of predictions · k-fold cross-validated MSE of linear regression · k-fold cross-validation fold sizes …Neural networks132-D convolution of an image with a kernel (deep-learning convention) · Backpropagation gradients of a one-hidden-layer network · Batch normalisation forward pass (training mode) …Optimizers6Adam optimizer: one update · AdamW optimizer (decoupled weight decay): one update · Cosine-annealing learning-rate schedule …Preprocessing4Label encoding and one-hot encoding of categories · Min-max scaling to [0, 1] · Robust scaling (median and interquartile range) …Ranking metrics4Mean average precision (MAP) of rankings · Mean reciprocal rank (MRR) · NDCG@k (normalised discounted cumulative gain) …Regression metrics5Adjusted R^2 · Mean absolute percentage error · Mean squared logarithmic error …

Try it

Worked examples

Each question below was put to data-science-ml-ai999x and the answer shown is the one it returned, with the steps it took.

Classification metricsm.ask(…)

Binary metrics from counts: TP = 40, FP = 10, FN = 5, TN = 45

accuracy = 0.85

precision = 0.8

recall = 0.8888888889

specificity = 0.8181818182

f1 = 0.8421052632

npv = 0.9

Show the steps

Binary classification metrics from confusion counts
P = \frac{TP}{TP+FP},\; R = \frac{TP}{TP+FN},\; F_1 = \frac{2PR}{P+R} = \frac{2TP}{2TP+FP+FN}
with TP = 40, FP = 10, FN = 5, TN = 45
→ accuracy = 0.85, precision = 0.8, recall = 0.8888888889, specificity = 0.8181818182, f1 = 0.8421052632, npv = 0.9, balanced_accuracy = 0.8535353535, fpr = 0.1818181818, fnr = 0.1111111111, jaccard = 0.7272727273

Capability: Binary classification metrics from confusion counts

Classification metricsm.ask(…)

ROC AUC: y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2]

auc = 0.8333333333

gini = 0.6666666667

n_pos = 3

n_neg = 2

Show the steps

ROC AUC from scores
\mathrm{AUC} = \frac{1}{n_+ n_-}\sum_{i \in +}\sum_{j \in -}\left[\mathbb{1}(s_i > s_j) + \tfrac12 \mathbb{1}(s_i = s_j)\right]
with y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2]
→ auc = 0.8333333333, gini = 0.6666666667, n_pos = 3, n_neg = 2

Capability: ROC AUC from scores

Classical algorithmsm.ask(…)

K-means clustering: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5], [3.5, 4.5]], C = [[1, 1], [5, 6]]

centroids = [[1.25, 1.5], [3.9, 5.1]]

labels = [0, 0, 1, 1, 1, 1, 1]

inertia = 8.525

Show the steps

k-means clustering from given initial centroids
\min \sum_k \sum_{i \in S_k} \|x_i - \mu_k\|^2
with X = [1 1; 1.5 2; 3 4; 5 7; 3.5 5; 4.5 5; 3.5 4.5], C = [1 1; 5 6]
→ centroids = [[1.25, 1.5], [3.9, 5.1]], labels = [0, 0, 1, 1, 1, 1, 1], inertia = 8.525

Capability: k-means clustering from given initial centroids

Neural networksm.ask(…)

Scaled dot-product attention: Q = [[1, 0], [0, 1]], K = [[1, 0], [0, 1], [1, 1]], V = [[1, 2], [3, 4], [5, 6]], causal = no

weights = [[0.4011120927, 0.1977758146, 0.4011120927], [0.1977758146, 0.4011120927, 0.4011120927]]

output = [[3, 4], [3.406672556, 4.406672556]]

output_sum = 14.81334511

Show the steps

Scaled dot-product attention
\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V
with Q = [1 0; 0 1], K = [1 0; 0 1; 1 1], V = [1 2; 3 4; 5 6], causal = False
→ weights = [[0.4011120927, 0.1977758146, 0.4011120927], [0.1977758146, 0.4011120927, 0.4011120927]], output = [[3, 4], [3.406672556, 4.406672556]], output_sum = 14.81334511

Capability: Scaled dot-product attention

Neural networksm.solve(…)

m.solve("ml_conv_output_size", n=224, kernel=7, stride=2, padding=3, dilation=1)

output_size = 112

effective_kernel = 7

Show the steps

Convolution output size
n_{\mathrm{out}} = \left\lfloor \frac{n + 2p - d(k-1) - 1}{s} \right\rfloor + 1
with n = 224, kernel = 7, stride = 2, padding = 3, dilation = 1
→ output_size = 112, effective_kernel = 7

Capability: Convolution output size

Clustering metricsm.ask(…)

Adjusted mutual information: labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1], method = arithmetic

ami = 0.6153846154

expected_mi = 0.266835672

mutual_information = 0.6365141683

Show the steps

Adjusted mutual information of two clusterings
\mathrm{AMI} = \frac{I - E[I]}{\mathrm{mean}(H(U), H(V)) - E[I]}
with labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1], method = arithmetic
→ ami = 0.6153846154, expected_mi = 0.266835672, mutual_information = 0.6365141683

Capability: Adjusted mutual information of two clusterings

Dimensionality reductionm.ask(…)

PCA: X = [[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3], [2.3, 2.7], [2, 1.6], [1, 1.1], [1.5, 1.6], [1.1, 0.9]]

explained_variance = [1.284027712, 0.04908339894]

explained_variance_ratio = [0.9631813143, 0.03681868565]

component_1 = [0.6778733985, 0.7351786555]

scores_1 = [0.8279701862, -1.777580325, 0.9921974944, 0.274210416, 1.675801419, 0.9129491032, -0.0991094375, -1.144572164, -0.4380461368, -1.223820555]

Show the steps

Principal component analysis (explained variance and first component)
\Sigma = \frac{1}{n-1}(X - \bar X)^\top (X - \bar X) = V \Lambda V^\top
with X = [2.5 2.4; 0.5 0.7; 2.2 2.9; 1.9 2.2; 3.1 3; 2.3 2.7; 2 1.6; 1 1.1; 1.5 1.6; 1.1 0.9]
→ explained_variance = [1.284027712, 0.04908339894], explained_variance_ratio = [0.9631813143, 0.03681868565], component_1 = [0.6778733985, 0.7351786555], scores_1 = [0.8279701862, -1.777580325, 0.9921974944, 0.274210416, 1.675801419, 0.9129491032, -0.0991094375, -1.144572164, -0.4380461368, -1.223820555]

Capability: Principal component analysis (explained variance and first component)

Gradientsm.ask(…)

Gradient descent step for linear regression: X = [[1], [2], [3], [4]], y = [2, 4, 6, 8], w = [1], b = 1, lr = 0.1

w_new = [2]

b_new = 1.3

loss_before = 3.5

loss_after = 1.69

Show the steps

One gradient-descent step for linear regression
w \leftarrow w - \eta \nabla_w L,\quad b \leftarrow b - \eta \frac{\partial L}{\partial b}
with X = [1; 2; 3; 4], y = [2, 4, 6, 8], w = [1], b = 1, lr = 0.1
→ w_new = [2], b_new = 1.3, loss_before = 3.5, loss_after = 1.69

Capability: One gradient-descent step for linear regression

Use it

From Python or the command line

Every snippet was run as shown; the output under it is what it printed.

import data_science_ml_ai999x as m

a = m.ask("Binary metrics from counts: TP = 40, FP = 10, FN = 5, TN = 45")
print(a.outcome)
print(a.values)
for step in a.steps:
    print(step["name"])

Output

SOLVED
{'accuracy': 0.85, 'precision': 0.8, 'recall': 0.888888888889, 'specificity': 0.818181818182, 'f1': 0.842105263158, 'npv': 0.9, 'balanced_accuracy': 0.853535353535, 'fpr': 0.181818181818, 'fnr': 0.111111111111, 'jaccard': 0.727272727273}
Binary classification metrics from confusion counts

import data_science_ml_ai999x as m

a = m.solve("ml_conv_output_size", n=224, kernel=7, stride=2, padding=3, dilation=1)
print(a.outcome)
print(a.values)

Output

SOLVED
{'output_size': 112, 'effective_kernel': 7}

import data_science_ml_ai999x as m

# an input is missing: the answer says what it needs instead of guessing
a = m.solve("ml_conv_output_size")
print(a.outcome)
print(a.message)

Output

NEEDS_INPUT
ml_conv_output_size needs n, kernel, stride, padding, dilation (input size; kernel size; stride; zero padding on each side; dilation (1 = ordinary)). Example: solve('ml_conv_output_size', n=32, kernel=3, stride=1, padding=1, dilation=1).

data-science-ml-ai999x "Binary metrics from counts: TP = 40, FP = 10, FN = 5, TN = 45"

Output

accuracy = 0.85, precision = 0.8, recall = 0.888889, specificity = 0.818182, f1 = 0.842105, npv = 0.9, balanced_accuracy = 0.853535, fpr = 0.181818, fnr = 0.111111, jaccard = 0.727273
1. Binary classification metrics from confusion counts: exact rational arithmetic on the four counts  with TP = 40, FP = 10, FN = 5, TN = 45  ->  accuracy = 0.85, precision = 0.8, recall = 0.888889, specificity = 0.818182, f1 = 0.842105, npv = 0.9, balanced_accuracy = 0.853535, fpr = 0.181818, fnr = 0.111111, jaccard = 0.727273
Assumptions: counts are whole numbers >= 0; every ratio is defined (TP + FP, TP + FN, TN + FP, TN + FN > 0)
…

Figures

Computed, not drawn by hand

Every value in these figures was computed by the package, each point a run of solve(); in a notebook, plot() and plot_relation() draw a capability's curves the same way.

ROC curve of a scored classifier

\mathrm{TPR}(t) = \frac{TP(t)}{P},\ \mathrm{FPR}(t) = \frac{FP(t)}{N}

The ROC curve the package returns for 12 scored examples (labels 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0; scores 0.95, 0.9, 0.85, 0.8, 0.7, 0.65, 0.6, 0.5, 0.4, 0.35, 0.3, 0.1): true against false positive rate at each of its thresholds, with area under the curve 0.777778.

Confusion matrix of a three-class classifier

C_{ij} = \#\{n : y_n = c_i,\ \hat y_n = c_j\}

The confusion matrix the package returns for true labels [0, 1, 2, 2, 1, 0, 2, 1] and predictions [0, 2, 2, 2, 1, 0, 1, 1]: rows are the true classes 0, 1, 2, columns the predicted ones; 6 of 8 on the diagonal, accuracy 0.75.

k-means clusters and their centroids

\min \sum_k \sum_{i \in S_k} \|x_i - \mu_k\|^2

k-means on 7 points from the starting centres [[1, 1], [5, 6]]: the package's final centroids (1.25, 1.5), (3.9, 5.1) (marked; arrows from the starting centres), cluster labels 1, 1, 2, 2, 2, 2, 2 and inertia 8.525.

Principal components: projection onto the first component

\Sigma = \frac{1}{n-1}(X - \bar X)^\top (X - \bar X) = V \Lambda V^\top

Principal components of 10 points: the package's first component (0.677873, 0.735179) through the mean of the points explains 96.3181 % of the variance (variance 1.28403 against 0.0490834); each point is joined to its projection at the score the package returns; the short arrow is two standard deviations (0.443095) along the second component.

Logistic regression: fitted probability and decision boundary

\min_{w,b}\ \tfrac12\|w\|^2 + C\sum_i \log\left(1 + e^{-\tilde y_i (w^\top x_i + b)}\right)

Logistic regression fitted by the package to x = 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4 with labels 0, 0, 1, 0, 1, 0, 1, 1 (C = 1): weight 0.738776, intercept -1.66225, training accuracy 0.75; the curve is the package's predicted probability at 81 points; marked: the decision boundary x = 2.25, P = 0.5.

k-nearest neighbours of a query point

\hat y = \arg\max_c \sum_{i \in N_k(x)} \mathbb{1}(y_i = c)

k-nearest-neighbour classification of the query point (3, 2.2) among 6 labelled points (k = 3, euclidean distance): the package's 3 nearest neighbours are points 2, 3, 5 at distances 1.28062, 1.56205, 2.00998 (dashed lines; the circle reaches the farthest of them); 2 of 3 votes give class 0.

Formulas

The mathematics inside

12 of its 107 formulas; every one is in the capability browser below.

Binary classification metrics from confusion counts

P = \frac{TP}{TP+FP},\; R = \frac{TP}{TP+FN},\; F_1 = \frac{2PR}{P+R} = \frac{2TP}{2TP+FP+FN}

ROC AUC from scores

\mathrm{AUC} = \frac{1}{n_+ n_-}\sum_{i \in +}\sum_{j \in -}\left[\mathbb{1}(s_i > s_j) + \tfrac12 \mathbb{1}(s_i = s_j)\right]

k-means clustering from given initial centroids

\min \sum_k \sum_{i \in S_k} \|x_i - \mu_k\|^2

Scaled dot-product attention

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Convolution output size

n_{\mathrm{out}} = \left\lfloor \frac{n + 2p - d(k-1) - 1}{s} \right\rfloor + 1

Adjusted mutual information of two clusterings

\mathrm{AMI} = \frac{I - E[I]}{\mathrm{mean}(H(U), H(V)) - E[I]}

Principal component analysis (explained variance and first component)

\Sigma = \frac{1}{n-1}(X - \bar X)^\top (X - \bar X) = V \Lambda V^\top

One gradient-descent step for linear regression

w \leftarrow w - \eta \nabla_w L,\quad b \leftarrow b - \eta \frac{\partial L}{\partial b}

Agglomerative hierarchical clustering merge heights

d(k, i \cup j) = \sqrt{\frac{(n_i + n_k)d_{ki}^2 + (n_j + n_k)d_{kj}^2 - n_k d_{ij}^2}{n_i + n_j + n_k}}\ \text{(Ward)}

Average precision and precision-recall AUC

\mathrm{AP} = \sum_k (R_k - R_{k-1})\,P_k

Calinski-Harabasz index

CH = \frac{\mathrm{tr}(B)}{\mathrm{tr}(W)}\cdot\frac{n - K}{K - 1}

Logistic regression loss and gradient (L2-regularised)

L = \frac{1}{n}\sum_i \left[\log(1 + e^{z_i}) - y_i z_i\right] + \frac{\lambda}{2}\|w\|^2,\quad \nabla_w L = \frac{1}{n}X^\top(\sigma(z) - y) + \lambda w

Capabilities

Every capability, with an example

Search by name, formula, input or topic. Each capability opens to show what it takes, what it returns, when it applies, and a question it answered.

107 capabilities

Classical algorithms 13

Agglomerative hierarchical clustering merge heightsml_hierarchical_clustering

Agglomerative clustering of points with single, complete, average or Ward linkage (Euclidean distance): the distance at which each of the n - 1 merges happens, in order, and the final merge height.

d(k, i \cup j) = \sqrt{\frac{(n_i + n_k)d_{ki}^2 + (n_j + n_k)d_{kj}^2 - n_k d_{ij}^2}{n_i + n_j + n_k}}\ \text{(Ward)}

Inputs

X points: one row per sample (2 to 40) matrix
method linkage: single, complete, average or ward choice

Returns

heights merge distances in merge order (ascending)
final_height height of the last merge

Applies when

2 to 40 points
no tied merge distances (complete, average, ward)

Example

m.ask("Hierarchical clustering: X = [[0, 0], [0, 1], [4, 0], [4, 1.5], [10, 10]], method = single")

→ heights = [1.0, 1.5, 4.0, 10.4043260233], final_height = 10.4043

Bernoulli naive Bayes prediction (binary features)ml_bernoulli_naive_bayes

Fit a Bernoulli naive Bayes classifier on 0/1 features with additive smoothing alpha (absent features count too) and give the predicted class and posteriors of a query vector.

P(x \mid c) = \prod_j p_{cj}^{x_j}(1 - p_{cj})^{1 - x_j}

Inputs

X training features (0 or 1) matrix
y training labels (whole numbers) vector
x query features (0 or 1) vector
alpha additive smoothing (> 0)

Returns

prediction class with the largest posterior
classes classes in sorted order
posteriors posterior probability of each class

Applies when

features 0 or 1
alpha > 0
at least two classes

Example

m.ask("Bernoulli naive Bayes: X = [[1, 0, 1], [1, 1, 0], [0, 1, 1], [0, 0, 1], [1, 1, 1]], y = [0, 0, 1, 1, 0], x = [1, 0, 0], alpha = 1")

→ prediction = 0, classes = [0, 1], posteriors = [0.860022396417, 0.139977603583]

Best decision-tree split (classification, Gini or entropy)ml_decision_stump_classification

The best single split x_j <= t of a classification tree's root by Gini impurity or entropy (bits): the feature, the threshold (midway between consecutive values), the impurity decrease and the sizes of both sides.

\Delta I = I(S) - \frac{|S_L|}{|S|}I(S_L) - \frac{|S_R|}{|S|}I(S_R)

Inputs

X features: one row per sample (whole numbers) matrix
y class labels (whole numbers) vector
criterion split criterion: gini or entropy choice

Returns

feature index (0-based column of X) of the best feature
threshold threshold t (go left when x_j <= t)
impurity_decrease parent impurity minus the size-weighted child impurities
n_left samples going left
n_right samples going right

Applies when

at least two classes
whole-number features
a unique best split

Example

m.ask("Decision tree best split: X = [[2, 7], [3, 1], [10, 8], [11, 2], [4, 9], [12, 0]], y = [0, 0, 1, 1, 0, 1], criterion = gini")

→ feature = 0, threshold = 7, impurity_decrease = 0.5, n_left = 3, n_right = 3

Best regression-tree split (squared error)ml_decision_stump_regression

The best single split x_j <= t of a regression tree's root by squared error: the feature, the threshold, the decrease of the mean squared error and the mean target on each side.

\Delta = \frac{1}{n}\left[SSE(S) - SSE(S_L) - SSE(S_R)\right]

Inputs

X features (whole numbers) matrix
y targets vector

Returns

feature index of the best feature
threshold threshold t (left when x_j <= t)
mse_decrease decrease of the node's mean squared error
left_mean mean target on the left
right_mean mean target on the right

Applies when

whole-number features
targets not all equal
a unique best split

Example

m.ask("Regression tree best split: X = [[1], [2], [3], [10], [11], [12]], y = [1, 1.2, 0.8, 5, 5.4, 4.6]")

→ feature = 0, threshold = 6.5, mse_decrease = 4, left_mean = 1, right_mean = 5

Distances and cosine similarity between two vectorsml_distance_metrics

The Euclidean, Manhattan and Chebyshev distances between two feature vectors, and their cosine similarity and cosine distance.

d_2 = \sqrt{\sum_i (x_i - z_i)^2},\quad \cos\theta = \frac{x \cdot z}{\|x\|\|z\|}

Inputs

x first vector vector
z second vector vector

Returns

euclidean sqrt(sum (x - z)^2)
manhattan sum |x - z|
chebyshev max |x - z|
cosine_similarity x.z / (|x| |z|)
cosine_distance 1 - cosine similarity

Applies when

equal lengths
non-zero vectors

Example

m.ask("Distance metrics: x = [1, 2, 3], z = [4, 6, 3]")

→ euclidean = 5, manhattan = 7, chebyshev = 4, cosine_similarity = 0.855482, cosine_distance = 0.144518

Gaussian naive Bayes predictionml_gaussian_naive_bayes

Fit a Gaussian naive Bayes classifier (class frequencies as priors, per-class feature means and population variances) and give the predicted class and posterior probabilities of a query point.

P(c \mid x) \propto P(c)\prod_j \mathcal{N}(x_j;\ \mu_{cj}, \sigma^2_{cj})

Inputs

X training features: one row per sample matrix
y training labels (whole numbers) vector
x query point vector

Returns

prediction class with the largest posterior
classes classes in sorted order
posteriors posterior probability of each class

Applies when

at least two classes
positive within-class variance of every feature

Example

m.ask("Gaussian naive Bayes: X = [[6, 180, 12], [5.92, 190, 11], [5.58, 170, 12], [5.92, 165, 10], [5, 100, 6], [5.5, 150, 8], [5.42, 130, 7], [5.75, 150, 9]], y = [0, 0, 0, 0, 1, 1, 1, 1], x = [6, 130, 8]")

→ prediction = 1, classes = [0, 1], posteriors = [1.54428667821e-07, 0.999999845571]

k-means clustering from given initial centroidsml_kmeans

Lloyd's k-means algorithm run to convergence from stated initial centroids: the final centroids, the cluster of each point and the inertia (within-cluster sum of squares).

\min \sum_k \sum_{i \in S_k} \|x_i - \mu_k\|^2

Inputs

X points: one row per sample matrix
C initial centroids: one row per cluster matrix

Returns

centroids final centroids
labels cluster (0-based row of C) of each point
inertia sum of squared distances to the final centroids

Applies when

no exact distance tie
no empty cluster during the iterations
convergence within 100 iterations

Example

m.ask("K-means clustering: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5], [3.5, 4.5]], C = [[1, 1], [5, 6]]")

→ centroids = [[1.25, 1.5], [3.9, 5.1]], labels = [0, 0, 1, 1, 1, 1, 1], inertia = 8.525

k-nearest-neighbour classification of a query pointml_knn_classify

Classify a query point by majority vote among its k nearest training points (stated metric), with the neighbours and their distances.

\hat y = \arg\max_c \sum_{i \in N_k(x)} \mathbb{1}(y_i = c)

Inputs

X training points: one row per sample matrix
y training labels (whole numbers) vector
x query point vector
k number of neighbours integer
metric distance: euclidean or manhattan choice

Returns

prediction majority class among the k neighbours
votes number of neighbours of that class
neighbors indices (0-based rows of X) of the k nearest points, ascending
neighbor_distances their distances, ascending

Applies when

no distance tie at the k-th neighbour
no tied vote
1 <= k <= n

Example

m.ask("kNN classification: X = [[1, 1], [2, 1], [4, 3], [5, 4], [1, 2], [6, 5]], y = [0, 0, 1, 1, 0, 1], x = [3, 2.2], k = 3, metric = euclidean")

→ prediction = 0, votes = 2, neighbors = [1, 2, 4], neighbor_distances = [1.28062484749, 1.56204993518, 2.00997512422]

k-nearest-neighbour regression of a query pointml_knn_regress

Predict a value at a query point as the mean (uniform) or inverse-distance-weighted mean of the targets of its k nearest training points.

\hat y = \frac{\sum_{i \in N_k} w_i y_i}{\sum_{i \in N_k} w_i},\quad w_i = 1 \text{ or } 1/d_i

Inputs

X training points matrix
y training targets vector
x query point vector
k number of neighbours integer
metric distance: euclidean or manhattan choice
weights uniform mean or inverse-distance weights choice

Returns

prediction predicted value
neighbors indices of the k nearest points, ascending
neighbor_distances their distances, ascending

Applies when

no distance tie at the k-th neighbour
distance weights: the query is not a training point

Example

m.ask("kNN regression: X = [[1], [2], [3], [10]], y = [1.5, 2.5, 3, 9], x = [2.4], k = 2, metric = euclidean, weights = uniform")

→ prediction = 2.75, neighbors = [1, 2], neighbor_distances = [0.4, 0.6]

Multinomial naive Bayes prediction (word counts)ml_multinomial_naive_bayes

Fit a multinomial naive Bayes classifier on count features (such as word counts) with additive (Laplace/Lidstone) smoothing alpha and give the predicted class and posteriors of a query count vector.

\hat\theta_{cj} = \frac{N_{cj} + \alpha}{N_c + \alpha d},\quad \log P(c \mid x) = \log P(c) + \sum_j x_j \log\hat\theta_{cj} + \text{const}

Inputs

X training counts: one row per document, one column per term matrix
y training labels (whole numbers) vector
x query counts vector
alpha additive smoothing (> 0; 1 = Laplace)

Returns

prediction class with the largest posterior
classes classes in sorted order
posteriors posterior probability of each class

Applies when

alpha > 0
counts >= 0
at least two classes

Example

m.ask("Multinomial naive Bayes: X = [[2, 1, 0, 0], [1, 1, 0, 1], [0, 0, 3, 1], [0, 1, 2, 2]], y = [0, 0, 1, 1], x = [1, 0, 1, 0], alpha = 1")

→ prediction = 0, classes = [0, 1], posteriors = [0.529780564263, 0.470219435737]

One k-means iteration (assignment and update)ml_kmeans_step

One Lloyd iteration of k-means from given centroids: assign every point to its nearest centroid, give the inertia of that assignment and the updated centroids (the means of the assigned points).

c(i) = \arg\min_k \|x_i - \mu_k\|^2,\quad \mu_k' = \frac{1}{|S_k|}\sum_{i \in S_k} x_i

Inputs

X points: one row per sample matrix
C current (or initial) centroids: one row per cluster matrix

Returns

labels index (0-based row of C) of the nearest centroid of each point
new_centroids mean of the points assigned to each centroid
inertia sum of squared distances of the points to their assigned (current) centroids

Applies when

no exact distance tie
no empty cluster
2 <= k <= n

Example

m.ask("K-means step: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5], [3.5, 4.5]], C = [[1, 1], [5, 6]]")

→ labels = [0, 0, 1, 1, 1, 1, 1], new_centroids = [[1.25, 1.5], [3.9, 5.1]], inertia = 19.25

Polynomial kernelml_polynomial_kernel

The polynomial kernel (gamma x.z + coef0)^degree between two vectors, as used by kernel SVMs.

k(x, z) = (\gamma\, x^\top z + c_0)^d

Inputs

x first vector vector
z second vector vector
gamma scale of the dot product
coef0 constant term
degree degree (1 to 10) integer

Returns

kernel (gamma x.z + coef0)^degree

Applies when

degree 1 to 10
equal lengths

Example

m.ask("Polynomial kernel: x = [1, 2], z = [3, 4], gamma = 1, coef0 = 1, degree = 2")

→ kernel = 144

RBF (Gaussian) kernel matrixml_rbf_kernel_matrix

The Gram matrix of the radial basis function kernel exp(-gamma ||x_i - x_j||^2) for a set of points, and the mean off-diagonal similarity.

K_{ij} = \exp\left(-\gamma\,\|x_i - x_j\|^2\right)

Inputs

X points: one row per point (at least two) matrix
gamma kernel width parameter (> 0)

Returns

K kernel matrix K_ij = exp(-gamma ||x_i - x_j||^2)
mean_offdiagonal mean of the off-diagonal entries

Applies when

gamma > 0
at least two points

Example

m.ask("RBF kernel matrix: X = [[0, 0], [1, 0], [0, 2]], gamma = 0.5")

→ K = [[1.0, 0.606530659713, 0.135335283237], [0.606530659713, 1.0, 0.0820849986239], [0.135335283237, 0.0820849986239, 1.0]], mean_offdiagonal = 0.27465

Classification metrics 20

Average precision and precision-recall AUCml_average_precision

Average precision (the step-wise area under the precision-recall curve, sum of (R_k - R_{k-1}) P_k over distinct score thresholds) and the trapezoidal area under the precision-recall curve.

\mathrm{AP} = \sum_k (R_k - R_{k-1})\,P_k

Inputs

y_true true labels (0 or 1) vector
scores classifier scores vector

Returns

average_precision average precision
pr_auc_trapezoid trapezoidal area under the precision-recall curve, from (recall 0, precision 1)

Applies when

labels 0 or 1
at least one positive

Example

m.ask("Average precision: y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2]")

→ average_precision = 0.916667, pr_auc_trapezoid = 0.902778

Averaged precision, recall and F1 (multi-class)ml_precision_recall_f1_averaged

Multi-class precision, recall and F1 averaged over classes: macro (unweighted), weighted (by support) or micro (pooled counts).

F_1^{\mathrm{macro}} = \frac{1}{K}\sum_k \frac{2P_kR_k}{P_k+R_k}

Inputs

y_true true class labels (whole numbers) vector
y_pred predicted class labels (whole numbers) vector
average averaging: macro, weighted or micro choice

Returns

precision averaged precision
recall averaged recall
f1 averaged F1 (macro and weighted: the average of the per-class F1)

Applies when

labels are whole numbers
equal lengths

Example

m.ask("Averaged precision recall F1: y_true = [0, 1, 2, 2, 1, 0, 2, 1], y_pred = [0, 2, 2, 2, 1, 0, 1, 1], average = macro")

→ precision = 0.777778, recall = 0.777778, f1 = 0.777778

Balanced accuracy (multi-class)ml_balanced_accuracy

Balanced accuracy, the mean recall over the classes present in the true labels, and its chance-adjusted form.

\mathrm{BA} = \frac{1}{K}\sum_{k} \frac{TP_k}{TP_k + FN_k}

Inputs

y_true true class labels (whole numbers) vector
y_pred predicted class labels (whole numbers) vector

Returns

balanced_accuracy mean per-class recall over the true classes
adjusted (balanced accuracy - 1/K) / (1 - 1/K), K = number of true classes

Applies when

at least two true classes
equal lengths

Example

m.ask("Balanced accuracy: y_true = [0, 0, 0, 0, 0, 0, 1, 1], y_pred = [0, 0, 0, 0, 0, 0, 0, 1]")

→ balanced_accuracy = 0.75, adjusted = 0.5

Balanced class weightsml_balanced_class_weights

Class weights that balance an imbalanced training set: n / (K n_c) for each class c.

w_c = \frac{n}{K\, n_c}

Inputs

y training labels (whole numbers) vector

Returns

classes classes in sorted order
weights weight of each class, n / (K n_c)

Applies when

labels are whole numbers

Example

m.ask("Balanced class weights: y = [0, 0, 0, 0, 0, 0, 1, 1]")

→ classes = [0, 1], weights = [0.666666666667, 2.0]

Binary classification metrics from confusion countsml_binary_metrics_from_counts

Accuracy, precision, recall (sensitivity), specificity, F1, negative predictive value, balanced accuracy, false positive and false negative rates and the Jaccard index from TP, FP, FN, TN.

P = \frac{TP}{TP+FP},\; R = \frac{TP}{TP+FN},\; F_1 = \frac{2PR}{P+R} = \frac{2TP}{2TP+FP+FN}

Inputs

TP true positives integer
FP false positives integer
FN false negatives integer
TN true negatives integer

Returns

accuracy (TP + TN) / n
precision TP / (TP + FP)
recall TP / (TP + FN), the sensitivity
specificity TN / (TN + FP)
f1 2 TP / (2 TP + FP + FN)
npv TN / (TN + FN)
balanced_accuracy (recall + specificity) / 2
fpr FP / (FP + TN)
fnr FN / (FN + TP)
jaccard TP / (TP + FP + FN)

Applies when

counts are whole numbers >= 0
every ratio is defined (TP + FP, TP + FN, TN + FP, TN + FN > 0)

Example

m.ask("Binary metrics from counts: TP = 40, FP = 10, FN = 5, TN = 45")

→ accuracy = 0.85, precision = 0.8, recall = 0.888889, specificity = 0.818182, f1 = 0.842105, npv = 0.9, balanced_accuracy = 0.853535, fpr = 0.181818, fnr = 0.111111, jaccard = 0.727273

Binary confusion matrix from labelsml_binary_confusion_matrix

Count true positives, false positives, false negatives and true negatives from true and predicted 0/1 labels, with the accuracy and error rate.

\mathrm{accuracy} = \frac{TP + TN}{TP + FP + FN + TN}

Inputs

y_true true labels (0 or 1, 1 = positive) vector
y_pred predicted labels (0 or 1) vector

Returns

TP true positives
FP false positives
FN false negatives
TN true negatives
accuracy (TP + TN) / n
error_rate (FP + FN) / n

Applies when

labels are 0 or 1
y_true and y_pred have the same length

Example

m.ask("Binary confusion matrix: y_true = [1, 0, 1, 1, 0, 0, 1, 0], y_pred = [1, 0, 0, 1, 0, 1, 1, 0]")

→ TP = 3, FP = 1, FN = 1, TN = 3, accuracy = 0.75, error_rate = 0.25

Binary log loss (cross-entropy)ml_log_loss_binary

The mean binary cross-entropy (log loss) of predicted probabilities of the positive class, in nats and in bits.

\mathcal{L} = -\frac{1}{n}\sum_i \left[y_i \ln p_i + (1-y_i)\ln(1-p_i)\right]

Inputs

y_true true labels (0 or 1) vector
p predicted probability of class 1 for each sample (0 < p < 1) vector

Returns

log_loss mean log loss in nats
log_loss_bits mean log loss in bits

Applies when

labels 0 or 1
0 < p < 1

Example

m.ask("Binary log loss: y_true = [1, 0, 1, 1], p = [0.9, 0.2, 0.7, 0.6]")

→ log_loss = 0.299001, log_loss_bits = 0.431367

Brier scoreml_brier_score

The Brier score, the mean squared difference between predicted probabilities of the positive class and the 0/1 outcomes.

\mathrm{BS} = \frac{1}{n}\sum_i (p_i - y_i)^2

Inputs

y_true true labels (0 or 1) vector
p predicted probability of class 1 (0 to 1) vector

Returns

brier mean squared error of the probabilities

Applies when

labels 0 or 1
0 <= p <= 1

Example

m.ask("Brier score: y_true = [1, 0, 1, 1], p = [0.9, 0.2, 0.7, 0.6]")

→ brier = 0.075

Cohen's kappa (unweighted, linear or quadratic)ml_cohen_kappa

Cohen's kappa, the agreement between two raters (or a classifier and the truth) corrected for chance, optionally weighted by the distance between classes.

\kappa = 1 - \frac{\sum_{ij} w_{ij} O_{ij}}{\sum_{ij} w_{ij} E_{ij}},\quad E_{ij} = \frac{r_i c_j}{n}

Inputs

y1 labels of the first rater (whole numbers) vector
y2 labels of the second rater (whole numbers) vector
weights disagreement weights: none, linear or quadratic (distance between class positions in sorted order) choice

Returns

kappa Cohen's kappa

Applies when

labels are whole numbers
at least two classes
equal lengths

Example

m.ask("Cohen's kappa: y1 = [0, 1, 2, 2, 1, 0, 2, 1], y2 = [0, 2, 2, 2, 1, 0, 1, 1], weights = none")

→ kappa = 0.619048

Confusion counts, precision and recall at a score thresholdml_threshold_metrics

Classify by a score threshold (score >= threshold is positive) and give the confusion counts, precision, recall and accuracy.

\hat y_i = \mathbb{1}(s_i \ge t)

Inputs

y_true true labels (0 or 1) vector
scores classifier scores vector
threshold decision threshold (a score at or above it is positive)

Returns

TP true positives
FP false positives
FN false negatives
TN true negatives
precision TP / (TP + FP)
recall TP / (TP + FN)
accuracy (TP + TN) / n

Applies when

labels 0 or 1
at least one score reaches the threshold
at least one positive

Example

m.ask("Metrics at a threshold: y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2], threshold = 0.25")

→ TP = 2, FP = 1, FN = 1, TN = 1, precision = 0.666667, recall = 0.666667, accuracy = 0.6

Expected calibration errorml_expected_calibration_error

The expected calibration error (support-weighted mean gap between accuracy and mean confidence over equal-width probability bins) and the maximum calibration error.

\mathrm{ECE} = \sum_b \frac{n_b}{n}\left|\bar y_b - \bar p_b\right|

Inputs

y_true true labels (0 or 1) vector
p predicted probability of class 1 vector
n_bins number of equal-width bins on [0, 1] integer

Returns

ece expected calibration error
mce maximum calibration error over non-empty bins
n_nonempty_bins number of non-empty bins

Applies when

labels 0 or 1
0 <= p <= 1
bins are [b/n, (b+1)/n); a probability on a bin edge goes to the upper bin

Example

m.ask("Expected calibration error: y_true = [1, 0, 1, 1, 0, 0, 1, 0], p = [0.92, 0.13, 0.67, 0.81, 0.36, 0.08, 0.44, 0.58], n_bins = 5")

→ ece = 0.14875, mce = 0.36, n_nonempty_bins = 5

F-beta scoreml_fbeta_score

The F-beta score, the weighted harmonic mean of precision and recall in which recall counts beta times as much as precision.

F_\beta = \frac{(1+\beta^2)\,P R}{\beta^2 P + R}

Inputs

TP true positives integer
FP false positives integer
FN false negatives integer
beta weight of recall relative to precision (beta > 0; 1 gives F1, 2 favours recall)

Returns

precision TP / (TP + FP)
recall TP / (TP + FN)
fbeta F-beta score

Applies when

beta > 0
TP > 0

Example

m.ask("F-beta score: TP = 40, FP = 10, FN = 5, beta = 2")

→ precision = 0.8, recall = 0.888889, fbeta = 0.869565

Hamming loss and subset accuracy (multi-label)ml_hamming_loss

For multi-label predictions (0/1 indicator matrices): the Hamming loss (fraction of wrong labels) and the subset accuracy (fraction of samples with every label right).

\mathrm{HL} = \frac{1}{nL}\sum_{i,j} \mathbb{1}(Y_{ij} \neq \hat Y_{ij})

Inputs

Y_true true label indicators: one row per sample, one column per label (0 or 1) matrix
Y_pred predicted label indicators, same shape matrix

Returns

hamming_loss wrong label entries / (samples x labels)
subset_accuracy fraction of rows predicted exactly
n_mismatches number of wrong label entries

Applies when

entries 0 or 1
same shape

Example

m.ask("Hamming loss: Y_true = [[1, 0, 1], [0, 1, 0]], Y_pred = [[1, 0, 0], [0, 1, 0]]")

→ hamming_loss = 0.166667, subset_accuracy = 0.5, n_mismatches = 1

Matthews correlation coefficientml_matthews_correlation

The Matthews correlation coefficient (phi coefficient) of a binary confusion matrix, a balanced measure between -1 and 1.

\mathrm{MCC} = \frac{TP\,TN - FP\,FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}

Inputs

TP true positives integer
FP false positives integer
FN false negatives integer
TN true negatives integer

Returns

mcc Matthews correlation coefficient

Applies when

no empty row or column of the confusion matrix

Example

m.ask("Matthews correlation coefficient: TP = 40, FP = 10, FN = 5, TN = 45")

→ mcc = 0.703526

Multi-class confusion matrixml_multiclass_confusion_matrix

The confusion matrix of a multi-class classifier (true classes in rows, predicted in columns) with the number correct and the accuracy.

C_{ij} = \#\{n : y_n = c_i,\ \hat y_n = c_j\}

Inputs

y_true true class labels (whole numbers) vector
y_pred predicted class labels (whole numbers) vector

Returns

classes classes in sorted order (the union of both label lists)
matrix confusion matrix: row = true class, column = predicted class
n_correct number of correct predictions (the trace)
accuracy fraction correct

Applies when

labels are whole numbers
equal lengths

Example

m.ask("Multiclass confusion matrix: y_true = [0, 1, 2, 2, 1, 0, 2, 1], y_pred = [0, 2, 2, 2, 1, 0, 1, 1]")

→ classes = [0, 1, 2], matrix = [[2, 0, 0], [0, 2, 1], [0, 1, 2]], n_correct = 6, accuracy = 0.75

Multi-class log loss (categorical cross-entropy)ml_log_loss_multiclass

The mean categorical cross-entropy of predicted class probabilities (rows of P), with the perplexity exp(loss).

\mathcal{L} = -\frac{1}{n}\sum_i \ln P_{i, y_i}

Inputs

y_true true class index of each sample (0 to K-1) vector
P predicted probabilities: one row per sample, one column per class, rows sum to 1 matrix

Returns

log_loss mean log loss in nats
perplexity exp(log loss)

Applies when

labels are column indices of P
rows of P sum to 1
P[i, y_i] > 0

Example

m.ask("Multiclass log loss: y_true = [0, 2, 1], P = [[0.7, 0.2, 0.1], [0.1, 0.3, 0.6], [0.25, 0.5, 0.25]]")

→ log_loss = 0.520216, perplexity = 1.68239

Per-class precision, recall, F1 and supportml_classification_report

A classification report: precision, recall, F1 and support for every class, with the macro-averaged F1.

P_k = \frac{TP_k}{TP_k + FP_k},\quad R_k = \frac{TP_k}{TP_k + FN_k}

Inputs

y_true true class labels (whole numbers) vector
y_pred predicted class labels (whole numbers) vector

Returns

classes classes in sorted order
precision precision of each class
recall recall of each class
f1 F1 of each class
support number of true members of each class
macro_f1 mean of the per-class F1

Applies when

labels are whole numbers
equal lengths

Example

m.ask("Classification report: y_true = [0, 1, 2, 2, 1, 0, 2, 1], y_pred = [0, 2, 2, 2, 1, 0, 1, 1]")

→ classes = [0, 1, 2], precision = [1.0, 0.666666666667, 0.666666666667], recall = [1.0, 0.666666666667, 0.666666666667], f1 = [1.0, 0.666666666667, 0.666666666667], support = [2, 3, 3], macro_f1 = 0.777778

ROC AUC from scoresml_roc_auc

The area under the ROC curve from 0/1 labels and classifier scores: the probability that a random positive scores above a random negative (ties count one half), with the Gini coefficient 2 AUC - 1.

\mathrm{AUC} = \frac{1}{n_+ n_-}\sum_{i \in +}\sum_{j \in -}\left[\mathbb{1}(s_i > s_j) + \tfrac12 \mathbb{1}(s_i = s_j)\right]

Inputs

y_true true labels (0 or 1) vector
scores classifier scores (higher = more likely positive) vector

Returns

auc area under the ROC curve
gini 2 AUC - 1
n_pos number of positives
n_neg number of negatives

Applies when

labels 0 or 1
at least one positive and one negative

Example

m.ask("ROC AUC: y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2]")

→ auc = 0.833333, gini = 0.666667, n_pos = 3, n_neg = 2

ROC curve pointsml_roc_curve

The ROC curve of a scored classifier: the false and true positive rates at every distinct score threshold (a score at or above the threshold is positive), and the trapezoidal area under it.

\mathrm{TPR}(t) = \frac{TP(t)}{P},\ \mathrm{FPR}(t) = \frac{FP(t)}{N}

Inputs

y_true true labels (0 or 1) vector
scores classifier scores vector

Returns

thresholds distinct scores, descending (one per curve point after the origin)
fpr false positive rates, starting at 0
tpr true positive rates, starting at 0
auc trapezoidal area under the curve

Applies when

labels 0 or 1
at least one positive and one negative

Example

m.ask("ROC curve: y_true = [0, 1, 1, 0, 1], scores = [0.1, 0.9, 0.8, 0.3, 0.2]")

→ thresholds = [0.9, 0.8, 0.3, 0.2, 0.1], fpr = [0.0, 0.0, 0.0, 0.5, 0.5, 1.0], tpr = [0.0, 0.333333333333, 0.666666666667, 0.666666666667, 1.0, 1.0], auc = 0.833333

Top-k accuracyml_top_k_accuracy

The fraction of samples whose true class is among the k highest-scored classes.

\mathrm{top}\text{-}k = \frac{1}{n}\sum_i \mathbb{1}\left[\mathrm{rank}_i(y_i) \le k\right]

Inputs

y_true true class index of each sample (0 to K-1) vector
S class scores: one row per sample, one column per class (K >= 3) matrix
k number of top classes that count as a hit integer

Returns

top_k_accuracy fraction of samples with the true class in the top k
n_hits number of such samples

Applies when

K >= 3 classes
1 <= k < K
distinct scores within each row

Example

m.ask("Top-k accuracy: y_true = [0, 1, 2, 2], S = [[0.5, 0.2, 0.2001], [0.3, 0.4, 0.2], [0.2, 0.4, 0.3], [0.7, 0.2, 0.1]], k = 2")

→ top_k_accuracy = 0.75, n_hits = 3

Clustering metrics 7

Adjusted mutual information of two clusteringsml_adjusted_mutual_information

The adjusted mutual information between two clusterings: the mutual information corrected for chance by its expected value under random labellings with the same cluster sizes (hypergeometric model).

\mathrm{AMI} = \frac{I - E[I]}{\mathrm{mean}(H(U), H(V)) - E[I]}

Inputs

labels_true reference labels (whole numbers) vector
labels_pred cluster labels (whole numbers) vector
method normaliser: arithmetic, geometric, min or max of the two entropies choice

Returns

ami (I - E[I]) / (normaliser - E[I])
expected_mi E[I] in nats
mutual_information I in nats

Applies when

each labelling has at least two clusters

Example

m.ask("Adjusted mutual information: labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1], method = arithmetic")

→ ami = 0.615385, expected_mi = 0.266836, mutual_information = 0.636514

Calinski-Harabasz indexml_calinski_harabasz_index

The Calinski-Harabasz (variance ratio) index of a clustering: between-cluster over within-cluster dispersion, scaled by (n - K)/(K - 1) (higher is better).

CH = \frac{\mathrm{tr}(B)}{\mathrm{tr}(W)}\cdot\frac{n - K}{K - 1}

Inputs

X points: one row per sample matrix
labels cluster label of each point (whole numbers) vector

Returns

calinski_harabasz CH index
between between-cluster dispersion B
within within-cluster dispersion W

Applies when

2 <= number of clusters <= n - 1

Example

m.ask("Calinski-Harabasz index: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5]], labels = [0, 0, 1, 1, 1, 1]")

→ calinski_harabasz = 14.6455, between = 28.8333, within = 7.875

Davies-Bouldin indexml_davies_bouldin_index

The Davies-Bouldin index of a clustering: the mean over clusters of the worst ratio of within-cluster scatter to between-centroid distance (lower is better).

DB = \frac{1}{K}\sum_k \max_{m \ne k}\frac{S_k + S_m}{\|c_k - c_m\|}

Inputs

X points: one row per sample matrix
labels cluster label of each point (whole numbers) vector

Returns

davies_bouldin Davies-Bouldin index

Applies when

2 <= number of clusters <= n - 1
distinct centroids

Example

m.ask("Davies-Bouldin index: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5]], labels = [0, 0, 1, 1, 1, 1]")

→ davies_bouldin = 0.374734

Homogeneity, completeness and V-measureml_homogeneity_completeness_v_measure

Entropy-based clustering scores against true classes: homogeneity (each cluster holds one class), completeness (each class sits in one cluster) and their weighted harmonic mean, the V-measure.

h = 1 - \frac{H(C \mid K)}{H(C)},\quad c = 1 - \frac{H(K \mid C)}{H(K)},\quad V_\beta = \frac{(1+\beta)hc}{\beta h + c}

Inputs

labels_true true class labels (whole numbers) vector
labels_pred cluster labels (whole numbers) vector
beta weight of completeness relative to homogeneity in the V-measure (1 = equal)

Returns

homogeneity 1 - H(C | K) / H(C) = I / H(C)
completeness I / H(K)
v_measure (1 + beta) h c / (beta h + c)

Applies when

each labelling has at least two clusters
beta > 0

Example

m.ask("V-measure: labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1], beta = 1")

→ homogeneity = 0.57938, completeness = 1, v_measure = 0.73368

Normalised mutual information of two clusteringsml_normalized_mutual_information

The normalised mutual information between two clusterings (or a clustering and the true classes), with the stated normaliser of the two entropies, and the mutual information in nats.

\mathrm{NMI} = \frac{I(U, V)}{\mathrm{mean}(H(U), H(V))}

Inputs

labels_true reference labels (whole numbers) vector
labels_pred cluster labels (whole numbers) vector
method normaliser: arithmetic, geometric, min or max of the two entropies choice

Returns

nmi I / normaliser(H_true, H_pred)
mutual_information I in nats

Applies when

each labelling has at least two clusters

Example

m.ask("Normalized mutual information: labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1], method = arithmetic")

→ nmi = 0.73368, mutual_information = 0.636514

Rand index, adjusted Rand index and Fowlkes-Mallows indexml_rand_index

Pair-counting agreement between two clusterings: the Rand index, the adjusted Rand index (corrected for chance) and the Fowlkes-Mallows index.

\mathrm{ARI} = \frac{\sum_{ij}\binom{n_{ij}}{2} - \frac{\sum_i\binom{a_i}{2}\sum_j\binom{b_j}{2}}{\binom{n}{2}}}{\frac12\left[\sum_i\binom{a_i}{2} + \sum_j\binom{b_j}{2}\right] - \frac{\sum_i\binom{a_i}{2}\sum_j\binom{b_j}{2}}{\binom{n}{2}}}

Inputs

labels_true reference labels (whole numbers) vector
labels_pred cluster labels (whole numbers) vector

Returns

rand_index fraction of sample pairs on which the two labellings agree
adjusted_rand_index ARI
fowlkes_mallows TP / sqrt((TP + FP)(TP + FN)) over pairs

Applies when

at least one within-cluster pair in each labelling

Example

m.ask("Adjusted Rand index: labels_true = [0, 0, 1, 1, 2, 2], labels_pred = [0, 0, 1, 1, 1, 1]")

→ rand_index = 0.733333, adjusted_rand_index = 0.444444, fowlkes_mallows = 0.654654

Silhouette scoreml_silhouette_score

The mean silhouette coefficient of a clustering (Euclidean distance) and the silhouette of each sample: (b - a) / max(a, b), a = mean distance within its cluster, b = mean distance to the nearest other cluster.

s_i = \frac{b_i - a_i}{\max(a_i, b_i)}

Inputs

X points: one row per sample matrix
labels cluster label of each point (whole numbers) vector

Returns

silhouette mean silhouette coefficient
samples silhouette of each sample (0 for a point alone in its cluster)

Applies when

2 <= number of clusters <= n - 1

Example

m.ask("Silhouette score: X = [[1, 1], [1.5, 2], [3, 4], [5, 7], [3.5, 5], [4.5, 5]], labels = [0, 0, 1, 1, 1, 1]")

→ silhouette = 0.613815, samples = [0.785495870518, 0.728161920035, 0.287385025256, 0.591063503361, 0.630078239981, 0.660704874448]

Dimensionality reduction 1

Principal component analysis (explained variance and first component)ml_pca

Principal component analysis of a data matrix: the variance explained by every component (sample covariance, n - 1), the explained variance ratios, the first principal direction and the first component score of every sample.

\Sigma = \frac{1}{n-1}(X - \bar X)^\top (X - \bar X) = V \Lambda V^\top

Inputs

X data: one row per sample, one column per feature (more samples than features, at most 8 features) matrix

Returns

explained_variance eigenvalues of the sample covariance, descending
explained_variance_ratio each eigenvalue / their sum
component_1 first principal direction (unit length, largest-magnitude entry positive)
scores_1 projection of each centred sample on component_1

Applies when

n > number of features
full rank
a unique first component

Example

m.ask("PCA: X = [[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3], [2.3, 2.7], [2, 1.6], [1, 1.1], [1.5, 1.6], [1.1, 0.9]]")

→ explained_variance = [1.28402771217, 0.0490833989383], explained_variance_ratio = [0.963181314349, 0.0368186856514], component_1 = [0.677873398528, 0.735178655544], scores_1 = [0.827970186201, -1.77758032528, 0.992197494415, 0.274210415975, 1.67580141864, 0.912949103159, -0.0991094374984, -1.1445721638, -0.438046136762, -1.22382055505]

Gradients 3

Logistic regression loss and gradient (L2-regularised)ml_logistic_loss_gradient

The mean logistic (binary cross-entropy) loss of a linear model with an L2 penalty (lam / 2) ||w||^2 and its gradient with respect to the weights and the bias.

L = \frac{1}{n}\sum_i \left[\log(1 + e^{z_i}) - y_i z_i\right] + \frac{\lambda}{2}\|w\|^2,\quad \nabla_w L = \frac{1}{n}X^\top(\sigma(z) - y) + \lambda w

Inputs

X feature matrix matrix
y labels (0 or 1) vector
w weights vector
b bias
lam L2 regularisation strength (>= 0; the bias is not penalised)

Returns

loss mean log loss + (lam/2)||w||^2
grad_w X^T (sigma(Xw + b) - y) / n + lam w
grad_b mean of sigma(Xw + b) - y

Applies when

labels 0 or 1
lam >= 0

Example

m.ask("Logistic regression gradient: X = [[1, 2], [2, 1], [3, 3]], y = [0, 1, 1], w = [0.5, -0.25], b = 0.1, lam = 0")

→ loss = 0.485376, grad_w = [-0.32406170005, -0.0492576850487], grad_b = -0.0246288

Mean squared error loss and gradient of a linear modelml_mse_gradient

The mean squared error of a linear model y_hat = X w + b and its gradient with respect to the weights and the bias.

L = \frac{1}{n}\|Xw + b - y\|^2,\quad \nabla_w L = \frac{2}{n}X^\top(Xw + b - y)

Inputs

X feature matrix: one row per sample matrix
y targets, one per row vector
w weights, one per column of X vector
b bias (intercept)

Returns

loss mean (X w + b - y)^2
grad_w dL/dw = (2/n) X^T (X w + b - y)
grad_b dL/db = (2/n) sum of residuals

Applies when

dimensions agree

Example

m.ask("MSE gradient of a linear model: X = [[1, 2], [3, 4], [5, 6]], y = [1, 2, 3], w = [0.1, 0.2], b = 0")

→ loss = 0.916667, grad_w = [-6.46666666667, -8.26666666667], grad_b = -1.8

One gradient-descent step for linear regressionml_gradient_descent_step_linear

One full-batch gradient-descent step on the mean squared error of a linear model: the new weights and bias and the loss before and after the step.

w \leftarrow w - \eta \nabla_w L,\quad b \leftarrow b - \eta \frac{\partial L}{\partial b}

Inputs

X feature matrix: one row per sample matrix
y targets, one per row vector
w weights, one per column of X vector
b bias (intercept)
lr learning rate (> 0)

Returns

w_new w - lr dL/dw
b_new b - lr dL/db
loss_before MSE before the step
loss_after MSE after the step

Applies when

lr > 0
dimensions agree

Example

m.ask("Gradient descent step for linear regression: X = [[1], [2], [3], [4]], y = [2, 4, 6, 8], w = [1], b = 1, lr = 0.1")

→ w_new = [2.0], b_new = 1.3, loss_before = 3.5, loss_after = 1.69

Information theory 7

Gini impurity decrease of a splitml_gini_split

The Gini impurity of a node, the size-weighted Gini impurity of its children after a split, and the decrease (the CART split criterion).

\Delta G = G(S) - \sum_v \frac{|S_v|}{|S|}G(S_v)

Inputs

y class labels (whole numbers) vector
g child (group) each sample goes to (whole numbers) vector

Returns

gini_parent Gini impurity before the split
gini_children weighted Gini impurity of the children
gini_decrease parent minus children

Applies when

at least two groups

Example

m.ask("Gini split: y = [0, 0, 0, 1, 1, 1, 1, 0], g = [0, 0, 0, 0, 1, 1, 1, 1]")

→ gini_parent = 0.5, gini_children = 0.375, gini_decrease = 0.125

Information gain and gain ratio of a splitml_information_gain

The information gain (in bits) of splitting class labels y by a discrete feature x, with the parent and conditional entropies, the split information and the gain ratio (C4.5).

IG = H(Y) - \sum_v \frac{|S_v|}{|S|} H(Y \mid X = v),\quad GR = \frac{IG}{H(X)}

Inputs

y class labels (whole numbers) vector
x feature value of each sample (whole numbers; each value is one branch) vector

Returns

parent_entropy H(y) in bits
conditional_entropy H(y | x) in bits
information_gain H(y) - H(y | x) in bits
split_info H(x) in bits
gain_ratio information gain / split information

Applies when

the feature takes at least two values

Example

m.ask("Information gain: y = [1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0], x = [0, 0, 1, 2, 2, 2, 1, 0, 0, 2, 0, 1, 1, 2]")

→ parent_entropy = 0.863121, conditional_entropy = 0.632482, information_gain = 0.230638, split_info = 1.57741, gain_ratio = 0.146214

Joint, marginal and conditional entropy and mutual informationml_joint_conditional_entropy

From a joint probability table of X (rows) and Y (columns): the joint entropy, both marginal entropies, both conditional entropies and the mutual information.

H(Y \mid X) = H(X, Y) - H(X),\quad I(X;Y) = \sum_{ij} P_{ij}\log\frac{P_{ij}}{P_i P_j}

Inputs

P joint probabilities P(X = i, Y = j): rows are values of X, columns values of Y; sums to 1 matrix
unit unit of information: nats (natural log), bits (log base 2) or bans (log base 10) choice

Returns

H_XY joint entropy
H_X entropy of X
H_Y entropy of Y
H_Y_given_X H(Y | X) = H(X, Y) - H(X)
H_X_given_Y H(X | Y)
mutual_information I(X; Y) = sum P log P / (P_X P_Y)

Applies when

probabilities >= 0 summing to 1

Example

m.ask("Joint and conditional entropy: P = [[0.25, 0.25], [0.5, 0]], unit = bits")

→ H_XY = 1.5, H_X = 1, H_Y = 0.811278, H_Y_given_X = 0.5, H_X_given_Y = 0.688722, mutual_information = 0.311278

KL divergence, Jensen-Shannon divergence and cross-entropy of discrete distributionsml_kl_divergence_discrete

The Kullback-Leibler divergence of two discrete distributions in both directions, the Jensen-Shannon divergence, the cross-entropy H(p, q) and the entropy H(p).

D_{KL}(p \| q) = \sum_i p_i \log\frac{p_i}{q_i},\quad H(p, q) = H(p) + D_{KL}(p \| q)

Inputs

p first distribution (positive, sums to 1) vector
q second distribution (positive, sums to 1) vector
unit unit of information: nats (natural log), bits (log base 2) or bans (log base 10) choice

Returns

kl_pq KL(p || q)
kl_qp KL(q || p)
js_divergence Jensen-Shannon divergence
cross_entropy H(p, q) = -sum p log q
entropy_p H(p)

Applies when

positive probabilities summing to 1
same length

Example

m.ask("KL divergence: p = [0.5, 0.5], q = [0.9, 0.1], unit = bits")

→ kl_pq = 0.736966, kl_qp = 0.531004, js_divergence = 0.146793, cross_entropy = 1.73697, entropy_p = 1

Mutual information between two label sequencesml_mutual_information_labels

The mutual information between two discrete variables observed together (two label lists), with both entropies and the conditional entropy H(x | y), from the empirical joint distribution.

I(X;Y) = \sum_{ij} \frac{n_{ij}}{n}\log\frac{n\,n_{ij}}{a_i b_j}

Inputs

x values of the first variable (whole numbers) vector
y values of the second variable (whole numbers) vector
unit unit of information: nats (natural log), bits (log base 2) or bans (log base 10) choice

Returns

mutual_information I(x; y)
H_x entropy of x
H_y entropy of y
H_x_given_y H(x | y) = H(x) - I

Applies when

labels are whole numbers
equal lengths

Example

m.ask("Mutual information of labels: x = [0, 0, 1, 1, 2, 2], y = [0, 0, 1, 1, 1, 1], unit = bits")

→ mutual_information = 0.918296, H_x = 1.58496, H_y = 0.918296, H_x_given_y = 0.666667

Node impurity from class counts (entropy, Gini, misclassification)ml_impurity_from_counts

The impurity of a decision-tree node from its class counts: entropy in bits, Gini impurity and misclassification error.

G = 1 - \sum_k p_k^2,\quad H = -\sum_k p_k \log_2 p_k

Inputs

counts number of samples of each class in the node (whole numbers) vector

Returns

entropy_bits entropy in bits
gini 1 - sum p_k^2
misclassification 1 - max p_k

Applies when

counts >= 0 with a positive total

Example

m.ask("Impurity from class counts: counts = [9, 5]")

→ entropy_bits = 0.940286, gini = 0.459184, misclassification = 0.357143

Shannon entropy and perplexity of a distributionml_entropy

The Shannon entropy of a discrete probability distribution in the stated unit, its perplexity (e^H in nats = 2^H in bits), the maximum entropy log K and the normalised entropy H / log K.

H(p) = -\sum_i p_i \log p_i

Inputs

p probabilities (>= 0, sum to 1) vector
unit unit of information: nats (natural log), bits (log base 2) or bans (log base 10) choice

Returns

entropy H(p) in the stated unit
perplexity exp(H in nats)
max_entropy log K in the stated unit
normalized_entropy H / log K (0 for K = 1)

Applies when

probabilities >= 0 summing to 1

Example

m.ask("Entropy of a distribution: p = [0.5, 0.25, 0.25], unit = bits")

→ entropy = 1.5, perplexity = 2.82843, max_entropy = 1.58496, normalized_entropy = 0.946395

Linear models 6

Lasso regression coefficients (stated alpha)ml_lasso_regression

Fit lasso regression: minimise ||y - X w - b||^2 / (2n) + alpha ||w||_1 (the intercept is not penalised); give the coefficients (some exactly zero), the intercept and the number of non-zero coefficients.

\min_{w,b}\ \frac{1}{2n}\|y - Xw - b\|^2 + \alpha\|w\|_1

Inputs

X features: one row per sample matrix
y targets vector
alpha L1 penalty strength (> 0)

Returns

w coefficients
b intercept
n_nonzero number of non-zero coefficients

Applies when

alpha > 0
non-constant features
a non-degenerate solution

Example

m.ask("Lasso regression: X = [[1, 0.5], [2, 1.5], [3, 1], [4, 3], [5, 2], [6, 3.5]], y = [1.1, 2.3, 2.9, 4.2, 5.1, 5.8], alpha = 0.1")

→ w = [0.914285714286, 0.0], b = 0.366667, n_nonzero = 1

Linear SVM decision values, classes and marginml_linear_svm_decision

For a linear SVM with given weights and bias: the decision value w.x + b of each sample, its class (+1 when the value is positive, else -1), the margin width 2 / ||w|| and each sample's distance to the hyperplane.

f(x) = w^\top x + b,\quad \text{margin} = \frac{2}{\|w\|}

Inputs

w weights vector
b bias
X samples: one row each matrix

Returns

decision w.x + b
classes +1 if w.x + b > 0, else -1
margin_width 2 / ||w||
distances |w.x + b| / ||w||

Applies when

w is not zero

Example

m.ask("Linear SVM decision: w = [2, -1], b = -1, X = [[1, 0], [0, 1], [2, 2]]")

→ decision = [1.0, -2.0, 1.0], classes = [1, -1, 1], margin_width = 0.894427, distances = [0.4472135955, 0.894427191, 0.4472135955]

Logistic regression fit (L2, stated C)ml_logistic_regression_fit

Fit binary logistic regression with an L2 penalty: minimise 0.5 ||w||^2 + C sum of log-losses (the intercept is not penalised); give the weights, the intercept, the objective and the training accuracy.

\min_{w,b}\ \tfrac12\|w\|^2 + C\sum_i \log\left(1 + e^{-\tilde y_i (w^\top x_i + b)}\right)

Inputs

X features: one row per sample matrix
y labels (0 and 1) vector
C inverse regularisation strength (> 0)

Returns

w fitted weights
b fitted intercept
objective minimised objective
train_accuracy fraction of training samples classified correctly (z > 0 -> 1)

Applies when

labels 0 and 1, both present
C > 0
Newton's method converges

Example

m.ask("Logistic regression fit: X = [[0.5], [1], [1.5], [2], [2.5], [3], [3.5], [4]], y = [0, 0, 1, 0, 1, 0, 1, 1], C = 1")

→ w = [0.73877634783], b = -1.66225, objective = 4.65311, train_accuracy = 0.75

Logistic regression prediction from given weightsml_logistic_regression_predict

The logits, predicted probabilities of class 1 and predicted classes of a logistic regression model with given weights and intercept.

P(y = 1 \mid x) = \sigma(w^\top x + b)

Inputs

w weights vector
b intercept
X samples to predict: one row each matrix

Returns

probabilities sigma(w.x + b) for each sample
classes 1 when w.x + b > 0, else 0
logits w.x + b

Applies when

w has one weight per feature

Example

m.ask("Logistic regression prediction: w = [0.8, -0.4], b = -0.2, X = [[1, 2], [3, 1], [0, 0]]")

→ probabilities = [0.450166002688, 0.8581489351, 0.450166002688], classes = [0, 1, 0], logits = [-0.2, 1.8, -0.2]

Perceptron training epochml_perceptron_epoch

One pass of the perceptron learning rule through the samples in order (labels -1/+1): whenever y (w.x + b) <= 0, update w += lr y x and b += lr y; the final weights, bias and number of updates.

\text{if } y_i(w^\top x_i + b) \le 0:\ w \leftarrow w + \eta y_i x_i,\ b \leftarrow b + \eta y_i

Inputs

X samples in presentation order matrix
y labels (-1 or +1) vector
w initial weights vector
b initial bias
lr learning rate (> 0)

Returns

w_new weights after the epoch
b_new bias after the epoch
n_updates number of updates (mistakes, including y f = 0)

Applies when

labels -1 and +1
lr > 0

Example

m.ask("Perceptron epoch: X = [[2, 1], [-1, -2], [1, -1], [-2, 1]], y = [1, -1, 1, -1], w = [0, 0], b = 0, lr = 1")

→ w_new = [2.0, 1.0], b_new = 1, n_updates = 1

Ridge regression coefficients (stated alpha)ml_ridge_regression

Fit ridge regression: minimise ||y - X w - b||^2 + alpha ||w||^2 (the intercept is not penalised); give the coefficients, the intercept and the training MSE.

\hat w = (X_c^\top X_c + \alpha I)^{-1} X_c^\top y_c

Inputs

X features: one row per sample matrix
y targets vector
alpha L2 penalty strength (>= 0; 0 is least squares)

Returns

w coefficients
b intercept
train_mse training mean squared error

Applies when

alpha >= 0
a non-singular system (alpha > 0 always is)

Example

m.ask("Ridge regression: X = [[1], [2], [3], [4], [5]], y = [1.2, 1.9, 3.1, 3.9, 5.2], alpha = 1")

→ w = [0.909090909091], b = 0.332727, train_mse = 0.0349289

Loss functions and activations 13

Binary cross-entropy with logits and its gradientml_bce_with_logits

The mean binary cross-entropy computed directly from logits (stable for large |z|; soft targets allowed) and its gradient with respect to each logit.

\ell(z, y) = \max(z, 0) - z y + \log(1 + e^{-|z|}),\quad \frac{\partial \ell}{\partial z} = \sigma(z) - y

Inputs

y targets between 0 and 1 vector
z logits vector

Returns

loss mean of max(z, 0) - z y + log(1 + e^-|z|)
gradient dL/dz_i = (sigma(z_i) - y_i) / n

Applies when

0 <= y <= 1

Example

m.ask("Binary cross-entropy with logits: y = [1, 0, 1], z = [2, -1, -0.5]")

→ loss = 0.471422, gradient = [-0.0397343073407, 0.0896471404567, -0.207486443734]

Binary focal loss and its gradientml_focal_loss

The mean binary focal loss -alpha_t (1 - p_t)^gamma log p_t from logits (it down-weights easy examples) and its gradient with respect to each logit.

FL = -\alpha_t (1 - p_t)^\gamma \log p_t

Inputs

y labels (0 or 1) vector
z logits vector
gamma focusing parameter (>= 0; 0 gives weighted cross-entropy)
alpha weight of the positive class (0 to 1; negatives get 1 - alpha)

Returns

loss mean focal loss
gradient dL/dz_i

Applies when

labels 0 or 1
gamma >= 0
0 <= alpha <= 1

Example

m.ask("Focal loss: y = [1, 0, 1], z = [2, -1, -0.5], gamma = 2, alpha = 0.25")

→ loss = 0.0372658, gradient = [-0.000405911682949, 0.0131452824944, -0.0438459851562]

Cross-entropy loss from logits and its gradientml_cross_entropy_logits

The cross-entropy loss of one sample from its logits and true class, the predicted probability of that class, and the gradient of the loss with respect to the logits (softmax minus one-hot).

L = -\log\frac{e^{z_y}}{\sum_j e^{z_j}},\quad \frac{\partial L}{\partial z_j} = p_j - \mathbb{1}(j = y)

Inputs

z logits of the classes vector
y index of the true class (0 to K-1) integer

Returns

loss logsumexp(z) - z_y
probability softmax probability of the true class
gradient dL/dz = softmax(z) - onehot(y)

Applies when

y is a class index

Example

m.ask("Cross-entropy from logits: z = [2, 1, 0.1], y = 0")

→ loss = 0.41703, probability = 0.659001, gradient = [-0.340998861114, 0.242432970705, 0.0985658904093]

Cross-entropy with label smoothingml_label_smoothing_cross_entropy

The cross-entropy of logits against a smoothed target (1 - epsilon on the true class plus epsilon spread evenly over all K classes) and its gradient with respect to the logits.

q_j = \frac{\varepsilon}{K} + (1-\varepsilon)\mathbb{1}(j=y),\quad L = -\sum_j q_j \log \sigma(z)_j

Inputs

z logits vector
y index of the true class (0 to K-1) integer
epsilon smoothing amount (0 <= epsilon < 1)

Returns

loss -sum_j q_j log softmax(z)_j
gradient softmax(z) - q

Applies when

0 <= epsilon < 1
y is a class index

Example

m.ask("Label smoothing cross-entropy: z = [2, 1, 0.1], y = 0, epsilon = 0.1")

→ loss = 0.513697, gradient = [-0.274332194447, 0.209099637371, 0.065232557076]

Hinge and squared hinge lossml_hinge_loss

The mean hinge loss max(0, 1 - y f) of a margin classifier with labels -1/+1, the mean squared hinge loss and the number of margin violations.

\ell(y, f) = \max(0,\ 1 - y f)

Inputs

y_true true labels (-1 or +1) vector
scores decision values f(x) vector

Returns

hinge mean max(0, 1 - y f)
squared_hinge mean max(0, 1 - y f)^2
n_violations samples with y f < 1

Applies when

labels -1 or +1, both present

Example

m.ask("Hinge loss: y_true = [1, -1, 1, -1], scores = [2, -0.5, 0.3, 0.8]")

→ hinge = 0.75, squared_hinge = 0.995, n_violations = 3

Huber, smooth L1 and pseudo-Huber lossml_huber_loss

The mean Huber loss with threshold delta (quadratic for small errors, linear for large), the smooth L1 loss (Huber / delta) and the pseudo-Huber loss.

L_\delta(r) = \begin{cases} \tfrac12 r^2 & |r| \le \delta \\ \delta\,(|r| - \tfrac12\delta) & |r| > \delta \end{cases}

Inputs

y_true true values vector
y_pred predicted values vector
delta threshold between the quadratic and linear parts (delta > 0)

Returns

huber mean Huber loss
smooth_l1 mean smooth L1 loss with beta = delta
pseudo_huber mean pseudo-Huber loss

Applies when

delta > 0

Example

m.ask("Huber loss: y_true = [3, -0.5, 2, 7], y_pred = [2.5, 0, 2, 8], delta = 1")

→ huber = 0.1875, smooth_l1 = 0.1875, pseudo_huber = 0.16257

Log-softmaxml_log_softmax

The log-softmax of logits, z_i - log sum exp(z), computed stably even for very large logits.

\log\sigma(z)_i = z_i - m - \log\sum_j e^{z_j - m}

Inputs

z logits vector

Returns

log_probabilities z_i - logsumexp(z)
log_sum_exp logsumexp(z)

Applies when

at least two logits

Example

m.ask("Log-softmax: z = [1, 2, 3]")

→ log_probabilities = [-2.40760596444, -1.40760596444, -0.407605964444], log_sum_exp = 3.40761

Multi-class hinge loss (Crammer-Singer and Weston-Watkins)ml_multiclass_hinge_loss

The multi-class hinge losses of a decision matrix: Crammer-Singer max(0, 1 - f_y + max_{j != y} f_j) and Weston-Watkins sum_{j != y} max(0, 1 - f_y + f_j) / K, averaged over samples.

\ell_{CS} = \max\left(0,\ 1 - f_y + \max_{j \ne y} f_j\right),\quad \ell_{WW} = \frac{1}{K}\sum_{j \ne y}\max(0,\ 1 - f_y + f_j)

Inputs

y_true true class index of each sample (0 to K-1) vector
D decision values: one row per sample, one column per class (K >= 3) matrix

Returns

crammer_singer mean Crammer-Singer hinge loss
weston_watkins mean Weston-Watkins hinge loss (sum over wrong classes divided by K)

Applies when

K >= 3 classes
labels are column indices

Example

m.ask("Multiclass hinge loss: y_true = [0, 1, 2], D = [[1.2, 0.3, -0.5], [0.4, 0.9, 1.1], [-0.2, 0.1, 2]]")

→ crammer_singer = 0.433333, weston_watkins = 0.2

Quantile (pinball) lossml_quantile_loss

The mean pinball loss of predictions of the alpha-quantile: under-predictions cost alpha per unit, over-predictions 1 - alpha.

L_\alpha(y, q) = \max\{\alpha (y - q),\ (\alpha - 1)(y - q)\}

Inputs

y_true true values vector
y_pred predicted quantiles vector
alpha quantile level (0 < alpha < 1)

Returns

pinball mean pinball loss

Applies when

0 < alpha < 1

Example

m.ask("Pinball loss: y_true = [1, 2, 3], y_pred = [0, 2, 3], alpha = 0.1")

→ pinball = 0.0333333

Sigmoid (logistic function) and its derivativeml_sigmoid

The logistic sigmoid 1 / (1 + e^-z), its derivative sigma(1 - sigma) and log sigma(z), computed stably for large |z|.

\sigma(z) = \frac{1}{1 + e^{-z}},\quad \sigma'(z) = \sigma(z)\,(1 - \sigma(z))

Inputs

z input (logit)

Returns

sigmoid 1 / (1 + exp(-z))
derivative sigma(z)(1 - sigma(z))
log_sigmoid log sigma(z)

Applies when

finite z

Example

m.ask("Sigmoid: z = 0")

→ sigmoid = 0.5, derivative = 0.25, log_sigmoid = -0.693147

Softmaxml_softmax

The softmax of a vector of logits (numerically stable: the largest logit is subtracted first), with log-sum-exp.

\sigma(z)_i = \frac{e^{z_i - m}}{\sum_j e^{z_j - m}},\ m = \max_j z_j

Inputs

z logits (at least two) vector

Returns

probabilities softmax probabilities exp(z_i) / sum exp(z_j)
log_sum_exp log sum exp(z_j)

Applies when

at least two logits

Example

m.ask("Softmax: z = [1, 2, 3]")

→ probabilities = [0.0900305731704, 0.244728471055, 0.665240955775], log_sum_exp = 3.40761

Softmax with temperatureml_softmax_temperature

The softmax of logits divided by a temperature T (T > 1 flattens, T < 1 sharpens the distribution) and the entropy of the result in nats.

p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

Inputs

z logits vector
T temperature (T > 0)

Returns

probabilities softmax(z / T)
entropy entropy of the distribution in nats

Applies when

T > 0

Example

m.ask("Softmax with temperature: z = [1, 2, 3], T = 2")

→ probabilities = [0.186323723226, 0.307195885718, 0.506480391056], entropy = 1.02019

Triplet margin lossml_triplet_loss

The triplet margin loss max(0, d(a, p) - d(a, n) + margin) of an anchor, a positive and a negative embedding, with both Euclidean distances.

L = \max\left(0,\ \|a - p\|_2 - \|a - n\|_2 + m\right)

Inputs

anchor anchor embedding vector
positive embedding of the same identity vector
negative embedding of a different identity vector
margin margin (>= 0)

Returns

loss max(0, d_ap - d_an + margin)
d_ap anchor-positive distance
d_an anchor-negative distance

Applies when

equal lengths
margin >= 0

Example

m.ask("Triplet loss: anchor = [0, 0], positive = [1, 1], negative = [2, 0], margin = 1")

→ loss = 0.414214, d_ap = 1.41421, d_an = 2

Model selection 5

Bias-variance decomposition of predictionsml_bias_variance_decomposition

For predictions of several models (one row each, e.g. trained on different samples) at the same test points with known noise-free values: the squared bias, the variance and the expected squared error, which equals their sum.

E[(\hat f - f)^2] = (E[\hat f] - f)^2 + E[(\hat f - E[\hat f])^2]

Inputs

P predictions: one row per trained model, one column per test point (at least two rows) matrix
f true (noise-free) value at each test point vector

Returns

bias_squared mean over points of (mean prediction - f)^2
variance mean over points of the variance of the predictions (divide by the number of models)
expected_mse mean squared error over models and points = bias^2 + variance

Applies when

at least two models

Example

m.ask("Bias-variance decomposition: P = [[1, 2], [3, 2.5], [2, 3.5]], f = [1.5, 2.5]")

→ bias_squared = 0.138889, variance = 0.527778, expected_mse = 0.666667

k-fold cross-validated MSE of linear regressionml_kfold_cv_linear_regression

The mean squared error of ordinary least squares (with intercept) on each of k consecutive, unshuffled folds, trained on the other folds, and the mean over folds.

\mathrm{CV}_k = \frac{1}{k}\sum_{i=1}^k \mathrm{MSE}_i

Inputs

X features: one row per sample matrix
y targets vector
k number of folds integer

Returns

fold_mse test MSE of each fold
mean_mse mean of the fold MSEs

Applies when

every training set has at least d + 2 samples and full rank

Example

m.ask("K-fold cross-validation of linear regression: X = [[1], [2], [3], [4], [5], [6]], y = [1.1, 1.9, 3.2, 3.9, 5.1, 5.8], k = 3")

→ fold_mse = [0.0925, 0.0243295847751, 0.01845], mean_mse = 0.0450932

k-fold cross-validation fold sizesml_kfold_sizes

The test and training set sizes of each fold when n samples are split into k consecutive folds (the first n mod k folds get one extra sample), and the index range of the first test fold.

|F_i| = \lfloor n/k \rfloor + \mathbb{1}(i < n \bmod k)

Inputs

n number of samples integer
k number of folds integer

Returns

test_sizes size of each test fold
train_sizes size of each training set
first_test_fold first and last index (0-based) of the first test fold

Applies when

2 <= k <= n

Example

m.ask("K-fold sizes: n = 10, k = 3")

→ test_sizes = [4, 3, 3], train_sizes = [6, 7, 7], first_test_fold = [0, 3]

Leave-one-out cross-validation of linear regression (PRESS)ml_loocv_linear_regression

The leave-one-out cross-validated MSE and the PRESS statistic of ordinary least squares with intercept, from one fit through the leverage shortcut e_i / (1 - h_ii).

\mathrm{PRESS} = \sum_i \left(\frac{e_i}{1 - h_{ii}}\right)^2,\quad h_{ii} = x_i^\top (X^\top X)^{-1} x_i

Inputs

X features: one row per sample matrix
y targets vector

Returns

press sum of squared leave-one-out errors
loocv_mse PRESS / n

Applies when

at least d + 3 samples
no leverage equal to 1

Example

m.ask("Leave-one-out cross-validation of linear regression: X = [[1], [2], [3], [4], [5], [6]], y = [1.1, 1.9, 3.2, 3.9, 5.1, 5.8]")

→ press = 0.210911, loocv_mse = 0.0351518

Train/test split sizesml_train_test_split_sizes

The number of test and training samples when n samples are split with a given test fraction (the test size is rounded up).

n_{\mathrm{test}} = \lceil t\,n \rceil,\quad n_{\mathrm{train}} = n - n_{\mathrm{test}}

Inputs

n number of samples integer
test_size fraction of the samples used for testing (0 to 1, or a percentage)

Returns

n_test ceil(test_size * n)
n_train n - n_test

Applies when

0 < test_size < 1
a non-empty training set

Example

m.ask("Train test split sizes: n = 100, test_size = 0.2")

→ n_test = 20, n_train = 80

Neural networks 13

2-D convolution of an image with a kernel (deep-learning convention)ml_conv2d_forward

The output feature map of a single-channel 2-D convolution as computed in deep learning (cross-correlation: the kernel is not flipped), with a stride and zero padding.

Y_{ij} = \sum_{a,b} X_{is + a,\ js + b}\, K_{ab}

Inputs

image input image (rows of pixel values) matrix
kernel kernel (filter) weights matrix
stride stride integer
padding zero padding on each side integer

Returns

output output feature map
output_sum sum of all outputs

Applies when

the kernel fits in the padded image

Example

m.ask("2D convolution: image = [[1, 2, 3, 0], [0, 1, 2, 3], [3, 0, 1, 2], [2, 3, 0, 1]], kernel = [[2, 0, 1], [0, 1, 2], [1, 0, 2]], stride = 1, padding = 0")

→ output = [[15.0, 16.0], [6.0, 15.0]], output_sum = 52

Backpropagation gradients of a one-hidden-layer networkml_backpropagation

One backpropagation pass of a dense network with one hidden layer: the loss (mean squared error with identity outputs, or cross-entropy of softmax outputs against a target distribution) and its gradients with respect to every weight and bias.

\delta_2 = \frac{\partial L}{\partial z_2},\ \nabla_{W_2} = \delta_2 h^\top,\ \delta_1 = (W_2^\top \delta_2) \odot \phi'(z_1),\ \nabla_{W_1} = \delta_1 x^\top

Inputs

x input vector vector
t target: values for mse, a probability distribution for cross-entropy vector
W1 hidden-layer weights (rows = hidden units, columns = inputs) matrix
b1 hidden-layer biases vector
activation hidden activation: relu, tanh or sigmoid choice
W2 output-layer weights (rows = outputs, columns = hidden units) matrix
b2 output biases vector
loss loss: mse (identity outputs, mean over outputs) or cross_entropy (softmax outputs) choice

Returns

loss loss value
dW1 dL/dW1
db1 dL/db1
dW2 dL/dW2
db2 dL/db2

Applies when

shapes agree
no ReLU pre-activation exactly 0
cross-entropy: target sums to 1, at least two outputs

Example

m.ask("Backpropagation gradients: x = [1, 2], t = [0.5], W1 = [[0.5, -0.25], [0.1, 0.3]], b1 = [0.1, -0.2], activation = relu, W2 = [[1, -1]], b2 = [0.05], loss = mse")

→ loss = 0.7225, dW1 = [[-1.7, -3.4], [1.7, 3.4]], db1 = [-1.7, 1.7], dW2 = [[-0.17, -0.85]], db2 = [-1.7]

Batch normalisation forward pass (training mode)ml_batch_norm_forward

Batch normalisation of a mini-batch in training mode: each feature is normalised with the batch mean and the biased batch variance plus eps, then scaled by gamma and shifted by beta.

\hat x = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \varepsilon}},\quad y = \gamma \hat x + \beta

Inputs

X mini-batch: one row per sample, one column per feature (at least two rows) matrix
gamma scale per feature vector
beta shift per feature vector
eps small constant added to the variance (e.g. 1e-5)

Returns

output gamma (x - mean)/sqrt(var + eps) + beta
batch_mean batch mean of each feature
batch_var biased batch variance of each feature

Applies when

at least two samples
var + eps > 0

Example

m.ask("Batch normalization: X = [[1, 2], [3, 6], [5, 10]], gamma = [1, 1], beta = [0, 0], eps = 0")

→ output = [[-1.22474487139, -1.22474487139], [0.0, 0.0], [1.22474487139, 1.22474487139]], batch_mean = [3.0, 6.0], batch_var = [2.66666666667, 10.6666666667]

Convolution output sizeml_conv_output_size

The output length (per spatial dimension) of a convolution or pooling window from the input size, kernel size, stride, zero padding and dilation.

n_{\mathrm{out}} = \left\lfloor \frac{n + 2p - d(k-1) - 1}{s} \right\rfloor + 1

Inputs

n input size integer
kernel kernel size integer
stride stride integer
padding zero padding on each side integer
dilation dilation (1 = ordinary) integer

Returns

output_size floor((n + 2p - d(k - 1) - 1) / s) + 1
effective_kernel d(k - 1) + 1

Applies when

the dilated kernel fits in the padded input

Example

m.ask("Convolution output size: n = 32, kernel = 3, stride = 1, padding = 1, dilation = 1")

→ output_size = 32, effective_kernel = 3

Forward pass of a one-hidden-layer dense networkml_dense_forward_pass

The forward pass of a dense network x -> activation(W1 x + b1) -> W2 h + b2 with an identity, sigmoid or softmax output: the hidden activations and the outputs.

h = \phi(W_1 x + b_1),\quad \hat y = g(W_2 h + b_2)

Inputs

x input vector vector
W1 hidden-layer weights (rows = hidden units, columns = inputs) matrix
b1 hidden-layer biases vector
activation hidden activation: relu, tanh or sigmoid choice
W2 output-layer weights (rows = outputs, columns = hidden units) matrix
b2 output biases vector
output output activation: identity, sigmoid or softmax choice

Returns

hidden hidden activations h
output network outputs
max_output largest output

Applies when

shapes agree (W1: hidden x inputs, W2: outputs x hidden)

Example

m.ask("Dense network forward pass: x = [1, 2], W1 = [[0.5, -0.25], [0.1, 0.3]], b1 = [0.1, -0.2], activation = relu, W2 = [[1, -1]], b2 = [0.05], output = identity")

→ hidden = [0.1, 0.5], output = [-0.35], max_output = -0.35

Max pooling and average poolingml_pooling

Max pooling and average pooling of a 2-D feature map with a square window, a stride and no padding.

Y_{ij} = \max_{a,b < k} X_{is+a,\ js+b}

Inputs

X feature map matrix
size window size integer
stride stride integer

Returns

max_pool maximum of each window
avg_pool mean of each window

Applies when

the window fits in the input

Example

m.ask("Max pooling: X = [[1, 3, 2, 4], [5, 6, 1, 2], [7, 2, 9, 0], [1, 8, 3, 4]], size = 2, stride = 2")

→ max_pool = [[6.0, 4.0], [8.0, 9.0]], avg_pool = [[3.75, 2.25], [4.5, 4.0]]

Multi-head attention (identity projections)ml_multihead_attention

Multi-head attention with identity input and output projections: the model dimension is split into equal heads, scaled dot-product attention runs in each head (scale 1/sqrt(head size)) and the head outputs are concatenated.

\mathrm{MHA} = [\mathrm{head}_1, \dots, \mathrm{head}_h],\quad \mathrm{head}_i = \mathrm{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{d/h}}\right)V_i

Inputs

Q queries (rows), width = model dimension matrix
K keys matrix
V values matrix
heads number of heads (divides the model dimension) integer

Returns

output concatenated head outputs
output_sum sum of all output entries

Applies when

equal widths
model dimension divisible by heads

Example

m.ask("Multi-head attention: Q = [[1, 0, 1, 0]], K = [[1, 0, 0, 1], [0, 1, 1, 0]], V = [[1, 2, 3, 4], [5, 6, 7, 8]], heads = 2")

→ output = [[2.32095380269, 3.32095380269, 5.67904619731, 6.67904619731]], output_sum = 18

Parameter count of a 2-D convolution layerml_conv_parameter_count

The number of weights, biases and total parameters of a 2-D convolution layer, including grouped and depthwise convolutions.

P = C_{\mathrm{out}}\,\frac{C_{\mathrm{in}}}{g}\,k_h k_w + C_{\mathrm{out}}

Inputs

in_channels input channels integer
out_channels output channels (filters) integer
kernel_h kernel height integer
kernel_w kernel width integer
groups groups (1 = ordinary; in_channels = depthwise) integer
bias whether the layer has a bias per filter boolean

Returns

weights C_out (C_in / g) k_h k_w
biases C_out or 0
total weights + biases

Applies when

channels divisible by groups

Example

m.ask("Convolution parameter count: in_channels = 3, out_channels = 64, kernel_h = 3, kernel_w = 3, groups = 1, bias = yes")

→ weights = 1728, biases = 64, total = 1792

Parameter count of a dense (fully connected) networkml_dense_parameter_count

The number of trainable parameters of each dense layer and of the whole network, from the layer sizes (input size first), with or without biases.

P = \sum_l (n_{l-1} n_l + n_l)

Inputs

sizes layer sizes from input to output, e.g. [784, 128, 10] vector
bias whether every layer has biases (yes or no) boolean

Returns

per_layer parameters of each layer: n_in n_out (+ n_out biases)
total total parameters

Applies when

positive whole-number sizes

Example

m.ask("Dense network parameter count: sizes = [784, 128, 10], bias = yes")

→ per_layer = [100480, 1290], total = 101770

Parameter count of a transformer encoderml_transformer_parameter_count

The parameters of a standard transformer encoder layer (multi-head self-attention with biases, a two-layer feed-forward block and two layer norms) and of a stack of such layers.

P_{\mathrm{layer}} = 4d^2 + 4d + 2 d\,d_{ff} + d_{ff} + d + 4d

Inputs

d_model model (embedding) dimension integer
n_heads attention heads (divide d_model) integer
d_ff feed-forward hidden size integer
n_layers number of layers integer

Returns

attention 4 d^2 + 4 d (Q, K, V and output projections with biases)
feed_forward 2 d d_ff + d_ff + d
layer_norms 4 d
per_layer parameters per layer
total parameters of the stack

Applies when

d_model divisible by n_heads

Example

m.ask("Transformer parameter count: d_model = 512, n_heads = 8, d_ff = 2048, n_layers = 6")

→ attention = 1.05062e+06, feed_forward = 2.09971e+06, layer_norms = 2048, per_layer = 3.15238e+06, total = 1.89143e+07

Parameter count of an LSTM, GRU or simple RNNml_rnn_parameter_count

The trainable parameters of each layer and of a stacked recurrent network (LSTM: 4 gates, GRU: 3, simple RNN: 1), with two bias vectors per gate as in PyTorch, optionally bidirectional.

P = G\,h\,(n_{\mathrm{in}} + h) + 2Gh

Inputs

cell cell type: lstm, gru or rnn choice
input_size input features integer
hidden_size hidden units integer
num_layers stacked layers integer
bidirectional bidirectional (yes or no) boolean

Returns

per_layer parameters of each layer (both directions)
total total parameters

Applies when

positive whole-number sizes

Example

m.ask("RNN parameter count: cell = lstm, input_size = 100, hidden_size = 128, num_layers = 1, bidirectional = no")

→ per_layer = [117760], total = 117760

Receptive field of stacked convolutionsml_receptive_field

The receptive field (how many input positions one output depends on, end to end) and the jump (input distance between adjacent outputs) of a stack of convolution or pooling layers with given kernel sizes, strides and dilations.

r = 1 + \sum_l (k_l - 1)\, d_l \prod_{i < l} s_i

Inputs

kernels kernel size of each layer (whole numbers) vector
strides stride of each layer vector
dilations dilation of each layer (1 = ordinary) vector

Returns

receptive_field 1 + sum (k_l - 1) d_l prod_{i<l} s_i
jump product of the strides

Applies when

one kernel, stride and dilation per layer
at most 20 layers

Example

m.ask("Receptive field: kernels = [3, 3, 3], strides = [1, 1, 1], dilations = [1, 1, 1]")

→ receptive_field = 7, jump = 1

Scaled dot-product attentionml_scaled_dot_product_attention

Scaled dot-product attention softmax(Q K^T / sqrt(d)) V for given query, key and value matrices, optionally with a causal mask (query i sees keys 0..i): the attention weights and the output.

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Inputs

Q queries: one row per query matrix
K keys: one row per key matrix
V values: one row per key matrix
causal apply a causal mask (yes or no) boolean

Returns

weights attention weights (rows sum to 1)
output weights V
output_sum sum of all output entries

Applies when

Q and K have the same width
K and V have the same rows
causal: as many queries as keys

Example

m.ask("Scaled dot-product attention: Q = [[1, 0], [0, 1]], K = [[1, 0], [0, 1], [1, 1]], V = [[1, 2], [3, 4], [5, 6]], causal = no")

→ weights = [[0.40111209268, 0.19777581464, 0.40111209268], [0.19777581464, 0.40111209268, 0.40111209268]], output = [[3.0, 4.0], [3.40667255608, 4.40667255608]], output_sum = 14.8133

Optimizers 6

Adam optimizer: one updateml_adam_step

One Adam update: the new first and second moment estimates and the parameters after the bias-corrected step w - lr m_hat / (sqrt(v_hat) + eps).

\hat m = \frac{m_t}{1 - \beta_1^t},\ \hat v = \frac{v_t}{1 - \beta_2^t},\quad w_t = w_{t-1} - \eta \frac{\hat m}{\sqrt{\hat v} + \varepsilon}

Inputs

w parameters vector
grad gradient vector
m first-moment estimate before the step (zeros at the start) vector
v second-moment estimate before the step (zeros at the start) vector
t step number after this update (1 for the first step) integer
lr learning rate
beta1 first-moment decay (e.g. 0.9)
beta2 second-moment decay (e.g. 0.999)
eps denominator constant (e.g. 1e-8)

Returns

m_new beta1 m + (1 - beta1) g
v_new beta2 v + (1 - beta2) g^2
w_new parameters after the step

Applies when

t >= 1
0 <= beta1, beta2 < 1
v >= 0
equal lengths

Example

m.ask("Adam step: w = [1, -2], grad = [0.5, -1], m = [0, 0], v = [0, 0], t = 1, lr = 0.001, beta1 = 0.9, beta2 = 0.999, eps = 1e-08")

→ m_new = [0.05, -0.1], v_new = [0.00025, 0.001], w_new = [0.99900000002, -1.99900000001]

AdamW optimizer (decoupled weight decay): one updateml_adamw_step

One AdamW update: the parameters first decay by the factor (1 - lr weight_decay), then take the bias-corrected Adam step.

w \leftarrow w(1 - \eta\lambda),\quad w_t = w - \eta \frac{\hat m}{\sqrt{\hat v} + \varepsilon}

Inputs

w parameters vector
grad gradient vector
m first-moment estimate before the step (zeros at the start) vector
v second-moment estimate before the step (zeros at the start) vector
t step number after this update (1 for the first step) integer
lr learning rate
beta1 first-moment decay (e.g. 0.9)
beta2 second-moment decay (e.g. 0.999)
eps denominator constant (e.g. 1e-8)
weight_decay decoupled weight decay (e.g. 0.01)

Returns

m_new beta1 m + (1 - beta1) g
v_new beta2 v + (1 - beta2) g^2
w_new parameters after decay and step

Applies when

t >= 1
0 <= beta1, beta2 < 1
v >= 0
weight_decay >= 0

Example

m.ask("AdamW step: w = [1, -2], grad = [0.5, -1], m = [0, 0], v = [0, 0], t = 1, lr = 0.001, beta1 = 0.9, beta2 = 0.999, eps = 1e-08, weight_decay = 0.01")

→ m_new = [0.05, -0.1], v_new = [0.00025, 0.001], w_new = [0.99899000002, -1.99898000001]

Cosine-annealing learning-rate scheduleml_cosine_lr_schedule

The learning rate at a given epoch under cosine annealing from lr_max down to eta_min over T_max epochs.

\eta_t = \eta_{\min} + \tfrac12(\eta_{\max} - \eta_{\min})\left(1 + \cos\frac{\pi t}{T}\right)

Inputs

lr_max initial (maximum) learning rate
eta_min final (minimum) learning rate
T_max length of the schedule in epochs (at most 1000) integer
epoch epoch number (0 to T_max) integer

Returns

lr eta_min + (lr_max - eta_min)(1 + cos(pi t / T_max)) / 2

Applies when

0 <= eta_min < lr_max
0 <= epoch <= T_max <= 1000

Example

m.ask("Cosine learning rate schedule: lr_max = 0.1, eta_min = 0, T_max = 100, epoch = 50")

→ lr = 0.05

Linear warm-up followed by cosine decayml_warmup_cosine_lr_schedule

The learning rate at a given step of the common transformer schedule: linear warm-up from start_factor lr_max to lr_max over the warm-up steps, then cosine decay to eta_min at the last step.

\eta_t = \begin{cases} \eta_{\max}\left(f + (1-f)\frac{t}{W}\right) & t < W \\ \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})\left(1 + \cos\frac{\pi (t - W)}{T - W}\right) & t \ge W \end{cases}

Inputs

lr_max peak learning rate
start_factor fraction of lr_max at step 0 (0 < f <= 1)
warmup warm-up steps integer
total total steps (at most 1000) integer
eta_min final learning rate
step step number (0 to total) integer

Returns

lr learning rate at the step

Applies when

1 <= warmup < total <= 1000
0 <= step <= total

Example

m.ask("Warmup cosine schedule: lr_max = 0.001, start_factor = 0.1, warmup = 10, total = 100, eta_min = 0, step = 5")

→ lr = 0.00055

SGD with momentum: one updateml_sgd_momentum_step

One step of stochastic gradient descent with momentum in PyTorch's form (optional Nesterov and weight decay): d = g + wd w, buf = mu buf + d, w = w - lr (buf, or d + mu buf with Nesterov).

b_t = \mu b_{t-1} + g_t,\quad w_t = w_{t-1} - \eta\, b_t

Inputs

w parameters vector
grad gradient vector
buf momentum buffer before the step (zeros at the start) vector
lr learning rate
momentum momentum mu (0 <= mu < 1)
nesterov use Nesterov momentum (yes or no) boolean
weight_decay L2 weight decay (>= 0)

Returns

buf_new momentum buffer after the step
w_new parameters after the step

Applies when

lr > 0
0 <= momentum < 1
weight_decay >= 0
equal lengths

Example

m.ask("SGD momentum step: w = [1, -2], grad = [0.5, -1], buf = [0.1, 0.2], lr = 0.1, momentum = 0.9, nesterov = no, weight_decay = 0")

→ buf_new = [0.59, -0.82], w_new = [0.941, -1.918]

Step learning-rate scheduleml_step_lr_schedule

The learning rate at a given epoch under step decay: multiplied by gamma every step_size epochs.

\eta_t = \eta_0\, \gamma^{\lfloor t / s \rfloor}

Inputs

lr0 initial learning rate
step_size epochs between decays integer
gamma decay factor (0 < gamma <= 1)
epoch epoch number (0 = start, at most 1000) integer

Returns

lr lr0 gamma^floor(epoch / step_size)
decays number of decays so far

Applies when

lr0 > 0
0 < gamma <= 1
0 <= epoch <= 1000

Example

m.ask("Step learning rate schedule: lr0 = 0.1, step_size = 30, gamma = 0.1, epoch = 65")

→ lr = 0.001, decays = 2

Preprocessing 4

Label encoding and one-hot encoding of categoriesml_one_hot_encoding

Encode a list of category values: the categories in sorted order, how often each occurs, the integer code (label encoding) of each value and its one-hot row.

\mathrm{onehot}(v)_j = \mathbb{1}(v = c_j)

Inputs

categories category values, written as a bracketed list of words: [red, green, red] text

Returns

categories distinct categories, sorted
counts occurrences of each category
codes label code of each value (position of its category)
one_hot one-hot row of each value

Applies when

non-empty category words

Example

m.ask("One-hot encoding: categories = [red, green, red, blue, green, red]")

→ categories = ['blue', 'green', 'red'], counts = [1, 2, 3], codes = [2, 1, 2, 0, 1, 2], one_hot = [[0, 0, 1], [0, 1, 0], [0, 0, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]]

Min-max scaling to [0, 1]ml_minmax_scaler

Scale each feature column linearly to the range [0, 1] using its minimum and maximum (a constant column scales to 0).

x' = \frac{x - \min_j}{\max_j - \min_j}

Inputs

X data: one row per sample, one column per feature matrix

Returns

mins column minima
maxs column maxima
scaled (x - min) / (max - min)

Applies when

a constant column scales to 0

Example

m.ask("Min-max scaler: X = [[1, 10], [2, 20], [3, 40], [5, 30]]")

→ mins = [1.0, 10.0], maxs = [5.0, 40.0], scaled = [[0.0, 0.0], [0.25, 0.333333333333], [0.5, 1.0], [1.0, 0.666666666667]]

Robust scaling (median and interquartile range)ml_robust_scaler

Scale each feature by subtracting its median and dividing by its interquartile range (quartiles by linear interpolation), which resists outliers; a column with zero IQR is only centred.

x' = \frac{x - \mathrm{median}}{Q_3 - Q_1}

Inputs

X data: one row per sample, one column per feature matrix

Returns

medians column medians
iqr column interquartile ranges Q3 - Q1
scaled (x - median) / IQR

Applies when

zero IQR: centred only

Example

m.ask("Robust scaler: X = [[1], [2], [3], [4], [100]]")

→ medians = [3.0], iqr = [2.0], scaled = [[-1.0], [-0.5], [0.0], [0.5], [48.5]]

Standardisation (z-scores of features)ml_standard_scaler

Standardise each feature column to mean 0 and unit variance: the column means, the population standard deviations and the scaled matrix (a constant column scales to 0).

z_{ij} = \frac{x_{ij} - \bar x_j}{s_j},\quad s_j^2 = \frac{1}{n}\sum_i (x_{ij} - \bar x_j)^2

Inputs

X data: one row per sample, one column per feature matrix

Returns

means column means
stds population standard deviations (divide by n)
scaled (x - mean) / std

Applies when

a constant column scales to 0

Example

m.ask("Standard scaler: X = [[1, 10], [2, 20], [3, 30], [4, 40]]")

→ means = [2.5, 25.0], stds = [1.11803398875, 11.1803398875], scaled = [[-1.3416407865, -1.3416407865], [-0.4472135955, -0.4472135955], [0.4472135955, 0.4472135955], [1.3416407865, 1.3416407865]]

Ranking metrics 4

Mean average precision (MAP) of rankingsml_mean_average_precision

The average precision of each query's ranked list (mean of precision at the rank of each relevant item, over the relevant items in the list) and their mean, MAP.

\mathrm{AP} = \frac{1}{|\mathrm{rel}|}\sum_{k:\ \mathrm{rel}_k = 1} P@k,\quad \mathrm{MAP} = \frac{1}{Q}\sum_q \mathrm{AP}_q

Inputs

R relevance (0 or 1) in rank order: one row per query, rows of equal length matrix

Returns

average_precision AP of each query
map mean of the APs

Applies when

relevance 0 or 1
every query has a relevant item

Example

m.ask("Mean average precision: R = [[1, 0, 1, 0, 0], [0, 1, 0, 0, 1]]")

→ average_precision = [0.833333333333, 0.45], map = 0.641667

Mean reciprocal rank (MRR)ml_mean_reciprocal_rank

The reciprocal rank of the first relevant item of each query's ranked list (0 when none is relevant) and their mean, the MRR.

\mathrm{MRR} = \frac{1}{Q}\sum_q \frac{1}{\mathrm{rank}_q}

Inputs

R relevance (0 or 1) in rank order: one row per query matrix

Returns

reciprocal_ranks 1 / rank of the first relevant item of each query
mrr mean reciprocal rank

Applies when

relevance 0 or 1

Example

m.ask("Mean reciprocal rank: R = [[0, 0, 1], [0, 1, 0], [1, 0, 0]]")

→ reciprocal_ranks = [0.333333333333, 0.5, 1.0], mrr = 0.611111

NDCG@k (normalised discounted cumulative gain)ml_ndcg_at_k

DCG@k of a ranked list with graded relevance (linear gain rel or exponential gain 2^rel - 1, discount 1/log2(rank + 1)), the ideal DCG@k of the same items sorted by relevance, and NDCG@k = DCG/IDCG.

\mathrm{DCG}@k = \sum_{i=1}^k \frac{g_i}{\log_2(i + 1)},\quad \mathrm{NDCG}@k = \frac{\mathrm{DCG}@k}{\mathrm{IDCG}@k}

Inputs

rel relevance grade of each item in rank order (>= 0) vector
k cut-off rank integer
gain gain: linear (rel) or exponential (2^rel - 1) choice

Returns

dcg DCG@k
idcg ideal DCG@k
ndcg DCG@k / IDCG@k

Applies when

grades >= 0
1 <= k <= list length
at least one relevant item

Example

m.ask("NDCG at k: rel = [3, 2, 3, 0, 1, 2], k = 6, gain = linear")

→ dcg = 6.86113, idcg = 7.141, ndcg = 0.960808

Precision@k and recall@k of a rankingml_precision_recall_at_k

Precision at k and recall at k of a ranked list (relevance 0/1 in rank order), with the number of relevant items in the top k and the hit indicator.

P@k = \frac{|\mathrm{rel} \cap \mathrm{top}_k|}{k},\quad R@k = \frac{|\mathrm{rel} \cap \mathrm{top}_k|}{|\mathrm{rel}|}

Inputs

rel relevance of each ranked item, in rank order (0 or 1) vector
n_relevant total number of relevant items for the query integer
k cut-off rank integer

Returns

precision_at_k relevant in top k / k
recall_at_k relevant in top k / n_relevant
hits relevant items in the top k
hit_rate 1 if any relevant item is in the top k

Applies when

relevance 0 or 1
1 <= k <= list length
n_relevant >= relevant items in the list

Example

m.ask("Precision at k: rel = [1, 0, 1, 1, 0, 0, 1], n_relevant = 5, k = 5")

→ precision_at_k = 0.6, recall_at_k = 0.6, hits = 3, hit_rate = 1

Regression metrics 5

Adjusted R^2ml_adjusted_r2

R^2 and the adjusted R^2 that penalises the number of predictors p of the model.

\bar R^2 = 1 - (1 - R^2)\frac{n-1}{n-p-1}

Inputs

y_true true values vector
y_pred fitted values vector
p number of predictors (excluding the intercept) integer

Returns

r2 R^2
adjusted_r2 1 - (1 - R^2)(n - 1)/(n - p - 1)

Applies when

n - p - 1 > 0
true values not all equal

Example

m.ask("Adjusted R squared: y_true = [10, 20, 30, 40, 50], y_pred = [12, 18, 33, 37, 52], p = 1")

→ r2 = 0.97, adjusted_r2 = 0.96

Mean absolute percentage errorml_mape

The mean absolute percentage error of predictions, as a fraction and in per cent.

\mathrm{MAPE} = \frac{1}{n}\sum_i \frac{|y_i - \hat y_i|}{|y_i|}

Inputs

y_true true values (none equal to 0) vector
y_pred predicted values vector

Returns

mape mean of |y - y_hat| / |y| (a fraction)
mape_percent the same in per cent

Applies when

no true value equal to 0

Example

m.ask("Mean absolute percentage error: y_true = [100, 200, 300], y_pred = [110, 190, 330]")

→ mape = 0.0833333, mape_percent = 8.33333

Mean squared logarithmic errorml_mean_squared_log_error

The mean squared logarithmic error ln(1 + y) - ln(1 + y_hat) and its square root (RMSLE), for non-negative targets such as counts and prices.

\mathrm{MSLE} = \frac{1}{n}\sum_i \left(\ln(1+y_i) - \ln(1+\hat y_i)\right)^2

Inputs

y_true true values (>= 0) vector
y_pred predicted values (>= 0) vector

Returns

msle mean squared log error
rmsle root mean squared log error

Applies when

values >= 0

Example

m.ask("Mean squared log error: y_true = [3, 5, 2.5, 7], y_pred = [2.5, 5, 4, 8]")

→ msle = 0.0397301, rmsle = 0.199324

Mean Tweedie deviance (normal, Poisson, gamma, compound Poisson-gamma)ml_tweedie_deviance

The mean Tweedie deviance of predictions for a stated power: 0 squared error, 1 Poisson, 2 gamma, 3 inverse Gaussian, between 1 and 2 compound Poisson-gamma.

d_p(y, \mu) = 2\left(\frac{y^{2-p}}{(1-p)(2-p)} - \frac{y\mu^{1-p}}{1-p} + \frac{\mu^{2-p}}{2-p}\right)

Inputs

y_true true values vector
y_pred predicted means (positive for power > 0) vector
power Tweedie power p (0, or p >= 1)

Returns

deviance mean Tweedie deviance

Applies when

power 0 or >= 1
positive predictions (power > 0)
y >= 0 (1 <= p < 2) or y > 0 (p >= 2)

Example

m.ask("Tweedie deviance: y_true = [2, 0, 1, 4], y_pred = [0.5, 0.5, 2, 2], power = 1")

→ deviance = 1.42602

Regression metrics (MAE, MSE, RMSE, R^2, explained variance)ml_regression_metrics

The standard regression error metrics of predictions against true values: mean absolute error, mean squared error, root mean squared error, R^2, explained variance, median absolute error and maximum error.

R^2 = 1 - \frac{\sum (y_i - \hat y_i)^2}{\sum (y_i - \bar y)^2},\quad \mathrm{RMSE} = \sqrt{\tfrac{1}{n}\sum (y_i - \hat y_i)^2}

Inputs

y_true true values vector
y_pred predicted values vector

Returns

mae mean absolute error
mse mean squared error
rmse root mean squared error
r2 coefficient of determination 1 - SS_res / SS_tot
explained_variance 1 - Var(y - y_hat) / Var(y)
median_ae median absolute error
max_error largest absolute error

Applies when

at least two values
the true values are not all equal

Example

m.ask("Regression metrics: y_true = [3, -0.5, 2, 7], y_pred = [2.5, 0, 2, 8]")

→ mae = 0.5, mse = 0.375, rmse = 0.612372, r2 = 0.948608, explained_variance = 0.957173, median_ae = 0.5, max_error = 1

Versions & files

Releases and repository files

Files

15 files, 44.4 MB in total. The largest:

File	Size
model.safetensors	18.9 MB
dist/data_science_ml_ai999x-1.0.1-py3-none-any.whl	15.4 MB
onnx/search/encoder.int8.onnx	4.9 MB
docs/data_science_ml_ai999x_capabilities.pdf	3.9 MB
onnx/search/index.json	403.3 KB
docs/data_science_ml_ai999x_capabilities.html	365 KB
docs/data_science_ml_ai999x_capabilities.md	283.8 KB
onnx/search/tokenizer.json	176.4 KB
README.md	25.3 KB
LICENSE	11.4 KB
evaluation/release_gate.md	2.3 KB
evaluation/KNOWLEDGE_CARD.md	1.4 KB

Browse every file ↗

Questions

Frequently asked

What does data-science-ml-ai999x do?

Data Science and Machine Learning answers questions in data science and machine learning: evaluation metrics, loss functions and gradients, classical machine-learning algorithms, dimensionality reduction, cross-validation, information theory and neural-network arithmetic. Each answer gives the value with its unit, the steps and the formula in LaTeX.

How do I install data-science-ml-ai999x?

With pip: pip install data-science-ml-ai999x. The same files are on Hugging Face (hf download AI999X/data-science-ml-ai999x --local-dir data-science-ml-ai999x) and GitHub.

Does it run offline?

Yes. It is a Python package that runs on an ordinary CPU without a network connection (Python >=3.10); no GPU and no API key are needed.

How many capabilities does data-science-ml-ai999x have?

107 capabilities in 14 topics, including Classification metrics, Classical algorithms, Loss functions and activations, Neural networks, Clustering metrics.

Which field and tasks does it cover?

Field: Data Science & Machine Learning. Tasks: Question Answering, Token Classification, Text Classification, Tabular Classification, Tabular Regression, Sentence Similarity, Feature Extraction.

What does an answer look like?

Asked "Binary metrics from counts: TP = 40, FP = 10, FN = 5, TN = 45", it answers: accuracy = 0.85, precision = 0.8, recall = 0.888889, specificity = 0.818182, f1 = 0.842105, npv = 0.9, balanced_accuracy = 0.853535, fpr = 0.181818, fnr = 0.111111, jaccard = 0.727273.

What happens when a question cannot be answered?

It says so and names what is missing or out of range, instead of guessing a number.

Explore every model.

More in Data Science & Machine Learning All models Capability orb