Kolmogorov-Arnold networks with interchangeable edge bases

library(thangldKANr)

One engine, many edge types

A Kolmogorov-Arnold network computes, in each layer, \(h^{(l+1)}_j = \sum_i \phi_{j,i}(h^{(l)}_i)\). The variants published since Liu et al. (2024) differ only in how an edge function \(\phi_{j,i}\) is written: a Chebyshev, Legendre or Jacobi expansion of \(\tanh x\), Gaussian radial basis functions (FastKAN), Mexican-hat wavelets (Wav-KAN), safe Pade rational functions, a hybrid SiLU + Chebyshev + RBF sum, or squared ReLU bumps (ReLU-KAN). thangldKANr keeps one engine (training loop with both backends, preprocessing, early stopping, prediction, edge plots, persistence, validation) and treats the edge type as an argument.

kan_bases()[, c("name", "label")]
#>        name                                     label
#> 1 chebyshev          Chebyshev polynomials (ChebyKAN)
#> 2       rbf Gaussian radial basis functions (FastKAN)
#> 3  legendre                      Legendre polynomials
#> 4   wavelet            Mexican-hat wavelets (Wav-KAN)
#> 5  rational              Safe Pade rational functions
#> 6    hybrid           SiLU + Chebyshev + RBF (hybrid)
#> 7    jacobi            Jacobi polynomials (JacobiKAN)
#> 8      relu             Squared ReLU bumps (ReLU-KAN)

Layers are stacked; interactions between inputs arise through composition, so a one-layer network is an additive model and should be reported as such.

Fitting and comparing bases

Inputs and response are standardized with constants estimated on the training rows; predictions are returned on the original scale. The base R backend (backend = "r") is used here so the vignette builds without libtorch; backend = "torch" (the default) uses R ‘torch’ autograd and follows the same trajectory in double precision.

set.seed(1)
n <- 400
x <- matrix(runif(2 * n, -2, 2), n, 2, dimnames = list(NULL, c("x1", "x2")))
f <- function(x) x[, "x1"] * x[, "x2"] + sin(x[, "x1"])
y <- f(x) + rnorm(n, sd = 0.05)
train <- 1:240
valid <- 241:300
test <- 301:400
fit_one <- function(basis, ...) {
  thangld_kan_fit(thangld_kan(c(2, 4, 1), basis = basis, ..., seed = 1),
                x[train, ], y[train],
                validation_data = list(x = x[valid, ], y = y[valid]),
                epochs = 150, batch_size = 32, lr = 0.02, patience = 30,
                seed = 1, backend = "r")
}
fits <- list(chebyshev = fit_one("chebyshev", degree = 4),
             rbf = fit_one("rbf"),
             relu = fit_one("relu"))
t(sapply(fits, function(m) thangld_kan_metrics(y[test], predict(m, x[test, ]))))
#>                rmse        mae        r2
#> chebyshev 0.1258298 0.09181367 0.9934847
#> rbf       0.5212386 0.35118095 0.8882005
#> relu      0.1213449 0.08555169 0.9939409

The epoch with the lowest validation loss is restored; the test rows were not used for any choice. This small example is not a benchmark: the ranking of bases depends on the data, the width, the basis size and the training budget.

plot(fits$chebyshev, type = "loss")

plot(fits$rbf, layer = 1, input = 1, output = 1, scale = "original")

An edge function is one component of the fitted network. In a multi-layer network it is not the partial effect of a predictor, and it is never a causal effect.

A different basis in each layer

basis may have one entry per layer. Basis-specific arguments go to the layers whose basis uses them; options sets values for one basis only.

m <- thangld_kan(c(2, 4, 3, 1), basis = c("rbf", "chebyshev", "jacobi"),
               degree = 3, options = list(jacobi = list(alpha = 0.5)),
               seed = 2)
m
#> <thangld_kan> Kolmogorov-Arnold network
#>   widths : 2 -> 4 -> 3 -> 1
#>   layers : RBF G=8 d=0.571 +LN +base; Chebyshev K=3; Jacobi K=3 (alpha=0.5, beta=1)
#>   params : 140
#>   fitted : no
kan_layer_bases(m)
#> [1] "rbf"       "chebyshev" "jacobi"

Adding a basis

Other packages add their edge type with register_kan_basis(); every function above then works with it, including thangld_kan_validate(). The interface is described in ?register_kan_basis. A toy linear edge:

lin <- list(
  label = "linear (toy)",
  layer_specs = function(widths) rep(list(list()), length(widths) - 1L),
  describe = function(spec) "linear",
  param_dims = function(p, q, spec) list(w = c(p, q)),
  init = function(p, q, spec) list(w = matrix(rnorm(p * q, sd = 0.3), p, q)),
  check_params = function(par, spec) invisible(TRUE),
  forward_r = function(h, par, spec) list(out = h %*% par$w, cache = h),
  backward_r = function(cache, g, par, spec)
    list(grads = list(w = crossprod(cache, g)), dh = g %*% t(par$w)),
  forward_torch = function(h, par, spec) torch::torch_matmul(h, par$w),
  edge_r = function(x, par, spec, i, j) data.frame(phi = par$w[i, j] * x))
register_kan_basis("toy_linear", lin)
res <- thangld_kan_validate("r", basis = "toy_linear", mixed = FALSE)
res[, c("group", "check", "pass")]
#>       group                                          check pass
#> 1     basis                          basis-specific checks   NA
#> 2   forward             oracle vs naive edge-by-edge loops   NA
#> 3  gradient parameters vs central difference O4 (h=0.0001) TRUE
#> 4  gradient      input vs central difference O4 (h=0.0001) TRUE
#> 5  gradient  parameters vs central difference O4 (h=1e-05) TRUE
#> 6  gradient       input vs central difference O4 (h=1e-05) TRUE
#> 7  gradient  parameters vs central difference O4 (h=1e-06) TRUE
#> 8  gradient       input vs central difference O4 (h=1e-06) TRUE
#> 9 save/load             predictions before vs after reload TRUE
unregister_kan_basis("toy_linear")

The missing basis-specific checks of the toy basis are reported as skipped (NA), not as passed.

Persistence and numerical checks

f <- tempfile(fileext = ".rds")
thangld_kan_save(fits$relu, f)
all.equal(predict(fits$relu, x[test, ]), predict(thangld_kan_load(f), x[test, ]))
#> [1] TRUE
res <- thangld_kan_validate("r")
aggregate(pass ~ basis, res, function(p) sprintf("%d/%d", sum(p), length(p)))
#>       basis  pass
#> 1 chebyshev 12/12
#> 2    hybrid 12/12
#> 3    jacobi 14/14
#> 4  legendre 13/13
#> 5     mixed   8/8
#> 6  rational 12/12
#> 7       rbf 13/13
#> 8      relu 14/14
#> 9   wavelet 16/16

thangld_kan_validate("both") with the torch runtime installed adds the torch forward, autograd and training checks; without the runtime they are reported as skipped, never as passed.