HighDimensionalModels.jl

Lasso estimators computed by coordinate descent. For a response $y \in \mathbb{R}^n$ and $n$ observations of $p$ predictors in the rows of $X$, the lasso is

\[\underset{\beta_0,\,\beta}{\text{minimize}} \quad \frac{1}{2n}\,\|y - \beta_0 - X\beta\|^2 + \lambda \sum_{j=1}^p \omega_j |\beta_j|,\]

with an unpenalized intercept $\beta_0$ and penalty loadings $\omega_j \ge 0$. HighDimensionalModels solves it at one penalty level $\lambda$ and along a path of levels, and solves the estimators built on it:

• the lasso, with an intercept, standardized columns, penalty loadings, observation weights and offsets;
• the square-root and the scaled lasso, whose $\lambda$ does not depend on the noise level, and the heteroscedasticity-robust (feasible) lasso;
• lasso paths, with screening, refitting and the functions of StatsAPI;
• the group lasso;
• varying-coefficient models by local polynomial regression, with the kernels of KernelSmoothing.jl, and sparse ones with a group lasso penalty.

The coordinate descent itself (the passes over the coordinates and the active set, the penalty paths with screening, the reweighting of the scaled and feasible lasso) is that of ProximalSolvers.jl, on the losses and penalties of ProximalBase.jl. coordinate_descent! runs it on other losses and on any penalty that separates across coordinates, including your own.

Installation

Before the first registration, use a checkout with registered dependencies:

git clone --branch master https://github.com/mlakolar/HighDimensionalModels.jl.git
cd HighDimensionalModels.jl
julia --project=. -e 'using Pkg; Pkg.instantiate()'
julia --project=.

HighDimensionalModels requires Julia 1.10 or later.

A first example

The lasso with an intercept at $\lambda = 0.1$, on 200 observations of 50 predictors of which the first five matter:

using HighDimensionalModels, Random

rng = Xoshiro(1)
n, p = 200, 50
X = randn(rng, n, p)
y = 1.0 .+ X[:, 1:5] * ones(5) + randn(rng, n)

sol = lasso(X, y, 0.1; intercept = true)
findall(!iszero, sol.x)

15-element Vector{Int64}:
  1
  2
  3
  4
  5
  7
  9
 23
 25
 26
 36
 40
 44
 48
 50

The coefficients, intercept first, and the fitted values of new observations come from the functions of StatsAPI:

coef(sol)

51-element SparseArrays.SparseVector{Float64, Int64} with 16 stored entries:
  [1 ]  =  0.952993
  [2 ]  =  0.890305
  [3 ]  =  0.906138
  [4 ]  =  0.974288
  [5 ]  =  0.973434
  [6 ]  =  0.959268
  [8 ]  =  0.0112604
        ⋮
  [24]  =  0.042762
  [26]  =  -0.00544152
  [27]  =  0.0402493
  [37]  =  0.054088
  [41]  =  0.0291767
  [45]  =  -0.0260445
  [49]  =  -0.0377111
  [51]  =  -0.064684

predict(sol, randn(rng, 3, p))

3-element Vector{Float64}:
 1.5493172119660075
 1.914213381111578
 0.6645143390165618

The lasso along 50 values of $\lambda$, from the smallest one whose solution is zero:

path = LassoPath(X, y; nλ = 50, intercept = true)
[count(!iszero, β) for β in path.βpath[1:10:end]]

5-element Vector{Int64}:
  0
  6
 37
 45
 50

The guide

• The lasso and its tuning-free variants: the mathematics of lasso, sqrt_lasso, scaled_lasso and feasible_lasso!, their conventions, and observation weights.
• Paths, screening and solver options: LassoPath, the options of coordinate descent and when each pays off, and the Gram form.
• The group lasso.
• Variable selection with error control: SLOPE, knockoff+, and complementary-pairs stability selection.
• Writing a loss or a penalty of your own.
• Varying-coefficient models.
• The examples, which go through the package and its options with their output.

The family

HighDimensionalModels is one of five Julia packages for high-dimensional statistics, three for estimation and inference and two for the optimization beneath them:

Most users need one of the first three. To solve a problem of your own, use ProximalSolvers; to write your own algorithm, ProximalBase.

Development tests

From the repository root, run julia scripts/test_registered.jl for the full suite with registered dependencies on Julia 1.10 and newer. In a separate checkout, julia scripts/test_master.jl checks integration with dependency master checkouts. The repository's CONTRIBUTING.md describes the naming convention and bootstrap scripts.