Varying-coefficient models

VaryingCoefficientCV

Result returned by varying_coefficient_cv and varying_coefficient_lasso_cv: the bandwidths tried, the cross-validated mean squared prediction error cvm at each (weighted by the observation weights), and the bandwidth that minimizes it.

VaryingCoefficientFit

Result returned by varying_coefficient.

Fields

• zgrid::Vector: the points z₀, sorted
• coefficients::Array{T,3}: coefficients[j, r + 1, g] estimates the r-th derivative βⱼ⁽ʳ⁾(zgrid[g]), r = 0, …, degree
• stderrors::Array{T,3}: their heteroskedasticity-robust standard errors (NaN where the local fit is singular), or an empty array without se = true
• bandwidth, degree, kernel: the settings of the fit
• rank::Vector{Int}: the rank of the local design at each point; below p(degree + 1) the local fit is not unique and the one of minimum norm is taken
• fitted::Vector, residuals::Vector: the fitted values ∑ⱼ Xᵢⱼ β̂ⱼ(zᵢ) of the local fits at the observed zᵢ, and the residuals, or empty without se = true
• nobs::Int: the number of observations

coef (coef(fit; derivative = r)), predict, stderror, confint, fitted, residuals and nobs apply to it.

VaryingCoefficientLassoFit

Result returned by varying_coefficient_lasso.

Fields

• zgrid::Vector: the points z₀, sorted
• coefficients::Array{T,3}: coefficients[j, r + 1, g], the penalized estimate of βⱼ⁽ʳ⁾(zgrid[g])
• refitted::Array{T,3}: the same from the unpenalized local fit on the selected variables, or an empty array without refit
• selected::BitMatrix: selected[j, g] when variable j has a nonzero coefficient at zgrid[g]
• λ, penalty, bandwidth, degree, kernel: the settings of the fit
• nobs::Int: the number of observations

coef (coef(fit; derivative = r, refit = false)), predict and nobs apply to it.

varying_coefficient(X, z, y, zgrid; kernel, bandwidth, degree = 1, weights = nothing, se = false)

The local polynomial estimate of the varying-coefficient model

yᵢ = ∑ⱼ Xᵢⱼ βⱼ(zᵢ) + εᵢ

at each point z₀ of zgrid. Near z₀ each βⱼ is approximated by a polynomial of degree degree in z - z₀, and the polynomials are fitted by weighted least squares,

minimize  ∑ᵢ wᵢ Kₕ(zᵢ - z₀) (yᵢ - ∑ⱼ Xᵢⱼ ∑ᵣ θⱼᵣ (zᵢ - z₀)ʳ)²,

with Kₕ(d) = k(d/h)/h for the kernel k and the bandwidth h, and the observation weights wᵢ (one when omitted). θⱼᵣ r! estimates the r-th derivative βⱼ⁽ʳ⁾(z₀). A column of ones in X gives a varying intercept.

The kernel is a function of the scaled distance, k(u), such as the kernels of KernelSmoothing.jl: EpanechnikovKernel(), GaussianKernel() (for which h is the standard deviation), UniformKernel(), and so on. With a compactly supported kernel only the observations within h of z₀ enter the fit at z₀; when they do not determine the polynomials (fewer than p(degree + 1) of them, or collinear), the fit of minimum norm is taken and rank records it. varying_coefficient_cv chooses the bandwidth by leave-one-out cross-validation.

With se = true, the fit also computes the fitted values and residuals of the local fits at the observed zᵢ (one more local fit per observation, which can cost far more than the fit on the grid), and the heteroskedasticity-robust (sandwich) standard errors of the estimates, (Z'WZ)⁻¹ Z'W diag(ε̂²) W Z (Z'WZ)⁻¹ for the local design Z and the kernel weights W (the observation weights count as frequencies: integer weights give the fit and the standard errors of replicated rows). They describe the variance of the local estimate, not its bias: a confidence interval from them covers βⱼ(z₀) only when the bandwidth is small enough for the bias to be negligible (undersmoothing).

using HighDimensionalModels, KernelSmoothing
fit = varying_coefficient(X, z, y, range(0, 1; length = 50); kernel = EpanechnikovKernel(), bandwidth = 0.2,
                         se = true)
coef(fit)                      # p × 50: the estimates of βⱼ at the points
coef(fit; derivative = 1)      # of their first derivatives
confint(fit)                   # pointwise intervals, from the robust standard errors
predict(fit, Xnew, znew)       # ∑ⱼ Xnewᵢⱼ βⱼ(znewᵢ), interpolated linearly between the points

varying_coefficient_cv(X, z, y, bandwidths; kernel, degree = 1, weights = nothing)

Leave-one-out cross-validation of the bandwidth of varying_coefficient: observation i is predicted from the local fit at zᵢ computed without it, and cvm is the mean squared error of these predictions, weighted by the observation weights. The local fit is linear in y, so the prediction without i comes from the fit with it, (ŷᵢ - Hᵢᵢ yᵢ) / (1 - Hᵢᵢ) with the leverage Hᵢᵢ, at the cost of one local fit per observation. An observation alone in its window, or in a window that does not determine the local polynomials, cannot be predicted, and its bandwidth gets cvm = Inf. Returns a VaryingCoefficientCV.

varying_coefficient_lasso(X, z, y, zgrid, λ, options = CDOptions(); kernel, bandwidth, degree = 1,
                        penalty = :group, refit = true, weights = nothing)

A sparse varying-coefficient model: at each point z₀ of zgrid, the local polynomial fit of varying_coefficient with a penalty that selects the variables Xⱼ whose coefficient functions are not zero near z₀,

minimize  ∑ᵢ vᵢ (yᵢ - ∑ⱼ Xᵢⱼ ∑ᵣ θⱼᵣ ((zᵢ - z₀)/h)ʳ)² / (2 ∑ᵢ vᵢ) + λ P(θ),

with the local weights vᵢ = wᵢ Kₕ(zᵢ - z₀). The loss is the mean squared error on the local sample, so λ is on the scale of an ordinary lasso.

• penalty = :group, the default, is the group lasso over the degree + 1 coefficients of each variable, P(θ) = ∑ⱼ √(degree + 1) ‖Z̃ⱼθⱼ‖ / √n, with the locally weighted columns Z̃ⱼ of variable j orthonormalized (the standardize = :orthonormal of group_lasso), so that the penalty is on the fitted values of each variable and a variable enters or leaves with all its polynomial terms.
• penalty = :lasso is the lasso on the expanded coefficients, with the loadings of the standardized local columns, as lasso with standardize_x; a variable is selected when any of its coefficients is not zero.

With refit, the selected variables are refitted at each point by the unpenalized local fit (all their polynomial terms), which removes the shrinkage of the penalty. Coordinate descent runs with options. The kernel and the bandwidth are as for varying_coefficient; varying_coefficient_lasso_cv chooses the bandwidth. Returns a VaryingCoefficientLassoFit.

varying_coefficient_lasso_cv(X, z, y, bandwidths, λ, options = CDOptions(); kernel, degree = 1,
                          penalty = :group, refit = true, weights = nothing)

Leave-one-out cross-validation of the bandwidth of varying_coefficient_lasso at the level λ: observation i is predicted from the penalized local fit at zᵢ computed without it (refitted on the selected variables with refit), and cvm is the mean squared error of these predictions, weighted by the observation weights. It solves one local lasso per observation and bandwidth. Returns a VaryingCoefficientCV.

coef(fit::VaryingCoefficientFit; derivative = 0)
stderror(fit::VaryingCoefficientFit; derivative = 0)
confint(fit::VaryingCoefficientFit; level = 0.95, derivative = 0)
predict(fit::VaryingCoefficientFit, X, z)

The p × length(zgrid) estimates of βⱼ⁽ʳ⁾ at the points of fit.zgrid (r = derivative), their robust standard errors, and the pointwise normal intervals estimate ± z stderror as a pair of matrices (lower, upper), which cover βⱼ⁽ʳ⁾(z₀) only when the bias of the local fit is negligible (see varying_coefficient). predict gives ∑ⱼ Xᵢⱼ βⱼ(zᵢ) with β interpolated linearly between the points of the grid (and constant outside it). fitted and residuals are those of the local fits at the observations.

coef(fit::VaryingCoefficientLassoFit; derivative = 0, refit = false)
predict(fit::VaryingCoefficientLassoFit, X, z; refit = false)

The p × length(zgrid) penalized estimates of βⱼ⁽ʳ⁾ (r = derivative) at the points of fit.zgrid, or with refit those of the unpenalized refit on the selected variables; and the predictions ∑ⱼ Xᵢⱼ βⱼ(zᵢ), interpolated linearly between the points, as for varying_coefficient.