API Library
Functions
The KernelFunctions API comprises the following four functions.
KernelFunctions.kernelmatrix — Functionkernelmatrix(κ::Kernel, x::AbstractVector)Compute the kernel κ for each pair of inputs in x. Returns a matrix of size (length(x), length(x)) satisfying kernelmatrix(κ, x)[p, q] == κ(x[p], x[q]).
kernelmatrix(κ::Kernel, x::AbstractVector, y::AbstractVector)Compute the kernel κ for each pair of inputs in x and y. Returns a matrix of size (length(x), length(y)) satisfying kernelmatrix(κ, x, y)[p, q] == κ(x[p], y[q]).
kernelmatrix(κ::Kernel, X::AbstractMatrix; obsdim)
+kernelmatrix(κ::Kernel, X::AbstractMatrix, Y::AbstractMatrix; obsdim)If obsdim=1, equivalent to kernelmatrix(κ, RowVecs(X)) and kernelmatrix(κ, RowVecs(X), RowVecs(Y)), respectively. If obsdim=2, equivalent to kernelmatrix(κ, ColVecs(X)) and kernelmatrix(κ, ColVecs(X), ColVecs(Y)), respectively.
KernelFunctions.kernelmatrix! — Functionkernelmatrix!(K::AbstractMatrix, κ::Kernel, x::AbstractVector)
+kernelmatrix!(K::AbstractMatrix, κ::Kernel, x::AbstractVector, y::AbstractVector)In-place version of kernelmatrix where pre-allocated matrix K will be overwritten with the kernel matrix.
kernelmatrix!(K::AbstractMatrix, κ::Kernel, X::AbstractMatrix; obsdim)
+kernelmatrix!(
+ K::AbstractMatrix,
+ κ::Kernel,
+ X::AbstractMatrix,
+ Y::AbstractMatrix;
+ obsdim,
+)If obsdim=1, equivalent to kernelmatrix!(K, κ, RowVecs(X)) and kernelmatrix(K, κ, RowVecs(X), RowVecs(Y)), respectively. If obsdim=2, equivalent to kernelmatrix!(K, κ, ColVecs(X)) and kernelmatrix(K, κ, ColVecs(X), ColVecs(Y)), respectively.
KernelFunctions.kernelmatrix_diag — Functionkernelmatrix_diag(κ::Kernel, x::AbstractVector)Compute the diagonal of kernelmatrix(κ, x) efficiently.
kernelmatrix_diag(κ::Kernel, x::AbstractVector, y::AbstractVector)Compute the diagonal of kernelmatrix(κ, x, y) efficiently. Requires that x and y are the same length.
kernelmatrix_diag(κ::Kernel, X::AbstractMatrix; obsdim)
+kernelmatrix_diag(κ::Kernel, X::AbstractMatrix, Y::AbstractMatrix; obsdim)If obsdim=1, equivalent to kernelmatrix_diag(κ, RowVecs(X)) and kernelmatrix_diag(κ, RowVecs(X), RowVecs(Y)), respectively. If obsdim=2, equivalent to kernelmatrix_diag(κ, ColVecs(X)) and kernelmatrix_diag(κ, ColVecs(X), ColVecs(Y)), respectively.
KernelFunctions.kernelmatrix_diag! — Functionkernelmatrix_diag!(K::AbstractVector, κ::Kernel, x::AbstractVector)
+kernelmatrix_diag!(K::AbstractVector, κ::Kernel, x::AbstractVector, y::AbstractVector)In place version of kernelmatrix_diag.
kernelmatrix_diag!(K::AbstractVector, κ::Kernel, X::AbstractMatrix; obsdim)
+kernelmatrix_diag!(
+ K::AbstractVector,
+ κ::Kernel,
+ X::AbstractMatrix,
+ Y::AbstractMatrix;
+ obsdim
+)If obsdim=1, equivalent to kernelmatrix_diag!(K, κ, RowVecs(X)) and kernelmatrix_diag!(K, κ, RowVecs(X), RowVecs(Y)), respectively. If obsdim=2, equivalent to kernelmatrix_diag!(K, κ, ColVecs(X)) and kernelmatrix_diag!(K, κ, ColVecs(X), ColVecs(Y)), respectively.
Input Types
The above API operates on collections of inputs. All collections of inputs in KernelFunctions.jl are represented as AbstractVectors. To understand this choice, please see the design notes on collections of inputs. The length of any such AbstractVector is equal to the number of inputs in the collection. For example, this means that
size(kernelmatrix(k, x)) == (length(x), length(x))is always true, for some Kernel k, and AbstractVector x.
Univariate Inputs
If each input to your kernel is Real-valued, then any AbstractVector{<:Real} is a valid representation for a collection of inputs. More generally, it's completely fine to represent a collection of inputs of type T as, for example, a Vector{T}. However, this may not be the most efficient way to represent collection of inputs. See Vector-Valued Inputs for an example.
Vector-Valued Inputs
We recommend that collections of vector-valued inputs are stored in an AbstractMatrix{<:Real} when possible, and wrapped inside a ColVecs or RowVecs to make their interpretation clear:
KernelFunctions.ColVecs — TypeColVecs(X::AbstractMatrix)A lightweight wrapper for an AbstractMatrix which interprets it as a vector-of-vectors, in which each column of X represents a single vector.
That is, by writing x = ColVecs(X), you are saying "x is a vector-of-vectors, each of which has length size(X, 1). The total number of vectors is size(X, 2)."
Phrased differently, ColVecs(X) says that X should be interpreted as a vector of horizontally-concatenated column-vectors, hence the name ColVecs.
julia> X = randn(2, 5);
+
+julia> x = ColVecs(X);
+
+julia> length(x) == 5
+true
+
+julia> X[:, 3] == x[3]
+trueColVecs is related to RowVecs via transposition:
julia> X = randn(2, 5);
+
+julia> ColVecs(X) == RowVecs(X')
+trueKernelFunctions.RowVecs — TypeRowVecs(X::AbstractMatrix)A lightweight wrapper for an AbstractMatrix which interprets it as a vector-of-vectors, in which each row of X represents a single vector.
That is, by writing x = RowVecs(X), you are saying "x is a vector-of-vectors, each of which has length size(X, 2). The total number of vectors is size(X, 1)."
Phrased differently, RowVecs(X) says that X should be interpreted as a vector of vertically-concatenated row-vectors, hence the name RowVecs.
Internally, the data continues to be represented as an AbstractMatrix, so using this type does not introduce any kind of performance penalty.
julia> X = randn(5, 2);
+
+julia> x = RowVecs(X);
+
+julia> length(x) == 5
+true
+
+julia> X[3, :] == x[3]
+trueRowVecs is related to ColVecs via transposition:
julia> X = randn(5, 2);
+
+julia> RowVecs(X) == ColVecs(X')
+trueThese types are specialised upon to ensure good performance e.g. when computing Euclidean distances between pairs of elements. The benefit of using this representation, rather than using a Vector{Vector{<:Real}}, is that optimised matrix-matrix multiplication functionality can be utilised when computing pairwise distances between inputs, which are needed for kernelmatrix computation.
Inputs for Multiple Outputs
KernelFunctions.jl views multi-output GPs as GPs on an extended input domain. For an explanation of this design choice, see the design notes on multi-output GPs.
An input to a multi-output Kernel should be a Tuple{T, Int}, whose first element specifies a location in the domain of the multi-output GP, and whose second element specifies which output the inputs corresponds to. The type of collections of inputs for multi-output GPs is therefore AbstractVector{<:Tuple{T, Int}}.
KernelFunctions.jl provides the following helper functions to reduce the cognitive load associated with working with multi-output kernels by dealing with transforming data from the formats in which it is commonly found into the format required by KernelFunctions. The intention is that users can pass their data to these functions, and use the returned values throughout their code, without having to worry further about correctly formatting their data for KernelFunctions' sake:
KernelFunctions.prepare_isotopic_multi_output_data — Methodprepare_isotopic_multi_output_data(x::AbstractVector, y::ColVecs)Utility functionality to convert a collection of N = length(x) inputs x, and a vector-of-vectors y (efficiently represented by a ColVecs) into a format suitable for use with multi-output kernels.
y[n] is the vector-valued output corresponding to the input x[n]. Consequently, it is necessary that length(x) == length(y).
For example, if outputs are initially stored in a num_outputs × N matrix:
julia> x = [1.0, 2.0, 3.0];
+
+julia> Y = [1.1 2.1 3.1; 1.2 2.2 3.2]
+2×3 Matrix{Float64}:
+ 1.1 2.1 3.1
+ 1.2 2.2 3.2
+
+julia> inputs, outputs = prepare_isotopic_multi_output_data(x, ColVecs(Y));
+
+julia> inputs
+6-element KernelFunctions.MOInputIsotopicByFeatures{Float64, Vector{Float64}, Int64}:
+ (1.0, 1)
+ (1.0, 2)
+ (2.0, 1)
+ (2.0, 2)
+ (3.0, 1)
+ (3.0, 2)
+
+julia> outputs
+6-element Vector{Float64}:
+ 1.1
+ 1.2
+ 2.1
+ 2.2
+ 3.1
+ 3.2See also prepare_heterotopic_multi_output_data.
KernelFunctions.prepare_isotopic_multi_output_data — Methodprepare_isotopic_multi_output_data(x::AbstractVector, y::RowVecs)Utility functionality to convert a collection of N = length(x) inputs x and output vectors y (efficiently represented by a RowVecs) into a format suitable for use with multi-output kernels.
y[n] is the vector-valued output corresponding to the input x[n]. Consequently, it is necessary that length(x) == length(y).
For example, if outputs are initial stored in an N × num_outputs matrix:
julia> x = [1.0, 2.0, 3.0];
+
+julia> Y = [1.1 1.2; 2.1 2.2; 3.1 3.2]
+3×2 Matrix{Float64}:
+ 1.1 1.2
+ 2.1 2.2
+ 3.1 3.2
+
+julia> inputs, outputs = prepare_isotopic_multi_output_data(x, RowVecs(Y));
+
+julia> inputs
+6-element KernelFunctions.MOInputIsotopicByOutputs{Float64, Vector{Float64}, Int64}:
+ (1.0, 1)
+ (2.0, 1)
+ (3.0, 1)
+ (1.0, 2)
+ (2.0, 2)
+ (3.0, 2)
+
+julia> outputs
+6-element Vector{Float64}:
+ 1.1
+ 2.1
+ 3.1
+ 1.2
+ 2.2
+ 3.2See also prepare_heterotopic_multi_output_data.
KernelFunctions.prepare_heterotopic_multi_output_data — Functionprepare_heterotopic_multi_output_data(
+ x::AbstractVector, y::AbstractVector{<:Real}, output_indices::AbstractVector{Int},
+)Utility functionality to convert a collection of inputs x, observations y, and output_indices into a format suitable for use with multi-output kernels. Handles the situation in which only one (or a subset) of outputs are observed at each feature. Ensures that all arguments are compatible with one another, and returns a vector of inputs and a vector of outputs.
y[n] should be the observed value associated with output output_indices[n] at feature x[n].
julia> x = [1.0, 2.0, 3.0];
+
+julia> y = [-1.0, 0.0, 1.0];
+
+julia> output_indices = [3, 2, 1];
+
+julia> inputs, outputs = prepare_heterotopic_multi_output_data(x, y, output_indices);
+
+julia> inputs
+3-element Vector{Tuple{Float64, Int64}}:
+ (1.0, 3)
+ (2.0, 2)
+ (3.0, 1)
+
+julia> outputs
+3-element Vector{Float64}:
+ -1.0
+ 0.0
+ 1.0See also prepare_isotopic_multi_output_data.
The input types returned by prepare_isotopic_multi_output_data can also be constructed manually:
KernelFunctions.MOInput — TypeMOInput(x::AbstractVector, out_dim::Integer)A data type to accommodate modelling multi-dimensional output data. MOInput(x, out_dim) has length length(x) * out_dim.
julia> x = [1, 2, 3];
+
+julia> MOInput(x, 2)
+6-element KernelFunctions.MOInputIsotopicByOutputs{Int64, Vector{Int64}, Int64}:
+ (1, 1)
+ (2, 1)
+ (3, 1)
+ (1, 2)
+ (2, 2)
+ (3, 2)As shown above, an MOInput represents a vector of tuples. The first length(x) elements represent the inputs for the first output, the second length(x) elements represent the inputs for the second output, etc. See Inputs for Multiple Outputs in the docs for more info.
MOInput will be deprecated in version 0.11 in favour of MOInputIsotopicByOutputs, and removed in version 0.12.
As with ColVecs and RowVecs for vector-valued input spaces, this type enables specialised implementations of e.g. kernelmatrix for MOInputs in some situations.
To find out more about the background, read this review of kernels for vector-valued functions.
Generic Utilities
KernelFunctions also provides miscellaneous utility functions.
KernelFunctions.nystrom — Functionnystrom(k::Kernel, X::AbstractVector, S::AbstractVector{<:Integer})Compute a factorization of a Nystrom approximation of the square kernel matrix of data vector X with respect to kernel k, using indices S. Returns a NystromFact struct which stores a Nystrom factorization satisfying:
\[\mathbf{K} \approx \mathbf{C}^{\intercal}\mathbf{W}\mathbf{C}\]
nystrom(k::Kernel, X::AbstractVector, r::Real)Compute a factorization of a Nystrom approximation of the square kernel matrix of data vector X with respect to kernel k using a sample ratio of r. Returns a NystromFact struct which stores a Nystrom factorization satisfying:
\[\mathbf{K} \approx \mathbf{C}^{\intercal}\mathbf{W}\mathbf{C}\]
nystrom(k::Kernel, X::AbstractMatrix, S::AbstractVector{<:Integer}; obsdim)If obsdim=1, equivalent to nystrom(k, RowVecs(X), S). If obsdim=2, equivalent to nystrom(k, ColVecs(X), S).
nystrom(k::Kernel, X::AbstractMatrix, r::Real; obsdim)If obsdim=1, equivalent to nystrom(k, RowVecs(X), r). If obsdim=2, equivalent to nystrom(k, ColVecs(X), r).
KernelFunctions.NystromFact — TypeNystromFactType for storing a Nystrom factorization. The factorization contains two fields: W and C, two matrices satisfying:
\[\mathbf{K} \approx \mathbf{C}^{\intercal}\mathbf{W}\mathbf{C}\]
Conditional Utilities
To keep the dependencies of KernelFunctions lean, some functionality is only available if specific other packages are explicitly loaded (using).
Kronecker.jl
https://github.com/MichielStock/Kronecker.jl
KernelFunctions.kronecker_kernelmatrix — Functionkronecker_kernelmatrix(
+ k::Union{IndependentMOKernel,IntrinsicCoregionMOKernel}, x::MOI, y::MOI
+) where {MOI<:IsotopicMOInputsUnion}Requires Kronecker.jl: Computes the kernelmatrix for the IndependentMOKernel and the IntrinsicCoregionMOKernel, but returns a lazy kronecker product. This object can be very efficiently inverted or decomposed. See also kernelmatrix.
KernelFunctions.kernelkronmat — Functionkernelkronmat(κ::Kernel, X::AbstractVector{<:Real}, dims::Int) -> KroneckerPowerReturn a KroneckerPower matrix on the D-dimensional input grid constructed by $\otimes_{i=1}^D X$, where D is given by dims.
Requires Kronecker.jl and for iskroncompatible(κ) to return true.
kernelkronmat(κ::Kernel, X::AbstractVector{<:AbstractVector}) -> KroneckerProductReturns a KroneckerProduct matrix on the grid built with the collection of vectors $\{X_i\}_{i=1}^D$: $\otimes_{i=1}^D X_i$.
Requires Kronecker.jl and for iskroncompatible(κ) to return true.
PDMats.jl
https://github.com/JuliaStats/PDMats.jl
KernelFunctions.kernelpdmat — Functionkernelpdmat(k::Kernel, X::AbstractVector)Compute a positive-definite matrix in the form of a PDMat matrix (see PDMats.jl), with the Cholesky decomposition precomputed. The algorithm adds a diagonal "nugget" term to the kernel matrix which is increased until positive definiteness is achieved. The algorithm gives up with an error if the nugget becomes larger than 1% of the largest value in the kernel matrix.
kernelpdmat(k::Kernel, X::AbstractMatrix; obsdim)If obsdim=1, equivalent to kernelpdmat(k, RowVecs(X)). If obsdim=2, equivalent to kernelpdmat(k, ColVecs(X)).
