Size Strategies
As shown in Advanced Tutorial size strategies give a high degree of flexibility when it comes to changing the size of vertices in the graph.
All size strategies are executed through the Δsize!
function.
Imported to namespace by
using NaiveNASlib.Advanced
NaiveNASlib.DefaultJuMPΔSizeStrategy
— TypeDefaultJuMPΔSizeStrategy <: AbstractJuMPΔSizeStrategy
DefaultJuMPΔSizeStrategy()
Default strategy which applies default constraints or objectives to a model.
Intended use is for strategies which apply special constraints or objectives to a subset of the vertices.
Also useful when configuring DecoratingJuMPΔSizeStrategies
.
Examples
Use to get add special constraints for a subset of vertices:
function NaiveNASlib.addsomeconstraint!(s::MySpecialStrategy, v::AbstractVertex, data)
if v === s.targetvertex
# Add some special constraints here
else
# Use the default
NaiveNASlib.addsomeconstraint!(DefaultJuMPΔSizeStrategy(), v::AbstractVertex, data)
end
end
Use with a DecoratingJuMPΔSizeStrategy
:
julia> using NaiveNASlib, NaiveNASlib.Advanced
julia> WithUtilityFun(defaultutility, DefaultJuMPΔSizeStrategy())
WithUtilityFun{typeof(defaultutility), DefaultJuMPΔSizeStrategy}(NaiveNASlib.defaultutility, DefaultJuMPΔSizeStrategy())
NaiveNASlib.ΔNinExact
— FunctionΔNinExact(args...; fallback)
Return a ΔNout{Exact}
configured to set nout
of the inputs to vertices in args
to the given sizes.
For vertices with more than one input, the size change must be expressed as a tuple with one element per input. Use missing
to indicate that no special treatment is needed for an input.
Accepts either a Dict
or arguments used to create a Dict
(e.g. a set of Pair
s or a generator).
By default, fallback
is set to give a warning and then try again with ΔNinRelaxed(args...)
.
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNinExact(v1, 3);
julia> ΔNinExact(v1 => -3, v2 => -2);
julia> ΔNinExact(Dict(v1 => 3, v2 => 2));
julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);
julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);
julia> ΔNinExact(v3, 3, missing, 2);
julia> ΔNinExact(v3 => (3, missing, 2), v4 => (-2, 0));
NaiveNASlib.ΔNinRelaxed
— FunctionΔNinRelaxed(args...; fallback)
Same as ΔNinExact
except Exact
is replaced by Relaxed
and fallback
set to ΔSizeFailNoOp
by default.
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNinRelaxed(v1, 3);
julia> ΔNinRelaxed(v1 => -3, v2 => -2);
julia> ΔNinRelaxed(Dict(v1 => 3, v2 => 2));
julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);
julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);
julia> ΔNinRelaxed(v3, 3, missing, 2);
julia> ΔNinRelaxed(v3 => (3, missing, 2), v4 => (-2, 0));
NaiveNASlib.ΔNin
— FunctionΔNin(args...;[fallback])
Splits args
into relaxed and exact size changes and creates the appropriate strategy (one of ΔNout{Exact}, ΔNout{Relaxed} or ΔNoutMix
).
Fallback strategy may be supplied through the fallback
keyword. Defaults to ΔNinRelaxed
for all given vertices.
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNin(v1, 3);
julia> ΔNin(v1 => -3, v2 => -2);
julia> ΔNin(Dict(v1 => 3, v2 => 2));
julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);
julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);
julia> ΔNin(v3, 3, missing, 2);
julia> ΔNin(v3 => (3, missing, 2), v4 => (-2, 0));
julia> ΔNin(v3, relaxed(3), missing, 2);
julia> ΔNin(v3 => (relaxed(3), missing, 2), v4 => relaxed(-2, 0));
NaiveNASlib.ΔNoutExact
— FunctionΔNoutExact(args...; fallback)
Return a ΔNout{Exact}
with fallback
set to give a warning and then try again with ΔNoutRelaxed(args...)
.
Accepts either a Dict
or arguments used to create a Dict
(e.g. a set of Pair
s or a generator).
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNoutExact(v1, 3);
julia> ΔNoutExact(v1 => -3, v2 => -2);
julia> ΔNoutExact(Dict(v1 => 3, v2 => 2));
NaiveNASlib.ΔNoutRelaxed
— FunctionΔNoutRelaxed(args...;fallback)
Return a ΔNout{Relaxed}
with fallback
set to ΔSizeFailNoOp
.
Accepts either a Dict
or arguments used to create a Dict
(e.g. a set of Pair
s or a generator).
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNoutRelaxed(v1, 3);
julia> ΔNoutRelaxed(v1 => -3, v2 => -2);
julia> ΔNoutRelaxed(Dict(v1 => 3, v2 => 2));
NaiveNASlib.ΔNout
— MethodΔNout(args...;[fallback])
Splits args
into relaxed and exact size changes and creates the appropriate strategy (one of ΔNout{Exact}, ΔNout{Relaxed} or ΔNoutMix
).
Fallback strategy may be supplied through the fallback
keyword. Defaults to ΔNoutRelaxed
for all given vertices.
Examples
julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced
julia> module TestNNLib
using NaiveNASlib, NaiveNASlib.Extend
mutable struct Affine{T} W::Matrix{T} end
NaiveNASlib.nout(a::Affine) = size(a.W, 1)
NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
export affine
end;
julia> using .TestNNLib;
julia> iv = affine(inputvertex("in", 3), 6);
julia> v1 = affine(iv, 4);
julia> v2 = affine(iv, 5);
julia> ΔNout(v1, 3);
julia> ΔNout(v1 => -3, v2 => -2);
julia> ΔNout(Dict(v1 => 3, v2 => 2));
julia> ΔNout(v1, relaxed(2));
julia> ΔNout(v1 => relaxed(2), v2 => -1);
NaiveNASlib.ΔNout
— TypeΔNout <: AbstractJuMPΔSizeStrategy
ΔNout{T, V, F}(Δs::Dict{V, Int}, fallback::F)
Strategy for changing nout of vertices.
For each key-value pair in v, Δ
in Δs
, change nout(v)
by Δ
, i.e new size is nout(v) + Δ
.
If T == Exact
, size change will be added as a constraint to the model which means that the operation will fail if it is not possible to change nout(v)
by exactly Δ
.
If T == Relaxed
, size change will be added as an objective to the model which means that nout(v)
might not change by exactly Δ
. In addition, a constraint that nout(v)
must change is also added.
NaiveNASlib.WithUtilityFun
— TypeWithUtilityFun{F, S} <: AbstractΔSizeStrategy
WithUtilityFun(utilityfun::F, strategy::S)
Applies neuron indices selection with strategy
and using utilityfun
to compute the utility of neurons indices.
Note that utilityfun
will override any utility function supplied in function call. Thus it is possible use WithUtilityFun
to change utility function e.g. when switching to a fallback strategy.
NaiveNASlib.UpperInsertBound
— TypeUpperInsertBound{V<:AbstractDict{<:AbstractVertex, <:Integer}, S, F} <: DecoratingJuMPΔSizeStrategy
UpperInsertBound(vbounds; strategy, fallback)
UpperInsertBound(vs, bound::Integer; strategy, fallback)
Sets upper bound for how many new neurons to insert of each vertex v
in vbounds
to vbounds[v]
and then applies neuron indices selection with strategy
(default DefaultJuMPΔSizeStrategy
).
Alternatively, sets the upper bound of each vertex in vs
to bound
. Applies bound
to all vertices if vs
is empty.
Mostly useful to minimize solver time in cases when one does not want any new neurons as disallowing this allows skipping a couple of constraints.
If it fails, the operation will be retried with the fallback
strategy (default ΔSizeFailNoOp
).
NaiveNASlib.WithKwargs
— TypeWithKWargs{KW, S} <: AbstractAfterΔSizeStrategy
WithKwargs(strategy; kwargs...)
Adds keyword argument kwargs
to any calls to Δsize!
after sizes have been determined.
NaiveNASlib.LogΔSizeExec
— TypeLogΔSizeExec <: AbstractJuMPΔSizeStrategy
LogΔSizeExec(msgfun)
LogΔSizeExec(msgfun, level::Logging.LogLevel)
LogΔSizeExec(msgfun, level::Logging.LogLevel, andthen::AbstractJuMPΔSizeStrategy)
Logs msgfun(v)
at log level level
, then executes AbstractJuMPΔSizeStrategy
andthen
for vertex v
.
NaiveNASlib.ThrowΔSizeFailError
— TypeThrowΔSizeFailError <: AbstractJuMPΔSizeStrategy
ThrowΔSizeFailError(msg::String)
Throws an ΔSizeFailError
with message msg
.
NaiveNASlib.ΔSizeFailNoOp
— TypeΔSizeFailNoOp <: AbstractJuMPΔSizeStrategy
ΔSizeFailNoOp()
Does not perform any action.
NaiveNASlib.AlignNinToNout
— TypeAlignNinToNout{S, F} <: DecoratingJuMPΔSizeStrategy
AlignNinToNout(;vstrat::S, fallback::F)
AlignNinToNout(vstrat::S, fallback::F)
Adds variables and constraints for nin(vi) == nout.(inputs(vi))
.
Generally useful when doing structural changes such are removing/adding vertices or edges.
If it fails, the operation will be retried with the fallback
strategy (default ΔSizeFailNoOp
).
NaiveNASlib.TruncateInIndsToValid
— TypeTruncateInIndsToValid{S} <: AbstractΔSizeStrategy
TruncateInIndsToValid()
TruncateInIndsToValid(s::S)
Ensures that all selected input indices are within range of existing input indices after applying s
(default DefaultJuMPΔSizeStrategy
).
Not needed in normal cases, but certain structural mutations (e.g create_edge!) may cause this to happen due to how constraints are (not) created when original sizes do not align in conjunction with how result of selection is interpreted.
An example of when it is needed is when adding an edge from vertex vi
to an invariant vertex vo
where nout(vi)
>
nout(vo)
. In this case it is expected that the result of the optimization is that the indices 1:nout(vi)
of vi
shall be kept. However, this will propagate to vo
which will be instructed to keep indices it does not have. With this strategy, all indices which are larger than nout(vo)
will be replaced by -1
(which indicates that new parameters shall be created)
While this may be considered a flaw in the output selection procedure, it is rare enough so that in most cases when it happens it is the result of a user error or lower level bug. Therefore this strategy is left optional to be used only in cases when mismatches are expected.
Note that in most normal cases, this has no effect since vertices capable of getting new input edges generally don't have parameters.
NaiveNASlib.TimeLimitΔSizeStrategy
— TypeTimeLimitΔSizeStrategy{S} <: DecoratingJuMPΔSizeStrategy
TimeLimitΔSizeStrategy(limit::Number)
TimeLimitΔSizeStrategy(limit::Number, base::S)
Sets the time limit for the solver to limit
. Use strategy base
for all other aspects.
NaiveNASlib.TimeOutAction
— TypeTimeOutAction{S,A,F} <: DecoratingJuMPΔSizeStrategy
TimeOutAction(action::A, base::S, fallback::F)
TimeOutAction(base; fallback)
Calls action(model)
if JuMP model model
has status MOI.TIME_LIMIT
after optimization stops. Use strategy base
for all other aspects.
Default action is to display a warning and then apply fallback
(default ΔSizeFailNoOp
).
NaiveNASlib.AfterΔSizeCallback
— TypeAfterΔSizeCallback{F, S} <: AbstractAfterΔSizeStrategy
AfterΔSizeCallback(cbfun::F, basestrat::S=ThrowΔSizeFailError())
Calls cbfun(v, Δ, isnout)
for all vertices which change size after having been asked to change their sizes as a result of basestrat
.
NaiveNASlib.logafterΔsize
— FunctionlogafterΔsize(printfun=nameorrepr;level=Logging.Info, base=DefaultJuMPΔSizeStrategy())
Return an AfterΔSizeCallback
configured to log size changes with log level level
.
For a given vertex v
, printfun(v)
will be used in the logged string.
Strategy base
will be used to change sizes (e.g if Δsize!(logafterΔsize(base))
is called).
NaiveNASlib.validateafterΔsize
— FunctionvalidateafterΔsize(printfun=nameorrepr; base=DefaultJuMPΔSizeStrategy())
Return an AfterΔSizeCallback
configured to validate that sizes (nin and nout) are consistent after a size change and throw a ΔSizeFailError
if validation fails.
For a given vertex v
, printfun(v)
will be used in the error message should the size validation fail.
Strategy base
will be used to change sizes (e.g if Δsize!(validateafterΔsize(base))
is called).