Size Strategies

DefaultJuMPΔSizeStrategy <: AbstractJuMPΔSizeStrategy
DefaultJuMPΔSizeStrategy()

Default strategy which applies default constraints or objectives to a model.

Intended use is for strategies which apply special constraints or objectives to a subset of the vertices.

Also useful when configuring DecoratingJuMPΔSizeStrategies.

Examples

Use to get add special constraints for a subset of vertices:

function NaiveNASlib.addsomeconstraint!(s::MySpecialStrategy, v::AbstractVertex, data)
    if v === s.targetvertex
        # Add some special constraints here
    else
        # Use the default
        NaiveNASlib.addsomeconstraint!(DefaultJuMPΔSizeStrategy(), v::AbstractVertex, data)
    end
end

Use with a DecoratingJuMPΔSizeStrategy:

julia> using NaiveNASlib, NaiveNASlib.Advanced

julia> WithUtilityFun(defaultutility, DefaultJuMPΔSizeStrategy())
WithUtilityFun{typeof(defaultutility), DefaultJuMPΔSizeStrategy}(NaiveNASlib.defaultutility, DefaultJuMPΔSizeStrategy())

ΔNinExact(args...; fallback)

Return a ΔNout{Exact} configured to set nout of the inputs to vertices in args to the given sizes.

For vertices with more than one input, the size change must be expressed as a tuple with one element per input. Use missing to indicate that no special treatment is needed for an input.

Accepts either a Dict or arguments used to create a Dict (e.g. a set of Pairs or a generator).

By default, fallback is set to give a warning and then try again with ΔNinRelaxed(args...).

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNinExact(v1, 3);

julia> ΔNinExact(v1 => -3, v2 => -2);

julia> ΔNinExact(Dict(v1 => 3, v2 => 2));

julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);

julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);

julia> ΔNinExact(v3, 3, missing, 2);

julia> ΔNinExact(v3 => (3, missing, 2), v4 => (-2, 0));

ΔNinRelaxed(args...; fallback)

Same as ΔNinExact except Exact is replaced by Relaxed and fallback set to ΔSizeFailNoOp by default.

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNinRelaxed(v1, 3);

julia> ΔNinRelaxed(v1 => -3, v2 => -2);

julia> ΔNinRelaxed(Dict(v1 => 3, v2 => 2));

julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);

julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);

julia> ΔNinRelaxed(v3, 3, missing, 2);

julia> ΔNinRelaxed(v3 => (3, missing, 2), v4 => (-2, 0));

ΔNin(args...;[fallback])

Splits args into relaxed and exact size changes and creates the appropriate strategy (one of ΔNout{Exact}, ΔNout{Relaxed} or ΔNoutMix).

Fallback strategy may be supplied through the fallback keyword. Defaults to ΔNinRelaxed for all given vertices.

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNin(v1, 3);

julia> ΔNin(v1 => -3, v2 => -2);

julia> ΔNin(Dict(v1 => 3, v2 => 2));

julia> v3 = conc(affine(iv, 4), affine(iv, 5), affine(iv, 6); dims=2);

julia> v4 = conc(affine(iv, 7), affine(iv, 8); dims=2);

julia> ΔNin(v3, 3, missing, 2);

julia> ΔNin(v3 => (3, missing, 2), v4 => (-2, 0));

julia> ΔNin(v3, relaxed(3), missing, 2);

julia> ΔNin(v3 => (relaxed(3), missing, 2), v4 => relaxed(-2, 0));

ΔNoutExact(args...; fallback)

Return a ΔNout{Exact} with fallback set to give a warning and then try again with ΔNoutRelaxed(args...).

Accepts either a Dict or arguments used to create a Dict (e.g. a set of Pairs or a generator).

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNoutExact(v1, 3);

julia> ΔNoutExact(v1 => -3, v2 => -2);

julia> ΔNoutExact(Dict(v1 => 3, v2 => 2));

ΔNoutRelaxed(args...;fallback)

Return a ΔNout{Relaxed} with fallback set to ΔSizeFailNoOp.

Accepts either a Dict or arguments used to create a Dict (e.g. a set of Pairs or a generator).

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNoutRelaxed(v1, 3);

julia> ΔNoutRelaxed(v1 => -3, v2 => -2);

julia> ΔNoutRelaxed(Dict(v1 => 3, v2 => 2));

ΔNout(args...;[fallback])

Splits args into relaxed and exact size changes and creates the appropriate strategy (one of ΔNout{Exact}, ΔNout{Relaxed} or ΔNoutMix).

Fallback strategy may be supplied through the fallback keyword. Defaults to ΔNoutRelaxed for all given vertices.

Examples

julia> using NaiveNASlib, NaiveNASlib.Extend, NaiveNASlib.Advanced

julia> module TestNNLib
           using NaiveNASlib, NaiveNASlib.Extend
           mutable struct Affine{T} W::Matrix{T} end
           NaiveNASlib.nout(a::Affine) = size(a.W, 1)
           NaiveNASlib.nin(a::Affine) = [size(a.W, 2)]
           NaiveNASlib.Δsize!(a::Affine, ins::AbstractVector, outs::AbstractVector) = a.W = parselect(a.W, 2=>ins[1], 1=>outs)
           affine(in, outsize) = absorbvertex(Affine(ones(outsize, nout(in))), in)
           export affine
       end;

julia> using .TestNNLib;

julia> iv = affine(inputvertex("in", 3), 6);

julia> v1 = affine(iv, 4);

julia> v2 = affine(iv, 5);

julia> ΔNout(v1, 3);

julia> ΔNout(v1 => -3, v2 => -2);

julia> ΔNout(Dict(v1 => 3, v2 => 2));

julia> ΔNout(v1, relaxed(2));

julia> ΔNout(v1 => relaxed(2), v2 => -1);

ΔNout <: AbstractJuMPΔSizeStrategy
ΔNout{T, V, F}(Δs::Dict{V, Int}, fallback::F)

Strategy for changing nout of vertices.

For each key-value pair in v, Δ in Δs, change nout(v) by Δ, i.e new size is nout(v) + Δ.

If T == Exact, size change will be added as a constraint to the model which means that the operation will fail if it is not possible to change nout(v) by exactly Δ.

If T == Relaxed, size change will be added as an objective to the model which means that nout(v) might not change by exactly Δ. In addition, a constraint that nout(v) must change is also added.

WithUtilityFun{F, S} <: AbstractΔSizeStrategy
WithUtilityFun(utilityfun::F, strategy::S)

Applies neuron indices selection with strategy and using utilityfun to compute the utility of neurons indices.

Note that utilityfun will override any utility function supplied in function call. Thus it is possible use WithUtilityFun to change utility function e.g. when switching to a fallback strategy.

UpperInsertBound{V<:AbstractDict{<:AbstractVertex, <:Integer}, S, F} <: DecoratingJuMPΔSizeStrategy
UpperInsertBound(vbounds; strategy, fallback)
UpperInsertBound(vs, bound::Integer; strategy, fallback)

Sets upper bound for how many new neurons to insert of each vertex v in vbounds to vbounds[v] and then applies neuron indices selection with strategy (default DefaultJuMPΔSizeStrategy).

Alternatively, sets the upper bound of each vertex in vs to bound. Applies bound to all vertices if vs is empty.

Mostly useful to minimize solver time in cases when one does not want any new neurons as disallowing this allows skipping a couple of constraints.

If it fails, the operation will be retried with the fallback strategy (default ΔSizeFailNoOp).

WithKWargs{KW, S} <: AbstractAfterΔSizeStrategy
WithKwargs(strategy; kwargs...)

Adds keyword argument kwargs to any calls to Δsize! after sizes have been determined.

LogΔSizeExec <: AbstractJuMPΔSizeStrategy
LogΔSizeExec(msgfun)
LogΔSizeExec(msgfun, level::Logging.LogLevel)
LogΔSizeExec(msgfun, level::Logging.LogLevel, andthen::AbstractJuMPΔSizeStrategy)

Logs msgfun(v) at log level level, then executes AbstractJuMPΔSizeStrategy andthen for vertex v.

ThrowΔSizeFailError <: AbstractJuMPΔSizeStrategy
ThrowΔSizeFailError(msg::String)

Throws an ΔSizeFailError with message msg.

ΔSizeFailNoOp <: AbstractJuMPΔSizeStrategy
ΔSizeFailNoOp()

Does not perform any action.

AlignNinToNout{S, F} <: DecoratingJuMPΔSizeStrategy
AlignNinToNout(;vstrat::S, fallback::F)
AlignNinToNout(vstrat::S, fallback::F)

Adds variables and constraints for nin(vi) == nout.(inputs(vi)).

Generally useful when doing structural changes such are removing/adding vertices or edges.

If it fails, the operation will be retried with the fallback strategy (default ΔSizeFailNoOp).

TruncateInIndsToValid{S} <: AbstractΔSizeStrategy
TruncateInIndsToValid()
TruncateInIndsToValid(s::S)

Ensures that all selected input indices are within range of existing input indices after applying s (default DefaultJuMPΔSizeStrategy).

Not needed in normal cases, but certain structural mutations (e.g create_edge!) may cause this to happen due to how constraints are (not) created when original sizes do not align in conjunction with how result of selection is interpreted.

An example of when it is needed is when adding an edge from vertex vi to an invariant vertex vo where nout(vi) > nout(vo). In this case it is expected that the result of the optimization is that the indices 1:nout(vi) of vi shall be kept. However, this will propagate to vo which will be instructed to keep indices it does not have. With this strategy, all indices which are larger than nout(vo) will be replaced by -1 (which indicates that new parameters shall be created)

While this may be considered a flaw in the output selection procedure, it is rare enough so that in most cases when it happens it is the result of a user error or lower level bug. Therefore this strategy is left optional to be used only in cases when mismatches are expected.

Note that in most normal cases, this has no effect since vertices capable of getting new input edges generally don't have parameters.

TimeLimitΔSizeStrategy{S} <: DecoratingJuMPΔSizeStrategy
TimeLimitΔSizeStrategy(limit::Number)
TimeLimitΔSizeStrategy(limit::Number, base::S)

Sets the time limit for the solver to limit. Use strategy base for all other aspects.

TimeOutAction{S,A,F} <: DecoratingJuMPΔSizeStrategy
TimeOutAction(action::A, base::S, fallback::F)
TimeOutAction(base; fallback)

Calls action(model) if JuMP model model has status MOI.TIME_LIMIT after optimization stops. Use strategy base for all other aspects.

Default action is to display a warning and then apply fallback (default ΔSizeFailNoOp).

AfterΔSizeCallback{F, S} <: AbstractAfterΔSizeStrategy
AfterΔSizeCallback(cbfun::F, basestrat::S=ThrowΔSizeFailError())

Calls cbfun(v, Δ, isnout) for all vertices which change size after having been asked to change their sizes as a result of basestrat.

logafterΔsize(printfun=nameorrepr;level=Logging.Info, base=DefaultJuMPΔSizeStrategy())

Return an AfterΔSizeCallback configured to log size changes with log level level.

For a given vertex v, printfun(v) will be used in the logged string.

Strategy base will be used to change sizes (e.g if Δsize!(logafterΔsize(base)) is called).

validateafterΔsize(printfun=nameorrepr; base=DefaultJuMPΔSizeStrategy())

Return an AfterΔSizeCallback configured to validate that sizes (nin and nout) are consistent after a size change and throw a ΔSizeFailError if validation fails.

For a given vertex v, printfun(v) will be used in the error message should the size validation fail.

Strategy base will be used to change sizes (e.g if Δsize!(validateafterΔsize(base)) is called).