AssertionError(msg="`Union{}` typed field should be strictly undefined")

[lgoettgens]

The Oscar.jl tests seem to be running into this assertion now after #57541 tried to fix oscar-system/Oscar.jl#4648:

 From worker 4:	Internal error: during type inference of
      From worker 4:	(::Type{Oscar.IntersectionTheory.AbstractVarietyMap{V1, V2} where V2<:Oscar.IntersectionTheory.AbstractVarietyT where V1<:Oscar.IntersectionTheory.AbstractVarietyT})(Oscar.IntersectionTheory.AbstractVariety, Oscar.IntersectionTheory.AbstractVariety, Array{Oscar.MPolyQuoRingElem{Oscar.MPolyDecRingElem{Nemo.QQFieldElem, Nemo.QQMPolyRingElem}}, 1}, AbstractAlgebra.Generic.FunctionalMap{Oscar.MPolyQuoRing{Oscar.MPolyDecRingElem{Nemo.QQFieldElem, Nemo.QQMPolyRingElem}}, Oscar.MPolyDecRing{Nemo.QQFieldElem, Nemo.QQMPolyRing}})
      From worker 4:	Encountered unexpected error in runtime:
      From worker 4:	AssertionError(msg="`Union{}` typed field should be strictly undefined")
      From worker 4:	PartialStruct at ./coreir.jl:62

https://github.com/oscar-system/Oscar.jl/actions/runs/13611555768/job/38059503585#step:11:677

Originally posted by @benlorenz in #57541 (comment)

cc @benlorenz @fingolfin

ping @aviatesk @serenity4 @topolarity as you guys worked on #57541

[serenity4]

The following case seems to still be unsupported:

julia> struct A{T}; x::Any; y::T; A{T}(x) where {T} = new{T}(x); end

julia> Core.PartialStruct(A{Union{}}, Union{Nothing,Bool}[false, nothing], Any[Int, Union{}])
ERROR: AssertionError: `Union{}` typed field should be strictly undefined
Stacktrace:
 [1] Core.PartialStruct(typ::Type, undefs::Vector{Union{Nothing, Bool}}, fields::Vector{Any})
   @ Base ./coreir.jl:62

Perhaps we should adjust the check to also allow undef === nothing, instead of requiring undef to be true for Union{} fields.

While a field with type Union{} implies undef === true, it seems a bit too much to me to require all callsites to check for Union{} in every field type to determine whether the corresponding undef entry should be set to true before constructing a PartialStruct.

[aviatesk]

I'm open to loosening the assertion if we need to, but I'd really like to understand why it's failing in the first place. When I ran the Oscar tests locally, I hit an inference stack overflow error and couldn't even get to the point where the assertion fails. Do we have a reliable way to reproduce this?

[serenity4]

I can try to reproduce and see if we can get to an understanding about what's going on

[serenity4]

Here it goes:

julia> using Oscar, Oscar.IntersectionTheory

julia> code_typed(degeneracy_locus, (Oscar.IntersectionTheory.AbstractBundle{Oscar.IntersectionTheory.AbstractVariety}, Oscar.IntersectionTheory.AbstractBundle{Oscar.IntersectionTheory.AbstractVariety}, Int64))

ERROR: AssertionError: `Union{}` typed field should be strictly undefined
Stacktrace:
  [1] Core.PartialStruct(typ::Type, undefs::Vector{Union{Nothing, Bool}}, fields::Vector{Any})
    @ Base ./coreir.jl:62
  [2] Core.PartialStruct(::Compiler.InferenceLattice{Compiler.ConditionalsLattice{…}}, typ::Any, undefs::Vector{Union{…}}, fields::Vector{Any})
    @ Compiler ./../usr/share/julia/Compiler/src/typelattice.jl:765
  [3] form_partially_defined_struct(𝕃ᵢ::Compiler.InferenceLattice{Compiler.ConditionalsLattice{Compiler.PartialsLattice{Compiler.ConstsLattice}}}, obj::Any, name::Any)
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/abstractinterpretation.jl:2162
  [4] abstract_call_builtin(interp::Compiler.NativeInterpreter, f::Core.Builtin, ::Compiler.ArgInfo, sv::Compiler.InferenceState)
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/abstractinterpretation.jl:2118
  [5] abstract_call_known(interp::Compiler.NativeInterpreter, f::Any, arginfo::Compiler.ArgInfo, si::Compiler.StmtInfo, sv::Compiler.InferenceState, max_methods::Int64)
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/abstractinterpretation.jl:2664
  [...] (truncated)

It errors when constructing a PartialStruct with

objt0 = Oscar.MPolyAnyMap{<:AbstractAlgebra.MPolyRing{T}, Union{}, Nothing} where T<:AbstractAlgebra.FieldElem
undefs = Union{Nothing, Bool}[0, 0, 0, 0, nothing, nothing, 0]
fields = Any[AbstractAlgebra.MPolyRing{T} where T<:AbstractAlgebra.FieldElem, Union{}, Nothing, Vector, Any, Vector{Int64}, Dict{Symbol, Any}]

where the Oscar.MPolyAnyMap type is defined here: https://github.com/oscar-system/Oscar.jl/blob/43891614c618991a89f198068f4d514d89a3e3d6/src/Rings/MPolyMap/Types.jl#L4-L42

We can see it being instantiated with a C = Union{} type parameter, and because the inner constructor calls new with 4 values here, datatype_min_ninitialized returns 4 and therefore partialstruct_init_undefs fills in the corresponding undefs entries with false.

Therefore, the fix I suggested would not even be enough. Even if we may wonder why it occurs, isn't that technically a valid Julia type that we may see in inference? (even if it would never be constructed at runtime)

I guess a possible fix would be to look at fields in partialstruct_init_undefs and set the corresponding entry to true for Union{} fields. In theory it might be enough so long as any construction of PartialStruct uses that function to define undefs.

[aviatesk]

Thanks so much for creating the reproducer!

From your explanation, it seems like the inference for new(OscarMPolyAnyMap{...}, ...) is skipping the check at

julia/Compiler/src/abstractinterpretation.jl

Lines 3092 to 3093 in d369c10

	at = tmeet(𝕃ᵢ, at, ft)
	at === Bottom && return RTEffects(Bottom, TypeError, EFFECTS_THROWS)

for some reason.

Because of that, an object type that shouldn't exist at runtime is being propagated around. This is likely causing form_partially_defiend_struct, which expects a valid object type, to throw an error.

However, I still can't reproduce the issue on my end though:

julia> using Oscar, Oscar.IntersectionTheory
  ___   ____   ____    _    ____
 / _ \ / ___| / ___|  / \  |  _ \   |  Combining ANTIC, GAP, Polymake, Singular
| | | |\___ \| |     / _ \ | |_) |  |  Type "?Oscar" for more information
| |_| | ___) | |___ / ___ \|  _ <   |  Manual: https://docs.oscar-system.org
 \___/ |____/ \____/_/   \_\_| \_\  |  Version 1.3.0

julia> code_typed(degeneracy_locus, (Oscar.IntersectionTheory.AbstractBundle{Oscar.IntersectionTheory.AbstractVariety}, Oscar.IntersectionTheory.AbstractBundle{Oscar.IntersectionTheory.AbstractVariety}, Int64))
Warning: detected a stack overflow; program state may be corrupted, so further execution might be unreliable.
ERROR: StackOverflowError:

• Julia version: Version 1.13.0-DEV.193 (2025-03-11)
• Oscar version: 1.3.0

For fixing the root cause, I think abstract_eval_new should correctly return Union{} in cases like this, since there might be other places that expect runtime-valid PartialStruct. What do you think?

[fingolfin]

@aviatesk that's odd. We had to apply a change to our code to work around a stack overflow in code inference in Julia (yet another regression on master), but this workaround should be in the Oscar 1.3.0 release... But perhaps just in case, try with latest Oscar master?

[aviatesk]

Sorry, it looks like the issue was caused by changes I had locally (I didn't notice because those changes didn't cause any error when running the base tests).

I looked into the problem, and there's nothing wrong with abstract_eval_new itself. It seems like Union{} can be contained within Type{...}, and form_partially_defined_struct needs to be adjusted to avoid creating a runtime invalid PartialStruct in those cases.

[serenity4]

I still can't reproduce the issue

perhaps just in case, try with latest Oscar master

It worked for me with Oscar master and a 3-day old Julia nightly version, in case that helps.

I managed to reproduce the pattern that generates an invalid type:

julia> struct A{C<:Union{Int,Float64}} val::C end

julia> f(a::A{<:String}) = a
f (generic function with 1 method)

julia> code_typed(f, (A,))
1-element Vector{Any}:
 CodeInfo(
1 ─     return a
) => A{Union{}}

which happens in Oscar when dispatching to this function with an instance of this type, which eventually branches on isdefined here which then leads to inference building a PartialStruct, triggering the error. Essentially this:

# make it mutable with at least one possibly undefined field so we later form a PartialStruct
julia> mutable struct MA{C<:Union{Int,String}} a::C; b::Any; MA(a::T) where {T} = new{T}(a); end

julia> f(a::MA{<:Symbol}) = isdefined(a, :b) ? invokelatest(identity, a) : a.a
f (generic function with 1 method)

julia> code_typed(f, (MA,))
ERROR: AssertionError: `Union{}` typed field should be strictly undefined
Stacktrace:
  [1] Core.PartialStruct(typ::Type, undefs::Vector{Union{Nothing, Bool}}, fields::Vector{Any})
    @ Base ./coreir.jl:62
  [2] Core.PartialStruct(::Compiler.InferenceLattice{Compiler.ConditionalsLattice{…}}, typ::Any, undefs::Vector{Union{…}}, fields::Vector{Any})
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/typelattice.jl:765
  [3] form_partially_defined_struct(𝕃ᵢ::Compiler.InferenceLattice{Compiler.ConditionalsLattice{Compiler.PartialsLattice{Compiler.ConstsLattice}}}, obj::Any, name::Any)
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/abstractinterpretation.jl:2164
  [4] abstract_call_builtin(interp::Compiler.NativeInterpreter, f::Core.Builtin, ::Compiler.ArgInfo, sv::Compiler.InferenceState)
    @ Compiler ~/.julia/juliaup/julia-nightly/share/julia/Compiler/src/abstractinterpretation.jl:2121
  [5] abstract_call_known(interp::Compiler.NativeInterpreter, f::Any, arginfo::Compiler.ArgInfo, si::Compiler.StmtInfo, sv::Compiler.InferenceState, max_methods::Int64)
  [...]

IIUC the root cause is that Union{} is the only type that satisfies <:Symbol and <:Union{Int,Float64}, and is therefore assumed when specializing the method f to a UnionAll A. Is that something we could change?