Design a unified target graph for projects and solutions

[Nirmal4G]

Goals

• Basic targets for Project dev loop support.
• Basic targets for Solution dev loop support.
• Integration targets for Project<->Project.
• Integration targets for Project<->Solution.
• Graph Build support for both project and solution.
• TBA

Targets List:

• Restore
• Build
• Clean
• Rebuild (wrapper around Clean + Build)
• Test
• Run
• Publish

NOTE: Design of the Target Graph is not final!

[jzabroski]

This is pretty neat, but I think it's less clear what the unified target graph means across various SDKs.

I always thought the Maven way of thinking about this was a lot cleaner and better followed the principles of dependency inversion by declaring an interface in the form of a goal that the plug-in targets had to achieve. I think that also aligns closely with what the original "The Build Master" textbook on MSBuild advocated for. If you think broadly about "what do we really want that other ecosystems have" and "what we have today (via binlog replays)" you can see that the main gap is still in traceability of inputs, mostly around goal-directed values passed into black boxes for these targets vs the targets themselves. There's no way to "observe" what the black box does in generating outputs/task items.

[Nirmal4G]

I think it's less clear what the unified target graph means across various SDKs.

May be I should come up with better naming. That said, with the current MSBuild target graph, although extensible, they are not what I call a concept based or more accurately—action based graph.

What I mean is, if you take a ProjectReference or any ~Reference item which can be a link/reference to other things in the project/solution or even outside of the project/solution cone, typically you would have certain behavior control on how the references are consumed by the project/solution. You would also have a way of "collecting" those item, do some "preparation" like transforms, de-duplication, etc... and then you would "resolve" those references into assets that the project/solution can consume.

This is just one example of an action based target graph design, I'm experimenting with. Likewise, you'd have a similar pattern of thought for other concepts that exist in the Build System (Restore, Build, Test, Publish, etc...). Remember that MSBuild is an orchestrator and not a build system. It was called XMake for a reason! The common tasks/targets that MSBuild ships with, is the build system or build definition but only designed for .NET only. That's why C++ team, when they migrated from the old system, they had to have their own Common tasks/targets and not use the ones that ship with MSBuild.

I always thought the Maven way of thinking about this was a lot cleaner and better followed the principles of dependency inversion by declaring an interface in the form of a goal that the plug-in targets had to achieve.

I personally don't like certain things about Maven, first is Java but that's subjective 😅. Then again, I have to say, when I was working on some java projects, Maven felt a lot cleaner, even though it was verbose, it had defined the language better. Once you know the language, the concepts, you can easily understand a POM file on its own. Maven and other systems like CMake, Eclipse IDE is where I saw specialization done best. But nowhere I saw generalization except in MSBuild and Visual Studio. The VS project system is one of the most extensible open platform I've ever come across. Nothing comes even close. What I mean is the design choices and philosophy of the older system. Not the garbage implementation we have today.

The main gap is still in traceability of inputs, mostly around goal-directed values passed into black boxes for these targets vs the targets themselves. There's no way to "observe" what the black box does in generating outputs/task items.

TRUE THAT. As you said binlog works but it's still a text log encoded in binary format. It depends on the caller to log the message as every other logging mechanisms work. But there's no evaluation level tracing in MSBuild orchestration process. There's also no scope or encapsulation concept applicable to properties, items or things that are either read-only or once-defined. But I for one, need a step-though debugger and tracer for MSBuild eval engine, badly! With that my life would be complete. 😇

TLDR; XMake design and philosophy was better.

You could literally have any kind of action to be done with MSBuild (at least in the XMake days). You still can but a lot of workarounds entered into the XMake and turned it into MSBuild, we all know and love today and made it bloat. Though the recent improvements to the Schema made it simpler, it suffers from things that every other "All-in-One" Swiss army tools suffer: Generalization and Specialization. Without the right balance of those, you would have all sorts of issues. These could stem from historical baggage that plagues the .NET Community or even issues that are a consequence of certain decisions that allowed specialization into an otherwise mostly a generalized product.

Even Miguel-de-Icaza who ported .NET Framework over to Mono has said some beautiful, yet sad things about XBuild (MSBuild on Mono). This is not the first instance. M$FT could get hands on some of the futuristic tech/concepts/ideas and they butcher it with "business" decisions. I have seen too many to list. All the products that ever produced by M$FT could have the potential to be better than anything at that time. In fact, every single thing was ahead of their time. But alas, it's fate.

AnyBuild is there too but I won't go too much into it. One thing I'll say is that AnyBuild could have been MSBuild Extended!? It was created for whole different set of requirements/use cases and even now it does work well in those scenarios. But what I'm hoping to have, is to learn from Build Systems, and to build a generalized orchestrator and a specialized declaration system to achieve what MSBuild couldn't. Everybody who loved and hated MSBuild, was hoping to see a XAML/POCO based system.

When I say XAML build system, I don't mean TFS XAML, NO, that was a horrible application of XAML. With XAML, you could have the most verbose doc or the most simplest doc. It's up to the writer. That's also why I'm experimenting with a XAML syntax based MSBuild fork privately. I'm still working up the design and MVP to check feasibility but it's just an experiment to learn about build systems.

Finally,

what do we really want that other ecosystems have?

I'm not even going there. What I want, not even a single existing build system offers. It's either too much generalization and too much specialization. Nothing that says, this is a declarative task, orchestrator, manager and runner. NOTHING! Believe me, I have looked all over.

When you are designing a solution for a problem through software, what you need is not just the right set of tools to create a solution at that time. NO! NAH! That only works when you're physically solving the problem. But when you are solving it via software, one of its core philosophy is Don't Repeat Yourself. Then you have Create once, innovate forever. So, What I need is a single solution that is created once for this kind of problem and can be innovated forever. It's like building the DNA of the software world. Playing God. That's what made me interested in software and friends.

User does not exist

Bots are not allowed to review

[jzabroski]

I think it's less clear what the unified target graph means across various SDKs.

May be I should come up with better naming. That said, with the current MSBuild target graph, although extensible, they are not what I call a concept based or more accurately—action based graph.

What I mean is, if you take a ProjectReference or any ~Reference item which can be a link/reference to other things in the project/solution or even outside of the project/solution cone, typically you would have certain behavior control on how the references are consumed by the project/solution. You would also have a way of "collecting" those item, do some "preparation" like transforms, de-duplication, etc... and then you would "resolve" those references into assets that the project/solution can consume.

@chethusk Related to our conversation on IncrementalClean behavior and where I feel MSBuild currently falls short in comparison to Google Bazel build process. I had forgotten @Nirmal4G and I discussed this 4 years ago, until I saw he recently began iterating on this idea again.

[Nirmal4G]

To be clear, I do have a private repo where I have fleshed out the design fully, and it's already in use in the production builds of our projects and solutions. But the private versions of these SDKs were hacked hastily in response to the 20 years of baggage in .NET!

I'm reviewing the changes commit by commit and extracting only the changes and designs that make sense in the long run. Recently I tried doing it with Copilot and other agents but damn they are so bad with deterministic changes and new ideas. I just wasted 2 months and $120 worth of token costs.

Coming to the Incremental Build/Clean with MSBuild, it has learned some optimizations, but it has a long way to go. The reason is well..., is because of the 20 years of baggage.

New Build schema and a modern orchestrator is in the works

We also have a trimmed-down version of MSBuild (affectionately called NSBuild) with a lot of changes—I mean, an entirely different XAML/XML hybrid schema. For example, there's a new Solution root element akin to the existing Project element along with a generic Build/Common (we haven't decided on the name for this one) root element.

You can have typed and hierarchical definitions of both Properties and Items. They also have access levels. You can save those definitions inside the Assembly along with Build Functions, Tasks and even Targets themselves and import them. The build logic inside SDKs is scoped to the SDKs themselves. You can also import a shared piece of logic multiple times.

Every single feature that a modern orchestrator is expected to have has been designed. And I'm still not finished. Copilot (Grok and Claude models) has been helpful in writing tests, code coverage and optimizing the C#/.NET code that was hastily written, but other than that it spits out slop for new ideas. We have to give it a skeleton to have it flesh it out so to speak.

[jzabroski]

Set-up a sponsorship and I am happy to contribute to help reimburse some of those costs. I'd love for this to get to a state where @baronfel could consider it as a MSBuild vNext.

[jzabroski]

We also have a trimmed-down version of MSBuild (affectionately called NSBuild) with a lot of changes—I mean, an entirely different XAML/XML hybrid schema. For example, there's a new Solution root element akin to the existing Project element along with a generic Build/Common (we haven't decided on the name for this one) root element.

Awhile ago Nick Craver (of StackOverflow.com fame) had proposed an alternative format for project files. StackOverflow.Redis uses the Microsoft.Build.Traversal SDK, but the problem historically is that Visual Studio could not read this format. Microsoft.Build.Traversal is more elegant than slnf (Solution Filters) as well.

[jzabroski]

Here is an example of how Nick does excludes which would fully handle solution filters: https://github.com/NickCraver/StackExchange.Exceptional/blob/main/Build.csproj

[Nirmal4G]

Set-up a sponsorship and I am happy to contribute to help reimburse some of those costs. I'd love for this to get to a state where @baronfel could consider it as a MSBuild vNext.

Thank you. It means so much. M$FT is notorious for its stance on backwards compat. I don't think they'll overhaul the entire MSBuild ecosystem just for structural and cosmetic improvements. Although broken and hard to learn, what we have now works.

Also, I'm learning and experimenting here. So, right now, I do not want to waste anyone's hard earned money on this without giving them a value for their support. But to see that someone on the internet is willing to sponsor a project that's not even started, incomplete and downright does not build on their system is in itself an achievement that no other project had the honor of, I think, as far as I know.

Once again, Thanks so much.

[Nirmal4G]

A while ago Nick Craver (of StackOverflow.com fame) had proposed an alternative format for project files... But the problem historically is that Visual Studio could not read this format.

Yeah, I knew about that. My schema also took some inspiration from those as well. Visual Studio is not even a M$FT product, originally. It was originally envisioned as a universal IDE for all workloads like Music/Film DAWs, including code.

There is still some proprietary software that still uses Visual Studio v1 Solution and Project files. I took some inspiration from those as well.

There's also controversial yet famous Visual Studio user-voice request that suggested to unify Solution, Project and Common build files concepts into serialized POCO that could be supported by any data format. They even suggested XAML as the declarative build language.

I was inspired by that idea. I want to see how far I can take this.

Microsoft.Build.Traversal is more elegant than slnf (Solution Filters) as well.

The slnf was an abomination that was hastily done to respond to the bad name that solution files started to accumulate after they simplified project files but not the solution files. So, yes, I can understand why you would think that. I hate them with passion as well! 😅