Question about downstream client usage of this project

[dbeavon]

Does the new "azure synapse workspaces" (public preview) make use of this project? They claim to offer the spark platform, along with language support for .Net. I've done a bit of research but haven't found any document that clearly makes a link between that product and this project.

While I haven't tried it, I read that they also support using C#.Net for doing Spark development. I would guess that they use software which was branched from this project (or is very similar). I'm wondering if they contribute to this project, and/or will ever import their changes back into this community project?

One reason for the question is to find if they are using Spark 3.0 yet. Perhaps they rely on a prior version, as we do in this community. I would like to start using .Net for my spark development, but I don't want to do so until spark 3.0 support is available (see this issue: #633 )

Another reason for the question is to determine if azure synapse would ever offer technical support for ".Net for Spark". If they are a downstream client of this project, then it seems clear that they would need to offer support for it as well. But if they have their own implementation of .Net for Spark, then they will not be supporting this project in any way.

I've already had many conversations with "databricks" and "azure-databricks" about this github project and, unfortunately, neither of them seem very keen on .Net. Nor will they support .Net for Spark (...except to the same extent that they support any third party extensions)

I'm hoping that "azure synapse workspaces" will become one of the beneficiaries of the work done in this project (assuming they don't have their own an entirely separate implementation of .net for spark). And I'm hoping that synapse will become an avenue for customers to receive formal support from Microsoft.

[rapoth]

@dbeavon Thanks for your question!

Azure Synapse Workspaces uses the latest version of .NET for Spark. Usually, the release first happens on GitHub and then tested/deployed onto Synapse. This is possible since the team responsible for Synapse also contributes quite heavily to the OSS version. Rest assured, I am not aware of any plans of "forking" the project in the context of any Microsoft offering including Azure Synapse Analytics. That being said, yes, at the moment, technical support for .NET for Spark comes when you use Azure Synapse Analytics Workspaces and a subset of the team that contributes to the OSS project are available to help through technical support as well.

In short, the OSS .NET for Spark team is a superset of the team that supports .NET for Spark in Synapse; you will get official and technical support when you use the product in Azure Synapse. And of course, in the spirit of open source, we will continue to make sure (via best-effort) that .NET for Spark can be run across all cloud providers (although supporting it as a first-class product is still within the respective cloud provider's control) - this is similar to any OSS library at the end of the day.

I hope this helps. If you have any other questions, please let us know.

PS: Regarding support for Spark 3.0, from an OSS standpoint, we are working aggressively towards enabling it in the next few weeks, after which, it will also be available via the Spark 3.0 offering of Azure Synapse Analytics (whenever it comes out).

[dbeavon]

This is really great news.

Having the option to pay someone (... anyone ...) for a support contract is pretty important to certain folks in corporate management. I would guess that for some companies, that is the main thing that is blocking them from using .Net with Spark at this time. I should set up a company called "Spark.Net Inc" and take all those checks!

I'm very eager to use .Net for Spark. This is going to be a big deal for data warehousing. I am a bit surprised why databricks/azure-databricks are willing to get left behind! They don't seem very eager to jump on board just yet...

I'm also surprised my google searches don't turn up a lot more information about .Net for Spark and Synapse Workspaces. There was quite a bit of fanfare when Spark 3.0 was released ... and I think the availability of the .Net language for Spark is even better news.

Maybe people will pay more attention after "Workspaces" is RTM'ed. Hopefully that happens soon. It would certainly be a nice Christmas present if it happens in 2020.

[rapoth]

Thank you for your kind words! And yes, we look forward to seeing a company named that way 👍 :)

The reason you are not finding a lot of documentation explaining the link between .NET for Spark and Synapse Workspace is primarily due to the fact that the latter is currently in preview. I'm sure we will button up the documentation. In fact, we are doing a doc refresh as part of #671. If you are looking for some specific documentation, please give us some feedback and we will add it to our backlog.

Thanks again for your interest! Hopefully, when you do try out the project, you will like it.