+1, if we don't remove this, we should rewrite it, this is currently blocking the itegration of the code base.

I generally agree with it, data quality is an important module of our project we not only have a data quality type task but also have a data quality result sub-module in UI, if we want to remove it we should earn enough +1 vote in our community

Yes, I think this operation should earn enough +1 vote.

I generally agree with it, data quality is an important module of our project we not only have a data quality type task but also have a data quality result sub-module in UI, if we want to remove it we should earn enough +1 vote in our community

+1

+1, I think data quality should be stripped of DS and maintained as an independent plug-in.

+1, The current data quality task type is barely used in our team

+1

-1 For me

We use data quality in our scene, I can maintain or rewrite this module

-1 For me

We use data quality in our scene, I can maintain or rewrite this module

You can create a new DSIP issue and put full design detail in it for discussion.

Reminder: this is not a vote now, it's currently a discussion. Please provide your detailed opinions as far as possible, and this will help the community make a better choice. By the way, voting should take place in the dev mailing list.

+1 ,In fact, some things can be added as plug-ins, and DS should focus on the core functions. If DS want to build an ecosystem, but DS do what other projects are good at, and then DS are not as good as them,DS will be dissed by users. Moreover, other projects will not play withDS because of the overlap of functions. This is not a good idea.

I have a question that users have used this module in their production environment. If we remove DQ module, how can they upgrade their environment?

I have a question that users have used this module in their production environment. If we remove DQ module, how can they upgrade their environment?

We'll add it to incompatible change docs.

+1

+1,Supporting the removal of this module. The code quality is not good and there are multiple CVE vulnerabilities which are fatal problems. At the same time, data quality management is closely related to business data and is not considered as the core advantage function point of our platform.

-1 For me
I think this is a more important part of big data, dolphins are mainly used by data developers, if you abandon this function, then they can only write their own programs to verify the data, but if the data developers can not write code?

+1
Scheduling should focus on scheduling functions itself, data quality should be part of the data middleware, and it is recommended to separate out a data quality platform independent of it. It should support various database engine or popular computing engine gateways, such as Kyuubi, and monitor table quality from the data middleware side. Then, it can feedback to the scheduling task through the DS API whether it is blocked or not, thus forming a complete data loop.

-1 For me
Data quality is a very good function for me. I don't need to integrate third-party open source plug-ins to do this, reducing the complexity of system use and operation and maintenance.

-1 For me
Data quality is a very good function for me. I don't need to integrate third-party open source plug-ins to do this, reducing the complexity of system use and operation and maintenance.

+1 Too dependent on external components, not belonging to its own field. Support removal.

Our team use data quality, although it's hard to use. So I wish a rewrite instead of a removal.

-1 For me

I agree to move codebase of DQ from dolphinscheduler main project into another ecological project just like the relationship of apache flink cdc, flink connector kafka and apache flink.

https://github.com/apache/flink-cdc
https://github.com/apache/flink-connector-kafka
https://github.com/apache/flink

The problem is NOT voting remove DQ module or not but we should refactor DQ module to improve code quality to make it extensible to support various DQ tasks.

It cannot be denied that DQ is a good application scenarios for dolphinscheduler.

Unfortunately current DQ task can NOT transfer result to downstream task, so it can NOT fulfil the requirement of automatic data reconciliation.

-1，depend on hadoop/spark framework and too many bugs

I think this module should not be removed, data quality is a very important part of big data, the current mainstream big data products have the data quality module, if removed, developers may need to write their own programs to complete the quality check task, which is a very painful thing for developers.our team is using data quality, although it's hard to use. So I hope it should be rewritten instead of deleted.