Skip to content

Spark 3.0 readiness part 2 #649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Sep 14, 2020

Conversation

Niharikadutta
Copy link
Collaborator

@Niharikadutta Niharikadutta commented Sep 2, 2020

This PR adds support for following APIs added in Spark 3.0:

Functions APIs

  • scala/python def xxhash64(cols: Column): Column
  • scala def split(str: Column, pattern: String, limit: Int): Column
  • scala/python def overlay(src: Column, replace: Column, pos: Column, len: Column): Column
  • scala def overlay(src: Column, replace: Column, pos: Column): Column
  • scala def add_months(startDate: Column, numMonths: Column): Column
  • scala def date_add(start: Column, days: Column): Column
  • scala def date_sub(start: Column, days: Column): Column
  • scala def schema_of_json(json: Column, options: java.util.Map[String, String]): Column
  • scala/python def map_entries(e: Column): Column
  • scala/python def from_csv(e: Column, schema: StructType, options: Map[String, String]): Column
  • scala def from_csv(e: Column, schema: Column, options: java.util.Map[String, String]): Column
  • scala def schema_of_csv(csv: String): Column
  • scala def schema_of_csv(csv: Column): Column
  • scala/python def schema_of_csv(csv: Column, options: java.util.Map[String, String]): Column
  • scala/python def to_csv(e: Column, options: java.util.Map[String, String]): Column
  • scala def to_csv(e: Column): Column
  • scala def years(e: Column): Column
  • scala def months(e: Column): Column
  • scala def days(e: Column): Column
  • scala def hours(e: Column): Column
  • scala def bucket(numBuckets: Column, e: Column): Column
  • scala def bucket(numBuckets: Int, e: Column): Column

This PR addresses part of #633

Niharikadutta and others added 30 commits April 19, 2020 22:31
This reverts commit a766146, reversing
changes made to 73db52b.

Reverting countvectorizer changes
…ie/spark"

This reverts commit ad6bced, reversing
changes made to 4c5d502.

reverting countvectorizer changes -2
@Niharikadutta Niharikadutta changed the title [WIP] Spark 3.0 readiness part 2 Spark 3.0 readiness part 2 Sep 4, 2020
@imback82 imback82 added the enhancement New feature or request label Sep 12, 2020
@imback82 imback82 added this to the 1.0.0 milestone Sep 12, 2020
Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for one nit comment, thanks @Niharikadutta!

Copy link
Member

@suhsteve suhsteve left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@imback82 imback82 merged commit 54a42cf into dotnet:master Sep 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants