Skip to content

Spark 3.0 readiness part 2 #649

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Sep 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
03b7939
Adding section for UDF serialization
Niharikadutta Apr 20, 2020
4ef693d
removing guides from master
Niharikadutta Apr 20, 2020
81145ca
Merge latest from master
Niharikadutta May 6, 2020
e4b81af
merging latest from master
Niharikadutta May 7, 2020
4c32173
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jun 2, 2020
4987a09
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jun 14, 2020
ca9612e
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jun 16, 2020
f581c86
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jun 20, 2020
086b325
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jun 23, 2020
2f72907
Merge remote-tracking branch 'upstream/master'
Niharikadutta Jul 25, 2020
6bab996
CountVectorizer
Jul 27, 2020
e2a566b
moving private methods to bottom
Jul 27, 2020
5f682a6
changing wrap method
Jul 28, 2020
31371db
setting min version required
Jul 31, 2020
60eb82f
undoing csproj change
Jul 31, 2020
ed36375
member doesnt need to be internal
Jul 31, 2020
c7baf72
too many lines
Jul 31, 2020
d13303c
removing whitespace change
Jul 31, 2020
f5b477c
removing whitespace change
Jul 31, 2020
73db52b
ionide
Jul 31, 2020
98f5e4d
Merge remote-tracking branch 'upstream/master'
Niharikadutta Aug 7, 2020
4c5d502
Merge remote-tracking branch 'upstream/master'
Niharikadutta Aug 10, 2020
a766146
Merge branch 'master' into ml/countvectorizer
GoEddie Aug 12, 2020
ad6bced
Merge branch 'ml/countvectorizer' of https://github.com/GoEddie/spark
Niharikadutta Aug 13, 2020
8e1685c
Revert "Merge branch 'master' into ml/countvectorizer"
Niharikadutta Aug 13, 2020
255515e
Revert "Merge branch 'ml/countvectorizer' of https://github.com/GoEdd…
Niharikadutta Aug 13, 2020
a44c882
Merge remote-tracking branch 'upstream/master'
Niharikadutta Aug 14, 2020
3c2c936
fixing merge errors
Niharikadutta Aug 14, 2020
88e834d
removing ionid
Niharikadutta Aug 20, 2020
a13de2d
Merge branch 'master' of github.com:Niharikadutta/spark
Niharikadutta Aug 21, 2020
13d0e4a
Merge remote-tracking branch 'upstream/master'
Niharikadutta Aug 24, 2020
595b141
Merge remote-tracking branch 'upstream/master'
Niharikadutta Aug 29, 2020
decfa48
Merge remote-tracking branch 'upstream/master'
Niharikadutta Sep 2, 2020
40470a4
Adding Function APIs for Spark 3.0
Niharikadutta Sep 2, 2020
b602b91
nit: function name
Niharikadutta Sep 3, 2020
1fe09b8
changes
Niharikadutta Sep 3, 2020
c12c341
Adding tests
Niharikadutta Sep 3, 2020
2031bbf
Merge branch 'master' into nidutta/spark3.0readiness_part2
imback82 Sep 8, 2020
49ea4e2
Merge branch 'master' into nidutta/spark3.0readiness_part2
imback82 Sep 12, 2020
968881e
Update src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/FunctionsTests.cs
Niharikadutta Sep 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions src/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/FunctionsTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -796,5 +796,55 @@ public void TestSignaturesV2_4_X()
col = MapConcat(col);
col = MapConcat(col, col);
}

/// <summary>
/// Test signatures for APIs introduced in Spark 3.0.*.
/// </summary>
[SkipIfSparkVersionIsLessThan(Versions.V3_0_0)]
public void TestSignaturesV3_0_X()
{
Column col = Column("col");

Assert.IsType<Column>(XXHash64());
Assert.IsType<Column>(XXHash64(col));
Assert.IsType<Column>(XXHash64(col, col));

Assert.IsType<Column>(Split(col, "\t", 1));
Assert.IsType<Column>(Split(col, "\t", -1));

Assert.IsType<Column>(Overlay(col, col, col));
Assert.IsType<Column>(Overlay(col, col, col, col));

Assert.IsType<Column>(AddMonths(col, col));

Assert.IsType<Column>(DateAdd(col, col));

Assert.IsType<Column>(DateSub(col, col));

var options = new Dictionary<string, string>() { { "hello", "world" } };
Assert.IsType<Column>(SchemaOfJson(col, options));

Assert.IsType<Column>(MapEntries(col));

Column schemaCol = SchemaOfCsv("[{\"col\":0}]");
Assert.IsType<Column>(FromCsv(col, schemaCol, options));

Assert.IsType<Column>(SchemaOfCsv(col));
Assert.IsType<Column>(SchemaOfCsv(col, options));

Assert.IsType<Column>(ToCsv(col));
Assert.IsType<Column>(ToCsv(col, options));

Assert.IsType<Column>(Years(col));

Assert.IsType<Column>(Months(col));

Assert.IsType<Column>(Days(col));

Assert.IsType<Column>(Hours(col));

Assert.IsType<Column>(Bucket(Lit(1), col));
Assert.IsType<Column>(Bucket(1, col));
}
}
}
Loading