Skip to content

Commit 304e021

Browse files
authored
Remove unsupported consecutive paramaters from doc (#1082)
Signed-off-by: Peng Huo <[email protected]>
1 parent c0c315f commit 304e021

File tree

1 file changed

+2
-22
lines changed

1 file changed

+2
-22
lines changed

docs/ppl-lang/ppl-dedup-command.md

+2-22
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@
88
- [Example 1: Dedup by one field](#example-1-dedup-by-one-field)
99
- [Example 2: Keep 2 duplicates documents](#example-2-keep-2-duplicates-documents)
1010
- [Example 3: Keep or Ignore the empty field by default](#example-3-keep-or-ignore-the-empty-field-by-default)
11-
- [Example 4: Dedup in consecutive document](#example-4-dedup-in-consecutive-document)
1211
- [Limitation](#limitation)
1312

1413
### Description
@@ -18,12 +17,11 @@ Using `dedup` command to remove identical document defined by field from the sea
1817
### Syntax
1918

2019
```sql
21-
dedup [int] <field-list> [keepempty=<bool>] [consecutive=<bool>]
20+
dedup [int] <field-list> [keepempty=<bool>]
2221
```
2322

2423
* int: optional. The ``dedup`` command retains multiple events for each combination when you specify <int>. The number for <int> must be greater than 0. If you do not specify a number, only the first occurring event is kept. All other duplicates are removed from the results. **Default:** 1
2524
* keepempty: optional. if true, keep the document if the any field in the field-list has NULL value or field is MISSING. **Default:** false.
26-
* consecutive: optional. If set to true, removes only events with duplicate combinations of values that are consecutive. **Default:** false.
2725
* field-list: mandatory. The comma-delimited field list. At least one field is required.
2826

2927

@@ -91,23 +89,6 @@ PPL query:
9189
+------------------+-----------------------+
9290

9391

94-
### Example 4: Dedup in consecutive document
95-
96-
The example show dedup the consecutive document.
97-
98-
PPL query:
99-
100-
os> source=accounts | dedup gender consecutive=true | fields account_number, gender;
101-
fetched rows / total rows = 3/3
102-
+------------------+----------+
103-
| account_number | gender |
104-
|------------------+----------|
105-
| 1 | M |
106-
| 13 | F |
107-
| 18 | M |
108-
+------------------+----------+
109-
110-
11192
### Additional Examples
11293

11394
- `source = table | dedup a | fields a,b,c`
@@ -122,7 +103,6 @@ PPL query:
122103
- `source = table | dedup 2 a,b | fields a,b,c`
123104
- `source = table | dedup 2 a keepempty=true | fields a,b,c`
124105
- `source = table | dedup 2 a,b keepempty=true | fields a,b,c`
125-
- `source = table | dedup 1 a consecutive=true| fields a,b,c` (Consecutive deduplication is unsupported)
126106

127107
### Limitation:
128108

@@ -154,4 +134,4 @@ Union
154134
+- UnresolvedRelation
155135
```
156136

157-
- this `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.4
137+
- this `dedup` command with `allowedDuplication > 1` feature needs spark version >= 3.4

0 commit comments

Comments
 (0)