You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Preprocessing operations can be applied to all nodes or specific nodes if the data format is different across nodes. This requires using the `nodes` argument, as follows:
* `target_column` : name of column to filter on (`str`)
98
-
* `comparator` : type of comparison (`ComparisonType` enum)
99
-
100
-
* equal
101
-
* nEqual
102
-
* greater
103
-
* greaterEq
104
-
* less
105
-
* lessEq
106
-
* in
107
-
108
-
* `value` : value with which to compare (`str`)
109
-
* `numerical` : whether the comparison is on numerical values, default = False (`bool`)
110
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
111
-
112
-
##### Counts
113
-
Concatenates a new column containing 1 for each row in order to count the number of rows
114
-
```
115
-
counts(output_column_name, nodes)
116
-
```
117
-
* `output_column_name` : name of the column to store the counts. If not specified, the name 'count' will be used. (`str`)
118
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
119
-
120
-
##### Transpose
121
-
Transpose index and columns
122
-
```
123
-
transpose(copy, nodes)
124
-
```
125
-
* `copy` : Whether to copy the data after transposing. default False (`bool`)
126
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
127
-
128
-
##### Set Index
129
-
Set the DataFrame index using existing columns.
130
-
```
131
-
set_index(cols, drop, append, nodes)
132
-
```
133
-
* `columns` : list of column names to set as index (`List[str]`)
134
-
* `drop` : Delete columns to be used as the new index. default True (`bool`)
135
-
* `append` : Whether to append columns to existing index. default False (`bool`)
136
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
137
-
138
-
##### Reset Index
139
-
Reset the index, or a level of it.
140
-
```
141
-
reset_index(level, drop, nodes)
142
-
```
143
-
* `level` : list of column names to remove from index (`List[str]`)
144
-
* `drop` : Do not try to insert index into dataframe columns. This resets the index to the default integer index. default False (`bool`)
145
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
146
-
147
-
##### Rename
148
-
Alter axes labels.
149
-
```
150
-
rename(mapper, axis, copy, errors, nodes)
151
-
```
152
-
* `mapper` : Dict of transformations to apply to that axis’ values. (`dict`)
153
-
* `axis` : Axis to target with `mapper`. Should be the axis name (‘index’, ‘columns’). The default is ‘index’. (`RenameAxis`)
154
-
* `copy` : Also copy underlying data. default True (`bool`)
155
-
* `errors` : If True raise a KeyError when a dict-like mapper, index, or columns contains labels that are not present in the Index being transformed. If False existing keys will be renamed and extra keys will be ignored.(`bool`)
156
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
157
-
158
-
##### As Type
159
-
Cast column types
160
-
```
161
-
astype(type_map, copy, errors, nodes)
162
-
```
163
-
* `mapper` : Dict which maps column names to dtypes. (`dict`)
164
-
* `copy` : Return a copy. default True (`bool`)
165
-
* `errors` : If True raise a KeyError when a dict-like mapper, index, or columns contains labels that are not present in the Index being transformed. If False existing keys will be renamed and extra keys will be ignored.(`bool`)
166
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
167
-
168
-
##### Extract Dict Field
169
-
Extract field value from dict-like columns
170
-
```
171
-
extract(field, columns, names, nodes)
172
-
```
173
-
* `field` : dict field to extract (`str`)
174
-
* `columns` : list of column names from which to extract field (`List[str]`)
175
-
* `names`: names of resulting columns, if None, no new columns are created (`List[str]`)
176
-
* `nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
Preprocessing operations can be applied to all nodes or specific nodes if the data format is different across nodes. This requires using the `nodes` argument, as follows:
*`target_column` : name of column to filter on (`str`)
69
-
*`comparator` : type of comparison (`ComparisonType` enum)
70
-
71
-
* equal
72
-
* nEqual
73
-
* greater
74
-
* greaterEq
75
-
* less
76
-
* lessEq
77
-
* in
78
-
79
-
*`value` : value with which to compare (`str`)
80
-
*`numerical` : whether the comparison is on numerical values, default = False (`bool`)
81
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
82
-
83
-
##### Counts
84
-
Concatenates a new column containing 1 for each row in order to count the number of rows
85
-
```
86
-
counts(output_column_name, nodes)
87
-
```
88
-
*`output_column_name` : name of the column to store the counts. If not specified, the name 'count' will be used. (`str`)
89
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
90
-
91
-
##### Transpose
92
-
Transpose index and columns
93
-
```
94
-
transpose(copy, nodes)
95
-
```
96
-
*`copy` : Whether to copy the data after transposing. default False (`bool`)
97
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
98
-
99
-
##### Set Index
100
-
Set the DataFrame index using existing columns.
101
-
```
102
-
set_index(cols, drop, append, nodes)
103
-
```
104
-
*`columns` : list of column names to set as index (`List[str]`)
105
-
*`drop` : Delete columns to be used as the new index. default True (`bool`)
106
-
*`append` : Whether to append columns to existing index. default False (`bool`)
107
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
108
-
109
-
##### Reset Index
110
-
Reset the index, or a level of it.
111
-
```
112
-
reset_index(level, drop, nodes)
113
-
```
114
-
*`level` : list of column names to remove from index (`List[str]`)
115
-
*`drop` : Do not try to insert index into dataframe columns. This resets the index to the default integer index. default False (`bool`)
116
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
117
-
118
-
##### Rename
119
-
Alter axes labels.
120
-
```
121
-
rename(mapper, axis, copy, errors, nodes)
122
-
```
123
-
*`mapper` : Dict of transformations to apply to that axis’ values. (`dict`)
124
-
*`axis` : Axis to target with `mapper`. Should be the axis name (‘index’, ‘columns’). The default is ‘index’. (`RenameAxis`)
125
-
*`copy` : Also copy underlying data. default True (`bool`)
126
-
*`errors` : If True raise a KeyError when a dict-like mapper, index, or columns contains labels that are not present in the Index being transformed. If False existing keys will be renamed and extra keys will be ignored.(`bool`)
127
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
128
-
129
-
##### As Type
130
-
Cast column types
131
-
```
132
-
astype(type_map, copy, errors, nodes)
133
-
```
134
-
*`mapper` : Dict which maps column names to dtypes. (`dict`)
135
-
*`copy` : Return a copy. default True (`bool`)
136
-
*`errors` : If True raise a KeyError when a dict-like mapper, index, or columns contains labels that are not present in the Index being transformed. If False existing keys will be renamed and extra keys will be ignored.(`bool`)
137
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
138
-
139
-
##### Extract Dict Field
140
-
Extract field value from dict-like columns
141
-
```
142
-
extract(field, columns, names, nodes)
143
-
```
144
-
*`field` : dict field to extract (`str`)
145
-
*`columns` : list of column names from which to extract field (`List[str]`)
146
-
*`names`: names of resulting columns, if None, no new columns are created (`List[str]`)
147
-
*`nodes` : which nodes to apply the preprocessing operation to, if `None` it will apply to all (`List[str]`)
0 commit comments