You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Adds a new enum `morpheus.config.ExecutionMode` with members `GPU` & `CPU` along with a new `morpheus.config.Config.execution_mode` attribute.
* For backwards compatibility, by default `Config.execution_mode` will always default to `GPU`
* Add new `supported_execution_modes` to `StageBase` which returns `ExecutionMode.GPU` by default. This ensures that building a pipeline with a stage not matching the execution mode will raise a reasonable error to the user.
* Add `CpuOnlyMixin` and `GpuAndCpuMixin` mixins to automate overriding this, and makes it easier for users to determine which execution modes a given stage supports at a glance.
* Since C++ Stage/Message impls can only support cuDF DataFrames, and RMM tensors, this PR re-purposes the existing Python stage/message impls mode to serve as CPU-only mode.
* CPU-only mode will center around pandas DataFrames and NumPy arrays for tensors, since the current Python code which expects cuDF/CuPy is already 99% compatible with pandas/NumPy.
* Avoid importing `cudf` or any other GPU based package which will fail on import at the top-level of a module. This is important for stage, message and modules which are automatically imported by the morpheus CLI tool.
* Add new utility methods to `morpheus.utils.type_utils` (ex: `get_df_pkg`, `is_cudf_type`) to help avoid importing cudf directly
* Add a new `Config.freeze` method which will make a config object immutable. This will be called the first time a config object is used to construct a pipeline or stage object. Prevents the possibility of config parameters from being changed in the middle of pipeline construction.
* `CudfHelper::load` is no longer called automatically on import, instead it is called manually on pipeline build when execution mode is GPU.
* Add Python implementation of `ControlMessage`
* To simulate a system without a GPU to test CPU-only mode, if the `CPU_ONLY` environment variable is defined `docker/run_container_dev.sh` will launch the container using the `runc` runtime.
* Remove automatic test parameterization of C++/Python mode, since supporting CPU-only mode will become the exception not the rule. Add a new `gpu_and_cpu_mode` test marker to explicitly indicate a test intended to be parameterized over execution modes.
* Fix copy constructor for `ControlMessage`
* `AppShieldSourceStage` now emits `ControlMessage`s, `AppShieldMessageMeta` is now deprecated
* `AutoencoderSourceStage` and thus `AzureSourceStage`, `CloudTrailSourceStage`, and `DuoSourceStage` now emit `ControlMessage`, `UserMessageMeta` is now deprecated.
* DFP production pipeline updated to remove `DFPMessageMeta`, pipeline now executes in C++ mode.
* Consolidate common logig in `docker/run_container_dev.sh` & `docker/run_container_release.sh` into `docker/run_container.sh`
* Remove inconsistent behavior in the Python impl of `TensorMemory.set_tensor` (nv-morpheus#1955)
Closesnv-morpheus#1646Closesnv-morpheus#1846Closesnv-morpheus#1852Closesnv-morpheus#1955
## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.
Authors:
- David Gardner (https://github.com/dagardner-nv)
- Yuchen Zhang (https://github.com/yczhang-nv)
Approvers:
- Michael Demoret (https://github.com/mdemoret-nv)
URL: nv-morpheus#1851
Our sink will function as a pass-through allowing the possibility of other sinks to be added to the pipeline. We could, hypothetically, have a pipeline where we emit the results to both RabbitMQ and a file. For this reason we will also be using the `PassThruTypeMixin`.
@@ -1032,14 +1032,15 @@ import pika
1032
1032
from morpheus.cli.register_stage import register_stage
1033
1033
from morpheus.config import Config
1034
1034
from morpheus.messages.message_meta import MessageMeta
1035
+
from morpheus.pipeline.execution_mode_mixins import GpuAndCpuMixin
1035
1036
from morpheus.pipeline.pass_thru_type_mixin import PassThruTypeMixin
1036
1037
from morpheus.pipeline.single_port_stage import SinglePortStage
Copy file name to clipboardexpand all lines: docs/source/developer_guide/guides/6_digital_fingerprinting_reference.md
+5-5
Original file line number
Diff line number
Diff line change
@@ -88,15 +88,15 @@ Defines a single column and type-cast.
88
88
| Argument | Type | Description |
89
89
| -------- | ---- | ----------- |
90
90
|`name`|`str`| Name of the column |
91
-
|`dtype`|`str` or Python type | Any type string or Python class recognized by [Pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
91
+
|`dtype`|`str` or Python type | Any type string or Python class recognized by [pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
92
92
93
93
#### Custom Column (`CustomColumn`)
94
94
Subclass of `ColumnInfo`, defines a column to be computed by a user-defined function `process_column_fn`.
95
95
96
96
| Argument | Type | Description |
97
97
| -------- | ---- | ----------- |
98
98
|`name`|`str`| Name of the column |
99
-
|`dtype`|`str` or Python type | Any type string or Python class recognized by [Pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
99
+
|`dtype`|`str` or Python type | Any type string or Python class recognized by [pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
100
100
|`process_column_fn`|`function`| Function which receives the entire `DataFrame` as its only input, returning a new [`pandas.Series`](https://pandas.pydata.org/docs/reference/api/pandas.Series.html) object to be stored in column `name`. |
101
101
|`input_column_types`|`dict[str, str]`| The input columns and the expected [`dtype` strings](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes) that are needed for this Column to successfully process. Setting this as `None` will pass all columns. Specifying which columns are needed improves performance. |
102
102
@@ -139,7 +139,7 @@ Subclass of `RenameColumn`, specific to casting UTC localized `datetime` values.
139
139
| Argument | Type | Description |
140
140
| -------- | ---- | ----------- |
141
141
|`name`|`str`| Name of the destination column |
142
-
|`dtype`|`str` or Python type | Any type string or Python class recognized by [Pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
142
+
|`dtype`|`str` or Python type | Any type string or Python class recognized by [pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
143
143
|`input_name`|`str`| Original column name |
144
144
145
145
#### String-Join Column (`StringJoinColumn`)
@@ -148,7 +148,7 @@ Subclass of `RenameColumn`, converts incoming `list` values to string by joining
148
148
| Argument | Type | Description |
149
149
| -------- | ---- | ----------- |
150
150
|`name`|`str`| Name of the destination column |
151
-
|`dtype`|`str` or Python type | Any type string or Python class recognized by [Pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
151
+
|`dtype`|`str` or Python type | Any type string or Python class recognized by [pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
152
152
|`input_name`|`str`| Original column name |
153
153
|`sep`|`str`| Separator string to use for the join |
154
154
@@ -158,7 +158,7 @@ Subclass of `ColumnInfo`, concatenates values from multiple columns into a new s
158
158
| Argument | Type | Description |
159
159
| -------- | ---- | ----------- |
160
160
|`name`|`str`| Name of the destination column |
161
-
|`dtype`|`str` or Python type | Any type string or Python class recognized by [Pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
161
+
|`dtype`|`str` or Python type | Any type string or Python class recognized by [pandas](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes)|
162
162
|`input_columns`|`List[str]`| List of columns to concatenate |
0 commit comments