rocAL Deserialize PR 5 : Add python support and examples #436

fiona-gladwin · 2025-12-24T14:37:26Z

Motivation

Adds deserialization testing capabilities to the rocAL library. Adds python binding for deserialize and python examples for the same.

Technical Details

Extends serialization test to verify both original and deserialized pipeline execution with comparative outputs
Introduce python deserialize API, and extend python serialization example to execute deserialize and compare outputs

Test Plan

Run serialization tests as mentioned in the README.md
The tests are also added as part of the make test

Test Result

The serialization pipeline must run successfully without any errors.

NOTE: To be merged after PR #435

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

…change

Remove the use of pipeline operator

…change

…ocAL into fg/ser_argument

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

…into fg/deser_pr3

Enable user to pass kwargs during pipeline deserialize

Copilot

Pull request overview

This PR adds comprehensive deserialization testing capabilities to rocAL, building on PR #435. It introduces Python bindings for pipeline deserialization, test files for both C++ and Python, and supporting infrastructure for dynamic node reconstruction.

Key Changes:

Added Python serialize() and deserialize() methods to the Pipeline class with file I/O support
Implemented C++ and Python test suites that validate both serialization and deserialization by comparing outputs from original and deserialized pipelines
Extended deserialization infrastructure with deserialize_args_from_protobuf() for argument reconstruction, node factory pattern for dynamic node creation, and initialize_args() methods across loader and augmentation nodes

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
tests/python_api/pipeline_serialize_test.py	Python test comparing original and deserialized pipeline outputs
tests/cpp_api/serialization_test/serialization_test.cpp	C++ test validating serialization and deserialization with image output comparison
tests/cpp_api/serialization_test/README.md	Documentation for serialization test usage and expected behavior
tests/cpp_api/serialization_test/CMakeLists.txt	Build configuration for C++ serialization tests
tests/cpp_api/CMakeLists.txt	Added CPU and GPU serialization test targets
rocAL_pybind/rocal_pybind.cpp	Python bindings for rocalSerialize and rocalDeserialize APIs
rocAL_pybind/amd/rocal/pipeline.py	Added serialize/deserialize methods with file I/O and parameter override support
rocAL_pybind/amd/rocal/fn.py	Fixed argument ordering in brightness_fixed function
rocAL/source/pipeline/pipeline_serializer.cpp	Implements argument deserialization from protobuf with type conversion and validation
rocAL/source/pipeline/master_graph.cpp	Reconstructs pipeline graph from deserialized data with tensor and node creation
rocAL/source/loaders/image/node_image_loader_single_shard.cpp	Added initialize_args for deserialization and node registration
rocAL/source/loaders/image/node_image_loader.cpp	Added initialize_args for deserialization and node registration
rocAL/source/augmentations/node_copy.cpp	Added node factory registration
rocAL/source/augmentations/color_augmentations/node_brightness.cpp	Added initialize_args with multiple overload support and node registration
rocAL/source/api/rocal_api.cpp	Implements rocalDeserialize with protobuf parsing and context creation
rocAL/include/pipeline/pipeline_serializer.h	Added deserialize_args_from_protobuf declaration
rocAL/include/pipeline/node_factory.h	Factory pattern for dynamic node creation with registration macros
rocAL/include/pipeline/node.h	Added virtual initialize_args methods and get_loader_module interface
rocAL/include/pipeline/master_graph.h	Added deserialization support with create_operator_output and add_node overload
rocAL/include/pipeline/enum_registry.h	Added convertIntToEnum for deserialization with std::any support
rocAL/include/pipeline/argument.h	Template get() method for type-safe argument extraction and init_args helper
rocAL/include/loaders/* (multiple)	Added override specifiers for get_loader_module methods
rocAL/include/augmentations/color_augmentations/node_brightness.h	Added initialize_args declaration
rocAL/include/api/rocal_api_types.h	Added RocalPipelineParams struct with std::optional fields
rocAL/include/api/rocal_api.h	Added rocalDeserialize API declaration
CHANGELOG.md	Updated with serialization test and Python support entries

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-05T21:13:28Z

rocAL_pybind/amd/rocal/pipeline.py

+    def deserialize(cls, serialized_pipeline=None, filename=None, **kwargs):
+        """
+        Deserialize the pipeline from the protobuffers and reconstruct the pipeline
+        The returned pipeline is deserialized and build


The documentation says "The returned pipeline is deserialized and build" but it should be "built" (past participle of "build" in this context).

Suggested change

The returned pipeline is deserialized and build

The returned pipeline is deserialized and built

Copilot · 2026-01-05T21:13:29Z

tests/cpp_api/serialization_test/README.md

+## Expected Output
+
+The test will:
+1. Create the rocAL pipeline
+2. Display pipeline information (dimensions, augmentation count)
+3. Serialize the pipeline and show the serialized string size
+4. Print the complete serialized pipeline string
+5. Run a few iterations to process images and display image names with labels
+6. Report successful completion


The README documentation is outdated and doesn't reflect the actual functionality of the test. The test now includes both serialization and deserialization validation (steps 4-6 in the "Expected Output" section should mention deserialization testing and comparison of outputs between original and deserialized pipelines).

Copilot · 2026-01-05T21:13:29Z

rocAL/include/pipeline/argument.h

+            // Handle scalar types - return the single stored value
+            else if (!is_vector) {
+                if (values.empty()) {
+                    THROW("Value not present for the given argument : " + arg_name + ".")


Missing semicolon at the end of the THROW statement. This will cause a compilation error.

Suggested change

THROW("Value not present for the given argument : " + arg_name + ".")

THROW("Value not present for the given argument : " + arg_name + ".");

Copilot · 2026-01-05T21:13:29Z

tests/python_api/pipeline_serialize_test.py

+    serialized_string = test_serialization(data_path, rocal_cpu, batch_size)
+
+    # Run the deserialization test
+    success = test_deserialization(serialized_string, rocal_cpu, batch_size)
+
+    if success:
+        print("SERIALIZATION AND DESERIALIZATION TESTS PASSED!")
+    else:
+        print("SERIALIZATION OR DESERIALIZATION TESTS FAILED")


The test will fail if serialization returns None, but the error handling doesn't exit or prevent the test from attempting deserialization with None. After line 260, if serialized_string is None, the test_deserialization function will receive None and fail, but the error reporting at line 265 will misleadingly report "SERIALIZATION OR DESERIALIZATION TESTS FAILED" when only serialization actually failed.

Copilot · 2026-01-05T21:13:30Z

rocAL/source/pipeline/pipeline_serializer.cpp

+                if (proto_arg.string_vectors_size() > 1) {
+                    THROW("Expected at most one string vector for argument " + arg.arg_name + ", but found " + std::to_string(proto_arg.string_vectors_size()));
+                }
+                const auto& string_vec = proto_arg.string_vectors(0);


Potential crash when accessing empty vector. The code checks if the vector size is greater than 1, but doesn't verify that there's at least one vector (size >= 1) before accessing index 0. If string_vectors_size() is 0, accessing string_vectors(0) will cause undefined behavior or a crash.

Copilot · 2026-01-05T21:13:31Z

rocAL/source/loaders/image/node_image_loader_single_shard.cpp

+void ImageLoaderSingleShardNode::initialize_args(std::vector<Argument> &arguments, std::shared_ptr<MetaDataReader> meta_data_reader) {
+    if (arguments.size() != INIT_ARGS_COUNT)
+        THROW("ImageLoaderSingleShardNode expected " + std::to_string(INIT_ARGS_COUNT) + " arguments, received " + std::to_string(arguments.size()) + 
+              "Ensure all arguments present in init are accounted for");


The error message is missing a period or newline separator. It should be "...received X. Ensure all arguments..." or include a newline for clarity.

Suggested change

"Ensure all arguments present in init are accounted for");

". Ensure all arguments present in init are accounted for");

Copilot · 2026-01-05T21:13:32Z

rocAL/source/loaders/image/node_image_loader.cpp

+void ImageLoaderNode::initialize_args(std::vector<Argument> &arguments, std::shared_ptr<MetaDataReader> meta_data_reader) {
+    if (arguments.size() != INIT_ARGS_COUNT)
+        THROW("ImageLoaderNode expected " + std::to_string(INIT_ARGS_COUNT) + " arguments, received " + std::to_string(arguments.size()) + 
+              "Ensure all arguments present in init are accounted for");


The error message is missing a period or newline separator. It should be "...received X. Ensure all arguments..." or include a newline for clarity.

Suggested change

"Ensure all arguments present in init are accounted for");

". Ensure all arguments present in init are accounted for");

Copilot · 2026-01-05T21:13:32Z

rocAL/source/pipeline/pipeline_serializer.cpp

+                if (proto_arg.int_vectors_size() > 1) {
+                    THROW("Expected at most one int vector for argument " + arg.arg_name + ", but found " + std::to_string(proto_arg.int_vectors_size()));
+                }
+                const auto& int_vec = proto_arg.int_vectors(0);


Potential crash when accessing empty vector. The code checks if the vector size is greater than 1, but doesn't verify that there's at least one vector (size >= 1) before accessing index 0. If int_vectors_size() is 0, accessing int_vectors(0) will cause undefined behavior or a crash.

Copilot · 2026-01-05T21:13:32Z

rocAL/source/pipeline/pipeline_serializer.cpp

+                if (proto_arg.float_vectors_size() > 1) {
+                    THROW("Expected at most one float vector for argument " + arg.arg_name + ", but found " + std::to_string(proto_arg.float_vectors_size()));
+                }
+                const auto& float_vec = proto_arg.float_vectors(0);


Potential crash when accessing empty vector. The code checks if the vector size is greater than 1, but doesn't verify that there's at least one vector (size >= 1) before accessing index 0. If float_vectors_size() is 0, accessing float_vectors(0) will cause undefined behavior or a crash.

Copilot · 2026-01-05T21:13:32Z

tests/python_api/pipeline_serialize_test.py

+            print(preview_text)
+            if len(serialized_string) > 500:
+                print("... (truncated)")
+        except:


Except block directly handles BaseException.

fiona-gladwin and others added 30 commits October 7, 2025 04:48

Merge branch 'develop' of https://github.com/ROCm/rocAL into fg/enum_…

bfb4b7e

…change

Merge branch 'fg/enum_change' into fg/enum_register

efe2b71

Merge branch 'fg/ser1_pipe_op' into fg/ser2_pipe_serialize

e736bfe

Add all serialization code to pipeline serializer

bef321b

Remove the use of pipeline operator

Merge branch 'fg/ser2_pipe_serialize' into fg/ser3_params_serialize

a82fb7f

Merge branch 'fg/ser3_params_serialize' into fg/serialize_test

5842b84

Revert fix

bbc96af

Update CHANGELOG.md

c4101fc

Update CHANGELOG.md

b668043

Update CHANGELOG.md

877bee0

Add comments and doc description

43636be

Support to serialize to file

64697e6

Update CMakeLists version for rocAL

ffdee4b

Merge branch 'fg/enum_change' into fg/enum_register

f9985cb

Merge branch 'develop' into fg/enum_register

01cc2d2

Merge branch 'develop' of https://github.com/ROCm/rocAL into fg/enum_…

7619bdd

…change

Merge branch 'fg/enum_change' into fg/enum_register

1d71cf9

Merge branch 'develop' into fg/enum_register

628ea36

Merge branch 'develop' into fg/enum_register

e620f0a

Merge branch 'develop' into fg/enum_register

5052fd6

Merge branch 'fg/enum_register' of https://github.com/fiona-gladwin/r…

aa91eed

…ocAL into fg/ser_argument

Update CHANGELOG

df3f598

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

1e88632

Update CHANGELOG.md

3077acb

Minor change

d2ccaab

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

6a7c102

Minor changes

59f7c0e

Update CHANGELOG.md

37fe4be

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

2be0388

Add const qualifier

29359b7

fiona-gladwin and others added 19 commits December 16, 2025 03:48

Update CHANGELOG.md

2506685

Add init args count in macro

4a7440f

Merge branch 'develop' into fg/serialize_python

12fe8b1

Merge branch 'fg/serialize_python' into fg/deser_pr1

0b0af1b

Merge branch 'develop' into fg/deser_pr1

af8b52d

Merge branch 'develop' into fg/deser_pr2

9f5bfa5

Merge branch 'fg/serialize_python' into fg/deser_pr1

530fb36

Merge branch 'fg/deser_pr1' into fg/deser_pr2

bf083ff

Merge branch 'fg/deser_pr2' into fg/deser_pr3

9544eed

Merge branch 'develop' into fg/deser_pr2

29e727a

Resolve review comments

8457b03

Merge branch 'fg/deser_pr2' of https://github.com/fiona-gladwin/rocAL …

e72948b

…into fg/deser_pr3

Merge branch 'fg/deser_pr3' into fg/deser_pr4

be44db1

Resolve PR comments

303f063

Merge branch 'fg/deser_pr4' into fg/deser_test

bff70c5

Minor fix

d78488e

Add pipeline params as optional

a153e25

Enable user to pass kwargs during pipeline deserialize

Minor fix unit test serialize

3fbbdb9

Minor changes - address review comments

01c0d0b

fiona-gladwin requested review from a team, kiritigowda and rrawther as code owners December 24, 2025 14:37

spolifroni-amd previously approved these changes Jan 5, 2026

View reviewed changes

kiritigowda self-assigned this Jan 5, 2026

kiritigowda requested a review from Copilot January 5, 2026 21:06

kiritigowda added the ci:precheckin label Jan 5, 2026

Copilot started reviewing on behalf of kiritigowda January 5, 2026 21:07 View session

Copilot AI reviewed Jan 5, 2026

View reviewed changes

Merge branch 'develop' into fg/deser_test

3c1cd0f

fiona-gladwin dismissed spolifroni-amd’s stale review via 3c1cd0f January 9, 2026 18:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rocAL Deserialize PR 5 : Add python support and examples #436

rocAL Deserialize PR 5 : Add python support and examples #436

fiona-gladwin commented Dec 24, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Copilot AI Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	The returned pipeline is deserialized and build
	The returned pipeline is deserialized and built

	THROW("Value not present for the given argument : " + arg_name + ".")
	THROW("Value not present for the given argument : " + arg_name + ".");

	"Ensure all arguments present in init are accounted for");
	". Ensure all arguments present in init are accounted for");

rocAL Deserialize PR 5 : Add python support and examples #436

Are you sure you want to change the base?

rocAL Deserialize PR 5 : Add python support and examples #436

Conversation

fiona-gladwin commented Dec 24, 2025

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants