Enable RAS manager for fatal and runtime errors #102

aasboddu · 2025-01-23T05:41:36Z

Enable RAS manager for fatal and runtime errors

APML RAS Manager Initialization:

Added initialization for APML RAS Manager.
Included conditional compilation for APML support.
Added a placeholder error log for PLDM RAS capabilities, indicating
that they are yet to be enabled.
The init function repeatedly attempts to get the BMC RAS OOB
configuration until successful.
The function initializes the platform with the block ID's that needs
to be harvested during a crashdump and sets up a D-Bus match to
monitor watchdog state changes to monitor BIOS post complete.
It reads CPU IDs for all CPUs and logs errors on failure.
The init function also creates separate polling threads for MCA,
DRAM CECC and PCIE AER error monitoring.
The function also handles BIOS post-completion by configuring PCIE
OOB settings and enabling PCIE error thresholds based on watchdog
timer changes.
It also clears SbrmiAlertMask register so APML_ALERT_L will be
asserted during a syncflood in the system.
The commit has oem_cper.h providing the outline of file format for
both runtime and crashdump CPER records.
Added additional json config parameters to enable OOB registers
during initialization.
Overall , this commit provides all the necessary preps needed to
enable the crashdump flow and runtime error monitoring.

Crashdump monitoring:

This commit introduces the handling of GPIO events for P0 and
P1 APML alerts
Binds the P0 alert event handler and P1 alert evernt handler
to manage these alerts.
Read RAS status register and check for errors.
Log and send alerts for various RAS errors including:
- SYS_MGMT_CTRL_ERR: Trigger cold reset based on policy.
- RESET_HANG_ERR: Suggest manual immediate reset.
- FATAL_ERROR: Harvest MCA data and reset based on policy.
- MCA_ERR_OVERFLOW: Log MCA runtime error counter overflow.
- DRAM_CECC_ERR_OVERFLOW: Log DRAM CECC runtime error
  counter overflow.
- PCIE_ERR_OVERFLOW: Log PCIE runtime error counter overflow.

CPER record generation:

Add functionality to generate Common Platform Error Record (CPER)
entries when a FATAL_ERROR is detected.
It also creates CPER record when a runtime MCA , DRAM CECC and
PCIE AER error is reported via polling or threshold overflow.
The system stores maximum of 10 CPER records in BMC including
fatal and runtime errors.
Create D-Bus object paths for each CPER file in the system,
allowing for download via redfish.
Update properties for each CPER object, including Filename,
Log, and Timestamp.

root@morocco-d89c:~# busctl tree com.amd.RAS
- /com - /com/amd
- /com/amd/RAS - /com/amd/RAS/0

Signed-off-by: aasboddu [email protected], Abinaya Dhandapani [email protected]

include/apml_manager.hpp

abinayaddhandapani · 2025-01-24T11:32:32Z

include/oem_cper.hpp

@@ -0,0 +1,61 @@
+#pragma once
+
+#include "libcper/Cper.h"


There is still open discussion to use libcper library or copy the contents of libcper/Cper.h to amd-ras repo is still under discussion.

@supven01 Can you please provide your inputs

Closing this comment as it is finalized to use the standaed cper.h file from libcper library

src/apml_manager.cpp

src/config_manager.cpp

src/error_monitor.cpp

config/ras_config.json

include/apml_manager.hpp

rajaganeshr · 2025-02-13T05:13:31Z

include/cper_util.hpp

+constexpr uint16_t PCIE_VENDOR_ID = 0x1022;
+constexpr int MINOR_REVISION = 0xB;
+
+/*template void dumpHeaderSection(const std::shared_ptr<FatalCperRecord>& data,


Is this commented code left out for debug purposes?

This is moved to cper_tul.cpp file.
Will delete these commented out lines.

include/oem_cper.hpp

src/error_monitor.cpp

src/utils/host_util.cpp

rajaganeshr · 2025-02-13T05:48:32Z

src/utils/host_util.cpp

+{
+    if (*systemRecovery == "WARM_RESET")
+    {
+        if ((buf & 0x4)) // SYS_MGMT_CTRL_ERR


pls add bitmask macro

mahkurap · 2025-02-27T08:39:47Z

include/oem_cper.hpp

+    CRASHDUMP_T CrashDumpData[32];
+    DF_DUMP DfDumpData;
+    UINT32 Reserved1[96];
+    UINT32 DebugLogIdData[12124];


couple of generic things:

please define macros for all the array sizes of various members of these structures in this file.

Is it OK to have the typdes as all capital letters?

Ack. Replaced all the array sizes with Macros.

The oem_cper.h is an extension of https://github.com/openbmc/libcper/blob/main/include/libcper/Cper.h with few additional structures appended as per the AMD requirement. So followed the same format as of Cper.h file in libcper library as it is defined under "extern C"

mahkurap · 2025-02-28T09:53:41Z

include/error_monitor.hpp

+    Manager(amd::ras::config::Manager&);
+
+    int errCount = 0;
+    uint8_t cpuCount;


my be late in the game. If I were you, I would make these members as private, and write get rouines for each of them, and initialize these variables as part of the constructor instead of initializing them in the getSocketInfo() method

Agreed. Moved these variables to protected and initialized these variables in the constructor.
I opted to keep the variables as protected instead of private because I need to use base class pointer to point to objects of the derived class.

mahkurap · 2025-02-28T09:54:50Z

include/error_monitor.hpp

+     *  @return On failure to create index file or read CPER index number
+     *  throw std::runtime_error exception.
+     */
+    void createCperIndexFile();


shouldn't this be part of cper_util.hpp?

mahkurap · 2025-02-28T09:56:44Z

include/crashdump_manager.hpp

+ */
+class CrashdumpInterface : public CrashdumpBase
+{
+  public:


what is the coding guideline? is it public first, or private first? not a major concern..

The other openbmc repo's follow public , protected and private order.
Opted the same here as well

mahkurap · 2025-02-28T10:05:08Z

include/apml_manager.hpp

+     * This function is invoked when an alert event occurs on P0. The function
+     * handles the event by processing the necessary response.
+     */
+    void p0AlertEventHandler();


p0AlertEventHandler(), and p1AertEventHandler() code will be similar, I am assuming. Instead, if we can have this Handler as a routine which would take the p0/p1 specifics as input, this code will be cleaner, and if we have to support p3, we are ready.

mahkurap · 2025-02-28T10:05:35Z

include/apml_manager.hpp

+     * This GPIO line is used to detect hardware alerts for P0 and trigger
+     * events for processing.
+     */
+    gpiod::line p0apmlAlertLine;


same comment as above.

RAS is not tested in 2P verona platforms as APML works only for 1 socket.
Not sure having one gpioline for both P0 and P1 will have any issues in the gpio monitoring.
I will mark this as TODO and change it once it is verified in Venice.

mahkurap · 2025-02-28T10:15:34Z

src/utils/host_util.cpp

+    }
+}
+
+void triggerWarmReset()


Adding a SEL log on any type of host reset might help during debug.

The SEL events will be logged for DC power off and ON as part of the x86-power-control application.

mahkurap · 2025-02-28T10:20:11Z

src/utils/cper_util.cpp

+{
+    bool ret = false;
+    int socNum = 0;
+    uint32_t tempVar[2][8];


shoud the socNum 2 be a macro/const?

mahkurap · 2025-02-28T10:23:05Z

src/main.cpp

+    errorMgr->configure();
+#endif
+
+#ifdef PLDM


see if you can make this as a variable based check? That is how it is going to be. if not now, may be later.

yes. This is a TODO item once the flow is finalized for PLDM. I will revert this and change it to variable.

ojayanth

Looks like lot of places coding guidelines is not aligned to openbmc . Please refer , #99 and #98 reveiw comments and update accordingly. Not looked Cpp files , will start reviewing cpp files only after addressing generic comments.

ojayanth · 2025-03-04T16:30:47Z

config/ras_config.json

@@ -4,7 +4,7 @@
            "ApmlRetries": {
                "Description": "Number of APML retry count",
                "Value": 10,
-                "MaxBoundLimit": "50"
+                "MaxBoundLimit": 50


Not a good practice to fix bugs in this commit , please add new commit and fix/change this if it is not introduced with this commit specific chnages.

Agreed. This is a bug in the previous code. The maxBoundLimit should be integer not a string.
I will revert this fix in this commit and create a new commit.

ojayanth · 2025-03-04T16:34:50Z

config/ras_config.json

+        },
+        {
+            "DramCeccPollingEn": {
+                "Description": "If this field is true, DRAM Cecc correctable errors will be polled.",


Cecc stands for Dynamic Random-Access Memory Correctable Error Correction Code, why you are using Cecc correctable errors here ? please repharse with Cecc exanded form.

Updated the expansion during first time or usage and acronym is used for further places

From PPR document:

DRAM CECC: Returns
Error count, counter info and
corresponding UMC_MCA
information from the valid
DRAM ECC Correctable
Error counter instance

Hence updating the description to "DRAM ECC Correctable Error" in the description

ojayanth · 2025-03-04T16:36:52Z

config/ras_config.json

+        },
+        {
+            "PcieAerPollingEn": {
+                "Description": "If this field is true, PCIE AER correctable errors will be polled.",


Nit inclued AER expaned form.

ojayanth · 2025-03-04T16:45:52Z

config/ras_config.json

+        },
+        {
+            "McaThresholdEn": {
+                "Description": "If this field is true, error thresholding is enable for MCa errors",


ojayanth · 2025-03-04T16:47:04Z

config/ras_config.json

+        },
+        {
+            "McaThresholdEn": {
+                "Description": "If this field is true, error thresholding is enable for MCa errors",


period is missing at end. Be consistent .

Rephrased this to "Polling interval in seconds for MCA errors ".
Removed period at the end for all the description to maintain the consistency with the previous commit.

ojayanth · 2025-03-04T16:52:59Z

config/ras_config.json

+        },
+        {
+            "DramCeccThresholdEn": {
+                "Description": "If this field is true, error thresholding is enable for DRAM CECC errors.",


"If this field is true, error thresholding for MCA errors will be enabled.",

ojayanth · 2025-03-04T16:58:35Z

include/apml_manager.hpp

+namespace ras
+{
+namespace apml
+{


General scope :

Please revisit the documentation and update according to OpenbMC guidelines.
Documentation: Improved documentation for clarity and consistency.
Parameter Names: Added parameter names in function declarations for better readability.

All classes are ordered in the public/protected/private sequence.
Replaced uint8_t with size_t wherever possible.
All variable names are in lowerCamelCase.
All functions are named in lowerCamelCase, except those called from the APML library, which use lower_snake_case.
All structs, classes, enums, and templates are in UpperCamelCase, except structures from the APML library, which follow lower_snake_case.
All constants are in lowerCamelCase.
APIs under a top-level namespace do not use the namespace names again in the API names.
Replaced all const int with fixed-width integer types.
Replaced all static const std::string with constexpr std::string_view.
Consistency is maintained across files in the comment descriptions of functions. This consistency is maintained file to file.

o/p of format-script.sh

Formatting code under /mnt/work2/adhandap/Code/amd-bmc-ras eslint: cannot find .eslintrc.json; using /mnt/work2/adhandap/Code/openbmc-build-scripts/config/eslint-global-config.json markdownlint: cannot find .markdownlint.yaml; using /mnt/work2/adhandap/Code/openbmc-build-scripts/config/markdownlint.yaml Running commit_gitlint Running commit_spelling

codespell-dictionary - misspelling count >> 0
generic-dictionary - misspelling count >> 0
Running clang_format
Running eslint
Running markdownlint
Running prettier
config/ras_config.json 54ms (unchanged)
config/gpio_config.json 2ms (unchanged)
README.md 34ms (unchanged)
.prettierrc.yaml 21ms (unchanged)
Result differences...
Format: PASSED

ojayanth · 2025-03-04T17:11:36Z

include/apml_manager.hpp

+     * This GPIO line is used to detect hardware alerts for P1 and trigger
+     * events for processing.
+     */
+    gpiod::line p1apmlAlertLine;


hard coding p0 /p1 approch is not scalable , Need better way to handle this instead of hard coding p0 and p1.

Like gpio line should be enabled based on config types.

Added gpio_config.json which contains list of ALERT_L gpio's and this is handled dynamically by reading the file.

ojayanth · 2025-03-04T17:22:12Z

include/cper_util.hpp

+static const std::string runtimeDramErr = "RUNTIME_DRAM_ERROR";
+static const std::string fatalErr = "FATAL";
+
+namespace amd


if you use namespace amd {
namespace ras {
namespace cper {
namespace util { approch you don't need to use cper name attachment in the below defintion . please reveiw and update .

ojayanth · 2025-03-04T17:22:37Z

include/cper_util.hpp

+namespace cper_util
+{
+
+constexpr int SEV_NON_FATAL_UNCORRECTED = 0;


General: Using int for constants that represent specific sizes or types can lead to ambiguity and potential issues. It's better to use fixed-width integer types from to ensure the correct size and type. Here's the corrected version:

ojayanth · 2025-03-13T14:34:36Z

config/ras_config.json

+        },
+        {
+            "DramCeccPollingEn": {
+                "Description": "If this field is true, Dynamic Random Accecc Memory Error Correction Code (DRAM ECC) correctable errors will be polled",


Fix typo" Accecc"

ojayanth · 2025-03-15T03:30:27Z

config/gpio_config.json

@@ -0,0 +1,3 @@
+{


Rename the file name to amd_ras_gpio_config.json.

ojayanth · 2025-03-15T03:44:15Z

include/apml_manager.hpp

@@ -0,0 +1,302 @@
+#include "error_monitor.hpp"
+
+extern "C"


Scope: All hpp/cpp files.
Include Only What You Need: Only include headers that are necessary for the file to compile. Avoid unnecessary includes to reduce compilation time.
Self-Sufficient Headers:Each header file should be able to compile on its own
Use Include Guards #pragma once ( hpp).

ojayanth · 2025-03-15T04:26:42Z

include/apml_manager.hpp

+{
+  public:
+    Manager() = delete;
+    ~Manager() = default;


Usually Constructors put in one block and the detsructor.

ojayanth · 2025-03-15T04:27:05Z

include/apml_manager.hpp

+{
+namespace apml
+{
+class Manager : public amd::ras::Manager


Class definition is missing.

ojayanth · 2025-03-15T04:28:47Z

include/apml_manager.hpp

+     * @details This function configures a GPIO line and stream descriptor to
+     * listen for events. It triggers the provided callback function upon event
+     * detection.
+     */


Incomplete function definitions.

ojayanth · 2025-03-15T04:29:04Z

include/apml_manager.hpp

+     *
+     * @details This function is invoked when an alert event occurs on P0 or P1.
+     * The function handles the event by processing the necessary response.
+     */


Incomplete function definition.

ojayanth · 2025-03-15T04:31:05Z

include/apml_manager.hpp

+    std::mutex dramErrorHarvestMtx;
+    std::mutex pcieErrorHarvestMtx;
+
+    /** @brief Update processor OOB configuration.


Nit: Better to follow the standard function descriptioms.consitent with public one.

ojayanth · 2025-03-15T04:39:26Z

include/apml_manager.hpp

+    sdbusplus::asio::object_server& objectServer;
+    std::shared_ptr<sdbusplus::asio::connection>& systemBus;
+
+    uint8_t progId;


Please group all mutex varibale toogether and and make sure these mutexes initialized as part of constructor.

ojayanth · 2025-03-15T05:28:06Z

meson.build

@@ -11,12 +11,27 @@ project(
 )

 config_file = '/var/lib/amd-bmc-ras/ras_config.json'


Move these to meson.options

ojayanth · 2025-03-16T06:27:53Z

src/utils/dbus_util.cpp

@@ -0,0 +1,83 @@
+#include "dbus_util.hpp"


source path is already under utils , file name rename to dbus.cpp

This is no longer valid as all the utils are clubbed togerther in util.hpp and cper_util.hpp
However, renamed the cper_util.hpp and platform_util.hpp to util/cper.hpp and util/platform.hpp

ojayanth · 2025-03-16T06:28:30Z

src/utils/dbus_util.cpp

@@ -0,0 +1,83 @@
+#include "dbus_util.hpp"
+
+namespace amd


Name space should be amd::ras::util::dbus

Renamed the gpio_config.json to amd_ras_gpio_config.json

Added documentation to all the API's

Only included headers that are necessary for the file to compile.

Moved the AMD CRB specifc changes to utils/platform.hpp

Changed to a common util file-based approach instead of having multiple util files.

APML RAS Manager Initialization: - Added initialization for APML RAS Manager. - Included conditional compilation for APML support. - Added a placeholder error log for PLDM RAS capabilities, indicating that they are yet to be enabled. - The init function repeatedly attempts to get the BMC RAS OOB configuration until successful. - The function initializes the platform with the block ID's that needs to be harvested during a crashdump and sets up a D-Bus match to monitor watchdog state changes to monitor BIOS post complete. - It reads CPU IDs for all CPUs and logs errors on failure. - The init function also creates separate polling threads for MCA errors, DRAM CECC errors and PCIE advanced error reporting. - The function also handles BIOS post-completion by configuring PCIE OOB settings and enabling PCIE error thresholds based on watchdog timer changes. - It also clears SbrmiAlertMask register so APML_ALERT_L will be asserted during a syncflood in the system. - The commit has oem_cper.h providing the outline of file format for both runtime and crashdump CPER records. - Added additional json config parameters to enable OOB registers during initialization. - Overall , this commit provides all the necessary preps needed to enable the crashdump flow and runtime error monitoring. Crashdump monitoring: - This commit introduces the handling of GPIO events for APML alerts by identifying the GPIO's from the gpio_config.json - Binds the P0 alert event handler and P1 alert evernt handler to manage these alerts. - Read RAS status register and check for errors. - Log and send alerts for various RAS errors including: - SYS_MGMT_CTRL_ERR: Trigger cold reset based on policy. - RESET_HANG_ERR: Suggest manual immediate reset. - FATAL_ERROR: Harvest MCA data and reset based on policy. - MCA_ERR_OVERFLOW: Log MCA runtime error counter overflow. - DRAM_CECC_ERR_OVERFLOW: Log DRAM CECC runtime error counter overflow. - PCIE_ERR_OVERFLOW: Log PCIE runtime error counter overflow. CPER record generation: - Add functionality to generate Common Platform Error Record (CPER) entries when a FATAL_ERROR is detected. - It also creates CPER record when a runtime MCA , DRAM CECC and PCIE advanced error reporting is reported via polling or threshold overflow. - The system stores maximum of 10 CPER records in BMC including fatal and runtime errors. - Create D-Bus object paths for each CPER file in the system, allowing for download via redfish. - Update properties for each CPER object, including Filename, Log, and Timestamp. root@morocco-d89c:~# busctl tree com.amd.RAS `- /com `- /com/amd `- /com/amd/RAS `- /com/amd/RAS/0 Verified in Venice SOC 1P system. Signed-off-by: aasboddu <[email protected]>, Abinaya Dhandapani <[email protected]>

supven01 · 2025-05-21T15:25:55Z

config/platform_default.json

@@ -0,0 +1,17 @@
+{
+  "CpuCount": 1,


For future, we should change it to 2.

supven01 · 2025-05-21T15:27:09Z

config/ras_config.json

+        {
+            "DramCeccPollingPeriod": {
+                "Description": "Polling interval in seconds for DRAM ECC correctable errors",
+                "Value": 5


In the next commit, please disable polling and enable threshold interrupts for DRAM CECC.

supven01 · 2025-05-21T15:27:28Z

config/ras_config.json

+            }
+        },
+        {
+            "McaPollingEn": {


In the next commit, please disable polling and enable threshold interrupts for MCA.

supven01 · 2025-05-21T15:27:45Z

config/ras_config.json

+            }
+        },
+        {
+            "PcieAerPollingPeriod": {


In the next commit, please disable polling and enable threshold interrupts for PCIe AER.

supven01 · 2025-05-21T15:51:10Z

src/main.cpp

@@ -19,6 +24,20 @@ int main()

    amd::ras::config::Manager manager(objectServer, systemBus);

+#ifdef APML


Please remove #ifdef in the future commit

supven01 · 2025-05-21T15:51:23Z

src/main.cpp

+    errorMgr->configure();
+#endif
+
+#ifdef PLDM


supven01

This commit has been a stellar example of how the reviews need to happen. Thanks Jayanth, Mahesh for the thorough reviews and Abinaya for addressing them diligently.

aasboddu requested review from ojayanth, modolaty and abinayaddhandapani January 23, 2025 05:42

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

include/apml_manager.hpp Outdated Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/apml_manager.cpp Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/config_manager.cpp Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/error_monitor.cpp Outdated Show resolved Hide resolved

aasboddu force-pushed the review branch from 30c3019 to e17018f Compare January 28, 2025 16:56

abinayaddhandapani force-pushed the review branch 3 times, most recently from 5ce505a to 6f25110 Compare February 10, 2025 19:30

abinayaddhandapani force-pushed the review branch 2 times, most recently from 1c1b01f to ad056d4 Compare February 11, 2025 10:47

abinayaddhandapani requested review from supven01 and srgovard February 11, 2025 10:49

abinayaddhandapani changed the title ~~Enable APML RAS Manager Initialization~~ Enable Fatal error monitoring and CPER file creation Feb 11, 2025

abinayaddhandapani self-requested a review February 11, 2025 11:01

abinayaddhandapani approved these changes Feb 11, 2025

View reviewed changes

abinayaddhandapani requested review from rajaganeshr and mahkurap February 13, 2025 02:23

rajaganeshr reviewed Feb 13, 2025

View reviewed changes

abinayaddhandapani force-pushed the review branch 2 times, most recently from a4d00b8 to 78a6f41 Compare February 18, 2025 12:15

abinayaddhandapani force-pushed the review branch 2 times, most recently from 83fae27 to ae7a5f1 Compare February 26, 2025 16:34

abinayaddhandapani changed the title ~~Enable Fatal error monitoring and CPER file creation~~ Enable RAS manager for fatal and runtime errors Feb 26, 2025

rajaganeshr approved these changes Feb 27, 2025

View reviewed changes

mahkurap reviewed Feb 27, 2025

View reviewed changes

mahkurap reviewed Feb 28, 2025

View reviewed changes

abinayaddhandapani force-pushed the review branch from ae7a5f1 to 7862321 Compare March 3, 2025 07:03

ojayanth suggested changes Mar 4, 2025

View reviewed changes

abinayaddhandapani force-pushed the review branch from 7862321 to 0e9b9c2 Compare March 6, 2025 13:32

ojayanth suggested changes Mar 16, 2025

View reviewed changes

ojayanth reviewed Mar 16, 2025

View reviewed changes

abinayaddhandapani force-pushed the review branch from 0e9b9c2 to ea0737b Compare April 1, 2025 10:30

abinayaddhandapani force-pushed the review branch from ea0737b to ac69d7b Compare April 18, 2025 06:25

abinayaddhandapani force-pushed the review branch from ac69d7b to 94a0e11 Compare April 25, 2025 14:39

abinayaddhandapani force-pushed the review branch 2 times, most recently from 588d9b8 to c7309af Compare May 19, 2025 11:35

abinayaddhandapani force-pushed the review branch from c7309af to 34a36c8 Compare May 19, 2025 11:36

supven01 approved these changes May 21, 2025

View reviewed changes

supven01 reviewed May 21, 2025

View reviewed changes

supven01 merged commit 5f87bcf into integ_sp7 May 21, 2025

supven01 deleted the review branch May 21, 2025 15:55

		@@ -11,12 +11,27 @@ project(
		)

		config_file = '/var/lib/amd-bmc-ras/ras_config.json'

		@@ -19,6 +24,20 @@ int main()

		amd::ras::config::Manager manager(objectServer, systemBus);

		#ifdef APML

Enable RAS manager for fatal and runtime errors #102

Enable RAS manager for fatal and runtime errors #102

Uh oh!

Conversation

aasboddu commented Jan 23, 2025 • edited by abinayaddhandapani Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abinayaddhandapani Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ojayanth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aasboddu commented Jan 23, 2025 •

edited by abinayaddhandapani

Loading

abinayaddhandapani Feb 14, 2025 •

edited

Loading