Enable RAS manager for fatal and runtime errors #102

aasboddu · 2025-01-23T05:41:36Z

Enable RAS manager for fatal and runtime errors

APML RAS Manager Initialization:

Added initialization for APML RAS Manager.
Included conditional compilation for APML support.
Added a placeholder error log for PLDM RAS capabilities, indicating
that they are yet to be enabled.
The init function repeatedly attempts to get the BMC RAS OOB
configuration until successful.
The function initializes the platform with the block ID's that needs
to be harvested during a crashdump and sets up a D-Bus match to
monitor watchdog state changes to monitor BIOS post complete.
It reads CPU IDs for all CPUs and logs errors on failure.
The init function also creates separate polling threads for MCA,
DRAM CECC and PCIE AER error monitoring.
The function also handles BIOS post-completion by configuring PCIE
OOB settings and enabling PCIE error thresholds based on watchdog
timer changes.
It also clears SbrmiAlertMask register so APML_ALERT_L will be
asserted during a syncflood in the system.
The commit has oem_cper.h providing the outline of file format for
both runtime and crashdump CPER records.
Added additional json config parameters to enable OOB registers
during initialization.
Overall , this commit provides all the necessary preps needed to
enable the crashdump flow and runtime error monitoring.

Crashdump monitoring:

This commit introduces the handling of GPIO events for P0 and
P1 APML alerts
Binds the P0 alert event handler and P1 alert evernt handler
to manage these alerts.
Read RAS status register and check for errors.
Log and send alerts for various RAS errors including:
- SYS_MGMT_CTRL_ERR: Trigger cold reset based on policy.
- RESET_HANG_ERR: Suggest manual immediate reset.
- FATAL_ERROR: Harvest MCA data and reset based on policy.
- MCA_ERR_OVERFLOW: Log MCA runtime error counter overflow.
- DRAM_CECC_ERR_OVERFLOW: Log DRAM CECC runtime error
  counter overflow.
- PCIE_ERR_OVERFLOW: Log PCIE runtime error counter overflow.

CPER record generation:

Add functionality to generate Common Platform Error Record (CPER)
entries when a FATAL_ERROR is detected.
It also creates CPER record when a runtime MCA , DRAM CECC and
PCIE AER error is reported via polling or threshold overflow.
The system stores maximum of 10 CPER records in BMC including
fatal and runtime errors.
Create D-Bus object paths for each CPER file in the system,
allowing for download via redfish.
Update properties for each CPER object, including Filename,
Log, and Timestamp.

root@morocco-d89c:~# busctl tree com.amd.RAS
- /com - /com/amd
- /com/amd/RAS - /com/amd/RAS/0

Signed-off-by: aasboddu [email protected], Abinaya Dhandapani [email protected]

include/apml_manager.hpp

abinayaddhandapani · 2025-01-24T11:32:32Z

include/oem_cper.hpp

@@ -0,0 +1,61 @@
+#pragma once
+
+#include "libcper/Cper.h"


There is still open discussion to use libcper library or copy the contents of libcper/Cper.h to amd-ras repo is still under discussion.

@supven01 Can you please provide your inputs

src/apml_manager.cpp

src/config_manager.cpp

src/error_monitor.cpp

config/ras_config.json

include/apml_manager.hpp

rajaganeshr · 2025-02-13T05:13:31Z

include/cper_util.hpp

+constexpr uint16_t PCIE_VENDOR_ID = 0x1022;
+constexpr int MINOR_REVISION = 0xB;
+
+/*template void dumpHeaderSection(const std::shared_ptr<FatalCperRecord>& data,


Is this commented code left out for debug purposes?

This is moved to cper_tul.cpp file.
Will delete these commented out lines.

include/oem_cper.hpp

src/error_monitor.cpp

src/utils/host_util.cpp

rajaganeshr · 2025-02-13T05:48:32Z

src/utils/host_util.cpp

+{
+    if (*systemRecovery == "WARM_RESET")
+    {
+        if ((buf & 0x4)) // SYS_MGMT_CTRL_ERR


pls add bitmask macro

APML RAS Manager Initialization: - Added initialization for APML RAS Manager. - Included conditional compilation for APML support. - Added a placeholder error log for PLDM RAS capabilities, indicating that they are yet to be enabled. - The init function repeatedly attempts to get the BMC RAS OOB configuration until successful. - The function initializes the platform with the block ID's that needs to be harvested during a crashdump and sets up a D-Bus match to monitor watchdog state changes to monitor BIOS post complete. - It reads CPU IDs for all CPUs and logs errors on failure. - The init function also creates separate polling threads for MCA, DRAM CECC and PCIE AER error monitoring. - The function also handles BIOS post-completion by configuring PCIE OOB settings and enabling PCIE error thresholds based on watchdog timer changes. - It also clears SbrmiAlertMask register so APML_ALERT_L will be asserted during a syncflood in the system. - The commit has oem_cper.h providing the outline of file format for both runtime and crashdump CPER records. - Added additional json config parameters to enable OOB registers during initialization. - Overall , this commit provides all the necessary preps needed to enable the crashdump flow and runtime error monitoring. Crashdump monitoring: - This commit introduces the handling of GPIO events for P0 and P1 APML alerts - Binds the P0 alert event handler and P1 alert evernt handler to manage these alerts. - Read RAS status register and check for errors. - Log and send alerts for various RAS errors including: - SYS_MGMT_CTRL_ERR: Trigger cold reset based on policy. - RESET_HANG_ERR: Suggest manual immediate reset. - FATAL_ERROR: Harvest MCA data and reset based on policy. - MCA_ERR_OVERFLOW: Log MCA runtime error counter overflow. - DRAM_CECC_ERR_OVERFLOW: Log DRAM CECC runtime error counter overflow. - PCIE_ERR_OVERFLOW: Log PCIE runtime error counter overflow. CPER record generation: - Add functionality to generate Common Platform Error Record (CPER) entries when a FATAL_ERROR is detected. - It also creates CPER record when a runtime MCA , DRAM CECC and PCIE AER error is reported via polling or threshold overflow. - The system stores maximum of 10 CPER records in BMC including fatal and runtime errors. - Create D-Bus object paths for each CPER file in the system, allowing for download via redfish. - Update properties for each CPER object, including Filename, Log, and Timestamp. root@morocco-d89c:~# busctl tree com.amd.RAS `- /com `- /com/amd `- /com/amd/RAS `- /com/amd/RAS/0 Signed-off-by: aasboddu <[email protected]>, Abinaya Dhandapani <[email protected]>

mahkurap · 2025-02-27T08:39:47Z

include/oem_cper.hpp

+    CRASHDUMP_T CrashDumpData[32];
+    DF_DUMP DfDumpData;
+    UINT32 Reserved1[96];
+    UINT32 DebugLogIdData[12124];


couple of generic things:

please define macros for all the array sizes of various members of these structures in this file.

Is it OK to have the typdes as all capital letters?

mahkurap · 2025-02-28T09:53:41Z

include/error_monitor.hpp

+    Manager(amd::ras::config::Manager&);
+
+    int errCount = 0;
+    uint8_t cpuCount;


my be late in the game. If I were you, I would make these members as private, and write get rouines for each of them, and initialize these variables as part of the constructor instead of initializing them in the getSocketInfo() method

mahkurap · 2025-02-28T09:54:50Z

include/error_monitor.hpp

+     *  @return On failure to create index file or read CPER index number
+     *  throw std::runtime_error exception.
+     */
+    void createCperIndexFile();


shouldn't this be part of cper_util.hpp?

mahkurap · 2025-02-28T09:56:44Z

include/crashdump_manager.hpp

+ */
+class CrashdumpInterface : public CrashdumpBase
+{
+  public:


what is the coding guideline? is it public first, or private first? not a major concern..

mahkurap · 2025-02-28T10:05:08Z

include/apml_manager.hpp

+     * This function is invoked when an alert event occurs on P0. The function
+     * handles the event by processing the necessary response.
+     */
+    void p0AlertEventHandler();


p0AlertEventHandler(), and p1AertEventHandler() code will be similar, I am assuming. Instead, if we can have this Handler as a routine which would take the p0/p1 specifics as input, this code will be cleaner, and if we have to support p3, we are ready.

mahkurap · 2025-02-28T10:05:35Z

include/apml_manager.hpp

+     * This GPIO line is used to detect hardware alerts for P0 and trigger
+     * events for processing.
+     */
+    gpiod::line p0apmlAlertLine;


same comment as above.

mahkurap · 2025-02-28T10:15:34Z

src/utils/host_util.cpp

+    }
+}
+
+void triggerWarmReset()


Adding a SEL log on any type of host reset might help during debug.

mahkurap · 2025-02-28T10:20:11Z

src/utils/cper_util.cpp

+{
+    bool ret = false;
+    int socNum = 0;
+    uint32_t tempVar[2][8];


shoud the socNum 2 be a macro/const?

mahkurap · 2025-02-28T10:23:05Z

src/main.cpp

+    errorMgr->configure();
+#endif
+
+#ifdef PLDM


see if you can make this as a variable based check? That is how it is going to be. if not now, may be later.

aasboddu requested review from ojayanth, modolaty and abinayaddhandapani January 23, 2025 05:42

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

include/apml_manager.hpp Outdated Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/apml_manager.cpp Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/config_manager.cpp Show resolved Hide resolved

abinayaddhandapani reviewed Jan 24, 2025

View reviewed changes

src/error_monitor.cpp Outdated Show resolved Hide resolved

aasboddu force-pushed the review branch from 30c3019 to e17018f Compare January 28, 2025 16:56

abinayaddhandapani force-pushed the review branch 3 times, most recently from 5ce505a to 6f25110 Compare February 10, 2025 19:30

abinayaddhandapani force-pushed the review branch 2 times, most recently from 1c1b01f to ad056d4 Compare February 11, 2025 10:47

abinayaddhandapani requested review from supven01 and srgovard February 11, 2025 10:49

abinayaddhandapani changed the title ~~Enable APML RAS Manager Initialization~~ Enable Fatal error monitoring and CPER file creation Feb 11, 2025

abinayaddhandapani self-requested a review February 11, 2025 11:01

abinayaddhandapani approved these changes Feb 11, 2025

View reviewed changes

abinayaddhandapani requested review from rajaganeshr and mahkurap February 13, 2025 02:23

rajaganeshr reviewed Feb 13, 2025

View reviewed changes

abinayaddhandapani force-pushed the review branch 2 times, most recently from a4d00b8 to 78a6f41 Compare February 18, 2025 12:15

abinayaddhandapani force-pushed the review branch from 78a6f41 to 83fae27 Compare February 26, 2025 16:32

abinayaddhandapani force-pushed the review branch from 83fae27 to ae7a5f1 Compare February 26, 2025 16:34

abinayaddhandapani changed the title ~~Enable Fatal error monitoring and CPER file creation~~ Enable RAS manager for fatal and runtime errors Feb 26, 2025

rajaganeshr approved these changes Feb 27, 2025

View reviewed changes

mahkurap reviewed Feb 27, 2025

View reviewed changes

mahkurap reviewed Feb 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable RAS manager for fatal and runtime errors #102

Enable RAS manager for fatal and runtime errors #102

aasboddu commented Jan 23, 2025 •

edited by abinayaddhandapani

Loading

abinayaddhandapani Jan 24, 2025

rajaganeshr Feb 13, 2025

abinayaddhandapani Feb 14, 2025 •

edited

Loading

rajaganeshr Feb 13, 2025

abinayaddhandapani Feb 14, 2025

mahkurap Feb 27, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

mahkurap Feb 28, 2025

Enable RAS manager for fatal and runtime errors #102

Are you sure you want to change the base?

Enable RAS manager for fatal and runtime errors #102

Conversation

aasboddu commented Jan 23, 2025 • edited by abinayaddhandapani Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abinayaddhandapani Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aasboddu commented Jan 23, 2025 •

edited by abinayaddhandapani

Loading

abinayaddhandapani Feb 14, 2025 •

edited

Loading