Skip to content

scraper, project, superblock: Implement automated greylisting #2778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f6245a3
Initial machinery in project.h/cpp and blockchain.cpp for greylist st…
jamescowens Apr 28, 2024
9cda503
Scraper greylisting machinery (Part 1)
jamescowens May 1, 2024
c3c2936
Implement NotifyProjectChanged core signal
jamescowens Nov 25, 2024
322520e
Extend listprojects
jamescowens Nov 25, 2024
178b055
Enhance GUI to understand new greylisting status
jamescowens Jan 3, 2025
0daf7c1
Baseline changes for superblock v3 contracts
jamescowens Jan 4, 2025
2d05817
Initial implementation of AutoGreylist
jamescowens Jan 6, 2025
4dc35e6
Modify scraper code to accommodate all cpid total credit computation …
jamescowens Jan 7, 2025
ebf7858
Modify scraper and superblock code to support automatic greylisting (…
jamescowens Jan 12, 2025
c7ab778
Specify contract version 2 for superblocks in superblock_tests.cpp
jamescowens Jan 12, 2025
6510349
Modify scraper and superblock code (part 3)
jamescowens Jan 12, 2025
d540691
Add hidden entry for superblockv3height startup parameter for testing
jamescowens Jan 13, 2025
dc6cd75
Enhance SuperblockToJson for superblock v3
jamescowens Jan 13, 2025
79d7c3f
Add sending ProjectsAllCpidTotalCredits part to ScraperSendFileManife…
jamescowens Jan 13, 2025
8062bdd
Implement FromDoubleToString that allows specification of precision
jamescowens Jan 13, 2025
892c93c
Improve CScraperManifest::ToJson() to display project all cpid total …
jamescowens Jan 13, 2025
9c5da17
Additional changes to scraper from debugging
jamescowens Jan 13, 2025
39138cb
Changes to make clang initializer order checker happy
jamescowens Jan 13, 2025
a865103
Ensure auto greylist override in Whitelist::Snapshot doesn't modify u…
jamescowens Jan 14, 2025
1f9393e
Fix m_meets_greylisting_criteria determination
jamescowens Jan 15, 2025
406ac4a
Suppress greylisted projects on explainmagnitude output.
jamescowens Jan 15, 2025
3f93bcf
Fix NetworkWideStatsEntry initialization
jamescowens Jan 15, 2025
bba131a
Ensure AutoGreylist correctly deals with newly whitelisted projects
jamescowens Jan 15, 2025
648d97f
Add AUTO_GREYLIST_OVERRIDE to project status
jamescowens Jan 19, 2025
3efc659
Modify addkey and project constructor to deal with auto greylist over…
jamescowens Jan 19, 2025
5b9bafd
Add auto greylist update history and make other improvements
jamescowens Jan 19, 2025
69c6c4c
Wire in greylist history to getautogreylist
jamescowens Jan 19, 2025
08c1904
Change order of precedence for excluded project status in UI.
jamescowens Jan 19, 2025
16ee46e
Add documentation and other minor adjustments
jamescowens Jan 19, 2025
181c374
Change around the AutoGreylist initialization
jamescowens Jan 19, 2025
0477325
Update copyright on changed/added files.
jamescowens Jan 19, 2025
be3cfa4
Modify UpdateGreylistCandidateEntry and AutoGreylist::Reset()
jamescowens Jan 20, 2025
bb27ab8
Implement auto greylist boost unit test
jamescowens Jan 23, 2025
87563c3
Adjust vGreylistedProjects and include in convergencereport
jamescowens Jan 26, 2025
c7bb045
Add m_requires_ext_adapter to project entry and associated machinery
jamescowens Jan 26, 2025
9a4ef8c
Add version validation for superblock and project contracts
jamescowens Jan 26, 2025
9236467
Correct missing beaconChanged and researcherChanged signals in resear…
jamescowens Jan 27, 2025
7c1420a
Change iterator to reference to silence clang warning
jamescowens Feb 2, 2025
482d81e
Minor adjustments and documentation cleanup after self review
jamescowens Feb 3, 2025
d708d05
Add automated_greylisting_design_highlights.md to doc folder
jamescowens Feb 3, 2025
b8bce75
Adjust locking on Whitelist::GetProjectsFirstActive() to avoid potent…
jamescowens Feb 3, 2025
bfe4317
Correct Quorum::Hash on v3 serialized superblocks
jamescowens Feb 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
270 changes: 270 additions & 0 deletions doc/automated_greylisting_design_highlights.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions src/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ GRIDCOIN_CORE_H = \
gridcoin/contract/registry.h \
gridcoin/contract/registry_db.h \
gridcoin/cpid.h \
gridcoin/fwd.h \
gridcoin/gridcoin.h \
gridcoin/magnitude.h \
gridcoin/mrc.h \
Expand Down
7 changes: 7 additions & 0 deletions src/chainparams.cpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
// Copyright (c) 2010 Satoshi Nakamoto
// Copyright (c) 2009-2020 The Bitcoin Core developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -73,6 +74,9 @@ class CMainParams : public CChainParams {
consensus.DefaultConstantBlockReward = 10 * COIN;
consensus.ConstantBlockRewardFloor = 0;
consensus.ConstantBlockRewardCeiling = 500 * COIN;
consensus.ProjectV4Height = std::numeric_limits<int>::max();
consensus.SuperblockV3Height = std::numeric_limits<int>::max();
// Immediately post zero payment interval fees 40% for mainnet
consensus.InitialMRCFeeFractionPostZeroInterval = Fraction(2, 5);
consensus.MRCZeroPaymentInterval = 14 * 24 * 60 * 60;
consensus.MaxMandatorySideStakeTotalAlloc = Fraction(1, 4);
Expand Down Expand Up @@ -191,6 +195,9 @@ class CTestNetParams : public CChainParams {
consensus.DefaultConstantBlockReward = 10 * COIN;
consensus.ConstantBlockRewardFloor = 0;
consensus.ConstantBlockRewardCeiling = 500 * COIN;
consensus.ProjectV4Height = std::numeric_limits<int>::max();
consensus.SuperblockV3Height = std::numeric_limits<int>::max();
// Immediately post zero payment interval fees 40% for testnet, the same as mainnet
consensus.InitialMRCFeeFractionPostZeroInterval = Fraction(2, 5);
consensus.MRCZeroPaymentInterval = 10 * 60;
consensus.MaxMandatorySideStakeTotalAlloc = Fraction(1, 4);
Expand Down
15 changes: 15 additions & 0 deletions src/chainparams.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
// Copyright (c) 2009-2010 Satoshi Nakamoto
// Copyright (c) 2009-2020 The Bitcoin Core developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -170,6 +171,20 @@ inline bool IsProjectV2Enabled(int nHeight)
return nHeight >= Params().GetConsensus().ProjectV2Height;
}

inline bool IsSuperblockV3Enabled(int nHeight)
{
// The argument driven override temporarily here to facilitate testing.

return nHeight >= gArgs.GetArg("-superblockv3height", Params().GetConsensus().SuperblockV3Height);
}

inline bool IsProjectV4Enabled(int nHeight)
{
// The argument driven override temporarily here to facilitate testing.

return nHeight >= gArgs.GetArg("-projectv4height", Params().GetConsensus().ProjectV4Height);
}

inline int GetSuperblockAgeSpacing(int nHeight)
{
return (fTestNet ? 86400 : (nHeight > 364500) ? 86400 : 43200);
Expand Down
9 changes: 8 additions & 1 deletion src/consensus/params.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
// Copyright (c) 2009-2010 Satoshi Nakamoto
// Copyright (c) 2009-2019 The Bitcoin Core developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -39,6 +40,8 @@ struct Params {
int PollV3Height;
/** Block height at which project v2 contracts are allowed */
int ProjectV2Height;
/** Block height at which project v4 contracts are allowed */
int ProjectV4Height;
/**
* @brief The default GRC paid for a constant block reward.
*
Expand All @@ -56,6 +59,10 @@ struct Params {
* for block v13+. Note that this is typed as int64_t rather than CAmount to avoid the extra include.
*/
int64_t ConstantBlockRewardCeiling;
/**
* @brief Block height at which superblock v3 contracts are allowed/required
*/
int SuperblockV3Height;
/** The fraction of rewards taken as fees in an MRC after the zero payment interval. Only consesnus critical
* at BlockV12Height or above. Note that this is typed as int64_t rather than CAmount to avoid the extra include.
*/
Expand Down Expand Up @@ -88,7 +95,7 @@ struct Params {
* for purposes of computing voting weight. Nominally 1 / 5.67 from Fern onwards.
*
* The magnitude weight factor can be set by an administrative protocol entry with the key name "magnitudeweightfactor" for
* V13+ blocks. The value is specified as a whole number or fraction. For example, 1 / 5.67 would be "100/567", 2 would be "2".
* V13+ blocks. The value is specified as a whole number or fraction. For example, 1 / 5.67 would be "100/567", 2 would be "2".
*/
Fraction DefaultMagnitudeWeightFactor;
/**
Expand Down
20 changes: 11 additions & 9 deletions src/gridcoin/beacon.cpp
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2014-2024 The Gridcoin developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -1239,14 +1239,16 @@ void BeaconRegistry::Deactivate(const uint256 superblock_hash)
//! \param recnum
//! \param key_type
//!
template<> void BeaconRegistry::BeaconDB::HandleCurrentHistoricalEntries(GRC::BeaconRegistry::BeaconMap& entries,
GRC::BeaconRegistry::PendingBeaconMap& pending_entries,
template<> void BeaconRegistry::BeaconDB::HandleCurrentHistoricalEntries(GRC::BeaconRegistry::BeaconMap& entries,
GRC::BeaconRegistry::PendingBeaconMap& pending_entries,
std::set<Beacon_ptr>& expired_entries,
const Beacon& entry,
entry_ptr& historical_entry_ptr,
const uint64_t& recnum,
const std::string& key_type)
{
GRC::BeaconRegistry::BeaconMap& first_entries,
const Beacon& entry,
entry_ptr& historical_entry_ptr,
const uint64_t& recnum,
const std::string& key_type,
const bool& populate_first_entries)
{
// Note that in this specialization, entry.m_cpid and entry.GetId() are used for the map keys. In the general template,
// entry.Key() is used (which here is the same as entry.m_cpid). No generalized method to implement entry.PendingKey()
// has been implemented up to this point, because the pending map is actually only used here in the beacon
Expand Down Expand Up @@ -1366,7 +1368,7 @@ template<> void BeaconRegistry::BeaconDB::HandleCurrentHistoricalEntries(GRC::Be

int BeaconRegistry::Initialize()
{
int height = m_beacon_db.Initialize(m_beacons, m_pending, m_expired_pending);
int height = m_beacon_db.Initialize(m_beacons, m_pending, m_expired_pending, m_beacon_first_entries, false);

LogPrint(LogFlags::BEACON, "INFO: %s: m_beacon_db size after load: %u", __func__, m_beacon_db.size());
LogPrint(LogFlags::BEACON, "INFO: %s: m_beacons size after load: %u", __func__, m_beacons.size());
Expand Down
4 changes: 3 additions & 1 deletion src/gridcoin/beacon.h
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2014-2024 The Gridcoin developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -830,6 +830,8 @@ class BeaconRegistry : public IContractHandler
//!
std::set<Beacon_ptr> m_expired_pending;

BeaconMap m_beacon_first_entries {}; //!< Not used here. Only to satisfy the template.

//!
//! \brief The member variable that is the instance of the beacon database. This is private to the
//! beacon registry and is only accessible by beacon registry functions.
Expand Down
22 changes: 16 additions & 6 deletions src/gridcoin/contract/registry_db.h
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) 2014-2024 The Gridcoin developers
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

Expand Down Expand Up @@ -66,10 +66,14 @@ class RegistryDB
//! specialization.
//! \param expired_entries. The map of expired pending entries. This is not used in the general template, only in the
//! beacons specialization.
//! \param first_entries: The map of the first entry for the given key. This is only currently used for the whitelist
//! (projects).
//!
//! \param populate_first_entries. This is a boolean that controls whether the first entries map is populated.
//!
//! \return block height up to and including which the entry records were stored.
//!
int Initialize(M& entries, P& pending_entries, X& expired_entries)
int Initialize(M& entries, P& pending_entries, X& expired_entries, M& first_entries, const bool& populate_first_entries)
{
bool status = true;
int height = 0;
Expand Down Expand Up @@ -173,8 +177,8 @@ class RegistryDB
m_historical[iter.second.m_hash] = std::make_shared<E>(entry);
entry_ptr& historical_entry_ptr = m_historical[iter.second.m_hash];

HandleCurrentHistoricalEntries(entries, pending_entries, expired_entries, entry,
historical_entry_ptr, recnum, key_type);
HandleCurrentHistoricalEntries(entries, pending_entries, expired_entries, first_entries, entry,
historical_entry_ptr, recnum, key_type, populate_first_entries);

number_passivated += (uint64_t) HandlePreviousHistoricalEntries(historical_entry_ptr);
} // storage_by_record_num iteration
Expand All @@ -198,14 +202,16 @@ class RegistryDB
//!
//! \param entries
//! \param pending_entries
//! \param expired_entries
//! \param first_entries
//! \param entry
//! \param historical_entry_ptr
//! \param recnum
//! \param key_type
//!
void HandleCurrentHistoricalEntries(M& entries, P& pending_entries, X& expired_entries, const E& entry,
void HandleCurrentHistoricalEntries(M& entries, P& pending_entries, X& expired_entries, M& first_entries, const E& entry,
entry_ptr& historical_entry_ptr, const uint64_t& recnum,
const std::string& key_type)
const std::string& key_type, const bool& populate_first_entries)
{
// The unknown or out of bound status conditions should have never made it into leveldb to begin with, since
// the entry contract will fail validation, but to be thorough, include the filter condition anyway.
Expand All @@ -226,6 +232,10 @@ class RegistryDB

// Insert or replace the existing map entry with the latest.
entries[entry.Key()] = historical_entry_ptr;

if (populate_first_entries && historical_entry_ptr->m_previous_hash.IsNull()) {
first_entries[entry.Key()] = historical_entry_ptr;
}
}
}

Expand Down
38 changes: 38 additions & 0 deletions src/gridcoin/fwd.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Copyright (c) 2014-2025 The Gridcoin developers
// Distributed under the MIT/X11 software license, see the accompanying
// file COPYING or https://opensource.org/licenses/mit-license.php.

#ifndef GRIDCOIN_FWD_H
#define GRIDCOIN_FWD_H

namespace GRC
{
//!
//! \brief Enumeration of project entry status. Unlike beacons this is for both storage
//! and memory.
//!
//! UNKNOWN status is only encountered in trivially constructed empty
//! project entries and should never be seen on the blockchain.
//!
//! DELETED status corresponds to a removed entry.
//!
//! ACTIVE corresponds to an active entry.
//!
//! GREYLISTED means that the project temporarily does not meet the whitelist qualification criteria.
//!
//! OUT_OF_BOUND must go at the end and be retained for the EnumBytes wrapper.
//!
enum class ProjectEntryStatus
{
UNKNOWN,
DELETED,
ACTIVE,
MAN_GREYLISTED,
AUTO_GREYLISTED,
AUTO_GREYLIST_OVERRIDE,
OUT_OF_BOUND
};
} // namespace GRC

#endif // GRIDCOIN_FWD_H

Loading
Loading