MetaRobotsChecker should use rules based on URLs instead of paths

[cristiingineru]

Currently, the rules of metaRobotsChecker are expressed using paths which is not consistent with other rules like, for example, used by ContentChecker or HreflangChecker.

Basically, metaRobotsChecker instead of using rules expressed like this:

[
    {
        "path": "/path1/.*",
        "index": true,
        "follow": true,
    },
    {
        "path": "/path2/.*",
        "index": true,
        "follow": true,
    }
]

should switch to:

[
    {
        "url": "https://www\.domain\.com/path1/.*",
        "index": true,
        "follow": true,
    },
    {
        "url": "https://www\.domain\.com/path2/.*",
        "index": true,
        "follow": true,
    }
]