Describe the bug
American Express scraper gets blocked by CloudFlare.
This is an israeli-bank-scrapers issue.
I created this issue for a library-update request when israeli-bank-scrapers updates.
To Reproduce
Regular usage with the AMEX scraper.
Expected behavior
Should get JSON data, instead it gets HTML for a "Blocked" screen (antibot).
Screenshots
Rendered HTML:
Additional context
Full error trace:
GENERIC: fetchPostWithinPage parse error: Unexpected token '<', "<!DOCTYPE "... is not valid JSON SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON at JSON.parse (<anonymous>) at fetchPostWithinPage (/Applications/caspion.app/Contents/Resources/app.asar/node_modules/israeli-bank-scrapers-core/lib/helpers/fetch.js:99:19) at async AmexScraper.login (/Applications/caspion.app/Contents/Resources/app.asar/node_modules/israeli-bank-scrapers-core/lib/scrapers/base-isracard-amex.js:263:28) at async AmexScraper.scrape (/Applications/caspion.app/Contents/Resources/app.asar/node_modules/israeli-bank-scrapers-core/lib/scrapers/base-scraper.js:30:21) at async zn (file:///Applications/caspion.app/Contents/Resources/app.asar/packages/main/dist/index.js:1036:7) at async tr (file:///Applications/caspion.app/Contents/Resources/app.asar/packages/main/dist/index.js:1141:23), url: https://he.americanexpress.co.il/services/ProxyRequestHandler.ashx?reqName=ValidateIdData, data: {"id":"<hidden>","cardSuffix":"<hidden>","countryCode":"212","idType":"1","checkLevel":"1","companyCode":"77"}, extraHeaders: {}, result: <!DOCTYPE html> <!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]--> <!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]--> <!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]--> <!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]--> <head> <title>Attention Required! | Cloudflare</title> <meta charset="UTF-8" /> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <meta http-equiv="X-UA-Compatible" content="IE=Edge" /> <meta name="robots" content="noindex, nofollow" /> <meta name="viewport" content="width=device-width,initial-scale=1" /> <link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" /> <!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" /><![endif]--> <style>body{margin:0;padding:0}</style> <!--[if gte IE 10]><!--> <script> if (!navigator.cookieEnabled) { window.addEventListener('DOMContentLoaded', function () { var cookieEl = document.getElementById('cookie-alert'); cookieEl.style.display = 'block'; }) } </script> <!--<![endif]--> </head> <body> <div id="cf-wrapper"> <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div> <div id="cf-error-details" class="cf-error-details-wrapper"> <div class="cf-wrapper cf-header cf-error-overview"> <h1 data-translate="block_headline">Sorry, you have been blocked</h1> <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> he.americanexpress.co.il</h2> </div><!-- /.header --> <div class="cf-section cf-highlight"> <div class="cf-wrapper"> <div class="cf-screenshot-container cf-screenshot-full"> <span class="cf-no-screenshot error"></span> </div> </div> </div><!-- /.captcha-container --> <div class="cf-section cf-wrapper"> <div class="cf-columns two"> <div class="cf-column"> <h2 data-translate="blocked_why_headline">Why have I been blocked?</h2> <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p> </div> <div class="cf-column"> <h2 data-translate="blocked_resolve_headline">What can I do to resolve this?</h2> <p data-translate="blocked_resolve_detail">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p> </div> </div> </div><!-- /.section --> <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300"> <p class="text-13"> <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">9f2c46c6df08f486</strong></span> <span class="cf-footer-separator sm:hidden">•</span> <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1"> Your IP: <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button> <span class="hidden" id="cf-footer-ip">hidden</span> <span class="cf-footer-separator sm:hidden">•</span> </span> <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span> </p> <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script> </div><!-- /.error-footer --> </div><!-- /#cf-error-details --> </div><!-- /#cf-wrapper --> <script> window._cf_translation = {}; </script> </body> </html>
Describe the bug
American Express scraper gets blocked by CloudFlare.
This is an israeli-bank-scrapers issue.
I created this issue for a library-update request when
israeli-bank-scrapersupdates.To Reproduce
Regular usage with the AMEX scraper.
Expected behavior
Should get JSON data, instead it gets HTML for a "Blocked" screen (antibot).
Screenshots
Rendered HTML:
Additional context
Full error trace: