`html-validator` is outputting more and more errors

[jfoclpf]

html-validator is outputting more and more errors, I suppose it depends on the overloading of the main server at W3.org

    at Request._callback (/home/jfolpf/autocosts/node_modules/html-validator/lib/validate.js:15:23)
    at Request.self.callback (/home/jfolpf/autocosts/node_modules/html-validator/node_modules/request/request.js:185:22)
    at Request.emit (events.js:223:5)
    at Request.<anonymous> (/home/jfolpf/autocosts/node_modules/html-validator/node_modules/request/request.js:1161:10)
    at Request.emit (events.js:223:5)
    at IncomingMessage.<anonymous> (/home/jfolpf/autocosts/node_modules/html-validator/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:312:28)
    at IncomingMessage.emit (events.js:228:7)
    at endReadableNT (_stream_readable.js:1185:12)
    at processTicksAndRejections (internal/process/task_queues.js:81:21)

There is any tip you can give to avoid overloading the sever? I do around 50 request in series with intervals of 1,1 seconds between requests, but still sometimes it blows up

[zrrrzzt]

Are you able to reproduce this error in a way I can test it?

I thought overloading the w3c-validator would result in a status code that would be handled.

You could run the validator as a webservice yourself here are instructions for running it locally as native code or through docker.

[jfoclpf]

It is very hard for me to present here now a test script that emulates the error, cause the errors appear as part of a large test script for html files which are loaded by a local http server, since I'm using handlebars to dynamically generate the html files. But the script which is failing is this one: https://github.com/jfoclpf/autocosts/blob/master/test/validateHtml.js#L115

[zrrrzzt]

Thank you. I'll see if I'm able to test it later.

However this looks like a module that might work for you: https://www.npmjs.com/package/html-validate. From the description https://html-validate.org/dev/using-api.html You could probably rewrite your testfile with minimum effort.
Disclaimer: I've not tested it myself but from the description it will validate your html locally.

Some day I might switch to this one instead of w3c for html-validator :-)

[jfoclpf]

Good tip, thanks a lot. Nonetheless I am not sure whether that validator applies all the rules of W3C, which is the main standard in the web. After reading its description, that is not clear to me, and there are a lot of html validation W3C subtleties that might slip from such validator.

…

On Wed, 12 Feb 2020, 14:14 Geir Gåsodden, ***@***.***> wrote: Thank you. I'll see if I'm able to test it later. However this looks like a module that might work for you: https://www.npmjs.com/package/html-validate. From the description https://html-validate.org/dev/using-api.html You could probably rewrite your testfile with minimum effort. Disclaimer: I've not tested it myself but from the description it will validate your html locally. Some day I might switch to this one instead of w3c for html-validator :-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#183?email_source=notifications&email_token=AA6M4DIK4NJT3DA6TAWMWNTRCPYZZA5CNFSM4KTLK4C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELQW36Y#issuecomment-585199099>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6M4DNCYLOY5TMOO5GG7XTRCPYZZANCNFSM4KTLK4CQ> .

[zrrrzzt]

I think the validation follows all the rules. Anyway, I've done some testing and will probably release a new version around this weekend #184

It will be a breaking change, but for your use I think it will be just a small rewrite.

[jfoclpf]

Great, thanks for the update. Indeed my test suite took ages cause I validated more than 40 html files in series with more than 1 second interval between them

…

On Wed, 12 Feb 2020, 18:35 Geir Gåsodden, ***@***.***> wrote: I think the validation follows all the rules. Anyway, I've done some testing and will probably release a new version around this weekend #184 <#184> It will be a breaking change, but for your use I think it will be just a small rewrite. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#183?email_source=notifications&email_token=AA6M4DLPFPRBTY7HWX3QZDDRCQXMZA5CNFSM4KTLK4C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELRVIBQ#issuecomment-585323526>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6M4DISAVLJNOC2MS2GW3DRCQXMZANCNFSM4KTLK4CQ> .

[XhmikosR]

There was an issue with the server, this should be fixed now, thanks to @sideshowbarker

I strongly advice against using anything else other than the official validator. It's been proven over all these years that this lead to poor results.

BTW I don't like the fact that the official validator needs Java, but that's how it is.

Just my 2 cents.

[jfoclpf]

@XhmikosR what I realised is that html-validate is much more stricter that the one from W3C. I'd say if passes the prior, it passes the latter.

[XhmikosR]

You, and everyone else, are free to use whatever you fits your needs. I know from experience (without having tried html-validate because I don't have any need to), is that every other package out there that claimed to validate HTML, failed badly. Maybe html-validate is the exception to this, I might try it if I ever have the need for this.

I don't really mind since I will just use another package if this one will switch to an unofficial validator. Unless that unofficial validator proves to be on par, with following the specs 100%.

Like I said, that's my personal opinion after years of working on this stuff, and I'm definitely not blaming html-validate which like I said I haven't used.

[jfoclpf]

@XhmikosR I'd love to have an official validator that works offline, because the server from W3C is very unreliable some times, and thus my tests suites on travis may output an error just because of that, failing the gits to be committed. And I'm validating 40 files in series with intervals of 1,1 second in-between just to avoid overloading the server, and still sometimes it fails.

[XhmikosR]

You can just use vnu-jar. Like I said it does require the Java runtime to be present, though.

Otherwise you are welcome to extend the npm package package to download the prebuilt binary based on the platform the user is on and submit a PR upstream :)

[jfoclpf]

According to the documentation of html-validate:

When rendering a document it is useful to try to correct malformed markup but a validator should be strict. No corrections, assumptions or guessing is done. If the markup is invalid the parser will tell you so. By ensuring the markup is strictly valid it reduces the amount of bugs where different browsers autocorrect the markup differently (this is especially true for mobile browsers).

As I said, my files were 100% ok on W3C and suddenly now have more than 300 errors with html-validate. If I have a critic to make to html-validate is that is too strict and picky, much more than necessary.

[jfoclpf]

Hi again :)
I was really fed up and about to uninstall this dependency, not your fault but I can imagine that the W3C server is bombarded with requests and they need to drop some. This module was constantly giving errors when the code was perfectly OK.

I eventually solved the issue with async.retry

const async = require('async')
const validator = require('html-validator')

// try calling apiMethod 10 times with exponential backoff
// (i.e. intervals of 100, 200, 400, 800, 1600, ... milliseconds)
async.retry({
  times: 10,
  interval: function(retryCount) {
    return 50 * Math.pow(2, retryCount);
  }
}, (callback) => {
    const options = {
      format: 'text',
      data: body // html body to be validated
    }

    validator(options)
      .then((result) => {
        if (result.toLowerCase().includes('error')) {
          callback(Error('Found html error'))
        } else if (result.toLowerCase().includes('warning')) {
          callback(Error('Found html warning'))
        } else {
          callback()
        }
      })
      .catch((err) => {
        callback(Error(err))
      })
}, function(err, result) {
    // do something with the result
})

I strongly recommend you to implement it

[jfoclpf]

the async.retry could be transparent to the user and you could allow it in the options

(async () => {
  const validator = require('html-validator')
  const options = {
    url: 'http://url-to-validate.com',
    format: 'text',
    retry: 10,
    retryInterval: (retryCount) => 50 * Math.pow(2, retryCount)
  }
  
  try {
    const result = await validator(options)
    console.log(result)
  } catch (error) {
    console.error(error)
  }
})()

what do you think @zrrrzzt ? Or maybe this simply goes totally beyond the scope of this module?