Refactor IOBuffer code

[jakobnissen]

I've been a little frustrated with the IOBuffer code. It contains a whole bunch of implicit invariants, and is poorly commented. It also has several bugs that ultimately stems from the code being unclear about its own assumptions.

This is a refactoring of IOBuffer. The primary goals are:

• Comment the code more heavily
• Test the code more thoroughly

The secondary goals are

• Fix a few outstanding bugs
• Add some minor performance improvements

This is a purely internal refactoring with be no change in behaviour of IOBuffer, except straight up bugfixes. However, note that previous code may have relied on buggy behaviour. Fixing bugs may therefore cause breakage.

Current changes

BEHAVIOUR CHANGES

• The following code used to not throw an error, but now does: IOBuffer(b"abc"; maxsize=2). I consider this a bugfix. It should not be possible to construct an IOBuffer with a buffersize larger than maxsize.
• It used to be possible to write to indices higher than maxindex, which could trigger a bug causing data loss. The bug has been fixed, but as a result, some IOBuffers may reach full capacity faster (really: reach it at the correct point), changing writing behaviour.

Bugfixes

• Do not corrupt data on copyline on a non-appending buffer
• Respect maxsize even after take! (fix IOBuffer does not respect maxsize after the first take! #57549)
• Fix bug when copying from an appending iobuffer to itself
• Fix bug where re-allocating the buffer may cause it to shrink, discarding data.
• Fix bug where truncate may throw a wrong BoundsError
• Fix a bug where truncating a buffer may not correctly removed mark at position that has been deleted
• Fix a bug where initializing an IOBuffer without an explicit buffer and with truncate=false makes it contain the full buffer
• Current behaviour for reset and position did not work for PipeBuffer. Fix that

Changes to brittle code

• Removed some tests that explicitly tested internal code and internal behaviour. Some of that behaviour has changed.
• Changed some internal PipeBuffer behaviour, which did not respect writable IOBuffer's ownership of their buffer and therefore failed spuriously

Performance improvements

• Writing to dense IOBuffer now uses memmove and is up to 10x faster for long writes.
• Minor optimisations (about ten percent) for writing to IOBuffers in general

Closes #57549

[vtjnash]

This is actually a bug in the current implementation. The ptr is not supposed to be restricted to be inside the file size, and any calls to write that happen should then automatically zero-pad the file from size:ptr (https://pubs.opengroup.org/onlinepubs/009695399/functions/lseek.html)

[jakobnissen]

Hmm how does that work? Would the user have to seek first to beyond size? That doesn't correspond to how seek normally works in Julia: Normally, you can't seek beyond the filesize. Should that be allowed? Then what happens if the user seeks beyond lastindex(buffer.data)?

[jakobnissen]

How is it supposed to work with IOStream? What file should the following script create?

f = open("/tmp/foo", "w+")
write(f, "abcd")
flush(f)
seek(f, 10)
write(f, 'a')
flush(f)
close(f)

[jakobnissen]

Let's stop the PR here - tests pass, the most pressing issues has been addressed, and it's already a fairly large (too large?) PR. I'll make a followup PR with some more tests and a few more optimisations and bugfixes after this is merged and I get code coverage.

[jakobnissen]

Adding @nhz2 because you've been a thorough reviewer on previous IO related PRs. Feel free to unassign yourself.

[vtjnash]

The removal of offset is a breaking change and performance regression. That probably shouldn't be mixed in with a PR that claims to be just bug fixes and refactoring for clarity

[jakobnissen]

I would argue against it being a performance regression, but surely, it's not a breaking change. How is it breaking to remove a non-documented field of a struct? One which was added only in the current release? Which documented behaviour does that break?

Edit: I see it now. The documentation says that the buffer only controls any input array if write is true.

[jakobnissen]

Thanks for the review. All comments has now been addressed (except the one about endianness which I don't understand)

[jakobnissen]

The build failure (segfault) on 32-bit linux looks unrelated.

[jakobnissen]

Bump. This is done now - a few people have reviewed it, but none have approved. It would be nice to get this in early in 1.13's cycle, so any bugs are detected early. Once this PR has landed, and #57668 has been fixed, I can make a followup PR to get test coverage closer to 100%

[jakobnissen]

Test failures seem unrelated