Add optional arguments to savetxt

[fiolj]

Motivation

Hello, I've been away from the project for a long time, and could not find discussions over some features of savetxt that I find useful, following mostly numpy savetxt behavior. I'd like to change the interface to something like. Please let me know
if there is any interest. As I said I could not find if it already was discussed and discarded.

I am proposing to add several arguments.

1. Adding the possibility of providing a unit number in place of a filename gives flexibility to add partial tables to a file.
2. fmt Gives the possibility of customize the output. In many cases complex calculations give only a few significant figures and I would like the number saved to express that. Should work in a similar way as it does in loadtxt().
3. header and footer give the possibility to add information to the data files (column names, information on parameters used in calculations or measurements, etc). I find this very important because it allows to keep information together with data.
4. Following numpy I find extremely useful if non-data information is commented or signaled in some form. This would be the use of the argument comments.
5. I would propose to change the length of delimiter from len=1 to arbitrary length. For intstance, we may want a *.cvs file to be separated by a comma and one or more spaces ', '.

The spec would be something like:

---

savetxt - save a 2D array into a text file

Status

Experimental

Description

Saves a rank-2 array into a text file.

Syntax

call [[stdlib_io(module):savetxt(interface)]] (array [, filename | unit] [, delimiter] [, fmt] [, header] [, footer] [, comments])

Arguments

array: Shall be a rank-2 array of type real, complex or integer.

filename (optional): Shall be a character expression containing the name of the file , that will contain the 2D array. Present at the same time than unit will give an error.

unit (optional): Shall be an integer containing the unit of an already opened file. Present at the same time than filename will give an error.

delimiter (optional): Shall be a character expression of any length that contains the delimiter used to separate the columns. The default is a single space ' '.

fmt (optional): Fortran format specifier for the text save. Defaults to the write format for the data type.

header (optional): Shall be a character expression that will be written at the beginning of the file.

footer (optional): Shall be a character expression that will be written at the end of the file.

comments (optional): Shall be a character expression of (length 1 that will be prepended to the header and footer strings to mark them as comments. Default: # .

---

Prior Art

• Python Numpy savetxt

• I've written a similar function in numfor

Additional Information

No response

[jalvesz]

Hi @fiolj,

I'm not aware of any discussions on this topic but I think it is a most welcomed contribution!

I've been thinking on a couple of enhancements for loadtxt:

1. add a shared option to enable opening the file in parallel (I've patched my own version to open in parallel with MPI)
2. change the internal logic to use access=stream and read the data with to_num_from_stream which would accelerate reading the whole file.

[fiolj]

Hi @jalvesz, thanks for the quick answer.
The proposal is to make the savetxt function more convenient, as I don't see (or in principle would use) it as a very optimized output for huge amounts of data.
I don't know what the rest of the community think about this proposal.

The implementation for these additions to are quite easy and straigthforward. If we would implement it I would suggest make it in stages:

1. Add header/footer (and comments)
2. Change to variable-length delimiters
3. Add option for format (fmt optional argument)
4. Add exclusive filename | unit (which would also allow output to stdout)

It is probably desirable to update loadtxt to make both functions compatible.

For instance, if we add header/footer we should also add the options comments to loadtxt.
May be it is a good idea to discuss your enhancements to loadtxt first, because we will have to verify that the line has numbers and does not start with a comment before attempting to convert them to the right type.

In any case, It would be good to have a discussion about this.

Juan

[jalvesz]

The proposal is to make the savetxt function more convenient, as I don't see (or in principle would use) it as a very optimized output for huge amounts of data.

Agreed, I just propose the enhancement because even if not advisable to use ASCII to store large amounts of data, it does happens, and also it might not be a question of the IO performance of a single file once, but reading many files many times. I'm convinced that enhancing loadtxt would be desirable.

I don't know what the rest of the community think about this proposal.

Just my experience so far, if you want a broader community input, you'll have to open a thread at the fortran-lang discourse. stdlib Github issues receive very little inputs.

It is probably desirable to update loadtxt to make both functions compatible.

Agree!

May be it is a good idea to discuss your enhancements to loadtxt first, because we will have to verify that the line has numbers and does not start with a comment before attempting to convert them to the right type.

With stream access, it would indeed be necessary to identify the starting position of the numerical section in the character pointer (which could be done by searching for EOL sequentially). Once the first line corresponding to the numerical block is found, then it should be quite straight forward to read through the rest of the file.

(just for reference, I had done a quick example of loading a file with ascii data here https://github.com/jalvesz/mini_app/blob/main/src/mini_app.f90 where load_mesh_stream is around 7 times faster than the plain load_mesh using Fortran intrinsic read)

[fiolj]

Just my experience so far, if you want a broader community input, you'll have to open a thread at the fortran-lang discourse. stdlib Github issues receive very little inputs.

I opened a thread now. Thanks

With stream access, it would indeed be necessary to identify the starting position of the numerical section in the character pointer (which could be done by searching for EOL sequentially). Once the first line corresponding to the numerical block is found, then it should be quite straight forward to read through the rest of the file.

Sounds interesting, and I agree that for many files could be useful.
In principle, if only header or footer are allowed, it could be parsed and determine the number of rows, as well as the beginning and end of data in the same function.
I'll play with your idea and code next week to get a more informed opinion.

[fiolj]

If we save or allow the files to be read with loadtxt to have commented lines, we should probably make loadtxt compatible.

One easy form of doing this is to

• Read the file
• Split into lines
• Split the lines at the comment-char
• Remove the left-side of comments-chars
• Then remove the line if empty

The problem with this approach is that we have to preprocess the full file, duplicating the memory usage.
Alternatively, we can process the file in chunks.

Any thoughts?
Should we start a new issue to discuss loadtxt?
Should we just preprocess or discuss also the use of stream access?

[jalvesz]

Should we just preprocess or discuss also the use of stream access?

I think it is the perfect moment to do so. With the stream access approach, the whole file is loaded in an initial large string. So then the analysis would be done by exploring the contents of the string. EOL characters would be registered inside as well (CR LF).

separating the header and footer sections would be quite straightforward by identifying the chunks between comments character and EOL.

Since the data is registered without explicit mention of the total amount of elements, the key analysis would be to count how many numerical values are in the numerical block before allocating the actual data buffer.

Couple of options that come to mind:

• parse once reading the values to simply count, then parse again after allocating the buffer and actually storing the values directly in it.
• Allocate the data buffer once with a size estimation coming from the file size, parse until either the numerical data section ends or the data buffer limit. Reallocate to continue or size down the buffer depending.

[fiolj]

Hi @jalvesz (and others). I've made a first attemp to change loadtxt() to make it compatible with the new optional arguments of savetxt() (header, footer, comments) and also use stream access to make it more compatible.
Currently I have only made a standalone module for real(dp) values for testing.
The intended spec would be

loadtxt(filename, array [, comments] [, delimiter] [, skiprows] [, max_rows])

This function is able to read data like:

`# header

# Other header
0.5000000000 1.0000000000000000E+000 5.0000000000000003E-002
1.5555555555 3.1111111111111112E+000 4.8395061728395061E-001
2.6111111111 2.6111111111111112E+000 1.3635802469135803E+000
3.6666666666 7.3333333333333339E+000 2.6888888888888891E+000
4.7222222222 9.4444444444444446E+000 4.4598765432098766E+000
5.7777777777 1.1555555555555555E+001 6.6765432098765434E+000
6.8333333333 1.3666666666666668E+001 9.3388888888888903E+000
7.8888888888 1.5777777777777779E+001 1.2446913580246916E+001
8.9444444444 1.7888888888888889E+001 1.6000617283950618E+001
10.000000000 2.0000000000000000E+001 2.0000000000000000E+001
# footer
`
and some superficial tests show improvements on performance with factors of about 3 (not orders of magnitude differences, at least with this first implementation).

However, this implementation fails if the files are not as clean (with empty lines and comments within the table); for instance fails for:

`# header

0.5000000000 1.0000000000000000E+000 5.0000000000000003E-002
# Other header

1.5555555555 3.1111111111111112E+000 4.8395061728395061E-001
2.6111111111 2.6111111111111112E+000 1.3635802469135803E+000 # inline comments
3.6666666666 7.3333333333333339E+000 2.6888888888888891E+000
4.7222222222 9.4444444444444446E+000 4.4598765432098766E+000
5.7777777777 1.1555555555555555E+001 6.6765432098765434E+000
6.8333333333 1.3666666666666668E+001 9.3388888888888903E+000
7.8888888888 1.5777777777777779E+001 1.2446913580246916E+001
8.9444444444 1.7888888888888889E+001 1.6000617283950618E+001
10.000000000 2.0000000000000000E+001 2.0000000000000000E+001
# footer
`

The changes for also accepting this kind of files imply to parse the whole file, and at least in this first attemp the performance worsens considerably. I've currently made an interface:

loadtxt(filename, array [, comments] [, delimiter] [, skiprows] [, max_rows] [,fast])

where the optional argument fast is a logical value indicating to use the fast version (which fails for the second class of files).
Default value should be .TRUE.

[jvdp1]

Thank you @fiolj . I support the idea. The API of loadtxt seems good to me.

• if fast is .false., does it fall back on the original implementation (without performance penalty )?
• Should there be an API with unit (instead of filename; similiarly to savetxt)?

[jalvesz]

@fiolj could you share the way you are doing the implementation? This will help us elucidate what's going on, I'm pretty sure there is a way of fixing the issue and not requiring an algorithmic switch.

[fiolj]

I've put a repository with the code in https://github.com/fiolj/demo-loadtxt.git

Both versions work with stream access.
The difference in execution times appears because the "fast" version only checks the first lines until data appears for a header and the last lines for footer, while the more versatile version checks all the lines. It would be very good if we can make it fast and versatile at the same time.

In the repository there is a module and a demo for testing the new routine (called loadtxt2), and comparing with the original loadtxt.

I am using make for compiling (with gfortran) but it is only a two-files code, should be easy to compile without the Makefile

In my system, current output (execution times) is:

For 10 files:
loadtxt took 0.16791 sec
loadtxt2-fast took 0.06319 sec
loadtxt2-versatile took 1.3775 sec
Each file has 30000 values

[fiolj]

Thank you @fiolj . I support the idea. The API of loadtxt seems good to me.

Thanks @jvdp1, I've upload the code in https://github.com/fiolj/demo-loadtxt.git

* if `fast` is `.false.`, does it fall back on the original implementation (without performance penalty )?

No, if fast is .false. it only checks a few lines for header and footer, making it compatible with the "new version" of savetxt and it is a factor of aprox 3 faster than the original implementation

* Should there be an API with `unit` (instead of `filename`; similiarly to `savetxt`)?

I did not think of it, but could be equally easily implemented if there is interest

[fiolj]

Another issue that I would raise is the name of the optional argument skiprows
In the new implementation, as in Numpy's loadtxt() the routine skips lines independently if they have comments (header), empty lines, or data values.
With this definition I would prefer the name skiplines