Describe proposed blob structure

[nikeshnazareth]

No description provided.

[dantaik]

Why header length is zero most of the time?

[nikeshnazareth]

because I suspect the 5 segments already described by the first chunk will be sufficient. We only need one segment per content type (eg. rollup, or compression mechanism). I just added this explanation to the doc

[dantaik]

if there are more than one blobs in a publication, will only the first blob contains the header (if the header is small enough to fit into a blob)?

[nikeshnazareth]

Yes, I think it makes sense to think of the whole structure as corresponding to one file. The header is the start of the file, which spans as many blobs as necessary.

[ernestognw]

Yes, so the header spans only at the beginning of the first blob since we're not expecting a significant amount of application_id. In theory, a header can even span more than 1 blob but I don't think that's realistic

[dantaik]

If we do not use binary search, will the order of data identifiers matter?

[nikeshnazareth]

I don't think the order matters, but my intuition is that data formats should be as deterministic/structured as possible by default. I think things like json are intentionally flexible because they're partially intended to be human readable.

If we insist on ascending order then:

• two publications can be checked for equality directly (by comparing their bytes or their hashes)
• we can reintroduce binary search if the sizes ever get large enough

This is a very mild preference because I don't see a particular reason to prefer one over the other.

[ernestognw]

I would support the ascending order approach, I would expect linear search to be simpler for small lists. Here we expect around 5 segments (or application_ids).

I don't think the order matters, but my intuition is that data formats should be as deterministic/structured as possible by default.

Does a structured data format also helps to identify invalid chunks? I also feel checking the order is not needed as long as the application knows how to read their own data

[dantaik]

there is no definition of "segment header"...

[nikeshnazareth]

I included a comment emphasising that it's the same format as the publication header

[dantaik]

If a rollup consists of multiple segments, should it concatenate these segments (excluding their headers) into a single byte array and validate them as a whole, or should each segment be validated individually, with the results (potentially structured) then concatenated together?

[nikeshnazareth]

I'm not sure what we're validating here but I think you wouldn't concatenate them in either case. Any data that should be processed together should be part of the same segment. If a rollup processes multiple segments, it will be because they're broken down into semantically different things like:

• the rollup transactions
• another rollup's transactions
• coordination information with other rollups
• potentially configuration information or updates that do not require releasing new node software

So the idea would be:

• use the headers to identify and locate the relevant segments
• if there are compressed segments, use the headers to ensure they are relevant and then decompress / locate the relevant segments
• you now have different segments that need to be processed independently

[dantaik]

I prefer not to use the next segment’s offset to determine the current segment’s size. This way, app A can use segment [100, 200], while app B can use segment [100, 300], allowing for greater flexibility.

[nikeshnazareth]

I think we're conceptualising "segments" differently, and perhaps that's relevant to this comment, whether it matters that the segments are ordered, and whether we need a length field.

I imagine each segment contains a particular type of information (where the type is identified by the data format id). In your example, something about the data needs to change at position 200 for app A to decide to stop processing it (eg. perhaps 100-200 represents A's transactions, while B embeds rollup A so it needs to know about A's transactions, but also needs to care about its own transactions in 200-300). In that case, I'd still call 100-200 one segment, and 200-300 another segment, and B would have registered that it wants to process both types.

Can you think of a scenario where rollups process partially overlapping information, but it does not make sense to conceptually divide the overlapping part as its own data segment?

[ernestognw]

Looking good. I reworded some sections, mainly aiming for clarity and to make the writing less personal

[ernestognw]

I would support the ascending order approach, I would expect linear search to be simpler for small lists. Here we expect around 5 segments (or application_ids).

I don't think the order matters, but my intuition is that data formats should be as deterministic/structured as possible by default.

Does a structured data format also helps to identify invalid chunks? I also feel checking the order is not needed as long as the application knows how to read their own data

[ernestognw]

Suggested change

[ernestognw]

LGTM