Skip to content

Skimming with coffea and writing out RNTuples takes forever to read the back up #1637

Description

@ikrommyd

Reproducer using the 40-event tests/samples/nano_dy.root file from the coffea repo:

import time
import awkward as ak
import uproot
from coffea.nanoevents import NanoEventsFactory, NanoAODSchema
NanoAODSchema.warn_missing_crossrefs = False


def is_rootcompat(a):
    """Is it a flat or 1-d jagged array?"""
    t = ak.type(a)
    if isinstance(t, ak.types.ArrayType):
        if isinstance(t.content, ak.types.NumpyType):
            return True
        if isinstance(t.content, ak.types.ListType) and isinstance(t.content.content, ak.types.NumpyType):
            return True
    return False


def uproot_writeable(events):
    """Restrict to columns that uproot can write compactly"""
    out = {}
    for bname in events.fields:
        if events[bname].fields:
            out[bname] = ak.zip(
                {
                    n: ak.to_packed(ak.without_parameters(events[bname][n]))
                    for n in events[bname].fields
                    if not n.endswith("IdxG")
                    if is_rootcompat(events[bname][n])
                }
            )
        else:
            out[bname] = ak.to_packed(ak.without_parameters(events[bname]))
    return out


filename = "tests/samples/nano_dy.root"
events = NanoEventsFactory.from_root({filename: "Events"}, mode="eager").events()

events = events[ak.sum(events.Electron.pt > 10, axis=1) > 0]
with uproot.recreate("skimmedevents.root") as fout:
    fout.mkrntuple("Events", uproot_writeable(events))

print("Finished skimming")
print("Reading the skimmed file with NanoEvents")
start = time.perf_counter()
events = NanoEventsFactory.from_root({"skimmedevents.root": "Events"}, mode="eager").events()
print("Elapsed time:", time.perf_counter() - start)
print("Done")

The above snippet takes 85 seconds for me in my machine to read the skimmed file skimmedevents.root.
If I swap mkrntuple to mktree it takes half a second.

This may not be an uproot issue directly but it may be an issue with how we read RNTuples in coffea.
Opening it for investigation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug (unverified)The problem described would be a bug, but needs to be triaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions