THE LINUX FOUNDATION PROJECTS
GSoCHPX

Hierarchical Collectives in HPX: The Story So Far

By September 11, 2026No Comments

Project: Hierarchical Collectives for HPX
Contributor: Anshuman Agrawal
Organization: The Ste||ar Group
Mentor: Alexander Strack

At a glance

This project completed HPX’s hierarchical collective family, hardened the communicator and distributed-test paths, and used multi-node measurements to replace a root-routed all_to_all bottleneck with a pairwise implementation selected by payload size. The report also records the benchmark corrections and cluster conditions needed to interpret the results.

The main collective work is merged upstream. The performance-attribution branch and GPU prototype described near the end are follow-up work, not part of the completed upstream contribution.

Official project: Hierarchical Collectives for HPX

Merged upstream work

How this started

In late March I emailed Alexander with one design question: for hierarchical collectives, should the tree topology be fixed (binomial, k-ary) or configurable, so it can adapt to the cluster it runs on?

I was coming from the GPU training side. My research internship had me profiling distributed training on DGX h100 nodes and tuning nccl collectives, and before that I had built a ring all reduce from scratch in cpp on top of mpi. So I knew collectives. I did not know HPX. HPX is an asynchronous runtime with its own communication layer, its own global addressing, and its own opinions about who talks to whom and when. Most of what I know about it now came from being wrong about it in public, in small doses, between April and July.

I also thought the topology question was the interesting one. It wasn’t. The real answer showed up in July, and it came out of a benchmark, not a design discussion. But that’s the end of the story.

April: small things first

Before I was selected I wanted to show I could work in the codebase, so I started with the smallest first thing I could find: fixing copy paste errors in the collectives documentation. Two doc fixes in, I moved to real code. HPX already had the hierarchical communicator machinery (subtrees of localities, each with a representative, plus a top level communicator connecting the representatives), so my job was to extend the family. Hierarchical all_reduce first, then all_gather, which is just a gather up the tree followed by a broadcast back down.

This is where HPX corrected me for the first time. Hierarchical collectives run two internal phases, so each user call has to consume two internal generation numbers. I mapped user generation k to internal generations 2k and 2k+1. Looked fine. But the communicator’s internal gate requires generations to increase monotonically with no gaps, and my mapping starts at 2. Generation 1 never happens, the gate rejects the sequence, nothing completes. The fix is the mapping the project has used ever since: 2k-1 and 2k, so user call 1 maps to internal 1 and 2. A four line change, and I understood the gate a lot better.

I also tried to be clever that month. I refactored the collectives to use nonblocking future continuations, decided within a day that the extra complexity was not buying anything yet, and reverted it. Then the useful stuff: a flat fallback so small site counts skip the tree, unit tests, tests for tree arities that don’t divide evenly, and benchmark support. By April 25 I had a design note written for the big one, hierarchical all_to_all, and that note became the spine of my proposal.

May: the plan got shorter

I joined the Ste||ar Discord in early May and posted the all_to_all design note. Hartmut Kaiser (hkaiser) replied the same day: “I think you have it nailed quite nicely. Let’s go for it as you outlined it, we can always correct the approach, if needed.” We ended up correcting the approach more than once, so that last part aged well.

Alexander and I had our first proper call in late May, after my university exams and some back and forth to find a slot that worked across IST and CEST. The call mostly deleted work. My design note assumed I first needed to generalize the generation arithmetic to stride three, since all_to_all has three phases. On the call we worked out that the phases naturally land on different communicators. The subtrees handle gather and scatter, the top communicator handles the exchange, so everything fits the existing stride two scheme without touching any other collective. A whole prerequisite disappeared.

The plan I posted after that call was three PRs: a basic hierarchical all_to_all, then flat fallback plus input validation plus full test coverage, then distributed regression tests and tests across collectives. hkaiser: “Ok, interesting analysis. Let’s try that!”

June, part one: all_to_all lands

The implementation is three phases. Every locality sends its blocks up to its subtree representative, the representatives exchange the merged blocks over the top communicator, and everything gets scattered back down. Fewer messages cross the slow links between nodes, which is the whole point of the hierarchy.

I opened the PR at the end of May. On June 1 hkaiser wrote “Excellent job on the all_to_all collective! Thanks!” and then left a pile of review comments. The next two days were review response: better names, a linear scan replaced with a binary search, asserts for the empty communicator edges, and one long argument between me and the formatter about what C++ should look like. The formatter won. The PR merged June 4, and the same day I started setting up HPX on the Rostam cluster with the LCI parcelport, because I wanted real numbers before claiming anything about performance.

June, part two: hardening

The second PR was the boring one, and it contained my favorite bug of the summer, because I had planted it myself. Back in April I added the flat fallback: below sixteen sites, skip the tree. Sensible feature. Except my own unit tests ran at small site counts, which means every “hierarchical” test had been quietly running the flat path. CI was green the whole time and testing the wrong thing. Fixing it meant rewriting tests across eight files and adding a new test across collectives so the tree path is actually pinned down, plus input validation on every entry point so bad arguments fail loudly instead of hanging.

Mid June brought a design discussion I’m still happy with. I had shipped a restriction: you couldn’t share one hierarchical communicator across different collectives, because different collectives consumed generations at different rates and mixing them would desync the gate. hkaiser asked whether we could drop the restriction by having every collective step by two. My answer was yes, and here is what it costs: the all_to_all top communicator and the standalone collectives with a single phase would all need padding to the same rate. I had kept each collective at its natural rate in the first PR on purpose, so I wouldn’t have to touch the others, and documented the sharing rule instead. That was the honest count at the time. But stepping everything by two is the cleaner long term call, so I changed it later. Everything steps by two now, the rate hides behind a generation_mode enum, sharing across collectives works, and there’s a test that mixes collectives on one communicator across tree shapes to keep it working.

Around then I also posted the first proper benchmark results as a GitHub discussion. They were bad. I’ll get to how bad.

Meanwhile, on the cluster

All of this ran on Rostam, the Ste||ar cluster, and Rostam turned into its own subplot in June. Brett Estrade runs it, and he heard from me a lot that month.

The first thing I brought him wasn’t even my problem. In early June, while poking at the HPX parcelports, I noticed that any MPI program asking for MPI_THREAD_MULTIPLE quietly lost UCX. The UCX modules on the cluster were all built single threaded, so OpenMPI silently fell back to plain TCP with one warning that was easy to miss. The gap is real: a two rank ping pong runs around 1 us single threaded and jumps to about 10 ms per op once a few threads pile onto the fallback, and asking for SERIALIZED instead brings UCX right back. I was going the LCI route myself, so it didn’t affect me, but it would quietly bite anyone running MPI with threads, so I flagged it to Brett with the module output and a suggested fix. Brett: “Thank you, good to know.”

Mid June the login node started dying. Charan hit it first, I confirmed and narrowed it: port 22 dead, the web portal on 443 fine, so sshd was hung, not the network. Brett rebooted and found a disk issue with no clear cause. The next day I couldn’t get in from India at all, and I had a nice theory that the reboot had reset a firewall rule against my mobile IP. Brett suggested the cheap test: try from a US host with a stable IP. I did, from a cloud box in Washington, and it couldn’t reach port 22 either while 443 worked from everywhere. So much for my theory. sshd was simply down again, for everyone. Brett rebooted once more and traced it toward the NFS storage backend, and when the node went partly down that same evening, accepting the TCP connection on 22 but resetting before the SSH banner, I could at least report it precisely enough that he knew which dependency was hanging. He added the machine to his monitoring so it would get caught in the act next time. None of this is glamorous, but the benchmarks don’t run without it.

Then the scare that mattered most for this report. On June 17 hkaiser pinged me at half past midnight: could I help with the MPI/IB issue? One MPI broadcast number, 8.6 ms where microseconds were expected, was ugly enough to implicate the InfiniBand fabric itself. I measured the 2 node, 40 rank broadcast again with completion aligned timing, taking the max elapsed across ranks instead of trusting any single rank, and the badness was gone. OpenMPI 5.0.5 and 5.0.7, default and forced UCX, all came in around 0.01 ms, and UCX over IB was healthy. The 8.6 ms was never the fabric. The benchmark had been recording only the root rank’s call return instead of collective completion. So I fixed the MPI benchmark to time completion aligned, matching the HPX side, and ran the full sweep again before publishing any comparison. Brett, watching all this land in his channel at 2 AM, asked whether it had anything to do with the cluster going down. Purely benchmark stuff, totally separate, and a fair question given the week he was having.

Usability first

In late June Alexander came back from the weekly HPX meeting with a course correction. My head was full of fancy algorithms. I had sketched a topology aware, pipelined, adaptive all_to_all and wanted to start on it. The feedback, from hkaiser through Alexander, was: usability first. And Alexander had the evidence in hand, because during his rework of the collective benchmark he had noticed the set wasn’t even complete. inclusive_scan and exclusive_scan were missing entirely. Finish the set before optimizing it.

He was right, and the scans had their own wrinkle anyway: exclusive_scan exists twice. The C++ standard version and the MPI version disagree about how the first element is treated, HPX has both, and the hierarchical versions have to handle both. I did a small cleanup PR first, the kind of tidying you only notice is needed after living in the code for six weeks, then built the hierarchical scans with tests and benchmark coverage. They merged on July 6 in PR #7343.

That week I also asked directly for feedback on how I was doing as a contributor. PR structure, communication, design approach, anything. hkaiser: “I personally am very happy with your work. Keep it that way!” I screenshotted that one.

July: the 30x wall

Now the thing I had been circling since June, and the first question was whether to believe the numbers at all. Getting them trustworthy was work in itself. Alexander reworked the collective benchmark on the HPX side, including how time was recorded, and the corrected results I posted at the end of June came out of his rework, not out of anything I fixed. He also pulled together data from three different machines, the older results on Medusa, the merged code on Buran, my runs on Loni, so we could tell machine effects from code effects. The MPI side had just had its own timing fixed after the IB scare. And one question that kept coming back in our meeting notes was whether the MPI baseline was even still fair. Everything below stands on that groundwork, and most of it was his.

The corrected numbers were still bad. HPX collectives over LCI were running at a geomean of 35.7x / 31.7x / 28.4x slower than MPI on 1, 2 and 4 nodes, twenty localities per node, with OpenMPI over UCX as the baseline. The worst case, large all_to_all, got worse as node count went up. So the gap was real, not a leftover from broken measurement. Nobody assigned me this problem. It was sitting in my own results, and I didn’t feel like I could write “hierarchical collectives” on anything while the flat ones were thirty times off.

So July turned into debugging, and the debugging had three findings.

The star. The existing all_to_all routed everything through a single root locality. Everyone sends their full vector to the root, the root rearranges, the root sends rows back out. At four nodes, one locality’s NIC becomes the bottleneck for the whole operation, which is why the gap grew with node count. Not a config issue, an algorithm and data path issue.

The chunk storm. The benchmark payload is a vector of vectors, and HPX serialization treated it as one zero-copy chunk per inner vector. Each all_to_all parcel exploded into a pile of separate transfers. At 80 ranks, thousands of little handshakes per iteration. As a diagnostic I forced the payload to coalesce, and the worst point went from 110x slower than MPI to 7.8x, with the 16K case on four nodes dropping from 655 ms to 193 ms. A useful clue, but not the final clean fix.

The cliff. After I built a direct pairwise all_to_all prototype, the leftover pattern was clearer: the cost was per message, not per byte. A size sweep showed a sharp cliff at about 8 KB, which is LCI’s default packet size. Below that, messages ride the cheap eager path. Above it they go through rendezvous, and in this setup LCI’s rendezvous cost about 1.5 ms per message where MPI’s comparable path cost about 0.33. The fix was small: raise the LCI packet size so the block sizes we care about stay eager.

The config that mattered:

  • hpx.parcel.lci.progress_type=worker

  • hpx.parcel.lci.progress_strategy=global

  • LCI_ATTR_PACKET_SIZE=73728

  • LCI_ATTR_NPACKETS=8192

With that, the pairwise all_to_all at 16K blocks ran 6.0 / 13.8 / 29.5 ms against MPI’s 4.9 / 11.75 / 24.8, so 1.23x / 1.18x / 1.15x. The full suite geomean gap fell from 35.7x / 31.7x / 28.4x to 5.69x / 6.51x / 7.84x. And the earlier workaround that coalesced chunks stopped being necessary once the packets were bigger, which made me more confident the diagnosis was right.

A lot of things did not help, and I kept that list too, since knowing what doesn’t help saves the next person time: other progress strategies, receive buffer knobs, restructuring how the prototype issued its sends. A few things actively caused trouble: an LCI protocol variant, a couple of background progress settings, and my own standalone audit binaries, which silently misbehaved without a runtime flag I didn’t know about. hkaiser pointed me at the definition that fixes that one at compile time.

Two side checks kept the analysis honest. hkaiser asked me to evaluate a pending performance PR, so instead of eyeballing stale binaries I built a three way comparison from one baseline: control, the PR, and an older variant of it. Result: geomean 1.01 vs control. Flat. His reply was “Yes, I know that #7345 has problems. We might abandon it after all. Thanks for checking.” So “we measured it and it does nothing” is a real, useful outcome, not a failure. He also worried that twenty localities per node might be causing thread oversubscription, so I checked the affinities directly: exactly 2 threads per locality, all 40 on distinct cores, with NUMA local placement, 10 per socket. Clean, not a placement artifact.

Then the definitive run, everything from one baseline in the same week: HPX went from 5.8 / 6.5 / 7.9x off MPI to 5.6 / 6.1 / 7.1x with pairwise all_to_all selected by size, and large all_to_all went from 7.1x behind to 1.2x, basically parity, with the crossover around 64 KB per rank. The honest split: large messages now sit around 2x MPI, small ones are still 10-13x off, and that residual is small message latency in the runtime’s round trip, not the data path. all_to_all went from the worst collective in the suite to one of HPX’s better ones.

vector2d, and the day the laptop died

The chunk storm finding led into the best conversation of the summer, and the ideas in it were hkaiser’s, so let me get the credit right. He started thinking out loud in the channel about fixing the serialization properly. First “we should consider specializing serialization for those,” then twenty minutes later “Or maybe not :/ – Let me think about this,” and then the idea that stuck: a 2D container that stores its elements contiguously, so the serializer sees one buffer instead of hundreds. He sketched a vector2d type built on mdspan, noted that our 2D payloads aren’t ragged anyway (every locality sends the same block size to each peer), and added that mdspan with a stride could even turn the all_to_all transpose into a view instead of an actual data shuffle. My part was small: confirm the row shapes from our side, then go build it.

I prototyped it that night and posted numbers at one in the morning. At 4 nodes, 64 KB blocks went from 662 ms to 228 ms (2.9x) and 16 KB blocks went from 709 ms to 59 ms (12x), with no giant eager pool needed, since one contiguous buffer serializes as one chunk. Only caveat: tiny blocks get slightly slower, because many cheap eager sends become one rendezvous. It helps the sizes that were hurting. hkaiser: “Excellent results!”

Then, that same evening, I spilled coffee on my only laptop, panicked, rushed it to a repair shop, and completely forgot Alexander and I had a meeting. When I messaged him from my phone to apologize, his reply was “no worries. I missed it too”, because he had forgotten one the week before, and earlier in June we had once spent half a week holding different beliefs about which day the sync was. We were not great at calendars. The work didn’t suffer for it.

The prototype has since grown into real code: two payload carriers, one for uniform rows and one for ragged ones, with sizing protected by overflow checks, validation, and their own serialization, wired through the all_to_all exchange and the gather and scatter paths, except on the leaf hops where flattening doesn’t pay. The gather and scatter carriers merged on July 14 in PR #7375, and the all_to_all exchange followed on July 16 in PR #7377. The hardening round merged in PR #7359, and the dangling communicator name fix merged in PR #7378.

hkaiser also flagged a CI failure in the new inclusive_scan. It turned out to be test phases reusing communicators they shouldn’t; the isolation fix merged on July 10 in PR #7364.

When one site throws

Late July found another collective correctness edge. A throwing reduction finalizer could consume the shared payload and leave later sites to re-enter the fold on moved-from state. A throwing step could strand the gate before a required segment was set. Different sites in one collective could therefore see different outcomes, or later generations could hang.

PR #7401, merged on July 24, caches the first operation exception inside the communicator and rethrows it for every site. The cache resets with the generation, so the same communicator can recover on the next call. The regression tests cover throwing finalizers in all_reduce, inclusive_scan, and both exclusive_scan forms, plus step failures in all_reduce and reduce.

A smaller all_reduce change merged two days later in PR #7398. The finalizer now moves the first payload into the reduction seed instead of copying it, with a regression test for move-only values.

Another July question: where does the time go?

The all_to_all work answered why one operation was bad. It did not explain why small HPX collectives were still ten or more times slower than MPI after the data path stopped being catastrophic.

So a second investigation started with a narrower promise: do not make the collectives faster yet. Put enough measurement into the runtime that a slow collective can no longer hide behind one total elapsed time number.

A collective can wait for its communicator lock, finish the last participant’s work, build the result, serialize parcels, acquire a connection, post to a transport, suspend on a future, wake on another worker, and then wait in a scheduler queue before the user sees completion. Timing only the collective function puts much of that story outside the frame.

The work was deliberately isolated from my main HPX checkout and from the cluster directories I was using for the other experiments. The branch starts from upstream commit fad6c8eff7 and currently has five local commits with DCO signatures:

  • b8454299c7 Add collective performance tracing

  • 7c93614910 Trace parcel serialization and transport

  • 73c6ce49d9 Trace future suspension and task wakeups

  • 043d5619f9 Extend collective attribution benchmarks

  • 690a5bd4d7 Bound scheduler performance trace volume

They are local commits. None has been pushed, no PR exists, and the later work described below is still uncommitted on top of them.

Putting stopwatches inside the runtime

The first pass added optional tracing behind HPX_WITH_PERFORMANCE_TRACING. With tracing disabled, which is the default, the measurement code is compiled out or reduced to small guards. With tracing enabled, HPX writes binary records that can be joined by operation, communicator, generation, participant, root, and payload size.

The communicator records cover mutex wait, time under the lock, time after the final participant arrives, finalization, result production, and generation advance. They also count the next generation arriving while the previous one is still finalizing.

The parcel records follow future preprocessing, encoding, archive size, parcel and chunk counts, connection or queue acquisition, transport posting, and transport completion. Each stage carries a flag saying whether it happened under the communicator lock. That means the data can answer the question directly instead of making me infer it from the call graph.

The future and scheduler records distinguish a future that was already ready from a real suspension. They record latency between suspension and wakeup, pending versus pending_boost wakeups, inline versus scheduled continuations, wakeups on the same worker versus another worker, and whether the destination worker was idle or backing off when the work arrived.

The runtime writes raw events. The parser derives categories afterwards. That keeps the recorded history stable even when the analysis changes its mind about how to group it.

The trace that ate the disk

The first cluster matrix found a failure in the measurement before it found a failure in the collectives.

I had added a task_execute event at scheduler dispatch. A pending HPX task can be redispatched many times, so one logical task emitted a record every time it went through the loop. On a TCP run across 16 nodes, one sparse trace reached a logical size of 22,310,835,736 bytes before the job ended on its own.

Nothing was cancelled. I treated the run as invalid and changed the producer. Commit 690a5bd4d7 removes the event at every dispatch while keeping the bounded events that answer the useful questions: suspend, wakeup, resume, worker transition, and idle state.

That was the first result of the attribution project. Instrumentation has its own performance bugs, and a profiler that changes the workload by writing tens of gigabytes is no longer measuring the original program.

The first attribution matrix

The local builds with tracing enabled and disabled both compiled the collectives benchmark. The distributed TCP suite passed 43 of 43 tests after excluding only concurrent_collectives, which has a documented AppleClang 15 limitation on this machine. A compatibility sweep with one locality covered ten operations across single use, multiple use, hierarchical arity 2, and hierarchical arity 4: 40 of 40 cases passed. Round trips for direct actions with two localities passed with tracing both enabled and disabled.

The benchmark writes one raw row at full precision for every measured iteration. Warmups stay out of the raw sample file. The summary contains mean, variance, standard deviation, minimum, maximum, median, and p95 calculated with nearest rank. The relevant diff passed the clang-format 20.1.7 and whitespace checks at that stage.

Rostam added one dependency lesson before it added numbers. The first LCI build used LCI_OPTIMIZE_FOR_NATIVE=ON because it was compiled on Medusa. When the binary ran on Buran, every rank trapped in LCT_init with an illegal instruction. The collectives had not started. The dependency had been built for the wrong CPU.

Job 173110 rebuilt LCI with LCI_OPTIMIZE_FOR_NATIVE=OFF, and I checked the generated CMake cache before using it. That is now a standing rule: a source change can imply a dependency or generated configuration change, and checking only the HPX revision is not enough.

The parcelport smoke gate passed as jobs 173305, 173306, and 173307 for TCP, MPI, and LCI. Focused runs with one locality per node followed from one to four nodes. A separate comparison with one node and four localities stayed separate because the endpoint count matched but the placement did not.

The corrected raw analysis contained 2,147 runs, 717 benchmark groups, 1,499 runs with traces, and 18,493,842 decoded records. Every trace that reached the parser had a valid footer. Twelve validation errors were expected markers for partial MPI jobs whose wrapper stopped before invoking the parser.

The first results were not subtle. At two nodes, a hierarchical broadcast with arity 2 and a 4 byte payload took about 76.92 us over TCP, 30.70 us over MPI, and 38.93 us over LCI. At four nodes, TCP rose to 114.07 us and MPI to 55.36 us. LCI jumped to 629.72 us with a large tail.

At four nodes, all_gather with a 4 byte payload was about 1,038.52 us over LCI and 248.02 us over TCP. At 64 KiB, LCI was about 4,995.68 us and TCP about 698.98 us. Arity also mattered. At four endpoints, arity 4 often beat arity 2 for all_gather and all_reduce; the TCP median for all_gather with a 4 byte payload and arity 4 was about 44 percent of arity 2. That is a configuration result, not permission to change the algorithm in this measurement work.

The lock, the wait, and the scheduler

The attribution changed the shape of the performance story.

Finalization, result production, and generation advance usually happened under the communicator lock. Parcel encoding and transport posting only sometimes did. The simple sentence “serialization happens while the communicator is locked” is therefore not generally true. It has to be supported record by record.

For small messages, the largest interval was usually not computation or transport. It was the time from a future waiter being enqueued until the task resumed. In TCP broadcast on four nodes, waiter wakeup was about 85.8 us median, compared with about 22.1 us of other lock time and 10.3 us of transport completion. In the LCI case on four nodes, waiter wakeup was about 611 us, while other lock time was about 9 us and transport about 2 us.

On the critical locality, the exclusive interval partition accounted for roughly 83.5 to 99.4 percent of broadcast latency for small messages, depending on transport and node count. That result comes from subtracting exclusive intervals sample by sample. The medians of overlapping categories cannot be added to make the same claim.

The matrix contained both futures that were already ready and suspended future access. Wakeups were pending rather than pending_boost, and cases on the same worker, another worker, and an idle worker all appeared. The evidence points toward suspension and wakeup as the next investigation. It does not justify a scheduler change in this unfinished measurement patch.

The queue wait time option had stopped compiling

HPX already had optional queue wait time counters, but normal builds did not exercise them. The queue entry member had been renamed to wait_time while seven accesses behind HPX_WITH_THREAD_QUEUE_WAITTIME still used waittime. Because the option is off by default, the stale name stayed invisible until the attribution work needed that path.

PR #7426, merged on August 2, corrects those accesses and enables the option in the Linux debug workflow. The option is now compiled by CI instead of depending on someone to remember a local build where it is enabled.

The bytes I was not counting

The first transport analysis recorded archive bytes and zero-copy chunk counts. I thought that was enough to discuss root byte concentration. It wasn’t.

Above the zero-copy threshold, the large payload lives outside the encoded archive. One chunk count says a transfer exists, but not whether it holds 8 KiB or 128 KiB. Any calculation of byte share based on archive size alone makes the large payload disappear.

The later work adds a record of transmitted size beside parcel encoding:

  • encoded archive bytes + transmitted zero-copy chunk bytes

Both tracing modes rebuilt the benchmark and zero_copy_parcel_test, and all eight zero-copy test configurations passed. A local TCP smoke with two localities and a direct action reported 131,348 bytes for each 128 KiB send: 131,072 payload bytes plus 276 encoded bytes, with one zero-copy chunk and two chunks that were not zero copy.

The separate Rostam byte build, job 173582, finished in 12 minutes 27 seconds. Jobs 173583, 173584, and 173585 passed smokes on two nodes for TCP, MPI, and LCI. Strict aggregation found 3 jobs, 9 runs, 27 samples, 9,176 raw records, and zero errors. For direct action, MPI and LCI each recorded:

  • 787,857 = 6 * 131,072 + 1,425 bytes

That was the point where the byte record became a measurement I trusted. It still has not been shipped.

MPI, and the crash after the finish line

The 128 KiB byte pass completed cleanly for TCP and LCI at two and four nodes. MPI behaved differently. Jobs 173588 and 173589 printed valid benchmark results and then crashed inside OpenMPI 5.0.5 during MPI_Finalize.

Two facts have to remain separate. The benchmark completed. The transport teardown failed. Calling the entire run a collective failure discards useful data; calling the Slurm job successful hides a real dependency problem.

Recovery runs sharpened the distinction. Some configurations produced complete valid traces before teardown. Others lost their final footer because srun --kill-on-bad-exit=1 killed the surviving rank as soon as its peer segfaulted. A new launcher can use HPX_SRUN_KILL_ON_BAD_EXIT=0 for isolated recovery cases so every surviving rank has a chance to flush. That setting does not hide the MPI failure; the teardown status still belongs in the metadata.

The cleaner dependency experiment, rebuilding with OpenMPI 5.0.7 on Marvin, did not start. Job 173593 was requeued and held with user env retrieval failed. Three earlier preliminary Marvin smokes, 173332, 173333, and 173335, were held for the same reason.

A later audit showed that the submit lines used --export=ALL; none requested --export=NONE or --get-user-env. Slurm had requeued each job once before holding it, and marvin04 was subsequently drained with the reason Diagnosing Lmod issues. The companion LCI smoke 173334 did reach marvin04 and marvin05, but its old v3 script failed because cmake was not available. Marvin exposes CMake as a module, while the old build script loaded only GCC and OpenMPI. Releasing the held jobs unchanged would therefore either repeat a cluster environment failure or proceed to a known script dependency failure.

On July 19, jobs 173332, 173333, 173335, and 173593 were cancelled. They were not resubmitted: they target the old 690a5bd4 attribution revision, whose trace contract was later rejected, and the preliminary smoke work has been superseded. The next immutable campaign must explicitly load CMake on Marvin, preflight the target partition and launch across multiple nodes, and avoid nodes Slurm has drained for module/environment diagnosis.

Several MPI byte cases therefore remain incomplete as historical results. Do not fill those gaps by rerunning the obsolete revision. The corrected campaign must keep benchmark completion and teardown failure explicit while using the new trace contract.

The teardown investigation did produce an HPX fix. PR #7404, merged on July 27, synchronizes ranks before MPI_Finalize, cancels and waits for the parcelport’s wildcard header receive, releases the duplicated communicator, and serializes receiver shutdown against late polling. That changes the failure path described above, but it does not retroactively make jobs 173588 and 173589 valid. A fresh campaign still has to verify the merged shutdown path on the affected OpenMPI configuration.

The same cleanup in late July fixed an unrelated source of distributed test flakiness. PR #7405 changed departed_locality_7384 to ask the OS for an unused port and made the remaining race between probe and bind explicit in the test. It merged on July 27.

The clean run that rejected every HPX trace

The next step was supposed to be a clean smoke before the full matrix on sixteen nodes. I built HPX revision 690a5bd4d7 on Rostam with tracing on and off as jobs 173574 and 173575. Native MPI parity job 173576 passed. All three HPX parcelports were enabled, the source was clean, and the harness current at that time passed all 47 tests.

The new smoke campaign ran as 173578 for HPX over LCI, 173579 for HPX over MPI, 173580 for HPX over TCP, and 173581 for native MPI. Native MPI passed 10 of 10 cases.

All 120 HPX benchmark processes also ran without a transport crash. Their return codes were zero, and the summaries and raw samples for collectives other than barrier were well formed. Strict validation still accepted 0 of 40 cases for each parcelport.

That failure was useful. The event table in the parser had drifted from the producer after event 17. The parser shifted waiter and scheduler events by one, invented an event for direct actions that did not exist, and still required task_execute after commit 690a5bd4d7 had deliberately stopped emitting it.

The old validator expected one object ID per correlation even though a causal path contains communicators, locks, parcels, futures, and scheduler threads. It expected uniform root, participant, generation, and payload metadata even though hierarchical collectives contain multiple communicators and phases. It looked for the public exclusive_scan name even when the benchmark called the form with an explicit initial value. And setup barriers and timing reductions were traced in the same broad attempt as the operation being measured.

Correcting the event numbers alone would have rescued only 53 of the 120 saved traces, even before requiring exact transmitted bytes. The validator did not have a reliable boundary for “this is the collective whose elapsed time is in this sample.”

Barriers found one more bug: the benchmark rejected test_size=0 globally, even though a barrier carries no payload. The current local change permits zero bytes for barrier and still requires a positive payload everywhere else.

The marker and tracing redesign are locally coherent

The later attribution work now carries two identities instead of overwriting one with the other. The outer fields identify the benchmark case, variant, warmup or measured iteration, and sample. Inner fields preserve the collective phase, communicator or runtime object, generation, payload, coordinates, topology level, and causal related object. A composite hierarchical all_gather can therefore retain its gather and broadcast phases while still joining every record to one timed raw sample.

The binary contract is explicitly versioned as format 3: a header of 64 bytes, records of 160 bytes, a footer of 64 bytes, and a summary of 96 bytes. Event IDs 1 through 26 are stable. Ready access and actual future suspension are alternatives; a suspension requires the matching wake and resume path, while a ready access does not. Local shortcuts do not invent remote parcel events.

The observer was redesigned as well. Every registered worker receives a bounded, preallocated record area. A measured path, or one that holds the communicator lock, only appends in memory; it does not open or flush a file. Finalization walks all registered buffers before transport teardown and writes footers and summaries with counters for dropped events and failures to open, write, or flush. The defaults are 8,192 records per thread, 512 threads, and 256 MiB, all configurable for a bounded campaign.

The benchmark marker and raw timing now use the same boundaries. The parser streams records incrementally and recomputes mean, population variance, standard deviation, minimum, maximum, median, and p95 with the nearest rank method from raw samples. It also distinguishes archive buffer bytes from transmitted bytes: the buffered path requires equality, while a zero-copy transfer includes external chunks and can be larger than its archive.

The public benchmark inventory now contains 11 semantic operations, including separate exclusive scans with explicit init and MPI semantics. The generated smoke has 143 cases and the complete expansion for equal comparison has 51,060 cases. Diagnostics and communicator infrastructure are labeled rather than counted as matched collectives.

Local evidence is substantial but not yet the publication gate. The focused suite for the trace format passed 14 tests. Native MPI integration passed every operation/algorithm pair. zero_copy_parcel_test passed 4/4 with tracing on and 4/4 with tracing off; distributed TCP collectives passed 43/43. A barrier with two localities and zero bytes passed, while a zero payload for a collective that moves data was rejected. Real traces with two and three localities joined measured samples to composite phases and demonstrated the rules for buffered and zero-copy transmitted bytes. A real C++20 module build compiled the core/full boundary and passed the two focused tracing tests.

The latest complete harness run passed 42 tests before the newest zero-copy fixture was added; the final suite is expected to discover 43 and must be run rather than assumed. HPX Inspect has been built but not yet run, final formatting and rebuilds from the final source remain, and matched data for tracing overhead has not yet been accepted. Nothing from this revision has been copied to Rostam. It is coherent local work, still uncommitted and not shipped.

August: pairwise all_to_all at sixteen nodes

The pairwise result is no longer only a prototype result. On August 1, Slurm job 178549 ran the implementation integrated into the branch on all 16 Buran nodes. The source was d298341e39, built on master with PR #7414 merged and the pairwise all_to_all branch applied. The job completed with exit code zero in 1:27:21.

The matrix covered 1, 2, 4, and 8 localities per node, for 16, 32, 64, and 128 localities in total. It swept eight block sizes from 4 bytes to 64 KiB per destination, not per locality in total. Every cell had five repetitions, three warmups, and ten measured iterations. Standard, forced pairwise, flat, and tree arities 2, 3, and 4 were remeasured from the same source in the same allocation. All 960 process executions were accepted, with no retries or entries in failures.txt.

The table reports the pairwise latency. Each entry is the median of five medians, one from each run.

  • 4 B: 0.373 ms (16 loc), 0.696 ms (32 loc), 1.281 ms (64 loc), 2.647 ms (128 loc)

  • 4 KiB: 0.424 ms (16 loc), 0.740 ms (32 loc), 1.381 ms (64 loc), 3.039 ms (128 loc)

  • 64 KiB: 0.698 ms (16 loc), 1.530 ms (32 loc), 4.405 ms (64 loc), 15.427 ms (128 loc)

Pairwise was not the best choice for the smallest payloads. Across all 32 combinations of layout and size, it beat the fastest HPX alternative run as one shot in the same campaign in 16. The crossover moved earlier as the locality count grew, and at 4 KiB per destination and above pairwise won in every layout. At 4 KiB it was 1.18x, 2.79x, 5.28x, and 13.82x faster than the best of standard, flat, and the three tree arities. At 64 KiB those ratios were 5.24x, 10.25x, 15.39x, and 22.47x. Across the complete matrix of 32 cells, the advantage in geometric mean over the best alternative selected independently in each cell was 1.79x.

The campaign measured the integrated branch rather than the earlier standalone prototype. PR #7431 merged that implementation on August 9. It adds the pairwise path to the synchronous and asynchronous overloads that use a basename and caches its channel communicator across generations. Automatic selection stays conservative: dynamically sized rows remain routed because independent local size decisions could make sites choose different algorithms. Callers that know their rows are uniform can explicitly request the pairwise path.

The campaign did not rerun native MPI, so this matrix supports the HPX algorithm comparison and should not be presented as a new comparison between HPX and MPI. An LSU Clang 17 and OpenMP build later exposed a portability problem with the compiler in the merged helper: a nested lambda captured a structured binding. The fix in one file merged on August 15 as PR #7449. It preserves result order and the public API.

August: the list gets shorter

The August list is shorter than the one I wrote in July. The scan collectives, payload carriers, hardening work, collective exception propagation, all_reduce seed move, MPI shutdown fix, parcelport fixes that use ports assigned by the OS (PR #7419 and PR #7449), queue wait time build fix, and pairwise all_to_all selected by size have merged. PR #7449 also added the collectives API documentation metadata. The last two open HPX items, PR #7449 and PR #7449, merged on August 15.

The remaining technical work is the attribution verification and a fresh, immutable Rostam campaign. The remaining handoff work is to open the Collective-Bench changes with the raw benchmark data, document the LCI hang cases and runtime flag footgun, and publish the plotting scripts with enough metadata that someone else can reproduce the comparisons. That work did not become an HPX PR before this report’s cutoff.

And the question from my March email has an answer, just not the one I expected. I asked whether the tree topology should be fixed or configurable. Four months later, the algorithm that actually mattered wasn’t chosen from a menu of topologies at all. The measurements forced it. The all_to_all based on topology is still on my wishlist, and when I get to it, it will be because the numbers say there’s still room for one.

A separate GPU question

The project started from my GPU background, and late July brought that thread back. I wrote a design for supporting GPU buffers in collectives and parcelports with two paths. P1 stages device buffers through bounded pinned host memory and works with every parcelport. P2 lets a qualified parcelport transfer device chunks directly, but only after a runtime transfer test proves that the actual nodes, GPUs, and transport can do it.

Then I built the smallest useful P1 prototype: a staged CUDA all_reduce using current HPX APIs. It compiled with CUDA 12.8 and passed with one locality, two localities on one V100 node, and two localities across two V100 nodes over TCP. Those are function tests, not performance results. The prototype still allocates pinned memory per call and leaves out the bounded pool, segmentation, global validation, cancellation, counters, HIP, and direct parcelport transfers.

Rostam also put a hard boundary around P2. The installed OpenMPI and UCX stack did not support CUDA buffers, nvidia_peermem was not loaded, and classic device registration failed. LCI could register A100 memory through DMA-BUF, but the first transfer failed with IBV_WC_LOC_PROT_ERR. Pinned A100 copies reached about 24 to 26 GB/s, enough to keep P1 worth pursuing, but not enough to call P2 supported. None of this code has merged. It is a later design and a working correctness prototype, not part of the completed collective work above.

What did not ship

The main algorithm thread has landed. Pairwise all_to_all with size dispatch merged in PR #7449, backed by the integrated branch run on 16 nodes above. PR #7449 removed the Clang 17 and OpenMP capture of a structured binding. PR #7412 updated the departed locality and launch process tests to use ports assigned by the OS, replaced inherited environment entries, and handled Windows environment names without case sensitivity. Both merged on August 15. At the status cutoff, no HPX code PR from this thread remains open.

The attribution thread has completed the branch inventory, benchmark coverage contract, and local producer/consumer redesign. Its active gate is now the last local verification pass: clang-format 20.1.7 and CMake formatting, HPX Inspect, diff/ASCII checks, final rebuilds with tracing enabled and disabled, the full discovered harness and distributed regressions, one final deciding composite trace, and a matched overhead measurement with tracing enabled and disabled, including the old stress case that flushed while holding the lock.

Only after that gate should the work return to Rostam. The next remote source and build must be new, immutable versions, not edits to the directories that produced earlier results. A fresh strict smoke has to pass HPX TCP, MPI, and LCI plus native MPI before the full allocation starts. The current generated smoke count is 143 cases, but it must be regenerated and reviewed from the final immutable manifest before submission.

The final campaign is sixteen nodes, normally one locality per node, over all supported hpx::collectives, all three parcelports, matched native MPI cases, and relevant payload and thread configurations. The requested arity sweep was 1, 2, and 4, but HPX rejects a hierarchical tree arity below 2. So arity 1 cannot be reported as a tree result: if it is meant as a flat or single level baseline, it needs a separate name. The valid tree comparison is arity 2 and 4, with arity 3 useful when an uneven grouping is needed. The campaign should expand in small batches, not as hundreds of jobs that all repeat the same hidden setup error.

Every accepted result needs a process status, raw samples, recomputed summary and p95, source and binary hashes, exact launch geometry, transport and module configuration, and strict trace evidence. Results with tracing enabled and disabled must stay separate so the observer’s overhead can be measured. The OpenMPI finalization crash must remain visible even when benchmark data completed before it. The merged shutdown fix in PR #7404 also needs a new run; a code change is not evidence that the historical cluster failure disappeared.

Only after that audit can the report say where the remaining latency lives with confidence. The current evidence points toward future suspension and task wakeup, especially for small messages and the LCI cases on four nodes. That is a question for the next investigation, not a scheduler patch hidden inside this one.

The benchmark harness and raw results also need their own handoff. Once the remaining campaign is complete, the Collective-Bench changes and data should be opened as a PR so the plotting scripts can be run independently. The final blog post should use only accepted results and keep the historical MPI configuration caveat attached to every comparison that depends on it.

Thanks

Alexander deleted work from my plan in May, rebuilt the benchmark so the numbers could be trusted, pointed me at usability in June when I wanted to build clever things, and shrugged off every scheduling mess on both sides. hkaiser reviews fast, thinks out loud in public, changes his mind in public, and sends you mdspan sketches at whatever hour he happens to be awake. Brett kept Rostam alive through a rough June, rebooting the login node at odd hours and adding monitoring so it would fail loudly next time, and still had patience for my reports. Thanks also to the rest of the Ste||ar folks in the channel and at the weekly meetings for taking a newcomer’s benchmark reports seriously. The main collective work is upstream; the attribution branch and GPU prototype remain later work.

For a broader look at all our community projects this summer, read the GSoC 2026 Wrap Up.

Emilios Tassios

About Emilios Tassios