Network-swarm generation implementation notes

Documentation index · Generation guide · Generation methods · Pinned network-swarm data

Outcome and claim boundary

Stage 4.4 is implemented as a reproducible, projection-aware generate-network-swarm pass. It detiles cumulative GeoJSON swarm features, preserves every raw properties.downloaders field, and renders one network-swarm atlas on each of the six production projections. The result is a density and access-characteristics map. It is not a graph: the input contains no source/destination pairs, routes, peer relationships, or measured edges, so the renderer does not invent them.

The confirmed design uses H3 parent cells to discover local groups and Izzi’s radial hexagon fill to make dense places legible. Clustering changes only display positions. Optional thin tethers point back to each feature’s true projected coordinate and must not be interpreted as traffic paths.

Feasibility evaluation and source decision

The supplied cumulative aggregate is a good fit for this pass because every feature has a geographic Point, a valid H3 index, a common H3 resolution, and the same downloader object. It is already spatially aggregated and requires no address-level plotting. The implementation pins the corrected source:

The source’s top-level swarm_features_size is retained verbatim as reported metadata, but it is not relabeled as the sum of downloaders.size; those are different values with different source semantics. H3 identifiers are parsed as unsigned 64-bit integers and serialized both as canonical H3 strings and decimal strings. They are never passed through an IEEE-754 JSON number.

The nine specialized fields overlap. In this snapshot, 210 features have a specialized-field sum greater than size. Consequently mobile, satellite, tor, tor_exit_nodes, vpn, relay, proxy, hosting, and service are independent observations, not pieces of a partition or a stacked total.

Variable input and bounded preparation

The committed ZIP is the offline default. scripts/prepare-network-swarm-data.sh accepts one local .zip, .geojson, or .json file, so another compatible source does not require a code change. For ZIP input it requires exactly one flat, safely named JSON member, validates the archive CRC, and caps expanded content at 64 MiB. Preparation is atomic and does not change the destination timestamp when the bytes are already current.

make prepare-network-swarm-data
make generate-network-swarm

Override the source or the prepared destination explicitly:

make NETWORK_SWARM_SOURCE=/absolute/path/cumulative.geojson.zip \
  prepare-network-swarm-data generate-network-swarm

NETWORK_SWARM_SOURCE may point directly to an already uncompressed .geojson or .json; the same bounded staging and content comparison still apply. NETWORK_SWARM_GEOJSON changes the staging destination rather than bypassing preparation.

The parser intentionally requires the implemented cumulative swarm contract: a FeatureCollection, hexagon partition metadata, a uniform valid H3 resolution, unique H3 cells, Point coordinates in range, and unsigned values for all ten downloader fields. Missing, malformed, duplicate, or inconsistent data stops generation rather than silently becoming zero. Stage 13 increases the marker radius from 0.026 to 0.042 inch; glyph internals and outlines were enlarged with it rather than merely increasing collision spacing.

Configuration profile

network-swarm-profile.json owns the decisions that may reasonably change between captures:

• source and parent H3 resolutions;
• marker radius (0.042 inch) and tether threshold in physical inches;
• label budget, nonzero opacity floor, and tether visibility;
• fixed per-field scale references; and
• archive/member names, both digests, repository commit, and license.

The default groups H3 resolution-5 cells under resolution-3 parents. The raw snapshot has 4,404 such parents, a median of two features per parent, a 90th-percentile group size of 14, and a largest group of 48. Resolution 4 mostly produces singletons, while resolution 2 creates groups as large as 266; resolution 3 was selected as the useful compromise.

Fixed scale references are snapshot nonzero 99th percentiles. A value v and reference q produce

u = clamp(log(1 + v) / log(1 + q), 0, 1)
opacity(v) = 0                         when v = 0
opacity(v) = m + (1 - m)u             otherwise

where the default nonzero floor m is 0.18. Fixed references make maps and future compatible snapshots visually comparable; one extreme city cannot rescale every other mark. Raw counts remain attached to SVG elements, so the perceptual transform does not replace source data.

H3 and projection-safe honeycomb placement

Clustering follows this deterministic sequence:

1. validate every source cell and verify its H3 resolution;
2. compute cellToParent(source, 3);
3. split each parent group by the selected projection’s native face/cell;
4. project all true coordinates and average each component in page space;
5. sort component members by descending downloaders.size, then H3 ID;
6. request center-filled positions from Izzi’s radiate_hexagon_honeycomb() and put the largest value at the center; and
7. shift an entire cluster just enough to keep its marks inside the frame.

Step 3 is essential for unfolded maps. Two geographically local cells can land on opposite sides of a projection cut; placing them in one page-space honeycomb would create a false cross-seam group. On Cahill-Keyes, the 4,404 raw parents become 4,418 projection-safe components.

Izzi’s routine walks neighbors with basis directions derived from (2r, 0) and (r, sqrt(3)r). Its current floating-point hash and epsilon equality policies can admit numerically equivalent centers through different BFS paths. The adapter requests a bounded surplus, canonicalizes returned coordinates onto the routine’s own r/sqrt(3)r lattice, rejects non-finite values, verifies uniqueness, and retains the nearest required centers. This integrates the shared radial-fill algorithm without allowing a duplicate display cell to hide a source feature.

Visual system and semantic layers

The visual research suggested a restrained atlas rather than a simulated network diagram. Arjen van Susteren’s Metropolitan World Atlas informed the consistent framed comparison and standard mark grammar. Lucille Tenazas’s MNL–SF–MINY–SF2–ROME–NY2 work informed the limited accent palette and structured date band. Stage 13 moves the plate to a WCAG-oriented light field so dense marks retain contrast in print and on ordinary displays. The concrete downloader glyph vocabulary begins with Alpha60’s augment_swarm_features_geojson prior art and extends it to all current object fields.

All projections contain the same discoverable layer contract:

network-swarm-background
terrestrial-land
cluster-tethers
downloaders-total
access
  downloaders-mobile
  downloaders-satellite
infrastructure
  downloaders-hosting
  downloaders-service
privacy-routing
  downloaders-vpn
  downloaders-tor
  downloaders-tor-exit-nodes
  downloaders-relay
  downloaders-proxy
labels
legend-and-provenance

The opaque #f2f4f3 ocean and subdued light-gray Natural Earth land preserve geographic context without competing with the darker marks. The main title is twice its former size (0.44 rather than 0.22 page units); metadata records the background, 0.042-inch marker radius, and data-title-scale="2". Labels are a deterministic, collision-grid bounded selection from the largest totals. Every base feature stores its H3 IDs, parent, projection cell, cluster size, country, city, GeoNames ID, and all ten raw downloader counts. Root metadata stores source digests, source commit, interval, data version, counts, cluster statistics, and projection.

Products

make generate-network-swarm writes the six layered SVGs. make generate-network-swarm-artifacts adds the matching PDF and 3840-pixel-long-side PNG files. Each PNG below links to the generated review artifact:

Per-projection targets use generate-network-swarm-PROJECTION. The canonical project-generation profile name is network-swarm; the former network name and short swarm name remain input aliases.

Verification and limitations

make check verifies both source digests, exact snapshot metadata and totals, overlapping category behavior, H3 parent statistics, one-to-one honeycomb positions for cluster sizes 1 through 48, and bounded finite layouts on all six projections. Every generator also reopens its SVG and checks the viewBox, complete layer vocabulary, provenance metadata, exact base feature count, and absence of non-finite output.

The atlas remains subject to source semantics and collection bias. A cumulative interval does not show simultaneity, rate, direction, bandwidth, or causation. City labels and H3 centroids are aggregate annotations, not precise endpoint locations. Specialized classifications can overlap, and their source methodology is not re-derived by this renderer. Comparisons across future snapshots are meaningful only when collection coverage, duration, minimum threshold, H3 resolution, and classification methods are compatible.

Future work worth investigating includes a time-aligned snapshot series, uncertainty and missing-coverage fields, configurable privacy suppression, and genuine origin/destination or autonomous-system flows when an input with explicit edge semantics is available. Those would be new data contracts, not inferred extensions of this cumulative Point collection.