Skip to content

Agent notes

Continuity notes for an AI agent picking this repo up. Read this before touching anything; it records the state, the non-negotiables, and the traps that have already cost real time.

Human-facing docs: README.md (catalog), CURRICULUM.md (module index and progress tracker), docs/MODULE_TEMPLATE.md (the content contract).

What this is

A GPU + ML teaching curriculum: 30 modules across four tracks, dual-vendor (AMD ROCm/HIP and NVIDIA CUDA) with a portable Triton path. Every module explains a concept twice - a plain-language analogy, then expert depth - with runnable labs, cited sources, and an honest tradeoffs section.

Current state (as of 2026-08-11)

Thing State
Modules 30 / 30 authored. A01-A10, B01-B07, C01-C07, D01-D06. All 9 sections, labs, exercises, solutions
Docs site Published, --strict, warning-free: https://gahan9.github.io/ParallelProgramming/
Cross-references All 30 modules linked to each other, to docs/GLOSSARY.md (65 ### terms) and to docs/REFERENCES.md (81 anchored sources). 2744 relative links across 223 files, all resolving. Contract lives in docs/MODULE_TEMPLATE.md
Read-aloud docs/javascripts/read-aloud.js + matching CSS, injected site-wide via extra_javascript. Browser speechSynthesis, no dependency. Skips code blocks and collapsed <details>; rewrites technical vocabulary before speaking
Site UX navigation.tabs + prune so a module page's sidebar shows only its own track, not all 30. docs/javascripts/panel-toggle.js hides the nav and TOC panels (persisted in localStorage); docs/javascripts/gfm-alerts.js upgrades GitHub > [!NOTE] blockquotes into Material admonitions, which keeps the source readable on both surfaces. Dark mode verified in a real browser, both schemes
Live pages live/ - study planner, 49 cited recall cards (with a Listen control), reference sheets, one concept player, 14-slide deck. Self-contained HTML
Reference sheets live/cheatsheets.html + live/data/reference.json - 4 track, 12 concept, 30 module and 30 analogy sheets, search + track filter, print CSS. Every bullet carries a cite; that is the contract, enforced by check_reference_data.py. The 30-lane rail is the signature and is load-bearing (filtering dims it). Data is fetched, so this page alone needs HTTP locally - it says so on screen
Analogies Every module teaches its concept twice. All 30 pairings live in reference.json under analogies (everyday phrasing, what it actually is, and where the module says the picture stops being true) and render as a tab. assets/analogy-bridge.svg is the ten-row summary. A01 and A02 state no break - a real gap against MODULE_TEMPLATE section 1, surfaced by the checker rather than hidden
Concept players live/concepts/ - interactive versions where a control teaches better than a drawing. coalescing.html is the pilot (stride + lockstep width). Bar for adding one: the geometry must genuinely change with the input, else it stays an SVG
Infographics assets/ - 17 hand-authored SVGs for what Mermaid cannot express, each with a brief: comment header. Five jobs now, not four: proportion, curve, sequence, simultaneity (two lanes busy at one instant - the data-flow assets) and identity. GIF is deliberately refused; the SVGs are already animated and scripts/export_visual.sh derives one into gitignored exports/ when an external surface truly needs a file
Content studio content/ - video/Shorts scripts. Packs exist for A01 and A02 only
Cursor canvas canvases/curriculum-dashboard.canvas.tsx (outside the repo, in the Cursor project dir)
Git main clean and pushed; every workflow run green

Layout

README.md CURRICULUM.md          catalog + module index/tracker
AGENTS.md                        this file
build_docs.py                    stages the publishable files into _docs/ (see Traps)
check_links.py                   resolves every relative link + anchor on both surfaces (Traps #8)
check_reference_data.py          reference.json: every bullet cited, every href + anchor resolving
check_live_js.py                 node --check on the inline script of every live/ page
build_reference_index.py         regenerates the "cited by" table in docs/REFERENCES.md
mkdocs.yml                       site config; nav lists all 30 modules
docs/                            SETUP, MODULE_TEMPLATE, BRAND, GLOSSARY, REFERENCES
docs/javascripts/                read-aloud, panel-toggle, gfm-alerts (all vanilla, no deps)
docs/stylesheets/brand.css       brand tokens + the Mermaid dark-mode pins (Traps #12)
tracks/A-gpu-programming/        A01-A10 (+ legacy 01/02/03 raw examples, superseded by A01)
tracks/B-gpu-ml-performance/     B01-B07
tracks/C-ml-system-design/       C01-C07 (design track: no GPU code, stdlib-only Python tools)
tracks/D-distributed-computing/  D01-D06 (below the ML layer: MPI, collectives, fabric, topology)
assets/                          SVG infographics + their system doc; sources/canva.md
live/                            dashboard, cheatsheets, flashcards, slides
live/data/reference.json         cited facts behind the reference sheets
live/concepts/                   interactive concept players (coalescing.html)
content/                         video scripts + brand overlay spec
outputs/                         sanitized lab transcripts (gfx950, ROCm 7) + manifest.json
scripts/                         collect_outputs.sh, sanitize_output.py, export_visual.sh
.cursor/skills/parallel-programming-brand/   brand skill (load when authoring visuals)

Non-negotiables

  1. Never publish a number the repo cannot reproduce. This is the spine of the whole project. Benchmark tables are omitted in favour of expected ordering and traffic ratios (which follow from the algorithm, not the silicon). Analytic tools print a banner marking output as a model, not a measurement. Published figures are attributed to their paper and its conditions. Hardware specs carry a source link and a retrieval date.
  2. Nine sections, exact headings, per docs/MODULE_TEMPLATE.md. Section 1 opens with a load-bearing layman analogy and says where the analogy breaks. Section 9 puts conceptual answers inside a <details markdown="1"><summary>Answers</summary> block - the attribute is required (Traps #11).
  3. Cite or hedge. Verify version-sensitive claims (hardware specs, tool flags, library APIs) with a web search and stamp the retrieval date. If you cannot verify it, say so in the text rather than asserting it. Never invent a citation, a DOI, or a benchmark.
  4. Dual-vendor. Where AMD and NVIDIA differ, show both (wavefront 64 vs warp 32). NVIDIA-only material is marked optional and names its AMD counterpart.
  5. Error checking in every GPU sample - HIP_CHECK / CUDA_CHECK on every runtime call, a launch-error check after every launch, bounds guards, and verification against a CPU reference.
  6. Brand tokens only, from docs/BRAND.md. Every Mermaid diagram starts with the init block from that file. Do not introduce new colours.
  7. Python tools in Tracks B and C are standard-library only - no numpy, no torch. Verify with python -S -I <tool>. They must exit non-zero with a one-line message on bad input, never a traceback.

Verify before you claim

# Links and anchors, both surfaces. Runs first in CI; a break fails the deploy
python check_links.py                          # relative links + anchors, offline, seconds
python check_links.py --external-only          # HTTP-checks every external URL, ~4 min
python build_reference_index.py --check        # fails if REFERENCES.md's index is stale

# Docs site (this is what CI runs; --strict means a warning fails the deploy)
python build_docs.py
python -m mkdocs build --clean --strict --site-dir ./site

# Inline JS in the live pages - a syntax error leaves a page silently dead, because
# nothing between the editor and the browser parses it
python check_live_js.py

# reference.json: every bullet cited, every href + anchor resolving, no dangling
# cluster or module ids. check_links.py cannot see inside JSON (Traps #14), so this is
# the only thing that catches a renamed heading breaking a cheatsheet citation.
python check_reference_data.py

# The reference sheets fetch their data, so exercise them over HTTP, not file://
python -m http.server 8000    # then open /live/cheatsheets.html and search for something

# Rendered layout: the checks above cannot see text running off a canvas or two labels
# landing on each other, both of which have shipped here before. Load each SVG in a
# headless browser and compare getBoundingClientRect against the viewBox. Chrome:
#   chrome --headless --disable-gpu --virtual-time-budget=15000 --dump-dom <harness url>
# Beware getBBox(): it ignores an enclosing transform, so it reports false collisions
# on any asset whose plot is translated. Use client rects.

# SVGs must be well-formed XML and pure ASCII
# (XmlDocument.LoadXml + a scan for [^\x09\x0A\x0D\x20-\x7E])

# Any file you wrote with typography: confirm it is really UTF-8 (see Traps #6)

# After pushing, confirm the deploy actually went green. Pushing is not shipping
$env:GITHUB_TOKEN = ""; gh run list --limit 1

Traps that have already cost time

  1. Auth. GITHUB_TOKEN is set in the environment to an AMD enterprise token (gsaraiya_amdeng) which has pull-only access to this repo, and it masks the gahan9 keyring account. Any git push or gh call fails with 403 / "Not Found" until you clear it: 
    $env:GITHUB_TOKEN = ""      # then gh resolves to gahan9 (admin, push)
  2. docs_dir cannot be . - MkDocs 1.6 hard-rejects a docs_dir that is the parent of mkdocs.yml, and rejects a site_dir nested inside it. The curriculum's Markdown must stay at the repo root so GitHub renders it with working relative links, so build_docs.py stages the publishable files into a gitignored _docs/. Always run it before mkdocs build.
  3. validation: uses the nested schema (links: / nav:). The flat pre-1.5 form is silently ignored, which once buried hundreds of real warnings under link noise.
  4. New asset types need two edits or they resolve on GitHub and 404 on the site: add the suffix to PUBLISH_SUFFIXES in build_docs.py, and add a path glob to .github/workflows/pages.yml (it only triggers on listed paths).
  5. PowerShell, not bash. No heredocs, no &&. For commit messages, write the message to a file with [System.IO.File]::WriteAllText(path, msg, (New-Object System.Text.UTF8Encoding $false)) then git commit -F. Set-Content -Encoding utf8 adds a BOM that ends up in the commit subject. Git operations here take 15-40s; budget for it.
  6. Writing non-ASCII can silently mangle it. Em dashes and arrows have landed as Windows-1252 or U+FFFD, which made six SVGs invalid XML and broke a --strict build on one Markdown file. After writing any file with typography, verify it decodes as UTF-8. SVGs are pure ASCII by policy - they use XML numeric entities (&#183;, &#8594;, &#215;). Keep them that way.
  7. Module data is duplicated in six places. Update all of them together: CURRICULUM.md (tables + totals), live/dashboard.html (M, PATHS), live/flashcards.html (CARDS), live/data/reference.json (modules, and clusters for a genuinely new concept), live/slides.html (headline counts), and the Cursor canvas. Each surface must stand alone with no shared runtime; six is still few enough to do by hand and a generator is not yet earned. Two things are deliberately not in that list and must stay derived: the reference sheets compute per-track module counts and hour totals from their own modules array, and the recall deck builds its track filter row from the cards that exist. Both were hand-written lists once, and both were silently wrong the moment Track D landed. If you find yourself typing a count, derive it instead.
  8. GitHub and MkDocs slugify headings differently, so a deep link can work on one surface and 404 on the other. ## 6. Challenges, Drawbacks & Tradeoffs is #6-challenges-drawbacks--tradeoffs on GitHub (the & leaves two hyphens) but ...-tradeoffs under stock Python-Markdown, which collapses runs. mkdocs.yml pins pymdownx.slugs.slugify to make the two agree - do not remove it, and run check_links.py (which models both) rather than trusting a rendered page.
  9. Link depth is the most common mistake. Three ../ from a module README to docs/, four from exercises/ or solutions/. GitHub silently renders a bad relative link as text; only the checker catches it.
  10. The reference index in docs/REFERENCES.md is generated, between <!-- BEGIN/END GENERATED: reference-index -->. Never hand-edit it; add the citation in the module and re-run build_reference_index.py.
  11. md_in_html needs markdown="1" on every raw HTML block, or the Markdown inside publishes as literal **bold** and backticks. This silently broke the Section 9 answers block in all 24 modules, and --strict does not catch it because it is valid HTML. Always write <details markdown="1">. GitHub ignores the attribute, so both surfaces stay correct.
  12. Never use MkDocs-only syntax for content GitHub must render. ???+ note "Title" with a 4-space-indented body looks right on the site but publishes on GitHub as a literal paragraph plus an indented code block - which silently turned the four syllabus tables into unclickable text. Use <details markdown="1"> + <summary>, which both surfaces honour. check_links.py cannot catch this: it reads raw Markdown, so it happily validates links GitHub will not render as links. Check a real GFM render (gh api -X POST /markdown) when adding new syntax.

  13. Do not re-theme Mermaid per diagram. Material rebinds Mermaid's label colour to --md-mermaid-label-fg-color, which it derives from --md-code-fg-color - tuned for dark code backgrounds, so in the slate scheme it resolves to a near-invisible grey on this curriculum's light classDef fills. docs/stylesheets/brand.css pins the --md-mermaid-* variables (flowchart and sequence) inside .mermaid. Fix contrast there, not in the diagrams.

  14. check_links.py does not read JSON or JavaScript. The hrefs inside live/data/reference.json are invisible to it, so a renamed heading breaks them silently. After touching that file or a section heading it cites, re-run the resolver in "Verify before you claim" below. The same blind spot applies to any URL built in a <script>.

  15. One key, one meaning, in shared data. reference.json briefly used modules for a count on a track and an array on a cluster. Searching then threw inside the render, and the page kept showing stale results instead of failing visibly - the worst kind of bug, because it looks like it works. The count is now derived and the reader guards with Array.isArray. Whenever you add a field, check no sibling shape already uses that name for something else.
  16. An SVG that validates can still be unreadable. Well-formed XML and pure ASCII say nothing about whether the text fits. Three shipped assets rendered their axis caption and top tick underneath the headline, and two more clipped a caption off the right edge - none of it visible in the source, because the overlap only exists once a font is applied. Measure in a real browser. And use getBoundingClientRect, not getBBox: getBBox reports coordinates in the element's own user space and ignores an enclosing transform, so every asset whose plot is translated reports collisions that are not there.

Both surfaces matter equally: the curriculum is read on GitHub and on the docs site, and Traps #8,

11 and #12 are all the same failure - a change verified on one surface only.

Do not claim this is verified

No GPU code in this repo has ever been compiled or run. There is no ROCm or CUDA toolchain on this machine. Every .cpp, .cu, and Triton kernel is hand-reviewed only. The Python tools and the docs build are verified. Be explicit about that boundary in any summary.

Track D sharpens the boundary rather than moving it, and is worth stating precisely: - D02's MPI C code is the one exception that was actually compiled. All ten .c/.h files build warning-free under -Wall -Wextra against genuine Open MPI 5.0.10 headers via a zig cc cross-compile. Compiled, not run: there is no mpirun here, so the deadlock lab's runtime behaviour is reasoned, not observed. - D03's RCCL/NCCL and D04's peer-copy sources are hand-reviewed only, like the rest. They were checked mechanically for the things a reviewer looks for (no unchecked runtime call, a GetLastError after every launch, a bounds guard in every kernel, a free for every allocation) but never compiled. - Every Python tool in Track D was executed, under python -S -I, with the exact invocations its module documents plus bad-input cases. That is real verification and can be claimed. - No multi-node measurement of any kind exists. Every inter-node figure is an analytic model and says so in its own output banner.

Known likely first-build friction: - HIP's warpSize stopped being constexpr in ROCm 7.0, and __AMDGCN_WAVEFRONT_SIZE__ is deprecated (A04 documents this). - B04's optional native-fp8 cross-check is behind __has_include with a -DB04_NO_NATIVE_FP8 escape hatch; it is the most likely place a build breaks. - D03/D04 link against RCCL or NCCL, which the other GPU modules do not; a missing -lrccl or a header path is the likely first failure there. - Makefiles are Linux-targeted (rm -rf); make is not installed here, so no recipe has been run.

Domain facts worth not re-deriving

  • rocprofv3 now defaults to rocpd SQLite, not CSV. Every profile-hip target passes --output-format csv explicitly for this reason.
  • CDNA 3 (gfx942) implements FNUZ fp8 (bias 8, max 240); Hopper implements OCP (bias 7, max 448). The same bits decode 2x apart, and 0x80 is NaN under FNUZ but -0.0 under OCP.
  • AMD's own CDNA 3 whitepaper (17.2 TB/s) and Hot Chips 2024 slides (14.7 TB/s) disagree on Infinity Cache bandwidth. B01 reports both rather than picking one; do not "fix" this.
  • Omniperf/Omnitrace were renamed in ROCm 6.3 (rocprof-compute, rocprof-sys-*).
  • 24 of the 310 external URLs cannot be machine-checked and this is not a defect. ACM, doi.org, MIT Press and the OCP spec site answer any scripted request with 403 no matter how the request is dressed; tensorflow.org sends browser agents around a locale-redirect loop. check_links.py reports these as "behind bot protection" and passes. Do not "fix" them by swapping in a mirror.

Open work, roughly by value

  1. Build-and-run pass on a real GPU. The highest-value next step: it converts every expected-shape table into a measured one and unblocks item 2.
  2. Content packs for A04, A06, A07, B02 - each needs a real measured number to lead with, which is why they are blocked on item 1. content/README.md now names the lead visual and the exact lab output each is waiting on, so a pack is a script plus a number, nothing else.
  3. A01 and A02 need a "where the analogy breaks" section. MODULE_TEMPLATE section 1 requires one and the other 28 modules have it; these two do not. A01's stadium and A02's warehouse are the two analogies a beginner meets first, so they are the worst two to leave unbounded. check_reference_data.py prints a note until they land.
  4. Recall cards for Track D. The deck is 49 cards over Tracks A, B and C; Track D was added after it was built and has none. The reference sheets link to flashcards.html?track=D and the deck says plainly that the track is empty, so the gap is visible rather than silent. Every card needs a real source.
  5. Two more concept players. live/concepts/coalescing.html proved the pattern. The ring all-reduce phase stepper (D03) and the roofline intensity slider (B02) are the two where a control genuinely beats the static asset. Same bar: the geometry has to change with the input.
  6. Bump Node 20 actions. CI warns that checkout@v4, configure-pages@v5, setup-python@v5, and upload-artifact@v4 are being forced onto Node 24. Nothing is broken; it is upkeep.
  7. Legacy cleanup. tracks/A-gpu-programming/01.hello_world/Makefile still calls bare rocprofv3 instead of $(ROCPROFV3), so it fails when ROCm is not on PATH - the bug fixed everywhere else. Kept as raw reference material, superseded by A01.

Session log

Rough effort record, kept because the operator asks for usage tracking.

Date Work
2026-07-18 Scaffold, shared docs, Module A01 as the gold standard; legacy folders migrated under Track A; brand system
2026-07-19 A02; A01 depth pass
2026-07-29 Modules A03-A10, B01-B07, C01-C07 authored via ~19 parallel subagents (~68k lines); content studio; canvas planner; wire-up. One subagent died of resource exhaustion at 19 concurrent - keep waves smaller
2026-08-02 Fixed the Pages deploy: enabled Pages (never switched on - the real cause of the 404), staged docs root, repaired validation:, turned on --strict. First successful publish
2026-08-04 assets/ infographic system (6 SVGs) + encoding repairs; live/flashcards.html (49 cited cards, three-box Leitner) and shared nav across the three live pages
2026-08-05 Read-aloud across the docs site and the recall cards. Chose browser speechSynthesis over pre-generated audio for the same reason SVG beat GIF: no unreviewable binaries, nothing to regenerate on a content edit
2026-08-09 Reconciled the two 2026-08-08 sessions and pushed both. The Pages deploy had been red for a day: splitting a half-finished tree into commits left docs/REFERENCES.md's generated index describing citations that were not in the same commit, and build_reference_index.py --check failed the build exactly as designed - but nobody looked at the run. Landing the rest of the rollout fixed it
2026-08-08 Docs-site UX pass in a parallel session: found that md_in_html had been silently publishing all 24 Section 9 answer blocks as raw Markdown (Traps #11); pinned the Mermaid dark-mode variables (Traps #12); added navigation.tabs/prune, the nav+TOC toggles and the GFM-alert upgrader. Its ??? note collapsibles then broke the syllabus tables on GitHub, migrated to <details markdown="1">
2026-08-10 Output cache on gfx950/ROCm 7: scripts/collect_outputs.sh + sanitize_output.py, 46 sanitized artifacts under outputs/, SETUP §6 + MODULE_TEMPLATE hook; A01/A02 exercise headers refreshed
2026-08-26 Analogy surface + animated data flow. Three assets: analogy-bridge.svg (ten everyday images against what they actually are, the two worlds separated by typeface rather than colour), dataflow-training-step.svg (B07's gradient all-reduce hiding under the backward pass, and the tail that structurally cannot hide) and dataflow-decode-loop.svg (B05's whole model dragged across the bus once per token). Simultaneity added as a fifth SVG job: a flowchart has no time axis and a bar chart has no second lane. All 30 module analogies extracted verbatim into reference.json with the point each module says its own breaks, rendering as a fourth tab. Found that A01 and A02 state no break at all, which MODULE_TEMPLATE section 1 requires - the card and the checker both say so rather than hiding it. Two verification steps that AGENTS.md documented as bash heredocs never ran on this repo's PowerShell, so they are now check_reference_data.py and check_live_js.py. Browser-tool access was gone this session; headless Chrome with --dump-dom replaced it, and caught two footers overflowing the canvas
2026-08-24 Visual learning system. Four new infographics closing the candidate list (reduction-tree A04, tile-reuse A06, attention-traffic A09, prefill-decode-split B05), each with an inline brief: header instead of a sidecar file; ring-allreduce.svg finally linked from B07. New reference-sheet surface: live/cheatsheets.html over a cited live/data/reference.json (4 track + 12 concept + 30 module sheets, search, track filter, print CSS, 30-lane rail). First concept player, live/concepts/coalescing.html. Two real bugs found by testing in a browser rather than by reading: reference.json used modules for both a count and an array, which threw inside the render and left stale results on screen (Traps #15); the recall deck's track filter was a hand-written A/B/C list, so Track D could never appear - both are now derived. Track D's #DB2777 added to BRAND.md and the brand skill, where it had been missing since the track landed
2026-08-11 Track D added (D01-D06, ~24k lines). Distributed computing below the ML layer, since B07/C03 already own sharding and fault tolerance: Amdahl/Gustafson/Hockney alpha-beta, MPI, NCCL/RCCL, RDMA and the fabric tiers, topology, and a scale-boundary capstone. 20 new sources (every DOI web-verified against the publisher before it was written), 17 glossary terms, 4 new animated SVGs, fourth track colour #DB2777. Authored D01 by hand as the gold standard, then five parallel subagents for D02-D06 and six more for the labs - one died on a network stall and was simply relaunched. Two subagents caught real arithmetic errors in prose they were implementing against (D01's m* crossover claim was backwards; D05's block-mapping rail claim was wrong), which is the argument for making every table reproducible by a tool rather than asserting it