Browser Harness

browser-harness

Direct browser control via CDP. For task-specific edits, use agent-workspace/agent_helpers.py. For setup, install, or connection problems, read https://github.com/browser-use/browser-harness/blob/main/install.md.

Domain skills are off by default. Set BH_DOMAIN_SKILLS=1 to enable them; see the bottom section.

If `BH_DOMAIN_SKILLS=1` and the task is site-specific, read every file in the matching `$BH_AGENT_WORKSPACE/domain-skills/<site>/` directory before inventing an approach.

Usage

browser-harness <<'PY'
print(page_info())
PY

• Invoke as browser-harness. Use heredocs for multi-line commands.
• Helpers are pre-imported. run.py calls ensure_daemon() before exec.
• First navigation is new_tab(url), not goto_url(url).
• The normal local flow attaches to the running Chrome/Chromium CDP endpoint. No browser ids or local profile selection.

Local Chrome

If the daemon cannot connect, run diagnostics:

browser-harness --doctor

If Chrome remote debugging is not enabled, the harness opens:

chrome://inspect/#remote-debugging

Ask the user to tick "Allow remote debugging for this browser instance" and click Allow if Chrome shows a permission popup. Then retry the same browser-harness command.

Remote Browsers

Use Browser Use cloud for headless servers, parallel sub-agents, or isolated work. Authenticate once:

browser-harness auth login

Or import a key safely:

browser-harness auth login --api-key-stdin

Pick a short made-up name; r7k2 below is just a placeholder:

browser-harness <<'PY'
start_remote_daemon("r7k2")
PY

BU_NAME=r7k2 browser-harness <<'PY'
new_tab("https://example.com")
print(page_info())
PY

When the task is done and a cloud browser is still running, ask directly: "Should I close this browser now?" If yes, run stop_remote_daemon(name). Remote daemons bill until they stop or time out.

Do not start a remote daemon and then keep using the default daemon. Use the same name for BU_NAME.

Cloud profile cookie sync reference: https://github.com/browser-use/browser-harness/blob/main/interaction-skills/profile-sync.md.

Page Workflow

• Screenshots first: use capture_screenshot() to understand visible state.
• Clicking: screenshot -> read pixel -> click_at_xy(x, y) -> screenshot again.
• After navigation, call wait_for_load().
• If the current tab is stale or internal, call ensure_real_tab().
• Use js(...) for DOM inspection or extraction when coordinates are the wrong tool.
• Login walls: stop and ask. Exception: use available SSO automatically when Chrome is already signed in; still stop for passwords, MFA, consent, or ambiguous account choice.
• Raw CDP is available with cdp("Domain.method", ...).

Interaction Skills

If you get stuck on a browser mechanic, check https://github.com/browser-use/browser-harness/tree/main/interaction-skills.

• connection.md
• cookies.md
• cross-origin-iframes.md
• dialogs.md
• downloads.md
• drag-and-drop.md
• dropdowns.md
• iframes.md
• network-requests.md
• print-as-pdf.md
• profile-sync.md
• screenshots.md
• scrolling.md
• shadow-dom.md
• tabs.md
• uploads.md
• viewport.md

Design Constraints

• Coordinate clicks default. CDP mouse events pass through iframes/shadow/cross-origin at the compositor level.
• Keep the connection model simple: use the default daemon, BU_NAME, BU_CDP_URL, BU_CDP_WS, or start_remote_daemon(...).
• Core helpers stay short. Put task-specific helper additions in $BH_AGENT_WORKSPACE/agent_helpers.py.

Gotchas

• chrome://inspect/#remote-debugging must be enabled for local Chrome control.
• Chrome may show an "Allow remote debugging?" popup; wait for the user to click Allow.
• Omnibox popups are not real work tabs.
• CDP target order is not Chrome's visible tab-strip order.
• BU_CDP_URL is an HTTP DevTools endpoint; the daemon resolves it to WebSocket.
• Ask before leaving cloud browsers running; stop them with stop_remote_daemon(name) or PATCH /browsers/{id} {"action":"stop"}.

Domain Skills

Only applies when BH_DOMAIN_SKILLS=1. Otherwise ignore domain skills.

When enabled, search $BH_AGENT_WORKSPACE/domain-skills/<host>/ before inventing an approach. goto_url(...) returns up to 10 skill filenames for the navigated host.

American Airlines — booking checkout (aa.com)

End-to-end guest checkout for a one-way revenue fare, through to the credit-card entry form. No antibot / CAPTCHA encountered on a clean, stock-Chrome CDP connection.

URL map (in order)

<slices> payload (URL-encoded):

[{"orig":"DFW","origNearby":false,"dest":"AUS","destNearby":false,"date":"2026-05-06"}]

The deep link skips the home-page React form entirely. The in-page search form is React-controlled; plain input.value = … does not propagate to the controlled state, so prefer the deep link.

Stable selectors & handles

Search results (/booking/choose-flights/1)

• button#flight-<N>-product-group-<CABIN> — top-level cabin card (e.g. flight-0-product-group-MAIN). Clicking this expands the fare tray in place, it does not navigate. Use .click() via js(...); the coordinate click has a tendency to scroll past the target.
• button#slice-<N>-MAIN-basic-economy / slice-<N>-MAIN-coach / -coach-plus / -coach-select / -first — the real "Select this fare" buttons inside the expanded tray. Visible only after the product-group button is clicked.
• button#carousel-<YYYY-MM-DD> — date carousel navigation.

Basic-Economy upgrade-upsell modal

After clicking slice-0-MAIN-basic-economy, an upsell dialog appears. The decline button is:

• button#btn-no-upgrade — text is "Accept restrictions" (not "No, thanks"). Click this to proceed with Basic Economy.

Trip summary

• button#login-continue-btn — "Log in and continue"
• button#continue-as-guest-btn — "Continue as guest" (use this)

Passenger details (/airfare-sales/ui/passenger-ui/)

This is a separate Angular app using custom elements (<adc-text-input>, <adc-select>, <app-passenger-page>, etc.) with open shadow roots. Inputs are addressed by formcontrolname on the host element; the real <input> / <select> is inside host.shadowRoot.

Top-level card:

• adc-button#paxCardButton0 — "Enter new passenger". Clicking it opens a <mat-dialog-container> with the passenger form.
• After save, it re-renders as "Edit, saved passenger information for, First Last".

Passenger form (hosts, inside the modal):

Modal buttons (both <adc-button>, no stable id — filter by inner text):

• "Cancel"
• "Save" — commits the passenger and closes the modal.

Contact form (appears on the main passenger page after at least one passenger is saved, NOT inside the modal):

Main Continue: <adc-button> with className containing save-button and innerText Continue. There is a second Continue ("Log in and continue") on the page — filter it out.

Seat map

• a#continueWithoutSeatsLink — "Skip seats for all flights" (use this to skip)
• button#nextFlightButton — "Continue" (only if seats are selected)

Trip extras (ancillaries)

• Single <adc-button> with text "Continue" — just click through.

Checkout / payment (/ecommerce/checkout-app/cart/<cart-id>)

The checkout page is plain HTML, no shadow DOM, no iframes for card fields. Fields are directly addressable by id. Sections are progressively disclosed.

1. Trip insurance section — two radios with name="allianz-insurance-selections":

• value="purchase" / value="decline" — pick decline, then click button#trip-insurance-continue-button.

2. Payment method radios, name="paymentMethod":

• value="CREDIT_CARD", AFFIRM, APPLE_PAY, GOOGLE_PAY, PAYPAL, HOLD.
• Selecting CREDIT_CARD expands the card form inline.

3. Credit card fields (all plain <input> / <select>):

• Submit: button with text "Pay now" near the bottom of the page.
• "Secure checkout" lock icon sits next to the Pay now button.

Fill recipe (shadow-piercing + native setter)

For both the passenger-details shadow inputs and the checkout inline inputs, React/Angular will ignore assignments via the prototype's value setter unless you dispatch the right composed events. Minimal pattern that works on both:

function nativeSet(el, v) {
  const proto = Object.getPrototypeOf(el);
  Object.getOwnPropertyDescriptor(proto, 'value').set.call(el, v);
  el.dispatchEvent(new Event('input',  {bubbles: true, composed: true}));
  el.dispatchEvent(new Event('change', {bubbles: true, composed: true}));
  el.dispatchEvent(new Event('blur',   {bubbles: true, composed: true}));
}

// Shadow-piercing helper for <adc-text-input> / <adc-select>:
function setHost(hostId, val) {
  const h = document.getElementById(hostId);
  const inner = h.shadowRoot.querySelector('input, select');
  inner.focus();
  nativeSet(inner, val);
}

For the PAN / CVV specifically, prefer the harness's type_text() (CDP keystrokes) — the checkout page's card-number tokenizer is happy with either, but keystrokes are a safer default if AA ever switches to a Spreedly/CyberSource iframe.

Traps

• React search form on the homepage is controlled state. Setting input.value = … then dispatching input/change on the native input does not take — the submitted URL has empty orig/dest. Use the deep-link URL above instead.
• Basic-Economy upsell. The decline button literally says "Accept restrictions" (id btn-no-upgrade). Easy to misread as an affirm-upgrade.
• The "Continue" on the passenger page has two meanings. Before you save a passenger, clicking the page-level save-button adc-button with text "Continue" silently opens the passenger modal instead of advancing. Save the passenger first (modal Save button), then hit the page-level Continue.
• `state` select clears when `country` is rewritten. Set country first, wait a tick, then set state. The Angular form control goes ng-invalid otherwise.
• `gender` values are single letters (M, F, U, X), not MALE/FEMALE. The a11y label shows "MaleFemaleUnspecifiedUndisclosed" concatenated, which is misleading.
• Formcontrolname `firstName` exists twice in the DOM — once on the passenger modal (<adc-text-input id="firstName">) and once on the checkout payment form (<input id="firstNameInput">). Target by the specific id to avoid collisions.
• Viewport emulation resets across `Emulation.setDeviceMetricsOverride` boundaries after navigation. Re-apply setDeviceMetricsOverride if the later page's page_info().w jumps back up.
• Tab title is prefixed with the harness's `🟢 ` marker on each real page, so page_info().title will start with that emoji — don't treat it as site content.

Waits

• wait_for_load(timeout=20) plus a time.sleep(3-5) after every page transition. The Angular apps (passenger-ui, ancillaries, ecommerce) hydrate lazily and load fires before the forms mount.
• After document.getElementById('slice-0-MAIN-basic-economy').click(), wait 2-3s for the upsell <mat-dialog-container> to mount before trying to query #btn-no-upgrade.
• After selecting a payment-method radio, wait ~3s for the credit-card subsection to expand (the cvvInput only appears after the first couple of fields hydrate).

Antibot posture (observed)

• No Akamai _abck / bm_sz challenge on the booking path with a vanilla user's Chrome.
• No PerimeterX. No interstitial. No CAPTCHA on search, fare-select, passenger, or checkout.
• Payment page is a first-party form — no Stripe/Braintree/Spreedly iframe on the CC fields at this stage. (Tokenization presumably happens on Pay-now submit; we did not submit.)

AgentList Discovery

https://agentlist.com — public directory of skills, agents, MCPs, configs, and paid services. Field-tested with browser-harness on 2026-05-03.

Do This First

Use the public API for read-only discovery. It is faster and returns full listing content without browser UI parsing.

import json

listings = json.loads(http_get("https://agentlist.com/api/listings?category=skill&q=github&limit=10"))
for item in listings:
    print(item["title"], item["id"], item["vote_count"])

The web UI is useful for visual verification, voting, signing in with passkeys, and submitting listings.

Public API

All read endpoints below worked without authentication.

import json

# Search everything
items = json.loads(http_get("https://agentlist.com/api/listings?q=github&limit=10"))

# Filter by category
skills = json.loads(http_get("https://agentlist.com/api/listings?category=skill&q=browser&limit=10"))
agents = json.loads(http_get("https://agentlist.com/api/listings?category=agent&limit=10"))
mcps = json.loads(http_get("https://agentlist.com/api/listings?category=mcp&limit=10"))
configs = json.loads(http_get("https://agentlist.com/api/listings?category=config&limit=10"))
paid = json.loads(http_get("https://agentlist.com/api/listings?category=paid&limit=10"))

# Sort and paginate
new_items = json.loads(http_get("https://agentlist.com/api/listings?sort=new&skip=0&limit=20"))
top_items = json.loads(http_get("https://agentlist.com/api/listings?sort=top&limit=20"))
trending_items = json.loads(http_get("https://agentlist.com/api/listings?sort=trending&limit=20"))

# Fetch one listing
listing = json.loads(http_get("https://agentlist.com/api/listings/6ea8f4f2-83cf-4625-a3cd-98b49d49a7b2"))

Useful fields seen on listing objects:

• id, category, title, description, content
• author_pubkey, vote_count, fetch_count, viewer_has_voted
• created_at, updated_at, reviewed_at, reviewed_by
• MCP/config/paid-specific fields may be present: repo_url, api_spec_url, transport, package, tools, target_tool, config_format, filename_hint, api_base_url, pricing_info, payment_method

Raw Skill Content

Use /raw/{id} when you only need the agent-readable content, not metadata.

skill_md = http_get("https://agentlist.com/raw/6ea8f4f2-83cf-4625-a3cd-98b49d49a7b2")

For skill-folder loaders, the hosted skills.agentlist.com path also works:

skill_md = http_get("https://skills.agentlist.com/skill/6ea8f4f2-83cf-4625-a3cd-98b49d49a7b2/SKILL.md")

Detail pages display a "Load in your local or cloud based agent" box with the folder URL:

https://skills.agentlist.com/skill/{id}/

Browser Navigation

new_tab("https://agentlist.com")
wait_for_load()
wait(1)
print(page_info())

The homepage is server-rendered enough to read immediately after load. It contains:

• Category buttons: Skills, Agents, MCPs, Configs, Paid For
• Sort controls: Top, New, Trending
• Search input: input[type=search]
• Main listing table with headers #, Title, Author, Date, Votes

Extract visible rows:

import json

rows = json.loads(js(r"""
JSON.stringify(Array.from(document.querySelectorAll("table:first-of-type tbody tr")).map(tr => {
  const cells = Array.from(tr.children).map(td => td.innerText.trim());
  return {
    rank: cells[0],
    title_description: cells[1],
    author: cells[2],
    date: cells[3],
    votes: cells[4]
  };
}))
"""))

Listing title links use /listing/{uuid}:

links = json.loads(js(r"""
JSON.stringify(Array.from(document.querySelectorAll('a[href^="/listing/"], a[href*="/listing/"]')).map(a => ({
  text: a.innerText.trim(),
  href: a.href
})))
"""))

UI Search Gotcha

fill_input("input[type=search]", "github") double-entered characters during testing (github became ggiitthhuubb). Prefer direct JS value assignment plus an input event, or use the API.

js("""
const input = document.querySelector('input[type=search]');
input.value = 'github';
input.dispatchEvent(new Event('input', {bubbles: true}));
""")
wait(1)
print(page_info())  # URL becomes https://agentlist.com/?q=github

Listing Detail Pages

Detail URL pattern:

https://agentlist.com/listing/{id}

On skill detail pages, the rendered content includes raw code blocks, API notes, discussion, and a load URL. Detail pages also keep the homepage table lower on the page, so use scoped selectors when extracting the detail body.

new_tab("https://agentlist.com/listing/6ea8f4f2-83cf-4625-a3cd-98b49d49a7b2")
wait_for_load()
wait(1)

data = json.loads(js(r"""
JSON.stringify({
  title: document.querySelector(".listing-title, h2")?.innerText || null,
  load_urls: Array.from(document.querySelectorAll("code")).map(e => e.innerText.trim()).filter(t => t.includes("skills.agentlist.com/skill/")),
  code_blocks: Array.from(document.querySelectorAll("pre, code")).map(e => e.innerText.slice(0, 300)).slice(0, 20)
})
"""))

The top-right user pill and Sign out button can appear when already authenticated. Do not assume a logged-out session.

Submit Page

Submit URL:

https://agentlist.com/submit

Observed fields:

• #title — title input
• #description — short description input
• input[type=url] — GitHub import URL
• first textarea — Markdown content
• #scripts_index_html — optional JavaScript skill payload

Observed category buttons:

buttons = json.loads(js(r"""
JSON.stringify(Array.from(document.querySelectorAll("button.category-btn")).map(b => ({
  text: b.innerText.trim(),
  selected: b.classList.contains("selected")
})))
"""))

The GitHub import button is button.btn.btn-secondary with text Import. The final submit button has text Submit listing. Submissions are signed with the user's Nostr identity; do not submit or vote unless the user explicitly asked for that exact action.

alaska — guest checkout to card-entry

Drives www.alaskaair.com from search through the credit-card entry form as a guest. No login needed. Reaches a filled payment form; stop before Book now.

URL patterns

• Results deep link (cash mode once toggled): https://www.alaskaair.com/search/results?O={orig}&D={dest}&OD={YYYY-MM-DD}&A=1&C=0&L=0&RT=false. The site sometimes lands in award/points mode. Flip via the Money/Points toggle (see selectors).
• Cart: https://www.alaskaair.com/search/cart?... — arrived via Add to cart from results.
• Guest info: https://www.alaskaair.com/book/guest-info — arrived via Continue as guest from cart.
• Seat selection: https://www.alaskaair.com/book/seat-selection.
• Review & pay: https://www.alaskaair.com/book/checkout — the payment page, card fields live here.

Framework — Auro design system (Alaska's web components)

The site is built on Alaska's Auro components. Every form control is a custom element whose real state lives on the wrapper, not the shadow-DOM internals. Setting `.value` programmatically does not satisfy the form validator. Validators trigger "Invalid X" errors unless the value was set by a real keystroke or a click-driven auro-menuoption selection.

Reliable patterns:

• Text fields (AURO-INPUT): click the wrapper, then type_text(...). Do not set .value and hope it sticks.
• Selects (AURO-SELECT): click the wrapper to open the dropdown, then click the matching auro-menuoption (has a value= attribute). A sibling native <select id="native-select-{id}"> exists but is not the validator source of truth.
• Buttons: the visible CTAs are FS-AURO-BUTTON or AURO-BUTTON. Coordinate-click them; their center is a stable target.

Card fields — CyberSource Flex Microform

PAN and Security Code are served from two separate cross-origin iframes from flex.cybersource.com/microform/bundle/v2.9.0/iframe.html. A coordinate click on the visible iframe box does not reliably focus the inner input (observed: click + type_text landed nowhere).

The approach that works:

1. Find the iframe targetIds via cdp("Target.getTargets") filtered on "cybersource" in url — there are exactly two. 2. Focus the input inside the iframe with js("document.getElementById('number').focus()", target_id=pan_frame_id) (CVV input id is securityCode). 3. Call type_text(...) — Input.insertText is routed to whatever element currently has focus, including cross-origin iframes, so the value lands in the Microform without touching frame coordinates.

PAN field id inside iframe: number. CVV id: securityCode. Both at the frame document root.

Stable selectors (guest-info + checkout)

• #firstName, #lastName — AURO-INPUT text fields (traveler 1).
• #gender, #dateOfBirthMonth — AURO-SELECT; pick option via auro-menuoption[value=...] after opening.
• #dateOfBirthDay, #dateOfBirthYear — AURO-INPUT digits.
• #email, #phone, #zipCode — AURO-INPUT under contact info.
• #saver-upsell-dialog — FS-AURO-DIALOG that appears after picking a Saver fare with a Continue with Saver FS-AURO-BUTTON inside. Find by walking into light-DOM descendants; click by rect, not selector.
• #expiration-month, #expiration-year — card exp AURO-SELECT; options are plain two-digit values like 12, 2029.
• #name, #addressLineOne, #billingInfoCity, #billingInfoZipCode — AURO-INPUT billing fields.
• #billingInfoState — AURO-SELECT; option values are 2-letter state codes (NY).
• Insurance: #TripInsurance_AWP0 (yes) / #TripInsurance_AWP1 (no). These radios render off-screen visually (e.g. x ≈ -15000). .click() via JS works; coordinate click does not.
• "Credit/debit card" payment-method radio: not an <input> — a plain BUTTON next to a SPAN with text Credit/debit card. Click the span/button by rect.

What doesn't work

• el.value = "..." on any Auro form element: display may update, but Auro's validator emits "Invalid X" on submit anyway. Always type or click-select.
• Coordinate clicks on CyberSource iframe boxes + type_text: focus doesn't land in the inner input. Use the iframe-target-focus pattern above.
• Deep-search document.querySelectorAll('button') for dialog CTAs: Auro buttons are FS-AURO-BUTTON custom elements, not BUTTON. Filter on tagName.includes('AURO-BUTTON') and walk shadowRoot chains.

Waits

• After clicking Continue with Saver: 3-5s for Svelte/Sapper transition to the trip-summary view.
• After Continue as guest: 5-6s to load /book/guest-info.
• After Continue on guest-info: 5-6s; re-check URL (/book/seat-selection) because validation errors keep you on the same URL with no thrown exception.
• After Skip seats: 5-6s to /book/checkout.

Traps

• Homepage (alaskaair.com) pops a credit-card promo modal on load; deep-linking straight to /search/results?... avoids it entirely.
• The results deep link lands in points/award mode by default. There is a single button[role=switch] with innerText="Money\nPoints\nPoints" — .click() toggles it.
• The "Protect your trip" insurance section is required and blocks advancing with no error until you select No. Use document.getElementById('TripInsurance_AWP1').click() — the radio is visually hidden at x ≈ -15000, so coordinate clicks won't find it.
• reCAPTCHA Enterprise badge appears on /book/checkout. It did not challenge in this run; behavior under heavy scripted sessions is unknown.

Known-working fake data shape

Cash fare SEA→PDX one-way, $249 Saver, April 29 2026 — smallest reachable price path. Swap O=/D= in the deep link for other Alaska hubs (LAX, SFO, PDX, SAN).

Amazon — Product Search & Data Extraction

Field-tested against amazon.com on 2025-04-18 using a logged-in Chrome session. No CAPTCHA or bot detection was triggered during any test run.

Navigation

Direct search URL (fastest, always use this)

goto_url("https://www.amazon.com/s?k=mechanical+keyboard")
wait_for_load()
wait(2)  # dynamic content needs ~2s after readyState=complete

Search box typing (use when you need category filtering)

goto_url("https://www.amazon.com")
wait_for_load()
wait(1)
js("document.querySelector('#twotabsearchtextbox').focus()")
js("document.querySelector('#twotabsearchtextbox').click()")
wait(0.3)
type_text("wireless mouse")
wait(0.3)
press_key("Enter")
wait_for_load()
wait(2)

Direct product page

# URL pattern: /dp/{ASIN}  or  /dp/{ASIN}?th=1 (Amazon may redirect to add ?th=1)
goto_url("https://www.amazon.com/dp/B08Z6X4NK3")
wait_for_load()
wait(2)

Session Gotcha

Always use `new_tab()` when opening Amazon for the first time in a harness session. goto_url() can silently fail to navigate if the current tab resists the navigation (observed when the daemon attached to a different real tab). The safe pattern:

tid = new_tab("https://www.amazon.com/s?k=mechanical+keyboard")
wait_for_load()
wait(2)

After that, goto_url() works fine within the same Amazon session.

Search Results Extraction

Container selector

[data-component-type="s-search-result"] — confirmed working, yields ~22 results per page.

Full extraction (field-tested)

results = js("""
  Array.from(document.querySelectorAll('[data-component-type="s-search-result"]')).map(el => ({
    asin: el.getAttribute('data-asin'),
    title: el.querySelector('h2 span')?.innerText?.trim(),
    price: el.querySelector('.a-price .a-offscreen')?.innerText,
    list_price: el.querySelector('.a-text-price .a-offscreen')?.innerText,
    rating: el.querySelector('[aria-label*="out of 5 stars"]')?.getAttribute('aria-label')?.split(' ')[0],
    reviews: el.querySelector('[aria-label*="ratings"]')?.getAttribute('aria-label'),
    is_sponsored: !!el.querySelector('.puis-sponsored-label-text'),
    url: el.querySelector('h2 a')?.href
  }))
""")

Field notes

• `asin`: data-asin attribute on the container div — always present, matches the /dp/{ASIN} URL.
• `title`: h2 span works consistently. h2 a.a-link-normal span also works.
• `price`: .a-price .a-offscreen returns the formatted string e.g. "$69.99". Use this, not .a-price-whole.
• `list_price`: .a-text-price .a-offscreen — only present when item is on sale (was/now pricing).
• `rating`: Use aria-label on [aria-label*="out of 5 stars"] — gives "4.5 out of 5 stars, rating details", split on space for the number.
• `reviews`: Use [aria-label*="ratings"] attribute — gives "1,514 ratings". Do NOT use .a-size-base.s-underline-text — that element exists on sponsored results and shows "Xbox" (a cross-sell widget text).
• `is_sponsored`: .puis-sponsored-label-text is present on sponsored listings; first 2-3 results are usually sponsored.
• `url`: h2 a href — contains the full /dp/{ASIN}/... URL.

Product Detail Page Extraction

Confirmed selectors (field-tested on B08Z6X4NK3)

detail = js("""
  ({
    title: document.querySelector('#productTitle')?.innerText?.trim(),
    price: (function() {
      var whole = document.querySelector('.a-price-whole')?.innerText?.replace(/[\\n.]/g,'');
      var frac  = document.querySelector('.a-price-fraction')?.innerText;
      return (whole && frac) ? '$' + whole + '.' + frac
           : document.querySelector('.a-price .a-offscreen')?.innerText || null;
    })(),
    list_price: document.querySelector('.basisPrice .a-offscreen')?.innerText,
    rating: document.querySelector('#acrPopover')?.getAttribute('title'),
    review_count: document.querySelector('#acrCustomerReviewText')?.innerText,
    availability: document.querySelector('#availability span')?.innerText?.trim(),
    brand: document.querySelector('#bylineInfo')?.innerText?.trim(),
    asin: document.querySelector('input[name="ASIN"]')?.value,
    bullet_points: Array.from(document.querySelectorAll('#feature-bullets li span.a-list-item'))
                       .map(e => e.innerText?.trim()).filter(t => t)
  })
""")

Price field notes

• #priceblock_ourprice and #priceblock_dealprice are legacy — they return null on modern product pages.
• Construct price from .a-price-whole + .a-price-fraction (both stripped of \n and .).
• As a fallback: first .a-price .a-offscreen on the page also works (confirmed $69.99).
• list_price from .basisPrice .a-offscreen shows the crossed-out "was" price when a discount exists.

Best Sellers Page

URL: https://www.amazon.com/Best-Sellers-{Category}/zgbs/{slug}/ e.g. https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics/

DOM structure (2025)

.zg-item-immersion does not exist — Amazon migrated to CSS modules. Use [data-asin] anchored on [id="gridItemRoot"]:

goto_url("https://www.amazon.com/Best-Sellers-Electronics/zgbs/electronics/")
wait_for_load()
wait(2)

items = js("""
  Array.from(document.querySelectorAll('[data-asin]')).map(el => {
    var container = el.closest('[id="gridItemRoot"]') || el;
    return {
      asin: el.getAttribute('data-asin'),
      rank: container.querySelector('[class*="zg-bdg-text"]')?.innerText,
      title: container.querySelector('img[alt]')?.getAttribute('alt'),
      price: container.querySelector('.p13n-sc-price, .a-size-base.a-color-price')?.innerText,
      url: 'https://www.amazon.com/dp/' + el.getAttribute('data-asin')
    }
  }).filter(r => r.rank)
""")

Note: Title comes from the product image alt attribute — the text title elements use obfuscated CSS module class names that change between deployments.

Pagination

# Get next page URL directly
next_url = js("document.querySelector('.s-pagination-next')?.href")
if next_url:
    goto_url(next_url)
    wait_for_load()
    wait(2)

# Or construct by page number
goto_url("https://www.amazon.com/s?k=wireless+mouse&page=2")

Result Count

count_text = js("document.querySelector('[data-component-type=\"s-result-info-bar\"] h1')?.innerText?.trim()")
# Returns e.g.: '1-16 of over 40,000 results for "wireless mouse"\nSort by:\n...'
# Extract just the count: count_text.split('\n')[0]

CAPTCHA Detection

No CAPTCHA was encountered during testing with a logged-in Chrome session. To detect defensively:

def check_captcha():
    text = js("document.body.innerText.slice(0,500)") or ""
    url  = page_info()["url"]
    return (
        "captcha" in text.lower()
        or "enter the characters" in text.lower()
        or "sorry, we just need to make sure" in text.lower()
        or "captcha" in url.lower()
        or "validateCaptcha" in url
    )

if check_captcha():
    raise RuntimeError("Amazon CAPTCHA hit — stop and notify user")

Amazon may serve a CAPTCHA on fresh/anonymous sessions. Using the browser's existing logged-in session avoids this in practice.

Gotchas

• `goto_url()` silent failure: On first visit, use new_tab(url) instead. After the tab is on Amazon, goto_url() works.
• `.zg-item-immersion` is gone: Best Sellers page uses CSS module classes (obfuscated). Use [data-asin] + img[alt] for title.
• `.a-size-base.s-underline-text` is unreliable for review count: On sponsored results it shows unrelated text (e.g. "Xbox"). Use [aria-label*="ratings"] instead.
• `#priceblock_ourprice` is legacy: Returns null on modern pages. Construct from .a-price-whole + .a-price-fraction.
• Sponsored results appear first: First 2-3 results are almost always is_sponsored: true. Filter them out with !el.querySelector('.puis-sponsored-label-text') when you need organic results.
• `data-asin` can be empty string on non-product rows: Filter with .filter(r => r.asin).
• Price split DOM: .a-price-whole innerText includes a trailing \n. — strip it: .replace(/[\n.]/g,'').
• ASIN from URL: Use /dp/([A-Z0-9]{10})/ regex on the product URL. data-asin on search results is always the canonical ASIN.
• `?th=1` redirect: Amazon appends ?th=1 (and sometimes ?psc=1) to product URLs after redirect. This is normal — input[name="ASIN"] always has the clean ASIN.
• Wait 2s after `wait_for_load()`: Amazon search results load the listing cards asynchronously. readyState=complete fires before cards render. A hard 2s wait is required.

Internet Archive / Wayback Machine — Scraping & Data Extraction

https://archive.org / https://web.archive.org — all public data, no auth required. Every workflow here is pure http_get — no browser needed.

Do this first

Use the CDX API for anything Wayback-related — it is the reliable workhorse. The Wayback Availability API (`/wayback/available`) is known to return empty `archived_snapshots` even for well-archived URLs and should not be used as a primary mechanism.

import json

# Find snapshots of any URL — primary entry point for Wayback data
r = http_get(
    "https://web.archive.org/cdx/search/cdx"
    "?url=iana.org&output=json&limit=5"
    "&fl=timestamp,original,statuscode,mimetype,length",
    timeout=40.0
)
rows = json.loads(r)
headers = rows[0]   # ['timestamp', 'original', 'statuscode', 'mimetype', 'length']
for row in rows[1:]:
    ts, orig, status, mime, length = row
    snap_url = f"https://web.archive.org/web/{ts}/{orig}"
    print(f"{ts}  {status}  {snap_url}")

For item metadata (books, video, audio, software), go straight to:

data = json.loads(http_get("https://archive.org/metadata/{identifier}", timeout=30.0))

Common workflows

Find the nearest archived snapshot to a target date

import json

# CDX sort=closest returns the single snapshot nearest to the given timestamp
r = http_get(
    "https://web.archive.org/cdx/search/cdx"
    "?url=iana.org&output=json&limit=1"
    "&fl=timestamp,original,statuscode"
    "&closest=20230601120000&sort=closest",
    timeout=60.0   # CDX can be slow — always use timeout >= 40s
)
rows = json.loads(r)
# rows[0] = header, rows[1] = closest snapshot
ts, orig, status = rows[1]
snap_url = f"https://web.archive.org/web/{ts}/{orig}"
# Result: ts='20230601114925', orig='https://www.iana.org/', status='200'
# snap_url: https://web.archive.org/web/20230601114925/https://www.iana.org/

Timestamp format is always 14-digit YYYYMMDDHHMMSS. Pass any prefix — 20230601 (day), 202306 (month), 2023 (year) — and CDX will match.

List all monthly snapshots for a URL (collapsed)

import json

r = http_get(
    "https://web.archive.org/cdx/search/cdx"
    "?url=iana.org&output=json"
    "&collapse=timestamp:6"   # :6 = dedupe by YYYYMM (one per month)
    "&from=20230101&to=20231231"
    "&fl=timestamp,original",
    timeout=60.0
)
rows = json.loads(r)
# rows[0] = header ['timestamp', 'original']
# rows[1:] = one row per month:
# ['20230101103807', 'https://www.iana.org/']
# ['20230201144829', 'https://www.iana.org/']
# ...12 rows for 2023

for ts, orig in rows[1:]:
    print(f"{ts[:4]}-{ts[4:6]}  https://web.archive.org/web/{ts}/{orig}")

collapse=timestamp:N deduplicates by the first N digits of the timestamp:

• :4 = one per year, :6 = one per month, :8 = one per day

List snapshots for an entire domain (all pages)

import json

# matchType=domain captures all URLs under that domain
r = http_get(
    "https://web.archive.org/cdx/search/cdx"
    "?url=iana.org&matchType=domain&output=json"
    "&limit=10&fl=timestamp,original,statuscode"
    "&collapse=timestamp:8",  # one capture per URL per day
    timeout=60.0
)
rows = json.loads(r)
for row in rows[1:]:
    print(row)
# ['19971210061738', 'http://www.iana.org:80/', '200']
# ['19980211065537', 'http://www.iana.org:80/', '200']
# ...

matchType options: exact (default), prefix (URL + subpaths), host (all subdomains), domain (host + all subdomains).

Filter snapshots by prefix path

import json

# All archived pages under /domains/ path
r = http_get(
    "https://web.archive.org/cdx/search/cdx"
    "?url=iana.org/domains/&matchType=prefix&output=json"
    "&limit=5&fl=timestamp,original,statuscode",
    timeout=40.0
)
rows = json.loads(r)
for row in rows[1:]:
    print(row)
# ['20080509121811', 'http://www.iana.org/domains/', '200']
# ['20080704174537', 'http://iana.org/domains/', '200']

Paginate CDX results with resumeKey

import json
from urllib.parse import quote

def cdx_all_snapshots(url, fl="timestamp,original,statuscode", page_size=500):
    """Iterate all CDX records for a URL, yielding rows (excluding header)."""
    base = (
        f"https://web.archive.org/cdx/search/cdx"
        f"?url={quote(url, safe='')}&output=json"
        f"&fl={fl}&limit={page_size}&showResumeKey=true"
    )
    resume_key = None
    while True:
        endpoint = base if resume_key is None else f"{base}&resumeKey={quote(resume_key)}"
        rows = json.loads(http_get(endpoint, timeout=60.0))
        # rows structure with showResumeKey=true:
        # [header, row1, row2, ..., [], [resume_key_string]]
        # The second-to-last row is [] (separator), last row is [resume_key]
        has_resume = len(rows) >= 2 and rows[-1] != [] and rows[-2] == []
        data_rows = rows[1:-2] if has_resume else rows[1:]
        for row in data_rows:
            yield row
        if not has_resume:
            break
        resume_key = rows[-1][0]

for row in cdx_all_snapshots("iana.org", fl="timestamp,original"):
    ts, orig = row
    # process...

Retrieve the actual archived page

# Direct snapshot URL: /web/{14-digit-timestamp}/{original-url}
snap_url = "https://web.archive.org/web/19971210061738/http://www.iana.org:80/"
content = http_get(snap_url, timeout=30.0)
# Returns the archived HTML with Wayback toolbar injected at top
# The toolbar is inside <!-- BEGIN WAYBACK TOOLBAR INSERT --> comments

# The calendar view URL pattern (for browser navigation, not http_get):
# https://web.archive.org/web/20230101000000*/python.org
# The * tells Wayback to show the calendar — returns HTML, not raw page

Item metadata (books, video, audio, software, collections)

import json
from urllib.parse import quote

identifier = "HardWonWisdomTrailer"
data = json.loads(http_get(f"https://archive.org/metadata/{identifier}", timeout=30.0))

# Top-level keys:
# alternate_locations, created, d1, d2, dir, files, files_count,
# is_collection, item_last_updated, item_size, metadata, server, uniq, workable_servers

meta = data['metadata']
# Common metadata fields (not all present on every item):
print(meta.get('identifier'))   # 'HardWonWisdomTrailer'
print(meta.get('title'))        # 'Hard Won Wisdom Trailer'
print(meta.get('mediatype'))    # 'movies' | 'texts' | 'audio' | 'software' | 'collection'
print(meta.get('creator'))      # 'jakemauz'
print(meta.get('date'))         # '2017-02-18'
print(meta.get('description'))  # HTML string — strip tags if needed
print(meta.get('subject'))      # str OR list of str depending on item
print(meta.get('publicdate'))   # '2017-02-18 11:51:16'
print(meta.get('collection'))   # parent collection identifier

files = data['files']
# Each file entry:
# name, source ('original'|'derivative'|'metadata'), format, size (bytes as str),
# md5, sha1, crc32, mtime
# For video/audio: length (seconds as str), height, width
# For derivative: original (name of source file)

# Find the primary original file
orig_files = [f for f in files if f.get('source') == 'original']
# orig_files[0]: {'name': 'Hard-won wisdom trailer.mp4', 'source': 'original',
#  'format': 'MPEG4', 'size': '7532153', 'length': '94.13',
#  'height': '360', 'width': '640', 'md5': 'aaeebe0481...', ...}

# Build download URL — two equivalent forms:
server = data['server']      # 'ia601405.us.archive.org'
dir_path = data['dir']       # '/2/items/HardWonWisdomTrailer'
fname = orig_files[0]['name']
from urllib.parse import quote as urlquote
# Form 1: direct storage server (fastest)
url1 = f"https://{server}{dir_path}/{urlquote(fname)}"
# Form 2: standard redirect URL (always works, resolved by CDN)
url2 = f"https://archive.org/download/{identifier}/{urlquote(fname)}"
# Both confirmed status 200, Content-Type: video/mp4

Search items (books, audio, video, software)

import json

# advancedsearch.php is the correct API — /search returns HTML
r = http_get(
    "https://archive.org/advancedsearch.php"
    "?q=artificial+intelligence+AND+mediatype:texts"
    "&fl[]=identifier&fl[]=title&fl[]=creator&fl[]=date&fl[]=downloads"
    "&rows=5&sort[]=downloads+desc&output=json",
    timeout=30.0
)
data = json.loads(r)
# data['responseHeader']['status'] = 0 (success)
# data['responseHeader']['QTime'] = query time ms
# data['response']['numFound'] = 25911 (total matches)
# data['response']['start'] = 0 (offset)
# data['response']['docs'] = list of item dicts

resp = data['response']
print(f"Total: {resp['numFound']}, showing: {len(resp['docs'])}")
for doc in resp['docs']:
    print(f"  {doc['identifier']}  {doc.get('title', '')[:50]}")
    # doc fields are only present if they have values — always use .get()

Pagination: use start= offset (not page=). Max rows= is not documented but 100 works reliably.

Search with all supported parameters

import json

r = http_get(
    "https://archive.org/advancedsearch.php"
    "?q=machine+learning+AND+mediatype:texts"  # Lucene query syntax
    "&fl[]=identifier&fl[]=title&fl[]=date&fl[]=year"
    "&fl[]=creator&fl[]=subject&fl[]=description&fl[]=downloads"
    "&rows=3"
    "&start=0"               # pagination offset
    "&sort[]=date+desc"      # sort field + direction
    "&output=json",
    timeout=30.0
)
data = json.loads(r)
# Confirmed fields in fl[]:
# identifier, title, date, year, creator, subject, description,
# downloads, mediatype, collection, language, avg_rating, num_reviews

# mediatype values: texts, audio, movies, software, image, etree, data, collection, account
# Sort fields: date, downloads, avg_rating, num_reviews, publicdate, addeddate

API reference

CDX field reference

The CDX API returns a JSON array of arrays. The first row is always the header when output=json.

Default fl= when omitted: urlkey,timestamp,original,mimetype,statuscode,digest,length (all 7 fields in that order).

Rate limits

No auth, no API key. In practice:

• CDX API: intermittently slow — individual queries time out at 20s and succeed at 40–60s. Always use timeout=40.0 minimum. 3 rapid sequential CDX calls in ~10s completed; 10 rapid calls produced 3 timeouts.
• Metadata API: Fast and reliable — 5 sequential calls completed in 3.0s with no errors.
• Search API: Fast — typically responds in 30–65ms (QTime in response header).
• No documented per-second or per-day limits. Archive.org's policy is to be respectful: add time.sleep(1) between CDX calls in loops.

Gotchas

• CDX times out — always set `timeout=40.0` or higher. The default 20s is often too short for CDX. Metadata and search APIs are fine at 20–30s. CDX slowness is backend-side and unpredictable; add retry logic for production use.

• Wayback Availability API is unreliable. GET /wayback/available?url=iana.org returns {"url": "iana.org", "archived_snapshots": {}} even for URLs confirmed archived via CDX. Tested 2026-04-18 across many URLs and timestamp combinations — consistently empty. Use CDX ?sort=closest&limit=1 instead (confirmed working).

• CDX first row is always the header when `output=json`. rows[0] is ['timestamp', 'original', ...], not a data row. Always slice rows[1:] for data. When showResumeKey=true, the last two rows are [] (separator) and ['<resume_key_string>'].

• CDX `fl=` must match exactly what you iterate. If you request &fl=timestamp,original you get 2-element rows; forgetting a field breaks destructuring. When in doubt, omit fl= entirely and get all 7 fields.

• `output=json` is required — there is no default JSON mode. Omitting output=json returns space-separated text. output=text also works and is slightly faster for simple queries.

• `timestamp` is a string, not an integer. Even in JSON, CDX returns all fields as strings: '1418' not 1418, '200' not 200. Cast explicitly: int(row[4]), int(row[6]).

• The `original` field preserves port numbers. Old crawls captured http://www.iana.org:80/ — the :80 is part of the URL. When building a playback URL, use original verbatim: f"https://web.archive.org/web/{ts}/{orig}" works correctly with the port included.

• Metadata `{}` means the item doesn't exist or is private. http_get("https://archive.org/metadata/nonexistent") returns '{}' (2-byte response) with HTTP 200. Always check if not data or if not data.get('metadata') before accessing fields.

• Metadata `subject` can be a string or a list. When a single subject tag is set, the API returns "subject": "short film". When multiple, it returns "subject": ["short film", "spoken word"]. Normalize with: subjects = [meta['subject']] if isinstance(meta.get('subject'), str) else meta.get('subject', []).

• File `size` and `length` are strings, not numbers. files[0]['size'] is '7532153' (bytes). files[0]['length'] is '94.13' (seconds for video/audio). Cast with int() and float() respectively.

• Use `archive.org/download/` not the raw storage server URL for reliability. The raw URL (ia601405.us.archive.org/2/items/...) is faster but server-specific. archive.org/download/{id}/{file} redirects to the correct storage node and remains stable as items migrate.

• `/search?output=json` returns HTML, not JSON. The /search endpoint is a React SPA — it ignores output=json. Always use advancedsearch.php for programmatic access.

• `collapse=timestamp:6` gives one row per month, but it keeps the FIRST capture of that month. If you want the last, you'd need to reverse and re-collapse, or fetch all and filter client-side. The collapse parameter de-duplicates by truncating the timestamp to N digits and keeping the first matching row.

• CDX `from=` / `to=` accept partial timestamps. from=20230101 means 20230101000000. to=20231231 means 20231231000000 (exclusive). To include all of 2023, use to=20240101.

articulate-rise

Articulate Rise 360 (rise.{instance}.articulate.com) — authoring + preview + the sandboxed code blocks. Notes are durable shape only, no per-task narration.

Authoring vs preview URL patterns

authoring  https://rise.{instance}.articulate.com/authoring/{courseId}/lesson/{lessonId}
preview    https://rise.{instance}.articulate.com/preview/{courseId}#/lessons/{lessonId}

Authoring has the editable React shell + Redux state. Preview is the rendered learner view. Both load the same code blocks but only authoring lets you mutate them. Use authoring for any edit workflow, preview for visual verification only.

The two-layer iframe (most important fact on this page)

A Rise "code block" is two iframes deep, not one:

parent page (rise.{instance}.articulate.com)
└── outer iframe[sandbox]            src = sandbox.articulateusercontent.eu/sandbox/sandbox.html#channel=…
    └── inner iframe                 src = about:srcdoc        ← the bespoke HTML lives HERE

• The outer is a thin sandbox shell. Minimal default CSS, no app fonts, no app state.
• The inner about:srcdoc is what the author's HTML actually renders into.
• The parent's document.querySelector('iframe[sandbox]') selects the outer.
• Anything that needs to read or test the content (computed styles, fonts, JS state, canvas measurements) must run inside the inner frame, never the outer.

If you forget this, you'll silently measure the wrong document and get false negatives — fonts will look unloaded, computed styles will look wrong, etc.

Walking to the inner frame (canonical pattern)

tree = cdp("Page.getFrameTree")
def find_inner(frame, depth=0):
    url = frame.get("frame", {}).get("url", "")
    children = frame.get("childFrames", []) or []
    if "about:srcdoc" in url and depth >= 2:
        return frame["frame"]["id"]
    for c in children:
        r = find_inner(c, depth + 1)
        if r: return r
    return None
inner_frame_id = find_inner(tree["frameTree"])

iso = cdp("Page.createIsolatedWorld", frameId=inner_frame_id, worldName="probe")
ctx = iso["executionContextId"]
result = cdp("Runtime.evaluate",
    expression="<your JS here>",
    contextId=ctx, returnByValue=True)

Don't try to attach to the outer iframe target and reach the inner from there — Rise's nesting + the sandbox attribute makes that flaky. Walk the frame tree from the parent.

Editing a code block in authoring view

The pencil icon on a block opens the "Add code" sidebar panel. Inside:

• The editor is Ace, not Monaco.
• Get the instance: ace.edit(document.querySelector('.ace_editor'))
• editor.getValue() / editor.setValue(newValue, -1) both work programmatically.
• The -1 arg keeps cursor at the top and avoids selecting the whole buffer.
• Programmatic edits do not trigger Rise's debounced change listener. The autosave fires on panel close, not on edit.
• Closing the panel commits the editor value to Redux. Closing it after a programmatic edit is what makes the change persist.
• There is no explicit Save button on a lesson in authoring. Rise autosaves on every panel close. Don't waste time looking for one.
• After save, the live iframe in authoring view does not re-render — it keeps showing the pre-edit state until you location.reload(). Always reload before verifying.

Closing the panel

Selector: .blocks-sidebar__close (an X button in the panel chrome). Click it via real CDP mouse, or el.click() works too — Rise listens to the DOM event, not just visual mouse.

Escape does not close it.

DOM gotcha: duplicate data-block-id

For each block, two elements carry the same data-block-id in the DOM:

• inner: .lesson-blocks__block-type-container[data-block-id=…]
• outer: .sparkle-fountain.block[data-block-id=…]

iframe.closest('[data-block-id]') picks the inner. The pencil/edit-controls toolbar is only attached to the outer. So if you need the controls (pencil, style, format), do a class-aware ancestor walk:

let el = iframe;
while (el && !el.classList?.contains('sparkle-fountain')) el = el.parentElement;
const outerBlockEl = el;  // pencil controls live on this one

Block-controls overlay needs a real mouse event

The pencil/style/format toolbar that floats above a block is rendered on a React hover state. A JS-dispatched mouseover event is not enough — Rise's listener is bound to the framework's synthetic events and only fires on real pointer movement.

Use CDP:

# Get block bounding rect, then:
cdp("Input.dispatchMouseEvent", type="mouseMoved",
    x=rect.x + rect.width/2, y=rect.y + 20, button="none")
# wait ~200ms for React to render the controls

After this the pencil button is in the DOM at:

.block-controls__btn-icon--type-content

Click via querySelector + element bounds + real mousePressed/mouseReleased.

Block IDs are NOT unique across cloned courses

When a course is cloned in Rise, the bespoke blocks keep their original IDs. So the same blockId can show up in two different courses, with completely different HTML in each. Each course holds its own copy of the block's srcdoc in its own Redux state — edits to one course do not propagate.

If you're iterating across courses and using blockId as a dedupe key, you'll under-count and skip blocks. Dedupe by (courseId, blockId) instead.

Fonts inside code blocks

The single most common bug in bespoke code blocks: the block's CSS uses

font-family: inherit;

…assuming inherit will pull the brand/theme font from the surrounding Rise page.

It can't. The sandbox iframe is cross-origin (sandbox.articulateusercontent.eu vs the Rise origin), so inherit resolves to the iframe's own root, which has nothing set, which falls through to UA default — Times New Roman on every browser.

To get a brand font rendering inside a code block:

1. Find the brand font's actual WOFF/WOFF2 URLs by walking the parent's stylesheets:

   for (const sheet of document.styleSheets) {
     try {
       for (const rule of sheet.cssRules || []) {
         if (rule instanceof CSSFontFaceRule && /YourFontName/i.test(rule.cssText)) {
           console.log(rule.cssText);  // → src: url('https://articulateusercontent.eu/rise/fonts/...')
         }
       }
     } catch (e) { /* cross-origin sheet */ }
   }

The URLs are typically served by articulateusercontent.eu/rise/fonts/{hash}.woff and have permissive CORS so they're reusable from inside the sandbox.

2. Inline matching @font-face rules at the top of the code block's own <style>:

   @font-face {
     font-family: 'YourFontName';
     src: url('https://articulateusercontent.eu/rise/fonts/{hash}.woff') format('woff');
     font-weight: 400;
     font-display: swap;
   }

3. Replace inherit in the block's CSS with an explicit family stack:

   font-family: 'YourFontName', Arial, Helvetica, sans-serif;

That fixes the rendering at its actual root cause. Native Rise blocks don't have this problem because they live in the parent document and inherit fonts there for free; bespoke code blocks are sandboxed and have to bring their own.

Tests that lie

• `document.fonts.check('1em FontName')` returns true even when FontName isn't registered. It only confirms the family-name string is parseable. Don't trust it.

• Running a font/style test in the outer shell instead of the inner srcdoc will report the outer shell's defaults, not the bespoke content's. This is the #1 false-negative trap.

• Reading `iframe.srcdoc` from the parent returns empty for the outer (the outer uses src=, not srcdoc=) and is cross-origin-blocked for the inner. Don't expect to read content via DOM attributes — go through CDP frame attach.

Ground-truth font check (canvas glyph width)

The only test that doesn't lie. Run inside the inner srcdoc frame via the isolated-world pattern above:

(() => {
  const c = document.createElement('canvas').getContext('2d');
  c.font = '700 24px "YourFontName", sans-serif';
  const named_w = c.measureText('Sample Text').width;
  c.font = '700 24px sans-serif';
  const fallback_w = c.measureText('Sample Text').width;
  return { named_w, fallback_w, loaded: named_w !== fallback_w };
})()

If named_w === fallback_w, the named font silently fell back. If they differ, the named font is the one actually being painted to pixels.

Useful endpoints

GET  /api/rise-runtime/course_fonts.css?typefaceIds={id1},{id2}    # @font-face for course-themed fonts
GET  /api/rise-runtime/fonts.css                                    # global font catalogue
POST /api/rise-runtime/ducks/rise/courses/GET_COURSE                # full course payload (auth required, exact request shape varies)

The course_fonts.css endpoint is the easiest place to discover the WOFF URLs for whatever brand font is themed onto the current course — just hit it in DevTools Network tab during a normal load and read the response.

Don'ts

• Don't try to inject CSS into the iframe from the parent. Cross-origin sandbox blocks all of it (style injection, parent.document.fonts.add(...) from inside, postMessage style protocols — none of it works).
• Don't measure anything in the outer sandbox shell. It's not the document the user sees.
• Don't trust document.fonts.check. Glyph-width measurement is the only honest test.
• Don't assume blockId is unique across courses.
• Don't expect a Save button. Panel-close is the save event.

arXiv Bulk Harvest + Semantic Scholar — OAI-PMH & Citation Enrichment

Companion to domain-skills/arxiv/scraping.md. Use the arxiv skill for search-and-fetch workflows. Use this skill when you need:

• Bulk-harvesting all papers in a subject area or date window (OAI-PMH)
• Citation counts, influential-citation scores, and cross-database IDs (Semantic Scholar)
• Per-paper version history and submitter info (arXivRaw metadata)

No API key required for either endpoint. Both return JSON or XML over plain HTTP.

---

OAI-PMH bulk harvest

Endpoint (confirmed 2026-04-19)

https://oaipmh.arxiv.org/oai

https://export.arxiv.org/oai2 is the old URL — it 301-redirects to the new one. Use the new URL directly to avoid the extra round-trip.

Harvest all cs papers from a date window

import xml.etree.ElementTree as ET
from helpers import http_get

OAI_NS = {
    'oai': 'http://www.openarchives.org/OAI/2.0/',
    'arXiv': 'http://arxiv.org/OAI/arXiv/',
}

def fetch_oai_page(url):
    """Fetch one OAI-PMH page; return (records_xml_list, next_token_or_None)."""
    xml = http_get(url)
    root = ET.fromstring(xml)
    records = root.findall('.//oai:record', OAI_NS)
    token_el = root.find('.//oai:resumptionToken', OAI_NS)
    token = token_el.text if token_el is not None and token_el.text else None
    return records, token

def parse_arxiv_record(rec):
    """Extract fields from one <record> element (metadataPrefix=arXiv)."""
    header = rec.find('oai:header', OAI_NS)
    meta   = rec.find('.//arXiv:arXiv', OAI_NS)
    if meta is None:
        return None   # deleted record (header has status="deleted")
    authors_el = meta.findall('arXiv:authors/arXiv:author', OAI_NS)
    authors = []
    for a in authors_el:
        fn = (a.findtext('arXiv:forenames', namespaces=OAI_NS) or '').strip()
        ln = (a.findtext('arXiv:keyname',   namespaces=OAI_NS) or '').strip()
        authors.append(f"{fn} {ln}".strip())
    return {
        'id':           meta.findtext('arXiv:id', namespaces=OAI_NS),
        'datestamp':    header.findtext('oai:datestamp', namespaces=OAI_NS),
        'created':      meta.findtext('arXiv:created',  namespaces=OAI_NS),
        'updated':      meta.findtext('arXiv:updated',  namespaces=OAI_NS),
        'title':        (meta.findtext('arXiv:title',    namespaces=OAI_NS) or '').strip(),
        'authors':      authors,
        'categories':   (meta.findtext('arXiv:categories', namespaces=OAI_NS) or '').split(),
        'abstract':     (meta.findtext('arXiv:abstract',   namespaces=OAI_NS) or '').strip(),
        'doi':          meta.findtext('arXiv:doi',         namespaces=OAI_NS),
        'journal_ref':  meta.findtext('arXiv:journal-ref', namespaces=OAI_NS),
        'license':      meta.findtext('arXiv:license',     namespaces=OAI_NS),
    }

# --- Main harvest loop ---
import time

BASE = 'https://oaipmh.arxiv.org/oai'
first_url = (
    f"{BASE}?verb=ListRecords"
    f"&metadataPrefix=arXiv"
    f"&set=cs"
    f"&from=2024-01-01"
    f"&until=2024-01-02"
)

papers = []
url = first_url
while url:
    records, token = fetch_oai_page(url)
    for rec in records:
        p = parse_arxiv_record(rec)
        if p:
            papers.append(p)
    print(f"  fetched {len(records)} records, total so far: {len(papers)}")
    if token:
        url = f"{BASE}?verb=ListRecords&resumptionToken={token}"
        time.sleep(5)   # OAI-PMH policy: >=5s between pages
    else:
        url = None

print(f"Done. {len(papers)} papers harvested.")
# Confirmed output for cs, 2024-01-01 to 2024-01-02:
# fetched 44 records, total so far: 44
# Done. 44 papers harvested.
# For 2024-01-01 to 2024-01-07 (cs): multiple pages, resumptionToken issued when >~200 records

Available verbs

Top-level sets (confirmed)

Subset sets use topic:topic:SUBCATEGORY notation, e.g. cs:cs:LG for Machine Learning. List all with verb=ListSets.

Available metadata formats

• arXiv — rich: id, created/updated dates, authors (keyname + forenames separately), categories, abstract, doi, journal-ref, license. Use this.
• arXivRaw — adds <submitter>, per-version history (<version version="v1"> with date and file size), author list as flat string. Use when you need version history.
• oai_dc — Dublin Core, minimal. Skip unless you need cross-system compatibility.
• arXivOld — legacy format pre-2007. Skip.

GetRecord + arXivRaw (version history)

import xml.etree.ElementTree as ET
from helpers import http_get

RAW_NS = {
    'oai': 'http://www.openarchives.org/OAI/2.0/',
    'raw': 'http://arxiv.org/OAI/arXivRaw/',
}

xml = http_get(
    "https://oaipmh.arxiv.org/oai"
    "?verb=GetRecord"
    "&metadataPrefix=arXivRaw"
    "&identifier=oai:arXiv.org:1706.03762"
)
root = ET.fromstring(xml)
meta = root.find('.//raw:arXivRaw', RAW_NS)

title     = meta.findtext('raw:title',     namespaces=RAW_NS)
submitter = meta.findtext('raw:submitter', namespaces=RAW_NS)
versions  = meta.findall('raw:version',    RAW_NS)
for v in versions:
    print(v.get('version'), v.findtext('raw:date', namespaces=RAW_NS))
# Confirmed output for 1706.03762 ("Attention Is All You Need"):
# v1 Mon, 12 Jun 2017 17:57:34 GMT
# v2 Mon, 19 Jun 2017 16:49:45 GMT
# ...
# v7 Wed, 02 Aug 2023 00:41:18 GMT
# submitter: Llion Jones

---

Semantic Scholar — citation enrichment for arXiv papers

No API key required (unauthenticated: 1 req/s, 5000 req/day). With a free key the limit rises to 100 req/s.

Base URL: https://api.semanticscholar.org/graph/v1/

Single paper lookup by arXiv ID

import json
from helpers import http_get

paper = json.loads(http_get(
    "https://api.semanticscholar.org/graph/v1/paper/arXiv:1706.03762"
    "?fields=title,year,venue,publicationDate,citationCount,"
    "influentialCitationCount,authors,abstract,externalIds"
))
print(paper['title'])                    # "Attention is All you Need"
print(paper['citationCount'])            # 173155  (confirmed 2026-04-19)
print(paper['influentialCitationCount']) # 19629
print(paper['venue'])                    # "Neural Information Processing Systems"
print(paper['externalIds']['ArXiv'])     # "1706.03762"
print(paper['externalIds']['DOI'])       # missing if no DOI
for a in paper['authors']:
    print(a['name'], a['authorId'])

The ID format arXiv:NNNN.NNNNN is accepted directly — no conversion needed.

Batch lookup (up to 500 IDs per POST)

import json
from helpers import http_get
import urllib.request

ids = ["arXiv:1706.03762", "arXiv:1810.04805", "arXiv:2005.14165"]
fields = "paperId,externalIds,title,year,citationCount,influentialCitationCount"

body = json.dumps({"ids": ids}).encode()
req = urllib.request.Request(
    f"https://api.semanticscholar.org/graph/v1/paper/batch?fields={fields}",
    data=body,
    headers={"Content-Type": "application/json"},
    method="POST",
)
with urllib.request.urlopen(req, timeout=20) as r:
    results = json.loads(r.read())

for p in results:
    print(p['externalIds'].get('ArXiv'), p['citationCount'], p['title'][:50])
# Confirmed output (2026-04-19):
# 1706.03762  173155  Attention is All you Need
# 1810.04805  113138  BERT: Pre-training of Deep Bidirectional Tran...
# 2005.14165  (varies)  Language Models are Few-Shot Learners

Note: helpers.http_get only does GET. For POST use urllib.request.Request directly as above.

Paper search

import json
from helpers import http_get

results = json.loads(http_get(
    "https://api.semanticscholar.org/graph/v1/paper/search"
    "?query=large+language+model"
    "&fields=paperId,externalIds,title,year,citationCount"
    "&limit=5"
))
total = results['total']   # e.g. 3473582 for "large language model"
for p in results['data']:
    arxiv_id = p['externalIds'].get('ArXiv', 'no-arxiv')
    print(arxiv_id, p['year'], p['citationCount'], p['title'][:50])
# next page: use offset=5, offset=10, etc.

Available fields (pass as comma-separated fields= query param)

---

Downloading PDFs

Direct PDF download — no auth, no redirect for versionless URLs (returns 200 + PDF body directly).

import urllib.request

def download_pdf(arxiv_id, dest_path, version=None):
    """
    arxiv_id: bare ID like '1706.03762' or versioned '1706.03762v7'
    version:  if given, appended as 'v{version}' — ignored if arxiv_id already has version
    dest_path: where to save, e.g. '/tmp/paper.pdf'
    """
    if 'v' not in arxiv_id.split('.')[-1] and version:
        arxiv_id = f"{arxiv_id}v{version}"
    url = f"https://arxiv.org/pdf/{arxiv_id}"
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=60) as r:
        with open(dest_path, 'wb') as f:
            f.write(r.read())
    print(f"Saved {r.headers.get('content-length', '?')} bytes to {dest_path}")

download_pdf('1706.03762', '/tmp/attention.pdf')
# Confirmed: saves 2215244 bytes, filename hint in header: '1706.03762v7.pdf'
# Versionless URL resolves to latest version server-side (no redirect, 200 direct)

---

Gotchas

• OAI-PMH endpoint moved. https://export.arxiv.org/oai2 301-redirects to https://oaipmh.arxiv.org/oai. Use the new URL. helpers.http_get (which uses urllib) does NOT follow redirects — you'll get an empty string or error. Either use urllib.request.urlopen with follow_redirects logic, or just use the canonical URL directly.

• OAI-PMH rate limit: 5 seconds between pages. The protocol requires a Retry-After interval. The server embeds an expirationDate on the resumptionToken. Violating the rate limit causes the token to be invalidated and the harvest fails silently. Always time.sleep(5) between pages.

• Resumption token is opaque but URL-encoded. The token looks like verb%3DListRecords%26...%26skip%3D247. Pass it verbatim as &resumptionToken=<token> — do not URL-encode it again.

• `datestamp` in OAI-PMH is last-modified date, not submission date. A paper submitted in 2008 can appear in a 2024 harvest window if it was revised then. The <created> and <updated> fields inside <arXiv> metadata are the actual submission/revision dates.

• Deleted records have no `<metadata>` element. The <header> will carry status="deleted". Always check meta is None after find('.//arXiv:arXiv', ...).

• Author structure differs between OAI-PMH formats. In arXiv metadata, authors are structured: <author><keyname>Vaswani</keyname><forenames>Ashish</forenames></author>. In arXivRaw, they're a flat comma-separated string: Ashish Vaswani, Noam Shazeer, .... In the Atom API, it's <name>Ashish Vaswani</name> (first-last order). Pick the source that matches your downstream use.

• Semantic Scholar 429 under unauthenticated bursts. The unauthenticated limit is ~1 req/s. Rapid parallel calls return {"code": "429"}. Add time.sleep(1) between single lookups or use the batch POST endpoint (up to 500 IDs, single request) to stay under the limit. The batch endpoint itself counts as 1 request.

• Semantic Scholar `externalIds` may lack `ArXiv` key. Not all papers have an arXiv preprint. When enriching an arXiv list with S2 data, always use .get('ArXiv') not ['ArXiv'].

• Atom API rate limit: 1 request per 3 seconds for sustained crawls. The API returns HTTP 429 "Rate exceeded." on rapid-fire requests. The OAI-PMH endpoint is designed for bulk and is more tolerant, but still requires the 5s sleep between resumption pages.

• OAI-PMH `set` param uses colon-separated hierarchy, not dot. The Atom API uses cat:cs.LG; OAI-PMH uses set=cs:cs:LG. Using set=cs.LG returns zero results.

• `http_get` in helpers.py does NOT follow HTTP redirects. If you must use it with the old OAI URL, you'll get an empty body. Either update the URL to the canonical one or use urllib.request.urlopen with a redirect handler.

---

How this complements the existing arxiv skill

ArXiv — Scraping & Data Extraction

https://arxiv.org — open-access preprint server. Never use the browser for ArXiv. All data is reachable via http_get using the Atom API or HTML meta tags. No API key required.

Do this first

Use the Atom API for any paper search or metadata fetch — one call, XML response, no auth.

import xml.etree.ElementTree as ET
from helpers import http_get

NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}

xml = http_get("http://export.arxiv.org/api/query?search_query=ti:transformer+AND+cat:cs.LG&max_results=5&sortBy=submittedDate&sortOrder=descending")
root = ET.fromstring(xml)
entries = root.findall('atom:entry', NS)

Use id_list for known paper IDs — supports comma-separated batch fetch in a single call.

Use http_get on https://arxiv.org/abs/{id} + regex for citation_* meta tags when you need the full abstract from an HTML page.

Common workflows

Search papers (API)

import xml.etree.ElementTree as ET
from helpers import http_get

NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}

xml = http_get(
    "http://export.arxiv.org/api/query"
    "?search_query=ti:transformer+AND+cat:cs.LG"
    "&max_results=5&sortBy=submittedDate&sortOrder=descending"
)
root = ET.fromstring(xml)
entries = root.findall('atom:entry', NS)
for e in entries:
    title     = e.find('atom:title', NS).text.strip().replace('\n', ' ')
    arxiv_id  = e.find('atom:id', NS).text.split('/')[-1]   # e.g. '2604.15259v1'
    published = e.find('atom:published', NS).text[:10]       # '2026-04-16'
    updated   = e.find('atom:updated', NS).text[:10]
    abstract  = e.find('atom:summary', NS).text.strip()
    authors   = [a.find('atom:name', NS).text for a in e.findall('atom:author', NS)]
    cats      = [c.get('term') for c in e.findall('atom:category', NS)]
    primary   = e.find('arxiv:primary_category', NS).get('term')
    comment   = e.find('arxiv:comment', NS)
    pdf_link  = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('title') == 'pdf'), None)
    abs_link  = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('rel') == 'alternate'), None)
    print(arxiv_id, published, title[:60])
    print("  Authors:", authors[:2])
    print("  PDF:", pdf_link)
# Confirmed output (2026-04-18):
# 2604.15259v1 2026-04-16 Stability and Generalization in Looped Transformers
#   Authors: ['Asher Labovich']
#   PDF: https://arxiv.org/pdf/2604.15259v1

Fetch single paper by ID (API)

import xml.etree.ElementTree as ET
from helpers import http_get

NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}

xml = http_get("http://export.arxiv.org/api/query?id_list=1706.03762")
root = ET.fromstring(xml)
e = root.find('atom:entry', NS)
title      = e.find('atom:title', NS).text.strip()
abstract   = e.find('atom:summary', NS).text.strip()
categories = [c.get('term') for c in e.findall('atom:category', NS)]
pdf_link   = next((l.get('href') for l in e.findall('atom:link', NS) if l.get('title') == 'pdf'), None)
print("Title:", title)
print("Categories:", categories)
print("PDF:", pdf_link)
print("Abstract:", abstract[:200])
# Confirmed output:
# Title: Attention Is All You Need
# Categories: ['cs.CL', 'cs.LG']
# PDF: https://arxiv.org/pdf/1706.03762v7
# Abstract: The dominant sequence transduction models are based on complex recurrent...

Batch fetch by comma-separated IDs (single call — fast)

Fetching 10 IDs in one call takes ~2s. Prefer this over parallel single-ID fetches.

import xml.etree.ElementTree as ET
from helpers import http_get

NS = {'atom': 'http://www.w3.org/2005/Atom'}

ids = ['1706.03762', '1810.04805', '2005.14165']  # Transformer, BERT, GPT-3
xml = http_get(f"http://export.arxiv.org/api/query?id_list={','.join(ids)}&max_results={len(ids)}")
root = ET.fromstring(xml)
for e in root.findall('atom:entry', NS):
    arxiv_id  = e.find('atom:id', NS).text.split('/')[-1]
    title     = e.find('atom:title', NS).text.strip()
    published = e.find('atom:published', NS).text[:10]
    print(arxiv_id, published, title[:60])
# Confirmed output:
# 1512.03385v1 2015-12-10 Deep Residual Learning for Image Recognition
# 1706.03762v7 2017-06-12 Attention Is All You Need
# 2005.14165v4 2020-05-28 Language Models are Few-Shot Learners
# 1810.04805v2 2018-10-11 BERT: Pre-training of Deep Bidirectional Transformers...
# Note: order returned may differ from order requested

Parallel fetch (ThreadPoolExecutor for independent IDs)

Use only when IDs are not known upfront or when mixing with other work. For pure batch, single comma-separated id_list call is faster.

import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
from helpers import http_get

NS = {'atom': 'http://www.w3.org/2005/Atom'}

def fetch_paper(arxiv_id):
    xml = http_get(f"http://export.arxiv.org/api/query?id_list={arxiv_id}")
    root = ET.fromstring(xml)
    e = root.find('atom:entry', NS)
    if e is None:
        return None
    return {
        'id': arxiv_id,
        'title': e.find('atom:title', NS).text.strip(),
        'published': e.find('atom:published', NS).text[:10],
    }

ids = ['1706.03762', '1810.04805', '2005.14165']
with ThreadPoolExecutor(max_workers=3) as ex:
    papers = list(ex.map(fetch_paper, ids))
for p in papers:
    print(p['id'], p['published'], p['title'][:60])
# Confirmed working — max_workers=3 is safe; don't exceed 5 for continuous crawling

HTML abstract page — citation_* meta tags

Use this when you want the full abstract or the versionless PDF URL without parsing Atom XML.

import re
from helpers import http_get

html = http_get("https://arxiv.org/abs/1706.03762", headers={"User-Agent": "Mozilla/5.0"})
# HTML page is ~48 KB, fully static, no JS required

title   = re.search(r'<meta name="citation_title" content="([^"]+)"', html)
pdf_url = re.search(r'<meta name="citation_pdf_url" content="([^"]+)"', html)
authors = re.findall(r'<meta name="citation_author" content="([^"]+)"', html)
date    = re.search(r'<meta name="citation_date" content="([^"]+)"', html)
arxiv_id = re.search(r'<meta name="citation_arxiv_id" content="([^"]+)"', html)
abstract = re.search(r'<meta name="citation_abstract" content="([^"]+)"', html)

print("Title:", title.group(1) if title else None)
print("PDF:", pdf_url.group(1) if pdf_url else None)
print("Authors:", authors[:3])
print("Date:", date.group(1) if date else None)
print("ID:", arxiv_id.group(1) if arxiv_id else None)
# Confirmed output for 1706.03762:
# Title: Attention Is All You Need
# PDF: https://arxiv.org/pdf/1706.03762   (no version suffix — always latest)
# Authors: ['Vaswani, Ashish', 'Shazeer, Noam', 'Parmar, Niki']
# Date: 2017/06/12
# ID: 1706.03762

All citation_* meta tags present on the abs page:

• citation_title — paper title
• citation_author — one tag per author, format "Last, First"
• citation_date — submission date YYYY/MM/DD
• citation_online_date — latest version date YYYY/MM/DD
• citation_pdf_url — versionless PDF URL (redirects to latest)
• citation_arxiv_id — bare ID without version suffix
• citation_abstract — full abstract text

Category search with pagination

import xml.etree.ElementTree as ET
from helpers import http_get

NS = {
    'atom': 'http://www.w3.org/2005/Atom',
    'opensearch': 'http://a9.com/-/spec/opensearch/1.1/',
}

# Page 1
xml = http_get(
    "http://export.arxiv.org/api/query"
    "?search_query=cat:cs.AI"
    "&max_results=10&start=0&sortBy=lastUpdatedDate&sortOrder=descending"
)
root = ET.fromstring(xml)
total   = root.find('opensearch:totalResults', NS).text   # e.g. '172726'
start_i = root.find('opensearch:startIndex', NS).text
per_pg  = root.find('opensearch:itemsPerPage', NS).text
print(f"Total cs.AI papers: {total}")  # Confirmed: 172726 (2026-04-18)

entries = root.findall('atom:entry', NS)

# Page 2: increment start
xml2 = http_get(
    "http://export.arxiv.org/api/query"
    "?search_query=cat:cs.AI"
    "&max_results=10&start=10&sortBy=lastUpdatedDate&sortOrder=descending"
)

URL and ID reference

API base URL

http://export.arxiv.org/api/query

HTTPS also works: https://export.arxiv.org/api/query

Query parameters

Search field prefixes

PDF and abstract URL construction

import re

arxiv_id = "1706.03762v7"                        # from API atom:id field
bare_id = re.sub(r'v\d+$', '', arxiv_id)          # strip version: '1706.03762'

pdf_versioned   = f"https://arxiv.org/pdf/{arxiv_id}"   # specific version
pdf_latest      = f"https://arxiv.org/pdf/{bare_id}"    # always redirects to latest
abs_versioned   = f"https://arxiv.org/abs/{arxiv_id}"
abs_latest      = f"https://arxiv.org/abs/{bare_id}"

The API's atom:link[@title='pdf'] href includes the version suffix. The HTML citation_pdf_url meta tag does not — it always resolves to the latest.

Category codes (confirmed paper counts, 2026-04-18)

Full category taxonomy: https://arxiv.org/category_taxonomy

Gotchas

• Never use the browser for ArXiv. The abstract page (/abs/) and search results are fully server-side rendered static HTML. http_get is sufficient for everything including full abstracts, author lists, and PDF URLs.

• Always define the namespace dict. Without NS = {'atom': 'http://www.w3.org/2005/Atom', 'arxiv': 'http://arxiv.org/schemas/atom'}, findall('atom:entry') silently returns []. All ArXiv Atom elements live in the http://www.w3.org/2005/Atom namespace; ArXiv-specific fields (comment, primary_category, journal_ref, doi) live in http://arxiv.org/schemas/atom.

• Batch single `id_list` call is faster than ThreadPoolExecutor. A comma-separated id_list with 10 IDs resolved in one call (1.91s) vs. 10 separate ThreadPoolExecutor calls (6.34s). Use the batch form when you already have the IDs.

• `atom:id` contains a URL, not a bare ID. The element text is http://arxiv.org/abs/1706.03762v7 — always split on / and take [-1] to get the bare ID with version. Strip version with re.sub(r'v\d+$', '', id) if needed.

• Batch `id_list` returns entries in unpredictable order. When fetching 1706.03762,1810.04805,2005.14165, entries came back ordered by publication date, not by the order given in the request. Index by ID, not position.

• `max_results` must be set explicitly when using `id_list` batches. If you request 10 IDs but omit max_results, the API defaults to 10, which happens to work — but set it explicitly to len(ids) to be safe.

• Nonexistent IDs return zero entries, not an error. id_list=9999.99999 gives totalResults=0 and an empty atom:entry list. Always check len(entries) > 0 before accessing entries[0].

• `arxiv:comment` and `arxiv:journal_ref` / `arxiv:doi` may be absent. Not all papers have these fields. Use e.find('arxiv:comment', NS) and check if el is not None and el.text.

• Rate limit: 3 seconds between requests recommended for bulk crawling. In practice, rapid bursts of 10 individual requests complete in ~6s (avg 0.63s/req) without being blocked. For sustained crawls over hundreds of papers, insert time.sleep(3) between requests. The API does not return rate limit headers — it just starts slowing responses or returns HTTP 503 silently.

• `citation_author` tags are in `"Last, First"` format, not "First Last" like the Atom API. The Atom atom:author/atom:name field gives "First Last" order. Pick the format that matches your downstream use.

• The `arxiv:affiliation` sub-element of `atom:author` is rarely populated. Most institutional affiliations are absent from the API response even when listed on the paper. The HTML abs page doesn't expose them in meta tags either.

• `sortBy=relevance` applies only with `search_query`. Using sortBy=relevance with id_list has no effect — results still come back in date order.

• `max_results` cap is 2000 per call. For bulk harvesting of a category, use start offset pagination and add 3s sleep between pages. opensearch:totalResults tells you the total so you can compute how many pages are needed.

• HTML `citation_abstract` meta tag contains the full abstract. Unlike the Atom atom:summary which can have trailing whitespace and embedded newlines, the meta tag version is a single clean string — no .strip() needed.

Atlas — my.recruitwithatlas.com

Gated recruitment SaaS. Auth via Google SSO (WebAuthn/passkey). GraphQL backend at /graphql (NextAuth session cookie, credentials: 'include' from the tab).

Routes

Filters in URL

BD opportunities uses ?filters=[JSON] (URL-encoded). Example "Me" filter:

[{"id":"opportunity_owner","selectedOptions":[{"id":"<USER_UUID>","title":"Me","excludeFromSearch":false}]}]

Filter IDs seen: opportunity_owner, stage, industry, segment, conversion_probability.

Finding your own user UUID

• Apply a filter like "owner = Me" in /business-development/opportunities, then read selectedOptions[0].id out of the URL filters= param.
• Or: query { me { id email } } via the GraphQL endpoint (see below).
• User UUIDs are tenant-stable; keep them in a local secret store, not in this shared skill.

Stages (BD funnel)

Identified → Initial Outreach → Late Stage → Converted → Archived. Seen as tab labels on /business-development/opportunities.

Auth quirks

• Google SSO flows through accounts.google.com/signin/oauth/id?... — passkey / WebAuthn only, no password fallback visible.
• Session state lives in multiple cookies (JWE session + CSRF). Injecting only the JWE into a fresh Chrome profile is not sufficient for UI access — you land in a login loop. For UI work: log in once inside a persistent Chrome profile and let all cookies settle. For backend-only GraphQL calls: the __Secure-authjs.session-token JWE alone is enough when sent with cookie: __Secure-authjs.session-token=<jwe> from an external HTTP client.

GraphQL endpoint

POST https://my.recruitwithatlas.com/graphql using the tab's own cookies:

js("""
fetch('/graphql', {
  method: 'POST',
  headers: {'Content-Type': 'application/json', 'apollo-require-preflight': 'true'},
  credentials: 'include',
  body: JSON.stringify({query: 'query { me { id email } }'})
}).then(r => r.json()).then(j => JSON.stringify(j))
""")

This reuses the session cookies of the current tab — no JWE juggling needed when browsing from inside browser-harness.

Known mutations (verified against production schema, April 2026): opportunityCreate, opportunityUpdate, companyCreate, projectCreate, projectUpdate, opportunityAddLead, createOpportunityNote. Create mutations return placeholder names; follow with an opportunityUpdate / projectUpdate to set the final name or description. opportunityAddLead side-effects Project.company onto Opportunity.targetCompany when the opp had none.

Page titles

The app sets a green-dot emoji prefix on titles: 🟢 Atlas Agency (sign-in), 🟢 Business development (BD overview), etc. Useful for wait_for conditions — the emoji is consistent across routes.

Big Bang (bigbang.hr) — Checkout & GTM DataLayer

Big Bang is a Croatian electronics retailer. The site is a Nuxt.js (Vue 3 SSR) SPA with jQuery UI for some widgets. GTM container GTM-5F34ZXDL, GA4 property G-QEEZK92T3P.

URL patterns

• Homepage: https://www.bigbang.hr
• Product: https://www.bigbang.hr/webshop/<slug>/<sku>
• Cart: https://www.bigbang.hr/webshop/kosarica/
• Checkout: https://www.bigbang.hr/webshop/kupac/ (all 3 steps live on this URL — SPA routing)

Checkout flow (3 steps, same URL)

1. Podaci kupca (Customer data) — name, address, location, email, phone 2. Način dostave (Delivery method) — package, pickup, parcel locker 3. Odabir plaćanja i završetak kupnje (Payment selection) — card, KEKS Pay, bank transfer

Framework quirks — form filling

Vue's reactive data model ignores programmatic .value = changes. Two approaches that work:

Native input value setter (works for most fields)

function setVal(id, value) {
    const el = document.getElementById(id);
    const setter = Object.getOwnPropertyDescriptor(
        window.HTMLInputElement.prototype, "value"
    ).set;
    setter.call(el, value);
    el.dispatchEvent(new Event("input", { bubbles: true }));
    el.dispatchEvent(new Event("change", { bubbles: true }));
}
setVal("first_name", "Test");
setVal("last_name", "Korisnik");
setVal("address", "Testna ulica 1");
setVal("email", "test@example.com");
setVal("phone", "00385911234567");

Location autocomplete (jQuery UI — needs CDP keyboard input)

The #location field uses jQuery UI Autocomplete (ul.ui-autocomplete.locations-ui-autocomplete). The native setter trick does NOT work here — it won't trigger the autocomplete dropdown or populate the dependent #zipcode and #city fields.

Instead, use real keyboard input via CDP:

js("document.getElementById('location').focus()")
cdp("Input.insertText", text="10000")
# Wait for autocomplete dropdown to appear (~500ms)
# Select from dropdown: li.ui-menu-item inside ul.locations-ui-autocomplete
# Use getBoundingClientRect() on the first li, then click_at_xy()
# This auto-populates #zipcode and #city

Stable form field IDs

#first_name, #last_name, #address, #location, #zipcode, #city, #email, #phone

Continue buttons

The continue buttons are button elements. JS .click() works but coordinate clicks can miss due to sidebar layout shifts. Prefer:

[...document.querySelectorAll("button")]
    .find(b => b.textContent.includes("Nastavi"))
    .click()

Step 1 button text: "Nastavi na odabir načina dostave" Step 2 button text: "Nastavi na plaćanje" Final button text: "Potvrdi i naruči"

GTM dataLayer events

The dataLayer fires standard GA4 ecommerce events:

`add_payment_info` quirk: The event fires when the payment step loads, before the user selects a payment method. The payment_type in the payload reflects the default pre-selected option ("opca_uplatnica_hr" = bank transfer), not the user's choice. The event also fires multiple times (observed 4x) — likely duplicate GTM triggers.

DataLayer interceptor pattern

The SPA preserves JS state across checkout steps (client-side routing), so a push interceptor installed once survives all 3 steps:

window.__dlEvents = [];
// Seed dataLayer first — GTM may not have loaded yet when the snippet runs,
// in which case window.dataLayer is undefined and .push.bind would throw.
// Seeding with [] is safe: GTM picks up an existing array on init.
window.dataLayer = window.dataLayer || [];
const origPush = window.dataLayer.push.bind(window.dataLayer);
window.dataLayer.push = function() {
    for (let i = 0; i < arguments.length; i++) {
        const entry = arguments[i];
        if (entry && entry.event) {
            window.__dlEvents.push({
                event: entry.event,
                ecommerce: entry.ecommerce
                    ? JSON.stringify(entry.ecommerce).substring(0, 500)
                    : undefined,
                timestamp: new Date().toISOString()
            });
        }
    }
    return origPush.apply(window.dataLayer, arguments);
};

Read captured events: js("JSON.stringify(window.__dlEvents)")

Payment methods (step 3)

• Plaćanje karticama — card payment
• KEKS Pay — mobile payment
• Virmansko plaćanje — bank transfer (default, pre-selected)
• HT vrijednosni bon — expandable voucher section

Bilibili — Site Navigation & Structure

Field-tested against bilibili.com on 2026-05-01. Requires login for personal-space features; public pages are accessible without auth.

---

URL Patterns

Core pages

Personal space sub-pages

Discovery

Video page

Content platforms

Creator

Other

---

Top Navigation Bar

Horizontal nav across the top of bilibili.com:

首页 | 番剧 | 直播 | 游戏中心 | 会员购 | 漫画 | 赛事

These are always visible regardless of login state.

Left sidebar channels (分区)

On the homepage, the left sidebar lists 30 content channels. Each maps to bilibili.com/c/{SLUG} or a top-level path:

动态  热门  番剧  电影  国创  电视剧  综艺  纪录片
动画  游戏  鬼畜  音乐  舞蹈  影视  娱乐  知识
科技数码  资讯  美食  小剧场  汽车  时尚美妆
体育运动  动物  vlog  绘画  人工智能  家装房产
户外潮流  健身

Channel URLs:

• bilibili.com/c/{slug} — e.g. /c/douga (动画), /c/game, /c/music, /c/ai
• Some use full words: /c/knowledge, /c/information, /c/food, /c/fashion, /c/sports, /c/animal, /c/painting, /c/home, /c/outdoors, /c/gym
• Others use abbreviations: /c/kichiku (鬼畜), /c/ent, /c/tech
• Short play: /c/shortplay
• Car: /c/car (no trailing slash in source)

---

User Menu (right side of top bar, login required)

Dropdown accessible via avatar in top-right corner:

---

Personal Space Tabs (space.bilibili.com/{UID})

Sub-navigation within the personal space:

主页 | 动态 | 投稿 | 合集和系列 | 收藏 | 追番追剧 | 设置

• 主页 — video grid + favorite folders + stats
• 动态 — this user's activity feed (distinct from t.bilibili.com which is the following feed)
• 投稿 — published videos, sortable by: 最新发布 / 最多播放 / 最多收藏
• 合集和系列 — curated video collections
• 收藏 — favorite folders (public or private)
• 追番追剧 — tracked anime/drama
• 设置 — space configuration

User Stats (visible on personal space)

Selector: .nav-statistics contains .nav-statistics__item children.

Values are in .nav-statistics__item-num.

---

Video Page — Interaction Features

When on a watch page (bilibili.com/video/{BV_ID}), the toolbar below the player offers:

三连 (triple-tap) is Bilibili's signature interaction — long-pressing the like button sends a like, donates 1 coin, and adds to favorites simultaneously.

Coins (硬币)

• Users receive coins daily by logging in
• Coins are spent by "投币" on videos (1 or 2 per video)
• Coin count appears in the header user area
• Separate from B币 (B-Coins) which are purchased with real money

Danmaku (弹幕)

Real-time comments that scroll across the video. Toggle with the danmaku button in the player controls. Danmaku data is loaded via XHR after the video player initializes.

Charging (充电)

Monthly subscription / tipping to support creators. Accessible from the creator's space page or below the video player.

---

Favorites (/favlist)

Each favorite folder displays: name, video count, visibility (公开/仅自己可见).

Two sections on the page:

• 我创建的收藏夹 — user's own folders
• 我追的合集/收藏夹 — followed collections from other users

Actions:

• Create new folder: click "新建收藏夹"
• Set visibility: per-folder setting (公开 / 仅自己可见)
• Default folder is created automatically and holds all quick-favorited videos

---

Watch History (/account/history)

Features:

• Pause/resume recording — "暂停记录历史" / "继续记录历史"
• Clear all — "清空历史"
• Date filters — 今天 / 昨天 / 近1周 / 1周前 / 1个月前
• Each entry shows: title, progress (看到 XX:XX), uploader name, category tag

History items are in .history-record elements.

Watch Later (稍后再看)

Located at bilibili.com/watchlater/#/list. Also appears as a default folder in the favorites section on the personal space homepage.

---

Search (search.bilibili.com)

Search results are tabbed:

综合 | 视频 | 番剧 | 影视 | 直播 | 专栏 | 用户

Each tab shows a count badge (e.g., "视频99+"). Query param: ?keyword={QUERY}.

Autocomplete suggestions appear when typing in the search input.

---

Popular / Hot Page (/v/popular/all)

Tab bar:

综合热门 | 每周必看 | 入站必刷 | 排行榜 | 全站音乐榜

• 综合热门 — trending right now
• 每周必看 — weekly curated picks, URL: /v/popular/weekly?num={ISSUE}
• 入站必刷 — all-time classic videos
• 排行榜 — redirects to /v/popular/rank/all
• 全站音乐榜 — music-specific chart

Ranking (/v/popular/rank/all)

24 category tabs: 全部, 番剧, 国创, 纪录片, 电影, 电视剧, 综艺, 动画, 游戏, 鬼畜, 音乐, 舞蹈, 影视, 娱乐, 知识, 科技, 数码, 美食, 汽车, 时尚, 美妆, 体育, 运动, 动物

Each entry shows: rank number, title, creator, view count, interaction count.

---

Detecting Login State

# Logged in: avatar element exists
avatar = js("document.querySelector('.header-entry-avatar')?.src || 'not logged in'")

# Logged out: login button visible
login_btn = js("document.querySelector('.header-login-entry')?.textContent?.trim() || 'no login btn'")

Extract the current user's UID:

uid = js("document.querySelector('.header-entry-avatar')?.closest('a')?.href?.match(/space\\.bilibili\\.com\\/(\\d+)/)?.[1]")

---

Gotchas

• 动态 has two meanings — t.bilibili.com is the following feed (content from people you follow), while space.bilibili.com/{UID}/dynamic is a specific user's activity. They are different pages.
• Favorites URL redirects — navigating to /favlist may redirect to /favlist?fid={DEFAULT_FOLDER_ID}&ftype=create, which opens the first folder automatically.
• History can be paused — if the history page says "历史功能暂停中", recording was paused by the user. Click "继续记录历史" to resume.
• UID is numeric — Bilibili user IDs are all-numeric, unlike YouTube handles. The UID appears in the space URL and is stable.
• BV vs AV IDs — modern video IDs use the BV format (e.g. BV1HeRKBdEoX). Legacy AV format (e.g. av170001) still resolves but all new content uses BV.
• Video URL gets `?vd_source=` appended — when navigating from an authenticated session, bilibili appends a vd_source tracking parameter. This can be stripped.
• Watch later is separate from history — /watchlater/#/list is a single-page app path, not a sub-page of /account/history.
• Coins are not B-Coins — 硬币 (coins) are earned daily for free. B币 (B-Coins) are purchased with real money and used for tipping/charging/premium.
• Some channel URL slugs use abbreviations — 鬼畜 is /c/kichiku, 娱乐 is /c/ent, 科技 is /c/tech. Not all are pinyin or translated.
• `wait_for_load()` is not enough on video pages — like YouTube, the video player and its toolbar components hydrate after the load event. Add a wait(3) before querying video toolbar selectors.

BOSS直聘 — Chat & Messaging

Field-tested against zhipin.com on 2026-05-01. Login required. Messages are loaded via WebSocket + REST API.

IMPORTANT: Never send messages without the user's explicit permission. This skill documents the read/retrieval mechanics only.

---

Architecture

BOSS直聘 uses a hybrid messaging architecture:

• Conversation list — loaded via WebSocket (ws6.zhipin.com) on page load, NOT via REST
• Message history — REST API /wapi/zpchat/geek/historyMsg
• Real-time messages — WebSocket push from ws6.zhipin.com

---

Chat Page (/web/geek/chat)

Page Structure

Left panel:
  .chat-user.v2                  — filter bar + search input
    .label-list > ul
      li.selected                — active filter tab
      li                         — "未读(N)" shows count badge in <i>
      li > .ui-dropmenu          — "更多" dropdown (仅沟通/有交换/有面试/不感兴趣)
      li.filter-item             — "AI筛选" dropdown with natural language input
    .boss-search-input           — contact search (placeholder: "搜索30天内的联系人")
  .user-list
    .user-list-content
      .friend-content-warp
        .friend-content           — conversation item (click to open)
          .friend-content.friend-top  — pinned/top conversation

Right panel (visible after clicking a conversation):
  .chat-record                   — message history container
    .message-item.item-myself    — message sent by user
      .item-time > .time         — timestamp
      .message-content > .text   — message body
      .message-status.status-read — read receipt ("已读")
    .message-item.item-friend    — message from recruiter
      .item-time > .time
      .message-content > .text

Filter Tabs

Top-level tabs (.chat-user.v2 .label-list li):

"更多" dropdown (.more-label li):

"AI筛选" (.filter-item > .ui-dropmenu): Opens a panel with a <textarea> for natural language filter input (e.g. "后端开发 上海 高薪").

Clicking Filter Tabs

def click_filter(label_text):
    """Click a filter tab by its text label."""
    js(f"""
    (function() {{
        var labels = document.querySelectorAll('.chat-user .label-list li .label-name');
        for (var i = 0; i < labels.length; i++) {{
            if (labels[i].textContent.trim().indexOf('{label_text}') === 0) {{
                labels[i].closest('li').click();
                return true;
            }}
        }}
        return false;
    }})()
    """)
    wait(1)

def click_more_filter(label_text):
    """Click an option inside the '更多' dropdown."""
    # First open the dropdown
    click_filter("更多")
    wait(0.5)
    js(f"""
    (function() {{
        var items = document.querySelectorAll('.more-label li span');
        for (var i = 0; i < items.length; i++) {{
            if (items[i].textContent.trim() === '{label_text}') {{
                items[i].closest('li').click();
                return true;
            }}
        }}
        return false;
    }})()
    """)
    wait(1)

Conversation Item (DOM)

Each .friend-content contains:

• Timestamp (e.g. "04月13日", "昨天")
• Recruiter name (e.g. "刘女士")
• Company name (e.g. "Soul App")
• Recruiter title (e.g. "招聘专家")
• Last message preview
• Unread count badge (numeric)

Read Conversation List (DOM)

def get_conversations():
    raw = js("""
    (function() {
        var items = document.querySelectorAll('.friend-content');
        var results = [];
        for (var i = 0; i < items.length; i++) {
            var el = items[i];
            var text = el.textContent;
            var badge = el.querySelector('[class*="badge"], [class*="unread"], [class*="count"]');
            var unread = badge ? parseInt(badge.textContent) || 0 : 0;
            results.push({
                text: text.trim().substring(0, 150),
                is_top: el.classList.contains('friend-top'),
                unread: unread
            });
        }
        return JSON.stringify(results);
    })()
    """)
    return json.loads(raw)

Open a Conversation

Click the .friend-content element:

def open_conversation(index=0):
    js(f"document.querySelectorAll('.friend-content')[{index}].click()")
    wait(2)

---

API: Message History

GET /wapi/zpchat/geek/historyMsg?bossId={bossId}&maxMsgId=0&c=20&page=1&src=0

Parameters

The bossId can be found in performance entries after clicking a conversation, or extracted from the WebSocket connection data on page load.

Response (zpData.messages[])

Each message has:

{
    "mid": 337069469603329,              # message ID (numeric, for pagination)
    "type": 3,                           # 3=regular message, 4=system message
    "received": true,                    # whether you received it
    "body": {
        "type": 1,                       # 1=text, 8=job card
        "text": "message text here...",  # present when body.type=1
        "jobDesc": { ... }               # present when body.type=8
    },
    "from": {
        "uid": 502838021,                # sender user ID
        "name": "张女士",
        "avatar": "https://img.bosszhipin.com/..."
    },
    "to": {
        "uid": 680839465                 # recipient user ID
    }
}

Message Body Types

Job Card Messages (body.type=8)

{
    "body": {
        "type": 8,
        "headTitle": "您正在与Boss刘女士直接沟通如下职位",
        "jobDesc": {
            "title": "AI Agent工程师",
            "salary": "35-60K·16薪",         # REAL salary — not font-encoded
            "company": "Soul App",
            "city": "上海 浦东新区 金桥",
            "experience": "经验不限",
            "education": "硕士",
            "stage": "D轮及以上",
            "positionCategory": "算法工程师",
            "boss": {
                "uid": 3872648,
                "name": "刘女士",
                "avatar": "https://img.bosszhipin.com/..."
            },
            "bossTitle": "招聘专家",
            "jobId": 509933581
        }
    }
}

Fetch Message History

def fetch_messages(boss_id, page=1, count=20):
    raw = js(f"""
    (async function() {{
        var url = '/wapi/zpchat/geek/historyMsg?bossId={boss_id}&maxMsgId=0&c={count}&page={page}&src=0';
        var r = await fetch(url);
        var d = await r.json();
        if (d.code !== 0 || !d.zpData) {{
            return JSON.stringify({{code: d.code, hasMore: false, count: 0, messages: [], error: d.msg || 'API error'}});
        }}
        var msgs = d.zpData.messages || [];
        return JSON.stringify({{
            code: d.code,
            hasMore: d.zpData.hasMore,
            count: msgs.length,
            messages: msgs.map(function(m) {{
                var b = m.body || {{}};
                return {{
                    mid: m.mid,
                    type: m.type,
                    body_type: b.type,
                    text: b.text || null,
                    job: b.jobDesc ? {{
                        title: b.jobDesc.title,
                        salary: b.jobDesc.salary,
                        company: b.jobDesc.company,
                        city: b.jobDesc.city,
                        boss_name: (b.jobDesc.boss || {{}}).name,
                        job_id: b.jobDesc.jobId
                    }} : null,
                    from_name: (m.from || {{}}).name,
                    from_uid: (m.from || {{}}).uid,
                    received: m.received
                }};
            }})
        }});
    }})()
    """)
    return json.loads(raw)

Pagination

Use maxMsgId (not page) for efficient pagination. Set maxMsgId to the smallest mid from the previous batch:

def fetch_all_messages(boss_id):
    all_msgs = []
    max_msg_id = 0
    while True:
        raw = js(f"""
        (async function() {{
            var r = await fetch('/wapi/zpchat/geek/historyMsg?bossId={boss_id}&maxMsgId={max_msg_id}&c=20&page=1&src=0');
            var d = await r.json();
            if (d.code !== 0 || !d.zpData) {{
                return JSON.stringify({{messages: [], hasMore: false}});
            }}
            return JSON.stringify(d.zpData);
        }})()
        """)
        data = json.loads(raw)
        msgs = data.get("messages", [])
        if not msgs:
            break
        all_msgs.extend(msgs)
        if not data.get("hasMore"):
            break
        max_msg_id = msgs[-1]["mid"]  # smallest mid
        wait(0.5)
    return all_msgs

---

Messages Read from DOM (after opening a conversation)

def read_messages_dom():
    raw = js("""
    (function() {
        var items = document.querySelectorAll('.message-item');
        var results = [];
        for (var i = 0; i < items.length; i++) {
            var el = items[i];
            var timeEl = el.querySelector('.time');
            var textEl = el.querySelector('.text');
            var statusEl = el.querySelector('.message-status');
            results.push({
                from_me: el.classList.contains('item-myself'),
                time: timeEl ? timeEl.textContent.trim() : '',
                text: textEl ? textEl.textContent.trim().substring(0, 300) : '',
                status: statusEl ? statusEl.textContent.trim() : ''
            });
        }
        return JSON.stringify(results);
    })()
    """)
    return json.loads(raw)

---

Extracting bossId from the Page

The bossId is embedded in WebSocket payloads and API calls. To discover it after clicking a conversation:

def get_current_boss_id():
    return js("""
    (function() {
        var entries = performance.getEntriesByType('resource');
        for (var i = entries.length - 1; i >= 0; i--) {
            var url = entries[i].name;
            if (url.indexOf('/wapi/zpchat/geek/historyMsg') === -1) continue;
            var match = url.match(/bossId=([^&]+)/);
            if (match) return match[1];
        }
        return null;
    })()
    """)

---

Navigating from Job Detail to Chat

Opening a job detail page and clicking "立即沟通" initiates a conversation with that job's recruiter. The API needed:

1. Navigate to /job_detail/{JOB_ID}.html 2. Find the chat button (.btn-startchat) element 3. The button's href or click handler contains the bossId and securityId

---

Gotchas

• Conversation list is WebSocket-loaded — no REST API for the list. Use DOM extraction (.friend-content) or monitor WebSocket frames to get the initial conversation list.
• Message history uses `bossId`, not `encryptBossId` — the bossId format is "9c833990a839f1251Hx92du5GA~~" (trailing ~~), different from the job list's encryptBossId.
• `maxMsgId` pagination — use the smallest mid from the current batch for the next page, not page parameter.
• Job cards in messages have real salary — body.jobDesc.salary returns "35-60K·16薪" unlike the DOM which uses font-encoded digits.
• System messages (type=4) — these include read receipts, file transfers ("对方已同意，您的附件简历已发送给对方"), and competitor analysis cards.
• After clicking a conversation, `wait(2)` — the message history needs time to render.
• `item-myself` vs `item-friend` — user messages have item-myself class, recruiter messages have item-friend.
• Contact search input — .boss-search-input searches within 30 days of contacts, not a general message compose box.

BOSS直聘 — Site Navigation & Structure

Field-tested against zhipin.com on 2026-05-01.

---

URL Patterns

Special channels

---

Top Navigation Bar

BOSS直聘 | 首页 | 职位 | 公司 | 校园 | 海归 | APP | 有了 | 海外 | 无障碍专区

Always visible. First item (BOSS直聘 logo) links to root domain.

---

User Menu (login required)

Dropdown on the right side of the top bar. Shows user's real name when logged in. Entries include:

• 消息 — chat with recruiters
• 简历 — resume management
• 升级VIP — paid membership
• 规则中心 — platform rules
• 切换为招聘者/切换为求职者 — dual-mode switch between job-seeker and recruiter

---

Home Page

Root URL redirects to city-specific page based on IP (e.g. /shanghai/). Shows industry category selector and recommended jobs.

Industry Categories (top-level)

互联网/AI, 电子/电气/通信, 产品, 客服/运营, 销售, 人力/行政/法务, 财务/审计/税务, 生产制造, etc.

Each expands to sub-specialties (e.g. 互联网/AI → Java, Python, 前端, AI工程师...).

Search Bar

"input[placeholder='搜索职位、公司']"

---

Gotchas

• Root URL redirects to city — zhipin.com → zhipin.com/{city}/ based on IP. Always check final URL after navigation.
• Dual-mode accounts — same account switches between job-seeker and recruiter. UI changes completely.
• Search is SPA-based — /web/geek/jobs uses client-side routing. URL params don't reflect active filters.
• city slug is pinyin — /shanghai/, /beijing/, /shenzhen/, /hangzhou/, etc. (English transliteration, not Chinese characters). Note: the job search API uses numeric city codes (e.g. city=101020100), not pinyin slugs — see the city code table in job-search.md.
• `wait_for_load()` may not be enough — heavy SPA, add wait(2) for hydration.

Shopify embedded apps run in iframes

Every Shopify app surfaced in the admin (first-party like Knowledge Base, third-party like Okendo) renders inside a sandboxed iframe. Your top-level document queries find the Shopify chrome (sidebar, header, search bar) but none of the app's UI.

How to target the iframe

from helpers import iframe_target, js, type_text

# 1. Find the iframe by URL substring
tid = iframe_target("qa-pairs-app")  # Knowledge Base App

# 2. Run JS inside the iframe by passing target_id
result = js("""
(() => {
  const button = Array.from(document.querySelectorAll('button')).find(b => b.textContent.trim() === 'Add FAQ');
  if (button) { button.click(); return {clicked: true}; }
  return {clicked: false};
})()
""", target_id=tid)

Finding the URL substring

The iframe's URL contains the app slug. Run:

import json
for t in cdp("Target.getTargets")["targetInfos"]:
    if t["type"] == "iframe" and "shopify" in t.get("url", "").lower():
        print(t["url"])

Then pick a substring unique to your target app.

Known Shopify app iframe slugs

Add to this table when you discover new ones.

Why iframes

Shopify uses App Bridge to embed third-party apps with isolation. Your top-level page CAN'T directly access app DOM for security reasons — you need iframe targeting (which the harness does via CDP Target.attachToTarget).

Coordinate clicks vs JS clicks

Coordinate clicks (click(x, y)) pass through iframes at the compositor level — they work. But JS clicks scoped to the iframe target are more reliable for routine button taps because:

• Element text content is stable across UI redesigns
• DPR scaling on retina is automatic
• React event handlers are guaranteed to fire (vs. CDP mouse events which sometimes hit a transparent layer above the button)

Gotcha — multiple iframes from same app

The Online Store editor renders the storefront preview AND the editor toolbar in two separate iframes. Pick the right one by URL substring; don't assume the first match is correct.

# WRONG — picks first match
tid = iframe_target("online-store-web")

# RIGHT — disambiguate
for t in cdp("Target.getTargets")["targetInfos"]:
    url = t.get("url", "")
    if "online-store-web" in url and "editor" in url:
        tid = t["targetId"]
        break