Product Analytics
Self-hosted posthog-js analytics on Cloudflare: insights, flags, replay, surveys, SQL, error tracking, LLM analytics, usage caps.
This is a web app spec on Spekkr, published by dwrtz. It is listed with the other specs. Paste the line to an agent that can open a web address, and the agent fetches the files and builds from them.
- Spec
dwrtz/product-analytics- Version
- 0.6.0
- Published by
- dwrtz
- Updated
- 20 Nov 2026
Build this spec
Build the spec at https://spekkr.com/dwrtz/product-analytics/start.md.
Paste this to your AI agent. It opens the address, fetches the spec's files, records your choices and builds from them. Nothing to install, no account.
Published by dwrtz.
What does the result look like?
What can I customize?
Your choices go into the line you copy. Anything else is yours to edit, but it leaves the tested path.
What do I get?
-
spec.mdRequirements R1-R32 (0.2: event shards, replay, experiments, surveys; 0.3: actions, cohorts, heatmaps, SQL, local evaluation, continuity, survey targeting, replay settings, no row limits; 0.4: optional Basin / Iceberg cold tier queried from SQL; 0.5: error tracking with source maps, usage caps, sampling, usage page, cost estimate and alerts; 0.6: LLM analytics (R31), public demo (R32), releases, merge/split, assignment, grouping and suppression rules, retroactive symbolication, server stacks, cold Trends, request and dropped-past-cap counters) with acceptance checks, scope and definition of done. -
protocol.mdThe normative SDK wire format: capture routes, body decoding, tokens, timestamps, v1 capture, persons and merges, feature flags, remote config, replay snapshots and settings, surveys and targeting, exposure, heatmaps, local evaluation, experience continuity, exceptions, quota limiting, error codes, LLM analytics capture (AI endpoints), 0.6 exception releases, server SDK stacks and suppression rules.SDK wire format
This is the part of PostHog's public ingestion and flags API that the posthog-js and posthog-node SDKs use, as this instance must implement it. It is normative: a build that follows it accepts unmodified SDKs.
test-vectors.jsonholds worked examples. Section references in brackets point to the PostHog MIT source the rule was read from (commit and paths inresearch.md); "observed" means seen in recorded SDK traffic.1. Endpoints
Path (trailing slash optional) Methods Used by Section /e,/i/v0/ePOST, GET posthog-js events (observed: /e/before remote config loads,/i/v0/e/after)2-5 /batch,/capture,/track,/engagePOST, GET older SDKs, server SDKs, imports 2-5 /i/v0/ai/batchPOST, GET PostHog AI SDKs; legacy bodies, $ai_*events only (0.6)2-5, 21 /i/v0/aiPOST PostHog AI SDKs, large payloads: multipart (0.6) 21 /i/v1/ai/eventsPOST v1 AI senders (0.6) 6, 21 /i/v1/analytics/eventsPOST posthog-node 6 and other v1 senders 6 /flags,/decidePOST, GET flag evaluation ( /flags/?v=2from posthog-js,?v=4from posthog-node)8 /array/<token>/config,/array/<token>/config.jsGET posthog-js remote config 9 /static/<file>,/static/<version>/<file>GET the snippet and lazy-loaded SDK bundles 9 /s,/i/v0/sPOST posthog-js session recorder snapshots (observed: /s/)11 /api/surveysGET posthog-js surveys ( ?token=)12 /flags/definitions,/api/feature_flag/local_evaluationGET posthog-node local evaluation (personal API key) 15 /api/web_experiments,/api/early_access_featuresany out-of-scope products 9 any of the above OPTIONS CORS preflight: 204 1 Every response to these paths carries
Access-Control-Max-Age: 86400,Access-Control-Allow-Origin: <request Origin or *>, and, when an Origin was sent,Access-Control-Allow-Credentials: trueandVary: Origin; preflights also allowGET, POST, OPTIONSand echoAccess-Control-Request-Headers.2. Legacy capture bodies [rust/capture/src/v0_request.rs]
After decoding (section 4) the body is JSON in one of these shapes:
- An array of events.
- An object with
batch(array of events) and optionallyapi_keyortoken,sent_at(ISO time) andhistorical_migration. - A single event object (has a string
event). - On
/engageonly: an object with$setor$set_onceand noevent, which is treated as one$identifyevent.
Anything else is 400
invalid_payload. An empty event list is 400empty_batch. An event is an object withevent(non-empty string, at most 200 characters), a distinct id (distinct_id, orproperties.distinct_id, or$distinct_id; a string of 1-200 characters, a number is turned into its string), and optionaluuid,timestamp,offset(ms),properties,$set,$set_once.$performance_event,$snapshotand$snapshot_itemsare dropped.The response is
{"status":1}with 200, or an empty 204 when the query hasbeacon=1.3. Project token [rust/capture/src/token.rs]
The token is found, in order, in the body's
api_keyortoken, the first event'sapi_key,tokenorproperties.token, then the query'stokenorapi_key. Missing: 401no_token. It is valid when it is 1-64 characters, ASCII, has no NUL byte and does not start withphx_(a personal API key); otherwise 401invalid_token. A valid token that names no project is accepted with the normal success response and its events are dropped (PostHog drops them later in its pipeline; SDKs see success).Error bodies are JSON
{"type", "code", "detail", "attr": null}wheretypeisauthentication_errorfor 401 andvalidation_errorotherwise. SDKs read only the status.4. Body decoding [rust/capture/src/payload/decompression.rs]
Apply these steps in order to the raw request bytes:
- If the bytes start with the gzip magic
1f 8b, orContent-Encodingisgzip, gunzip them, whatever thecompressionhint says. (posthog-js 1.450 sends gzip withContent-Type: text/plainand nocompressionparameter: observed.) Acompression=gzip/gzip-jshint on bytes without the magic is ignored (plain text).Content-Encoding: deflateinflates. - Decode as UTF-8 (invalid: 400
invalid_payload). - If
Content-Typeisapplication/x-www-form-urlencodedor the text starts withdata=, parse it as a form and continue with thedatafield; a formcompressionfield replaces the query hint. - If the hint is
lz64orlz-stringand the text does not start with{or[, decompress with LZStringdecompressFromBase64. - If the text still does not start with
{or[and is base64 (standard or URL alphabet), decode it. - Parse JSON (failure: 400
invalid_payload).
For GET, the bytes are the
dataquery parameter. Decoded bodies over 20 MB are 413payload_too_large. The query parameter_is the client's send time in ms and is used assent_atwhen the body has none and it is after 2021-01-01.5. Event normalization
For each event, with
nowthe server's receive time:- uuid: kept if it is a UUID (lower-cased), otherwise a new UUIDv7 from the event time. A uuid already stored for the project is dropped (deduplication).
Timestamp [rust/common/types/src/timestamp.rs]:
- Parse
timestampas RFC 3339 / ISO 8601. A trailing+HHor-HHzone gets:00added; a time with no zone is UTC; digits after milliseconds are dropped. - If it parsed and
sent_atis known (and not ignored): event time = timestamp − (sent_at − now) (clock-skew correction). If it parsed withoutsent_at: the timestamp. Otherwise:now. - If
offset(ms) is present andnow − offsetis a representable date (chrono's range, years −262143 to 262142), event time =now − offset; otherwise the offset is ignored. - More than 23 hours after
now: usenow. A year outside 0-9999: the Unix epoch.sent_atis ignored when the event hasproperties.$ignore_sent_at: true(legacy) oroptions.disable_skew_correction: true(v1).
- Parse
- Person processing:
options.process_person_profile(v1) orproperties.$process_person_profile(falsemeans no person). Stored back on the event as$process_person_profile. - Person updates:
$setand$set_onceare read from the event's top level and fromproperties(top level wins key by key);$unset(array of names) fromproperties. They are kept on the event inpropertiestoo. - v1 fields:
session_idandwindow_idbecome$session_idand$window_idif those are not already set. properties.tokenandproperties.distinct_idare removed.- IP and location:
$ipis set fromCF-Connecting-IPunless the event has one or the project discards IPs (then$ipis removed). Unless the event has$geoip_disable: trueor already carries any$geoip_*property, Cloudflare's request data adds$geoip_country_code,$geoip_country_name,$geoip_city_name,$geoip_subdivision_1_name,$geoip_subdivision_1_code,$geoip_postal_code,$geoip_continent_code,$geoip_time_zone,$geoip_latitude,$geoip_longitude(only those present). - Library: when
$libis missing and the request hasPostHog-Sdk-Info: <lib>/<version>, set$liband$lib_version.
6. Capture v1 [rust/capture/src/v1/README.md]
POST /i/v1/analytics/eventsAuthorization: Bearer <project token>. Missing or unparseable header: 401invalid_api_key; a scheme other than Bearer: 400invalid_auth_scheme; an invalid token (section 3): 401.Content-Encodingabsent,gzipordeflate; anything else is 415unsupported_encoding.- Body:
{"created_at": ISO time, "batch": [event…], "capture_internal"?: bool, "historical_migration"?: bool}. Each event:event,uuid(must be a UUID if present, else 400invalid_uuid),distinct_id,timestamp,properties,options(process_person_profile,cookieless_mode,disable_skew_correction,product_tour_id),session_id,window_id.created_atis the batch'ssent_at. An empty or missing batch, or a body that is not JSON, is 400invalid_payload. capture_internal: truesets the event IP to127.0.0.1.- Response 200:
{"results": {"<uuid>": {"result": "ok" | "drop", "details"?: string, "duplicates_dropped"?: n}}}. A repeated uuid in the same batch is reported once, withduplicates_dropped. posthog-node treats any other body as "unreadable response" (observed with{"status":1}).
Case Status codeempty body, or GET without data400 no_dataempty event list (legacy) 400 empty_batchbody not decodable / not JSON 400 invalid_payloadno token 401 no_tokeninvalid token 401 invalid_token(v1:invalid_api_key)v1 scheme other than Bearer 400 invalid_auth_schemev1 uuid not a UUID 400 invalid_uuiddecoded body over 20 MB 413 payload_too_largev1 Content-Encoding not gzip/deflate 415 unsupported_encoding7. Persons and groups [nodejs/src/ingestion/common/persons, PoE.md]
Processing happens per project, in event order within a delivered batch, only for events with person processing on (section 5).
Merge events.
$identifymergesproperties.$anon_distinct_idinto the event's distinct id;$create_aliasand$merge_dangerouslymergeproperties.alias. With target = event distinct id and other = that id (both legal, different):- neither has a person: create one person with both ids, identified;
- one has a person: add the other id to it, mark it identified;
- both have the same person: mark it identified;
- different persons: if the other's person is identified and the event is not
$merge_dangerously, do not merge (mark target identified); otherwise move all of the other person's ids to the target person, merge properties (target's values win), keep the earliercreated_atand the laterlast_seen_at, delete the other person. $identifywith no anonymous id: mark the target's person identified, creating it if needed.
- Illegal ids never take part in a merge: the empty or all-space string; case-insensitively
anonymous,guest,distinctid,distinct_id,id,not_authenticated,email,undefined,true,false; case-sensitively[object Object],NaN,None,none,null,0,undefined; and each of these wrapped in single or double quotes. - Properties. Then, on the target's person (created if missing):
$set_oncekeys not yet present are set,$setkeys are set,$unsetkeys are removed. - Personless events (
$process_person_profile: false) create nothing, but once their distinct id is linked to a person (for example by a later$identifywith that$anon_distinct_id), queries count them for that person. - Groups.
$groupidentifywith$group_typeand$group_keycreates or updates the group, merging$group_setinto its properties. A project has at most 5 group types (indexes 0-4, in first-seen order); more are ignored. Group membership of other events is read from theirproperties.$groups({"company": "acme"}), which the SDKs add. posthog-node sends$groupidentifywith distinct id$<type>_<key>and person processing on, so such a person exists, as in PostHog. - Every event updates its person's
last_seen_at.
8. Feature flags [rust/feature-flags/src/flags, properties]
Request
POST /flags(queryvany value;config=truealso returns remote config) or/decide. The body is decoded as in section 4 (posthog-js sends it gzipped astext/plain). Fields:token(orapi_key, or the query),distinct_id,person_properties,groups,group_properties,flag_keys_to_evaluate,$anon_distinct_id,$device_id,timezone. Missing or invalid token, or a token for no project: 401{"type":"authentication_error","code":"invalid_api_key",...}. Undecodable body: 400malformed_data.Evaluation
Properties = the stored properties of the distinct id's person, overlaid by
person_propertiesfrom the request. Only active flags are evaluated (onlyflag_keys_to_evaluatewhen given). For each flag, condition sets in order:- Every property filter in the set must match (below).
- Rollout:
rollout_percentagemissing or null means 100. 100 matches; otherwisehash(key + ".", distinct_id, "") <= pct / 100. - The first set that matches both decides: enabled,
condition_match,condition_index= its index. A set that matched its filters but not the rollout leavesout_of_rollout_boundas the reason if no later set matches; otherwise the reason isno_condition_match. - Variant (multivariate flags): the set's
variantoverride if it names a variant; else withh = hash(key + ".", distinct_id, "variant"), walk the variants in order summingrollout_percentage / 100and take the first whose running sum is greater thanh.
hash(prefix, id, salt)= take SHA-1 of the UTF-8 stringprefix + id + salt, read the first 15 hex digits as an integer and divide by0xfffffffffffffff(vectorsflag_hash).Property filters (
{key, value, operator, type}), with values compared as their string form (JSON for objects) unless stated:operator matches when exact(default)the value equals, case-insensitively, any of value(a list or one value)is_notit equals none of them icontains/not_icontainsit contains / does not contain value, case-insensitivelyregex/not_regexit matches / does not match valueas a regular expression (an invalid pattern never matches)gtgteltlteboth are numbers (or numeric strings) and compare so; otherwise no match is_set/is_not_setthe property is present / absent A missing property matches only
is_not,not_icontains,not_regexandis_not_set.Response (200)
{ "flags": { "<key>": { "key": "<key>", "enabled": true, "variant": "test", "reason": { "code": "condition_match", "condition_index": 0, "description": "Matched condition set 1" }, "metadata": { "id": 7, "version": 3, "description": "<flag name>", "payload": "{\"color\":\"green\"}" } } }, "featureFlags": { "<key>": "test" }, "featureFlagPayloads": { "<key>": "{\"color\":\"green\"}" }, "errorsWhileComputingFlags": false, "requestId": "<uuid v7>", "evaluatedAt": 1795010769773 }featureFlags[key]is the variant,true, orfalse.- Payloads are JSON strings: the flag's
payloads[variant], orpayloads["true"]for a flag without variants; only for enabled flags. errorsWhileComputingFlagsis true when there was no distinct id./decide, and/flags?config=true, add the remote config fields of section 9 andconfig: {"enable_collect_everything": <autocapture on>}.- posthog-node 6 asks for
v=4and accepts this v2-shaped answer (it logs an info line once: observed).
9. Remote config, SDK files and stubs
GET /array/<token>/config(JSON) and/array/<token>/config.js(JavaScript that setswindow._POSTHOG_REMOTE_CONFIG[<token>] = {config: …, siteApps: []}), cacheable for 5 minutes; an unknown token is 404.{ "token": "<token>", "supportedCompression": ["gzip", "gzip-js"], "hasFeatureFlags": true, "analytics": { "endpoint": "/i/v0/e/" }, "elementsChainAsString": true, "defaultIdentifiedOnly": true, "autocapture_opt_out": false, "autocaptureExceptions": false, "errorTracking": { "autocaptureExceptions": false, "suppressionRules": [], "captureExtensionExceptions": false }, "captureDeadClicks": false, "capturePerformance": false, "sessionRecording": false, "heatmaps": false, "surveys": false, "productTours": false, "siteApps": [] }hasFeatureFlagsis whether the project has an active flag.autocapture_opt_outis the project setting.surveysistruewhile the project has a running survey (12).sessionRecordingisfalse, or, when the project records sessions, the recorder config posthog-js reads (fields from posthog-jspackages/browser-common/src/types/remote-config.tsSessionRecordingRemoteConfig; observed working with posthog-js 1.450.1):{ "endpoint": "/s/", "consoleLogRecordingEnabled": true, "recorderVersion": "v2", "sampleRate": null, "minimumDurationMilliseconds": null, "linkedFlag": null, "networkPayloadCapture": null, "masking": null, "urlTriggers": [], "urlBlocklist": [], "eventTriggers": [], "triggerMatchType": null, "scriptConfig": null }The
/flagsresponse withconfig=true(and/decide) carries the same two fields. 0.5:autocaptureExceptionsanderrorTracking.autocaptureExceptionsare the project's error tracking switch (19); both, andsessionRecording, arefalsewhile the project's events (or replay) are capped in drop mode (20). 0.3: the project's replay settings fill these fields (18);heatmapsandcaptureDeadClicksare the project's heatmaps setting (14);capturePerformanceis{"network_timing": true, "web_vitals": false}when network capture is on (18), elsefalse./static/<file>serves the posthog-jsdist/*.jsfiles of the version the build installed (plus its licence), with CORS and a one-hour cache;/static/<version>/<file>serves the same file for any version (posthog-js asks for/static/<its version>/surveys.jsfirst: observed). posthog-js loads extension bundles such assurveys.jseven when the remote config turns the product off, so every bundle must be present.Stubs:
/api/web_experiments→{"experiments":[]},/api/early_access_features→{"earlyAccessFeatures":[]}.10. SDK behaviour to expect (informative, observed)
- posthog-js drops all events from user agents it classifies as bots, including headless Chrome, unless
opt_out_useragent_filter: true. Tests that drive a headless browser must set it. - posthog-js loads
config.js, then sends events (batched, gzip,text/plain) to/e/until remote config arrives and to the config'sanalytics.endpoint(/i/v0/e/) after. $identifyevents carry$setand$set_onceat the top level andproperties.$anon_distinct_id;people.setsends a$setevent withproperties.$set;group()sends$groupidentifyand adds$groupsto later events;alias()sends$create_aliaswithproperties.alias.- With
person_profiles: 'identified_only'(the default), events beforeidentify()have$process_person_profile: false. - Autocapture sends
$elements_chainas a string.$pageleaveon navigation is sent withfetch(…, {keepalive: true}), notsendBeacon. - posthog-node 6 posts v1 batches (gzip,
application/json) and calls/flags/?v=4.
11. Session recording snapshots [posthog-js packages/browser/src/extensions/replay; observed]
posthog-js loads the recorder from
/static/<version>/lazy-recorder.js(scriptConfig.scriptwould rename it; this instance sendsnull) once remote config has asessionRecordingobject, then POSTs batches to the config'sendpoint(/s/). Observed with posthog-js 1.450.1: gzip body,Content-Type: text/plain, no query string; decode as section 4.The body is a JSON array of events (a
{"batch": [...]}object or a single event is also accepted):[{ "uuid": "01a1ef4e-…", "event": "$snapshot", "timestamp": "2026-…Z", "properties": { "$snapshot_data": [ <rrweb event>, … ], "$snapshot_bytes": 27654, "$session_id": "01a1ef4e-98a2-…", "$window_id": "01a1ef4e-98a2-…", "distinct_id": "user-42", "token": "<project token>", "$lib": "web", "$lib_version": "1.450.1", "$snapshot_host": "…" } }]- The token is
properties.token(else the body'sapi_key/token, else?token=).$snapshot_itemsis accepted as an older name for$snapshot_data. Events other than$snapshot/$snapshot_itemsare ignored.$session_idmust match[A-Za-z0-9._:-]{1,200}. - The answer is always
{"status":1}(200) once the body decodes, also when the project does not record sessions (the data is dropped); a body that does not decode is 400invalid_payload, over 20 MB 413. - rrweb events are
{type, data, timestamp}: 2 full snapshot, 3 incremental (data.source: 0 mutation, 1 mouse move, 2 mouse interaction wheredata.type2 is a click, 3 scroll, 5 input, 8 stylesheet rule), 4 meta (data.href,width,height), 5 custom (posthog-js sends$remote_config_received,$session_options,$posthog_config,$recording_started, and$pageviewon navigation), 6 plugin (rrweb/console@1withpayload.level). - Field compression (observed on every full snapshot and mutation): an event with
"cv": "2024-10"carries some fields as a string whose characters are the bytes (0-255) of a gzip stream of the field's JSON:dataof a type 2 event;texts,attributes,removesandaddsof a type 3 source 0 event;addsandremovesof a type 3 source 8 event. Fields that are already arrays or objects are left as they are. A reader gunzips them, parses the JSON and dropscv. - Inputs are masked by posthog-js before sending (observed:
"text": "*******"for a password and for text fields); the server stores what it receives. - Regular events carry
$session_idand$window_id, which link them to the recording.
12. Surveys [posthog-js packages/browser-common/src/types/surveys.ts, packages/core/src/surveys/events.ts; observed]
posthog-js fetches
GET /api/surveys/?token=<token>&_=<ms>(observed) when remote config sayssurveys: true, and lazy-loads/static/<version>/surveys.js. A missing or malformed token is 401missing_api_key, an unknown one 401invalid_api_key. The answer:{ "surveys": [ { "id": "<uuid>", "name": "NPS", "description": "", "type": "popover", "questions": [ { "id": "q-rating", "type": "rating", "question": "How likely…?", "description": "", "optional": false, "display": "number", "scale": 10, "lowerBoundLabel": "Unlikely", "upperBoundLabel": "Very likely" }, { "id": "q-choice", "type": "single_choice", "question": "What brought you here?", "choices": ["Search", "Friend", "Ad"], "hasOpenChoice": false, "shuffleOptions": false }, { "id": "q-open", "type": "open", "question": "Anything else?" } ], "appearance": { "whiteLabel": true }, "conditions": { "events": { "values": [ { "name": "opened_pricing" } ] } }, "start_date": "2026-…Z", "end_date": null, "linked_flag_key": null, "targeting_flag_key": null, "internal_targeting_flag_key": null, "feature_flag_keys": [], "current_iteration": null, "current_iteration_start_date": null, "schedule": "once", "enable_partial_responses": false } ] }Only running surveys (launched, not stopped, not archived) are listed; posthog-js itself applies
conditions(urlwithurlMatchType,selector,events,deviceTypes) and remembers per browser which surveys it showed.appearance.whiteLabel: truehides posthog-js's "Survey by PostHog" footer.Events posthog-js sends (observed), on the normal capture path:
survey shown:$survey_id,$survey_name,$survey_iteration,$survey_iteration_start_date.survey dismissed: the same, plus$survey_partially_completedwhen some questions were answered.survey sent: the same, plus$survey_submission_id(one id per person's pass through the survey),$survey_completed,$survey_questions([{id, question, response}]) and one$survey_response_<question id>per answer (a number for ratings, a string for single choice and open, an array for multiple choice), and a top-level$set: {"$survey_responded/<survey id>": true}. Older SDKs send$survey_responsefor the first question and$survey_response_<index>for later ones; readers accept both.
13. Experiment exposure [observed]
posthog.getFeatureFlag(key)andisFeatureEnabled(key)send$feature_flag_calledthe first time a flag is read in a page load (and again if its value changes), with$feature_flag(the key),$feature_flag_response(variant string or boolean),$feature_flag_id,$feature_flag_version,$feature_flag_reason,$feature_flag_request_id,$feature_flag_payloadand$feature_flag_bootstrapped_response(observed). posthog-node'sgetFeatureFlag/evaluateFlags().getFlagsends the same event from the server. Experiments use the first such event per person as the exposure (design.md "Experiments").14. Heatmaps [posthog-js packages/browser/src/heatmaps.ts; observed]
With
heatmaps: truein remote config, posthog-js (observed 1.450.1) buffers pointer data and sends it about every 5 s, and on page leave, as a normal event on the capture path (section 1):{ "event": "$$heatmap", "properties": { "$heatmap_data": { "<page url>": [ { "x": 312, "y": 188, "target_fixed": false, "type": "click" } ] }, "$viewport_width": 1280, "$viewport_height": 760, "$current_url": "<page url>", "$pathname": "/", … } }typeisclick,rageclick(3 clicks within 1 s and 30 px),deadclick(from the lazily loadeddead-clicks-autocapture.jswhencaptureDeadClicksis on) ormousemove.x,yare CSS pixels; the page scroll is already added toyunlesstarget_fixed(the target is in a fixed-position element).- The instance aggregates
x / $viewport_width(0-1, 1 % steps) andyin 16 px rows pertype, URL and viewport width range.
Scroll depth needs no heatmap setting:
$pageleave(and the next$pageviewin single-page apps) carries$prev_pageview_pathname,$prev_pageview_last_scroll,$prev_pageview_last_scroll_percentage,$prev_pageview_max_scroll,$prev_pageview_max_scroll_percentage,$prev_pageview_last_content(_percentage)and$prev_pageview_max_content(_percentage); percentages are 0-1.15. Local flag evaluation [posthog-node src/extensions/feature-flags; observed with posthog-node 6.0.3]
With a personal API key posthog-node polls (default every 30 s):
GET /flags/definitions?token=<phc_…>&send_cohorts Authorization: Bearer <phx_…> If-None-Match: "<etag of the last 200>"(older SDKs call
/api/feature_flag/local_evaluation?token=…&send_cohorts; both are served, and/api/projects/:id/feature_flags/local_evaluationfor API users). No or a bad personal key is 401{"type": "authentication_error", …}; an unknown token 401. 200:{ "flags": [ { "id": 1, "team_id": null, "key": "new-cart", "name": "…", "active": true, "deleted": false, "ensure_experience_continuity": false, "version": 3, "filters": { "aggregation_group_type_index": null, "groups": [ … ], "multivariate": null, "payloads": {} } } ], "group_type_mapping": { "0": "company" }, "cohorts": { "7": { "type": "OR", "values": [ { "type": "AND", "values": [ { "key": "plan", "type": "person", "value": ["pro"], "operator": "exact" } ] } ] } } }with an
ETag(SHA-1 of the body) and 304 for a matchingIf-None-Match.cohorts(only withsend_cohorts) holds the property-only cohorts flags reference, in flag-property form. Flags that reference a behavioural or static cohort name a cohort id missing fromcohorts, and flags withensure_experience_continuityare flagged by the SDK itself; posthog-node then evaluates those through/flags(section 8). Locally evaluated$feature_flag_calledevents carrylocally_evaluated: trueand$feature_flag_reason: "Evaluated locally".16. Experience continuity [posthog-js; rust/feature-flags hash key overrides]
After
identify(), posthog-js's next/flagsbody carries"$anon_distinct_id": "<previous anonymous id>"once (observed). For each active flag withensure_experience_continuity, the instance stores a hash key override(distinct_id, flag_key) → hash_key = $anon_distinct_idfor both the new and the anonymous distinct id, unless one exists. Evaluating such a flag for any distinct id of a person uses the override's hash key in place of the distinct id in the rollout and variant hashes (section 8). Overrides follow person merges and are deleted with the person.17. Survey targeting, iterations and partial responses [posthog-js packages/browser/src/extensions/surveys; observed]
GET /api/surveys/?token=(section 12) adds per survey:{ "linked_flag_key": "new-cart", "targeting_flag_key": "survey-targeting-9cadf132b610", "internal_targeting_flag_key": "survey-targeting-9cadf132b610-custom", "feature_flag_keys": [ { "key": "flag", "value": "new-cart" }, { "key": "targeting_flag", "value": "…" }, { "key": "internal_targeting_flag", "value": "…-custom" } ], "conditions": { "url": "/pricing", "urlMatchType": "icontains", "selector": "#cta", "deviceTypes": ["Desktop"], "deviceTypesMatchType": "exact", "seenSurveyWaitPeriodInDays": 14, "linkedFlagVariant": "test", "events": { "values": [ { "name": "opened_pricing" } ], "repeatedActivation": false }, "cancelEvents": { "values": [ { "name": "closed_cart" } ] }, "actions": { "values": [ { "id": 3, "name": "Added to cart", "steps": [ … ] } ] } }, "current_iteration": 2, "current_iteration_start_date": "…", "schedule": "recurring", "enable_partial_responses": true }- posthog-js checks the three flags itself through
/flags, so the instance keeps them as real flags, hidden from the flag list: the targeting flag has the survey'stargeting_flag_filters; the internal flag matches persons where$survey_dismissed/<id>[/<iteration>]and$survey_responded/<id>[/<iteration>]are not set (posthog-js sets them with$seton dismiss and send).urlMatchTypeis one ofexact,is_not,icontains,not_icontains,regex,not_regex. - Iterations:
current_iteration = min(iteration_count, floor((now - start_date) / iteration_frequency_days) + 1); posthog-js adds$survey_iterationand$survey_iteration_start_dateto survey events. - Partial responses: posthog-js sends one
survey sentper answered question with the same$survey_submission_id, cumulative$survey_response_<question id>values,$survey_completed: falseuntil the last; closing early sendssurvey abandonedwith$survey_partially_completed: true.
18. Replay settings [posthog-js packages/browser-common/src/types/remote-config.ts; observed]
The
sessionRecordingobject of section 9, filled from the project's replay settings:setting field sample rate 0-1 sampleRateas a two-decimal string ("0.25"),null= allminimum duration minimumDurationMillisecondslinked flag linkedFlag:"key"or{"flag": "key", "variant": "v"}console logs consoleLogRecordingEnablednetwork headers / bodies networkPayloadCapture: {"recordHeaders", "recordBody"}network timing top-level capturePerformance: {"network_timing": true, …}URL triggers / blocklist urlTriggers/urlBlocklist:[{"url": "<regex>", "matching": "regex"}]event triggers eventTriggers: ["<event>"], `triggerMatchType: "any""all"` masking masking: {"maskAllInputs", "maskTextSelector", "blockSelector"}canvas recordCanvas: true,canvasFps,canvasQuality("0.4")Resulting rrweb events (observed): console as type 6
{"plugin": "rrweb/console@1", "payload": {"level", "trace": [], "payload": [<JSON strings>]}}; network as type 6{"plugin": "rrweb/network@1", "payload": {"requests": [PerformanceResourceTiming fields + "method", "status", headers and bodies when enabled]}}; posthog-js turns the network recorder off when the page's host islocalhostor127.0.0.1. Canvas as type 3 source 9 (canvas mutation) with base64 webp image data; the player needs canvas replay turned on.19. Exceptions [posthog-js packages/browser/src/extensions/exception-autocapture, posthog-node src/extensions/error-tracking; observed with posthog-js 1.450.1 and posthog-node 6.0.3]
$exceptionis an ordinary event on the capture endpoints (posthog-js:/i/v0/e/; posthog-node:/i/v1/analytics/events). WithautocaptureExceptionstruthy in the remote config posthog-js loads/static/exception-autocapture.jsand captureswindow.onerrorandunhandledrejection;captureException(error, props)works either way. Observed properties (tests/fixtures/v05-errors.json):{ "$exception_list": [ { "type": "TypeError", "value": "Price must be a number, got string", "mechanism": { "type": "generic", "handled": false, "synthetic": false, "exception_id": 0 }, "stacktrace": { "type": "raw", "frames": [ { "platform": "web:javascript", "filename": "http://host/app.min.js", "function": "r", "in_app": true, "lineno": 1, "colno": 206 }, { "platform": "web:javascript", "filename": "http://host/app.min.js", "function": "n", "in_app": true, "lineno": 1, "colno": 49 } ] } } ], "$exception_level": "error", "$exception_capture_enabled_server_side": true, "$exception_fingerprint": "qty-range" }- The list's first entry is the outer exception; frames are ordered outermost first, so the frame that threw is the last one. Line and column are 1-based.
- posthog-node frames carry
"platform": "node:javascript", amodule,in_app: falsefornode:internals and dependencies, and already includepre_context/context_line/post_contextread from disk. $exception_fingerprintis optional (string; a list is accepted).
The server adds to the stored event:
$exception_fingerprint(computed unless sent,test-vectors.json"fingerprints"),$exception_issue_id,$exception_type/$exception_messagefrom the first entry when missing, and, for frames resolved with an uploaded source map,stacktrace.type: "resolved"and per frameresolved: true, the originalfilename/lineno/colno/function, context lines andminified: {filename, function, lineno, colno}.20. Quota limiting [posthog-js src/rate-limiter.ts; observed]
A capture response may carry
quota_limited, a list of batch keys. posthog-js then drops new events for those keys for 60 seconds without sending them (observed: after one limited answer, four further clicks and acaptureExceptionsent nothing). Batch keys:events(ordinary events),exceptionEvent($exception),recordings(/s/snapshots).endpoint, project capped in drop mode response /e/,/i/v0/e/,/batch/, …{"status": 1, "quota_limited": ["events", "exceptionEvent"]}/i/v1/analytics/events{"results": {…}, "quota_limited": ["events", "exceptionEvent"]}/s/{"status": 1, "quota_limited": ["recordings"]}In sample mode the answer stays
{"status": 1}and a deterministic share is kept (test-vectors.json"sampling"). Events dropped because of a cap are accepted with status 200 like events of an unknown project.0.6: events, requests and replay uploads refused this way are counted per project ("Dropped past cap" on the Usage screen); the answer is unchanged.
21. LLM analytics capture [PostHog docs "LLM analytics", posthog-js packages/ai; observed shapes]
AI events are ordinary events with reserved names and properties. Names:
$ai_generation(one model call),$ai_embedding,$ai_span(a step),$ai_trace(the whole run); other$ai_*names are stored as sent. Properties read by this instance (all optional):property meaning $ai_trace_idgroups a run; a call without one is its own trace $ai_span_id,$ai_parent_idtree links; a parent equal to the trace id is top level $ai_span_name,$ai_trace_namedisplay name $ai_model,$ai_providere.g. gpt-4o-mini,openai$ai_input_tokens,$ai_output_tokenstoken counts (OpenAI-style input includes cached tokens) $ai_cache_read_input_tokens,$ai_cache_creation_input_tokenscache tokens (Anthropic reports them apart from input) $ai_total_cost_usd,$ai_input_cost_usd,$ai_output_cost_usdSDK-computed cost; wins over list prices $ai_latencyseconds $ai_is_error,$ai_error,$ai_http_statuserrors (status ≥ 400 counts as an error) $ai_input,$ai_output_choicesmessages in and out (any JSON) $ai_input_state,$ai_output_statea trace's or span's input and output Endpoints (in addition to every capture path in section 1):
POST /i/v0/ai/batch/: legacy bodies (section 2); events whose name does not start with$ai_are dropped (counted as misrouted), the rest are normalized like/batch. Answer{"status": 1}(orquota_limited, section 20).POST /i/v1/ai/events: v1 body (section 6); a non-AI event's result is{"result": "drop", "details": "misrouted_event"}.POST /i/v0/ai:Authorization: Bearer <project token>(401invalid_api_keyotherwise),Content-Type: multipart/form-data, optionalContent-Encoding: gzip, at most 25 MB. Exactly one part namedevent(JSON object withevent,distinct_id, optionaluuid,timestamp,properties) and at most one part namedevent.properties(JSON object merged over the event's properties). Any other part name, a secondevent, a non-AI event (misrouted_event) or a bad uuid (invalid_uuid) is 400 with the section 6 error shape. Success is{"status": 1}.
22. Exceptions 0.6: releases, server SDKs, suppression rules [posthog-js @posthog/browser-common remote-config types; posthog-python, posthog-go, posthog-rs]
- Release: the first non-empty of
$release,release,$app_version,app_version(test-vectors.json"releases"). - Server SDK
$exception_listframes carry their ownplatform(python,go,rust, …),filename,abs_path,function,module,lineno, optionalcolno,in_appand, from posthog-python,pre_context/context_line/post_context. They arrive symbolicated; the server never applies source maps to them. - Remote config
errorTracking.suppressionRulesis a list of{"type": "AND" | "OR", "values": [{"key": "$exception_types" | "$exception_values", "operator": "exact" | "is_not" | "icontains" | "not_icontains" | "regex" | "not_regex", "value": string | string[], "type": "error_tracking_issue_property"}]}. posthog-js evaluates it before sending an exception; the server applies the same rules at ingest (test-vectors.json"suppression_rules").
# SDK wire format This is the part of PostHog's public ingestion and flags API that the posthog-js and posthog-node SDKs use, as this instance must implement it. It is normative: a build that follows it accepts unmodified SDKs. `test-vectors.json` holds worked examples. Section references in brackets point to the PostHog MIT source the rule was read from (commit and paths in `research.md`); "observed" means seen in recorded SDK traffic. ## 1. Endpoints | Path (trailing slash optional) | Methods | Used by | Section | |---|---|---|---| | `/e`, `/i/v0/e` | POST, GET | posthog-js events (observed: `/e/` before remote config loads, `/i/v0/e/` after) | 2-5 | | `/batch`, `/capture`, `/track`, `/engage` | POST, GET | older SDKs, server SDKs, imports | 2-5 | | `/i/v0/ai/batch` | POST, GET | PostHog AI SDKs; legacy bodies, `$ai_*` events only (0.6) | 2-5, 21 | | `/i/v0/ai` | POST | PostHog AI SDKs, large payloads: multipart (0.6) | 21 | | `/i/v1/ai/events` | POST | v1 AI senders (0.6) | 6, 21 | | `/i/v1/analytics/events` | POST | posthog-node 6 and other v1 senders | 6 | | `/flags`, `/decide` | POST, GET | flag evaluation (`/flags/?v=2` from posthog-js, `?v=4` from posthog-node) | 8 | | `/array/<token>/config`, `/array/<token>/config.js` | GET | posthog-js remote config | 9 | | `/static/<file>`, `/static/<version>/<file>` | GET | the snippet and lazy-loaded SDK bundles | 9 | | `/s`, `/i/v0/s` | POST | posthog-js session recorder snapshots (observed: `/s/`) | 11 | | `/api/surveys` | GET | posthog-js surveys (`?token=`) | 12 | | `/flags/definitions`, `/api/feature_flag/local_evaluation` | GET | posthog-node local evaluation (personal API key) | 15 | | `/api/web_experiments`, `/api/early_access_features` | any | out-of-scope products | 9 | | any of the above | OPTIONS | CORS preflight: 204 | 1 | Every response to these paths carries `Access-Control-Max-Age: 86400`, `Access-Control-Allow-Origin: <request Origin or *>`, and, when an Origin was sent, `Access-Control-Allow-Credentials: true` and `Vary: Origin`; preflights also allow `GET, POST, OPTIONS` and echo `Access-Control-Request-Headers`. ## 2. Legacy capture bodies [rust/capture/src/v0_request.rs] After decoding (section 4) the body is JSON in one of these shapes: 1. An array of events. 2. An object with `batch` (array of events) and optionally `api_key` or `token`, `sent_at` (ISO time) and `historical_migration`. 3. A single event object (has a string `event`). 4. On `/engage` only: an object with `$set` or `$set_once` and no `event`, which is treated as one `$identify` event. Anything else is 400 `invalid_payload`. An empty event list is 400 `empty_batch`. An event is an object with `event` (non-empty string, at most 200 characters), a distinct id (`distinct_id`, or `properties.distinct_id`, or `$distinct_id`; a string of 1-200 characters, a number is turned into its string), and optional `uuid`, `timestamp`, `offset` (ms), `properties`, `$set`, `$set_once`. `$performance_event`, `$snapshot` and `$snapshot_items` are dropped. The response is `{"status":1}` with 200, or an empty 204 when the query has `beacon=1`. ## 3. Project token [rust/capture/src/token.rs] The token is found, in order, in the body's `api_key` or `token`, the first event's `api_key`, `token` or `properties.token`, then the query's `token` or `api_key`. Missing: 401 `no_token`. It is valid when it is 1-64 characters, ASCII, has no NUL byte and does not start with `phx_` (a personal API key); otherwise 401 `invalid_token`. A valid token that names no project is accepted with the normal success response and its events are dropped (PostHog drops them later in its pipeline; SDKs see success). Error bodies are JSON `{"type", "code", "detail", "attr": null}` where `type` is `authentication_error` for 401 and `validation_error` otherwise. SDKs read only the status. ## 4. Body decoding [rust/capture/src/payload/decompression.rs] Apply these steps in order to the raw request bytes: 1. If the bytes start with the gzip magic `1f 8b`, or `Content-Encoding` is `gzip`, gunzip them, whatever the `compression` hint says. (posthog-js 1.450 sends gzip with `Content-Type: text/plain` and **no** `compression` parameter: observed.) A `compression=gzip`/`gzip-js` hint on bytes without the magic is ignored (plain text). `Content-Encoding: deflate` inflates. 2. Decode as UTF-8 (invalid: 400 `invalid_payload`). 3. If `Content-Type` is `application/x-www-form-urlencoded` or the text starts with `data=`, parse it as a form and continue with the `data` field; a form `compression` field replaces the query hint. 4. If the hint is `lz64` or `lz-string` and the text does not start with `{` or `[`, decompress with LZString `decompressFromBase64`. 5. If the text still does not start with `{` or `[` and is base64 (standard or URL alphabet), decode it. 6. Parse JSON (failure: 400 `invalid_payload`). For GET, the bytes are the `data` query parameter. Decoded bodies over 20 MB are 413 `payload_too_large`. The query parameter `_` is the client's send time in ms and is used as `sent_at` when the body has none and it is after 2021-01-01. ## 5. Event normalization For each event, with `now` the server's receive time: - **uuid**: kept if it is a UUID (lower-cased), otherwise a new UUIDv7 from the event time. A uuid already stored for the project is dropped (deduplication). - **Timestamp** [rust/common/types/src/timestamp.rs]: 1. Parse `timestamp` as RFC 3339 / ISO 8601. A trailing `+HH` or `-HH` zone gets `:00` added; a time with no zone is UTC; digits after milliseconds are dropped. 2. If it parsed and `sent_at` is known (and not ignored): event time = timestamp − (sent_at − now) (clock-skew correction). If it parsed without `sent_at`: the timestamp. Otherwise: `now`. 3. If `offset` (ms) is present and `now − offset` is a representable date (chrono's range, years −262143 to 262142), event time = `now − offset`; otherwise the offset is ignored. 4. More than 23 hours after `now`: use `now`. A year outside 0-9999: the Unix epoch. `sent_at` is ignored when the event has `properties.$ignore_sent_at: true` (legacy) or `options.disable_skew_correction: true` (v1). - **Person processing**: `options.process_person_profile` (v1) or `properties.$process_person_profile` (`false` means no person). Stored back on the event as `$process_person_profile`. - **Person updates**: `$set` and `$set_once` are read from the event's top level and from `properties` (top level wins key by key); `$unset` (array of names) from `properties`. They are kept on the event in `properties` too. - **v1 fields**: `session_id` and `window_id` become `$session_id` and `$window_id` if those are not already set. - `properties.token` and `properties.distinct_id` are removed. - **IP and location**: `$ip` is set from `CF-Connecting-IP` unless the event has one or the project discards IPs (then `$ip` is removed). Unless the event has `$geoip_disable: true` or already carries any `$geoip_*` property, Cloudflare's request data adds `$geoip_country_code`, `$geoip_country_name`, `$geoip_city_name`, `$geoip_subdivision_1_name`, `$geoip_subdivision_1_code`, `$geoip_postal_code`, `$geoip_continent_code`, `$geoip_time_zone`, `$geoip_latitude`, `$geoip_longitude` (only those present). - **Library**: when `$lib` is missing and the request has `PostHog-Sdk-Info: <lib>/<version>`, set `$lib` and `$lib_version`. ## 6. Capture v1 [rust/capture/src/v1/README.md] `POST /i/v1/analytics/events` - `Authorization: Bearer <project token>`. Missing or unparseable header: 401 `invalid_api_key`; a scheme other than Bearer: 400 `invalid_auth_scheme`; an invalid token (section 3): 401. - `Content-Encoding` absent, `gzip` or `deflate`; anything else is 415 `unsupported_encoding`. - Body: `{"created_at": ISO time, "batch": [event…], "capture_internal"?: bool, "historical_migration"?: bool}`. Each event: `event`, `uuid` (must be a UUID if present, else 400 `invalid_uuid`), `distinct_id`, `timestamp`, `properties`, `options` (`process_person_profile`, `cookieless_mode`, `disable_skew_correction`, `product_tour_id`), `session_id`, `window_id`. `created_at` is the batch's `sent_at`. An empty or missing batch, or a body that is not JSON, is 400 `invalid_payload`. - `capture_internal: true` sets the event IP to `127.0.0.1`. - Response 200: `{"results": {"<uuid>": {"result": "ok" | "drop", "details"?: string, "duplicates_dropped"?: n}}}`. A repeated uuid in the same batch is reported once, with `duplicates_dropped`. posthog-node treats any other body as "unreadable response" (observed with `{"status":1}`). | Case | Status | `code` | |---|---|---| | empty body, or GET without `data` | 400 | `no_data` | | empty event list (legacy) | 400 | `empty_batch` | | body not decodable / not JSON | 400 | `invalid_payload` | | no token | 401 | `no_token` | | invalid token | 401 | `invalid_token` (v1: `invalid_api_key`) | | v1 scheme other than Bearer | 400 | `invalid_auth_scheme` | | v1 uuid not a UUID | 400 | `invalid_uuid` | | decoded body over 20 MB | 413 | `payload_too_large` | | v1 Content-Encoding not gzip/deflate | 415 | `unsupported_encoding` | ## 7. Persons and groups [nodejs/src/ingestion/common/persons, PoE.md] Processing happens per project, in event order within a delivered batch, only for events with person processing on (section 5). - **Merge events.** `$identify` merges `properties.$anon_distinct_id` into the event's distinct id; `$create_alias` and `$merge_dangerously` merge `properties.alias`. With target = event distinct id and other = that id (both legal, different): - neither has a person: create one person with both ids, identified; - one has a person: add the other id to it, mark it identified; - both have the same person: mark it identified; - different persons: if the other's person is identified and the event is not `$merge_dangerously`, do not merge (mark target identified); otherwise move all of the other person's ids to the target person, merge properties (target's values win), keep the earlier `created_at` and the later `last_seen_at`, delete the other person. - `$identify` with no anonymous id: mark the target's person identified, creating it if needed. - **Illegal ids** never take part in a merge: the empty or all-space string; case-insensitively `anonymous`, `guest`, `distinctid`, `distinct_id`, `id`, `not_authenticated`, `email`, `undefined`, `true`, `false`; case-sensitively `[object Object]`, `NaN`, `None`, `none`, `null`, `0`, `undefined`; and each of these wrapped in single or double quotes. - **Properties.** Then, on the target's person (created if missing): `$set_once` keys not yet present are set, `$set` keys are set, `$unset` keys are removed. - **Personless events** (`$process_person_profile: false`) create nothing, but once their distinct id is linked to a person (for example by a later `$identify` with that `$anon_distinct_id`), queries count them for that person. - **Groups.** `$groupidentify` with `$group_type` and `$group_key` creates or updates the group, merging `$group_set` into its properties. A project has at most 5 group types (indexes 0-4, in first-seen order); more are ignored. Group membership of other events is read from their `properties.$groups` (`{"company": "acme"}`), which the SDKs add. posthog-node sends `$groupidentify` with distinct id `$<type>_<key>` and person processing on, so such a person exists, as in PostHog. - Every event updates its person's `last_seen_at`. ## 8. Feature flags [rust/feature-flags/src/flags, properties] ### Request `POST /flags` (query `v` any value; `config=true` also returns remote config) or `/decide`. The body is decoded as in section 4 (posthog-js sends it gzipped as `text/plain`). Fields: `token` (or `api_key`, or the query), `distinct_id`, `person_properties`, `groups`, `group_properties`, `flag_keys_to_evaluate`, `$anon_distinct_id`, `$device_id`, `timezone`. Missing or invalid token, or a token for no project: 401 `{"type":"authentication_error","code":"invalid_api_key",...}`. Undecodable body: 400 `malformed_data`. ### Evaluation Properties = the stored properties of the distinct id's person, overlaid by `person_properties` from the request. Only active flags are evaluated (only `flag_keys_to_evaluate` when given). For each flag, condition sets in order: 1. Every property filter in the set must match (below). 2. Rollout: `rollout_percentage` missing or null means 100. 100 matches; otherwise `hash(key + ".", distinct_id, "") <= pct / 100`. 3. The first set that matches both decides: enabled, `condition_match`, `condition_index` = its index. A set that matched its filters but not the rollout leaves `out_of_rollout_bound` as the reason if no later set matches; otherwise the reason is `no_condition_match`. 4. Variant (multivariate flags): the set's `variant` override if it names a variant; else with `h = hash(key + ".", distinct_id, "variant")`, walk the variants in order summing `rollout_percentage / 100` and take the first whose running sum is greater than `h`. `hash(prefix, id, salt)` = take SHA-1 of the UTF-8 string `prefix + id + salt`, read the first 15 hex digits as an integer and divide by `0xfffffffffffffff` (vectors `flag_hash`). Property filters (`{key, value, operator, type}`), with values compared as their string form (JSON for objects) unless stated: | operator | matches when | |---|---| | `exact` (default) | the value equals, case-insensitively, any of `value` (a list or one value) | | `is_not` | it equals none of them | | `icontains` / `not_icontains` | it contains / does not contain `value`, case-insensitively | | `regex` / `not_regex` | it matches / does not match `value` as a regular expression (an invalid pattern never matches) | | `gt` `gte` `lt` `lte` | both are numbers (or numeric strings) and compare so; otherwise no match | | `is_set` / `is_not_set` | the property is present / absent | A missing property matches only `is_not`, `not_icontains`, `not_regex` and `is_not_set`. ### Response (200) ```json { "flags": { "<key>": { "key": "<key>", "enabled": true, "variant": "test", "reason": { "code": "condition_match", "condition_index": 0, "description": "Matched condition set 1" }, "metadata": { "id": 7, "version": 3, "description": "<flag name>", "payload": "{\"color\":\"green\"}" } } }, "featureFlags": { "<key>": "test" }, "featureFlagPayloads": { "<key>": "{\"color\":\"green\"}" }, "errorsWhileComputingFlags": false, "requestId": "<uuid v7>", "evaluatedAt": 1795010769773 } ``` - `featureFlags[key]` is the variant, `true`, or `false`. - Payloads are JSON **strings**: the flag's `payloads[variant]`, or `payloads["true"]` for a flag without variants; only for enabled flags. - `errorsWhileComputingFlags` is true when there was no distinct id. - `/decide`, and `/flags?config=true`, add the remote config fields of section 9 and `config: {"enable_collect_everything": <autocapture on>}`. - posthog-node 6 asks for `v=4` and accepts this v2-shaped answer (it logs an info line once: observed). ## 9. Remote config, SDK files and stubs `GET /array/<token>/config` (JSON) and `/array/<token>/config.js` (JavaScript that sets `window._POSTHOG_REMOTE_CONFIG[<token>] = {config: …, siteApps: []}`), cacheable for 5 minutes; an unknown token is 404. ```json { "token": "<token>", "supportedCompression": ["gzip", "gzip-js"], "hasFeatureFlags": true, "analytics": { "endpoint": "/i/v0/e/" }, "elementsChainAsString": true, "defaultIdentifiedOnly": true, "autocapture_opt_out": false, "autocaptureExceptions": false, "errorTracking": { "autocaptureExceptions": false, "suppressionRules": [], "captureExtensionExceptions": false }, "captureDeadClicks": false, "capturePerformance": false, "sessionRecording": false, "heatmaps": false, "surveys": false, "productTours": false, "siteApps": [] } ``` `hasFeatureFlags` is whether the project has an active flag. `autocapture_opt_out` is the project setting. `surveys` is `true` while the project has a running survey (12). `sessionRecording` is `false`, or, when the project records sessions, the recorder config posthog-js reads (fields from posthog-js `packages/browser-common/src/types/remote-config.ts` `SessionRecordingRemoteConfig`; observed working with posthog-js 1.450.1): ```json { "endpoint": "/s/", "consoleLogRecordingEnabled": true, "recorderVersion": "v2", "sampleRate": null, "minimumDurationMilliseconds": null, "linkedFlag": null, "networkPayloadCapture": null, "masking": null, "urlTriggers": [], "urlBlocklist": [], "eventTriggers": [], "triggerMatchType": null, "scriptConfig": null } ``` The `/flags` response with `config=true` (and `/decide`) carries the same two fields. 0.5: `autocaptureExceptions` and `errorTracking.autocaptureExceptions` are the project's error tracking switch (19); both, and `sessionRecording`, are `false` while the project's events (or replay) are capped in drop mode (20). 0.3: the project's replay settings fill these fields (18); `heatmaps` and `captureDeadClicks` are the project's heatmaps setting (14); `capturePerformance` is `{"network_timing": true, "web_vitals": false}` when network capture is on (18), else `false`. `/static/<file>` serves the posthog-js `dist/*.js` files of the version the build installed (plus its licence), with CORS and a one-hour cache; `/static/<version>/<file>` serves the same file for any version (posthog-js asks for `/static/<its version>/surveys.js` first: observed). posthog-js loads extension bundles such as `surveys.js` even when the remote config turns the product off, so every bundle must be present. Stubs: `/api/web_experiments` → `{"experiments":[]}`, `/api/early_access_features` → `{"earlyAccessFeatures":[]}`. ## 10. SDK behaviour to expect (informative, observed) - posthog-js drops all events from user agents it classifies as bots, including headless Chrome, unless `opt_out_useragent_filter: true`. Tests that drive a headless browser must set it. - posthog-js loads `config.js`, then sends events (batched, gzip, `text/plain`) to `/e/` until remote config arrives and to the config's `analytics.endpoint` (`/i/v0/e/`) after. - `$identify` events carry `$set` and `$set_once` at the **top level** and `properties.$anon_distinct_id`; `people.set` sends a `$set` event with `properties.$set`; `group()` sends `$groupidentify` and adds `$groups` to later events; `alias()` sends `$create_alias` with `properties.alias`. - With `person_profiles: 'identified_only'` (the default), events before `identify()` have `$process_person_profile: false`. - Autocapture sends `$elements_chain` as a string. `$pageleave` on navigation is sent with `fetch(…, {keepalive: true})`, not `sendBeacon`. - posthog-node 6 posts v1 batches (gzip, `application/json`) and calls `/flags/?v=4`. ## 11. Session recording snapshots [posthog-js packages/browser/src/extensions/replay; observed] posthog-js loads the recorder from `/static/<version>/lazy-recorder.js` (`scriptConfig.script` would rename it; this instance sends `null`) once remote config has a `sessionRecording` object, then POSTs batches to the config's `endpoint` (`/s/`). Observed with posthog-js 1.450.1: gzip body, `Content-Type: text/plain`, no query string; decode as section 4. The body is a JSON array of events (a `{"batch": [...]}` object or a single event is also accepted): ```json [{ "uuid": "01a1ef4e-…", "event": "$snapshot", "timestamp": "2026-…Z", "properties": { "$snapshot_data": [ <rrweb event>, … ], "$snapshot_bytes": 27654, "$session_id": "01a1ef4e-98a2-…", "$window_id": "01a1ef4e-98a2-…", "distinct_id": "user-42", "token": "<project token>", "$lib": "web", "$lib_version": "1.450.1", "$snapshot_host": "…" } }] ``` - The token is `properties.token` (else the body's `api_key`/`token`, else `?token=`). `$snapshot_items` is accepted as an older name for `$snapshot_data`. Events other than `$snapshot`/`$snapshot_items` are ignored. `$session_id` must match `[A-Za-z0-9._:-]{1,200}`. - The answer is always `{"status":1}` (200) once the body decodes, also when the project does not record sessions (the data is dropped); a body that does not decode is 400 `invalid_payload`, over 20 MB 413. - rrweb events are `{type, data, timestamp}`: 2 full snapshot, 3 incremental (`data.source`: 0 mutation, 1 mouse move, 2 mouse interaction where `data.type` 2 is a click, 3 scroll, 5 input, 8 stylesheet rule), 4 meta (`data.href`, `width`, `height`), 5 custom (posthog-js sends `$remote_config_received`, `$session_options`, `$posthog_config`, `$recording_started`, and `$pageview` on navigation), 6 plugin (`rrweb/console@1` with `payload.level`). - **Field compression** (observed on every full snapshot and mutation): an event with `"cv": "2024-10"` carries some fields as a string whose characters are the bytes (0-255) of a gzip stream of the field's JSON: `data` of a type 2 event; `texts`, `attributes`, `removes` and `adds` of a type 3 source 0 event; `adds` and `removes` of a type 3 source 8 event. Fields that are already arrays or objects are left as they are. A reader gunzips them, parses the JSON and drops `cv`. - Inputs are masked by posthog-js before sending (observed: `"text": "*******"` for a password and for text fields); the server stores what it receives. - Regular events carry `$session_id` and `$window_id`, which link them to the recording. ## 12. Surveys [posthog-js packages/browser-common/src/types/surveys.ts, packages/core/src/surveys/events.ts; observed] posthog-js fetches `GET /api/surveys/?token=<token>&_=<ms>` (observed) when remote config says `surveys: true`, and lazy-loads `/static/<version>/surveys.js`. A missing or malformed token is 401 `missing_api_key`, an unknown one 401 `invalid_api_key`. The answer: ```json { "surveys": [ { "id": "<uuid>", "name": "NPS", "description": "", "type": "popover", "questions": [ { "id": "q-rating", "type": "rating", "question": "How likely…?", "description": "", "optional": false, "display": "number", "scale": 10, "lowerBoundLabel": "Unlikely", "upperBoundLabel": "Very likely" }, { "id": "q-choice", "type": "single_choice", "question": "What brought you here?", "choices": ["Search", "Friend", "Ad"], "hasOpenChoice": false, "shuffleOptions": false }, { "id": "q-open", "type": "open", "question": "Anything else?" } ], "appearance": { "whiteLabel": true }, "conditions": { "events": { "values": [ { "name": "opened_pricing" } ] } }, "start_date": "2026-…Z", "end_date": null, "linked_flag_key": null, "targeting_flag_key": null, "internal_targeting_flag_key": null, "feature_flag_keys": [], "current_iteration": null, "current_iteration_start_date": null, "schedule": "once", "enable_partial_responses": false } ] } ``` Only running surveys (launched, not stopped, not archived) are listed; posthog-js itself applies `conditions` (`url` with `urlMatchType`, `selector`, `events`, `deviceTypes`) and remembers per browser which surveys it showed. `appearance.whiteLabel: true` hides posthog-js's "Survey by PostHog" footer. Events posthog-js sends (observed), on the normal capture path: - `survey shown`: `$survey_id`, `$survey_name`, `$survey_iteration`, `$survey_iteration_start_date`. - `survey dismissed`: the same, plus `$survey_partially_completed` when some questions were answered. - `survey sent`: the same, plus `$survey_submission_id` (one id per person's pass through the survey), `$survey_completed`, `$survey_questions` (`[{id, question, response}]`) and one `$survey_response_<question id>` per answer (a number for ratings, a string for single choice and open, an array for multiple choice), and a top-level `$set: {"$survey_responded/<survey id>": true}`. Older SDKs send `$survey_response` for the first question and `$survey_response_<index>` for later ones; readers accept both. ## 13. Experiment exposure [observed] `posthog.getFeatureFlag(key)` and `isFeatureEnabled(key)` send `$feature_flag_called` the first time a flag is read in a page load (and again if its value changes), with `$feature_flag` (the key), `$feature_flag_response` (variant string or boolean), `$feature_flag_id`, `$feature_flag_version`, `$feature_flag_reason`, `$feature_flag_request_id`, `$feature_flag_payload` and `$feature_flag_bootstrapped_response` (observed). posthog-node's `getFeatureFlag` / `evaluateFlags().getFlag` sends the same event from the server. Experiments use the first such event per person as the exposure (design.md "Experiments"). ## 14. Heatmaps [posthog-js packages/browser/src/heatmaps.ts; observed] With `heatmaps: true` in remote config, posthog-js (observed 1.450.1) buffers pointer data and sends it about every 5 s, and on page leave, as a normal event on the capture path (section 1): ```json { "event": "$$heatmap", "properties": { "$heatmap_data": { "<page url>": [ { "x": 312, "y": 188, "target_fixed": false, "type": "click" } ] }, "$viewport_width": 1280, "$viewport_height": 760, "$current_url": "<page url>", "$pathname": "/", … } } ``` - `type` is `click`, `rageclick` (3 clicks within 1 s and 30 px), `deadclick` (from the lazily loaded `dead-clicks-autocapture.js` when `captureDeadClicks` is on) or `mousemove`. - `x`, `y` are CSS pixels; the page scroll is already added to `y` unless `target_fixed` (the target is in a fixed-position element). - The instance aggregates `x / $viewport_width` (0-1, 1 % steps) and `y` in 16 px rows per `type`, URL and viewport width range. Scroll depth needs no heatmap setting: `$pageleave` (and the next `$pageview` in single-page apps) carries `$prev_pageview_pathname`, `$prev_pageview_last_scroll`, `$prev_pageview_last_scroll_percentage`, `$prev_pageview_max_scroll`, `$prev_pageview_max_scroll_percentage`, `$prev_pageview_last_content(_percentage)` and `$prev_pageview_max_content(_percentage)`; percentages are 0-1. ## 15. Local flag evaluation [posthog-node src/extensions/feature-flags; observed with posthog-node 6.0.3] With a personal API key posthog-node polls (default every 30 s): ``` GET /flags/definitions?token=<phc_…>&send_cohorts Authorization: Bearer <phx_…> If-None-Match: "<etag of the last 200>" ``` (older SDKs call `/api/feature_flag/local_evaluation?token=…&send_cohorts`; both are served, and `/api/projects/:id/feature_flags/local_evaluation` for API users). No or a bad personal key is 401 `{"type": "authentication_error", …}`; an unknown token 401. 200: ```json { "flags": [ { "id": 1, "team_id": null, "key": "new-cart", "name": "…", "active": true, "deleted": false, "ensure_experience_continuity": false, "version": 3, "filters": { "aggregation_group_type_index": null, "groups": [ … ], "multivariate": null, "payloads": {} } } ], "group_type_mapping": { "0": "company" }, "cohorts": { "7": { "type": "OR", "values": [ { "type": "AND", "values": [ { "key": "plan", "type": "person", "value": ["pro"], "operator": "exact" } ] } ] } } } ``` with an `ETag` (SHA-1 of the body) and 304 for a matching `If-None-Match`. `cohorts` (only with `send_cohorts`) holds the property-only cohorts flags reference, in flag-property form. Flags that reference a behavioural or static cohort name a cohort id missing from `cohorts`, and flags with `ensure_experience_continuity` are flagged by the SDK itself; posthog-node then evaluates those through `/flags` (section 8). Locally evaluated `$feature_flag_called` events carry `locally_evaluated: true` and `$feature_flag_reason: "Evaluated locally"`. ## 16. Experience continuity [posthog-js; rust/feature-flags hash key overrides] After `identify()`, posthog-js's next `/flags` body carries `"$anon_distinct_id": "<previous anonymous id>"` once (observed). For each active flag with `ensure_experience_continuity`, the instance stores a hash key override `(distinct_id, flag_key) → hash_key = $anon_distinct_id` for both the new and the anonymous distinct id, unless one exists. Evaluating such a flag for any distinct id of a person uses the override's hash key in place of the distinct id in the rollout and variant hashes (section 8). Overrides follow person merges and are deleted with the person. ## 17. Survey targeting, iterations and partial responses [posthog-js packages/browser/src/extensions/surveys; observed] `GET /api/surveys/?token=` (section 12) adds per survey: ```json { "linked_flag_key": "new-cart", "targeting_flag_key": "survey-targeting-9cadf132b610", "internal_targeting_flag_key": "survey-targeting-9cadf132b610-custom", "feature_flag_keys": [ { "key": "flag", "value": "new-cart" }, { "key": "targeting_flag", "value": "…" }, { "key": "internal_targeting_flag", "value": "…-custom" } ], "conditions": { "url": "/pricing", "urlMatchType": "icontains", "selector": "#cta", "deviceTypes": ["Desktop"], "deviceTypesMatchType": "exact", "seenSurveyWaitPeriodInDays": 14, "linkedFlagVariant": "test", "events": { "values": [ { "name": "opened_pricing" } ], "repeatedActivation": false }, "cancelEvents": { "values": [ { "name": "closed_cart" } ] }, "actions": { "values": [ { "id": 3, "name": "Added to cart", "steps": [ … ] } ] } }, "current_iteration": 2, "current_iteration_start_date": "…", "schedule": "recurring", "enable_partial_responses": true } ``` - posthog-js checks the three flags itself through `/flags`, so the instance keeps them as real flags, hidden from the flag list: the targeting flag has the survey's `targeting_flag_filters`; the internal flag matches persons where `$survey_dismissed/<id>[/<iteration>]` and `$survey_responded/<id>[/<iteration>]` are not set (posthog-js sets them with `$set` on dismiss and send). `urlMatchType` is one of `exact`, `is_not`, `icontains`, `not_icontains`, `regex`, `not_regex`. - Iterations: `current_iteration = min(iteration_count, floor((now - start_date) / iteration_frequency_days) + 1)`; posthog-js adds `$survey_iteration` and `$survey_iteration_start_date` to survey events. - Partial responses: posthog-js sends one `survey sent` per answered question with the same `$survey_submission_id`, cumulative `$survey_response_<question id>` values, `$survey_completed: false` until the last; closing early sends `survey abandoned` with `$survey_partially_completed: true`. ## 18. Replay settings [posthog-js packages/browser-common/src/types/remote-config.ts; observed] The `sessionRecording` object of section 9, filled from the project's replay settings: | setting | field | |---|---| | sample rate 0-1 | `sampleRate` as a two-decimal string ("0.25"), `null` = all | | minimum duration | `minimumDurationMilliseconds` | | linked flag | `linkedFlag`: `"key"` or `{"flag": "key", "variant": "v"}` | | console logs | `consoleLogRecordingEnabled` | | network headers / bodies | `networkPayloadCapture: {"recordHeaders", "recordBody"}` | | network timing | top-level `capturePerformance: {"network_timing": true, …}` | | URL triggers / blocklist | `urlTriggers` / `urlBlocklist`: `[{"url": "<regex>", "matching": "regex"}]` | | event triggers | `eventTriggers: ["<event>"]`, `triggerMatchType: "any" | "all"` | | masking | `masking: {"maskAllInputs", "maskTextSelector", "blockSelector"}` | | canvas | `recordCanvas: true`, `canvasFps`, `canvasQuality` ("0.4") | Resulting rrweb events (observed): console as type 6 `{"plugin": "rrweb/console@1", "payload": {"level", "trace": [], "payload": [<JSON strings>]}}`; network as type 6 `{"plugin": "rrweb/network@1", "payload": {"requests": [PerformanceResourceTiming fields + "method", "status", headers and bodies when enabled]}}`; posthog-js turns the network recorder off when the page's host is `localhost` or `127.0.0.1`. Canvas as type 3 source 9 (canvas mutation) with base64 webp image data; the player needs canvas replay turned on. ## 19. Exceptions [posthog-js packages/browser/src/extensions/exception-autocapture, posthog-node src/extensions/error-tracking; observed with posthog-js 1.450.1 and posthog-node 6.0.3] `$exception` is an ordinary event on the capture endpoints (posthog-js: `/i/v0/e/`; posthog-node: `/i/v1/analytics/events`). With `autocaptureExceptions` truthy in the remote config posthog-js loads `/static/exception-autocapture.js` and captures `window.onerror` and `unhandledrejection`; `captureException(error, props)` works either way. Observed properties (`tests/fixtures/v05-errors.json`): ```json { "$exception_list": [ { "type": "TypeError", "value": "Price must be a number, got string", "mechanism": { "type": "generic", "handled": false, "synthetic": false, "exception_id": 0 }, "stacktrace": { "type": "raw", "frames": [ { "platform": "web:javascript", "filename": "http://host/app.min.js", "function": "r", "in_app": true, "lineno": 1, "colno": 206 }, { "platform": "web:javascript", "filename": "http://host/app.min.js", "function": "n", "in_app": true, "lineno": 1, "colno": 49 } ] } } ], "$exception_level": "error", "$exception_capture_enabled_server_side": true, "$exception_fingerprint": "qty-range" } ``` - The list's first entry is the outer exception; frames are ordered outermost first, so the frame that threw is the **last** one. Line and column are 1-based. - posthog-node frames carry `"platform": "node:javascript"`, a `module`, `in_app: false` for `node:` internals and dependencies, and already include `pre_context` / `context_line` / `post_context` read from disk. - `$exception_fingerprint` is optional (string; a list is accepted). The server adds to the stored event: `$exception_fingerprint` (computed unless sent, `test-vectors.json` "fingerprints"), `$exception_issue_id`, `$exception_type` / `$exception_message` from the first entry when missing, and, for frames resolved with an uploaded source map, `stacktrace.type: "resolved"` and per frame `resolved: true`, the original `filename` / `lineno` / `colno` / `function`, context lines and `minified: {filename, function, lineno, colno}`. ## 20. Quota limiting [posthog-js src/rate-limiter.ts; observed] A capture response may carry `quota_limited`, a list of batch keys. posthog-js then drops new events for those keys for 60 seconds without sending them (observed: after one limited answer, four further clicks and a `captureException` sent nothing). Batch keys: `events` (ordinary events), `exceptionEvent` (`$exception`), `recordings` (`/s/` snapshots). | endpoint, project capped in drop mode | response | |---|---| | `/e/`, `/i/v0/e/`, `/batch/`, … | `{"status": 1, "quota_limited": ["events", "exceptionEvent"]}` | | `/i/v1/analytics/events` | `{"results": {…}, "quota_limited": ["events", "exceptionEvent"]}` | | `/s/` | `{"status": 1, "quota_limited": ["recordings"]}` | In sample mode the answer stays `{"status": 1}` and a deterministic share is kept (`test-vectors.json` "sampling"). Events dropped because of a cap are accepted with status 200 like events of an unknown project. 0.6: events, requests and replay uploads refused this way are counted per project ("Dropped past cap" on the Usage screen); the answer is unchanged. ## 21. LLM analytics capture [PostHog docs "LLM analytics", posthog-js packages/ai; observed shapes] AI events are ordinary events with reserved names and properties. Names: `$ai_generation` (one model call), `$ai_embedding`, `$ai_span` (a step), `$ai_trace` (the whole run); other `$ai_*` names are stored as sent. Properties read by this instance (all optional): | property | meaning | |---|---| | `$ai_trace_id` | groups a run; a call without one is its own trace | | `$ai_span_id`, `$ai_parent_id` | tree links; a parent equal to the trace id is top level | | `$ai_span_name`, `$ai_trace_name` | display name | | `$ai_model`, `$ai_provider` | e.g. `gpt-4o-mini`, `openai` | | `$ai_input_tokens`, `$ai_output_tokens` | token counts (OpenAI-style input includes cached tokens) | | `$ai_cache_read_input_tokens`, `$ai_cache_creation_input_tokens` | cache tokens (Anthropic reports them apart from input) | | `$ai_total_cost_usd`, `$ai_input_cost_usd`, `$ai_output_cost_usd` | SDK-computed cost; wins over list prices | | `$ai_latency` | seconds | | `$ai_is_error`, `$ai_error`, `$ai_http_status` | errors (status ≥ 400 counts as an error) | | `$ai_input`, `$ai_output_choices` | messages in and out (any JSON) | | `$ai_input_state`, `$ai_output_state` | a trace's or span's input and output | Endpoints (in addition to every capture path in section 1): - `POST /i/v0/ai/batch/`: legacy bodies (section 2); events whose name does not start with `$ai_` are dropped (counted as misrouted), the rest are normalized like `/batch`. Answer `{"status": 1}` (or `quota_limited`, section 20). - `POST /i/v1/ai/events`: v1 body (section 6); a non-AI event's result is `{"result": "drop", "details": "misrouted_event"}`. - `POST /i/v0/ai`: `Authorization: Bearer <project token>` (401 `invalid_api_key` otherwise), `Content-Type: multipart/form-data`, optional `Content-Encoding: gzip`, at most 25 MB. Exactly one part named `event` (JSON object with `event`, `distinct_id`, optional `uuid`, `timestamp`, `properties`) and at most one part named `event.properties` (JSON object merged over the event's properties). Any other part name, a second `event`, a non-AI event (`misrouted_event`) or a bad uuid (`invalid_uuid`) is 400 with the section 6 error shape. Success is `{"status": 1}`. ## 22. Exceptions 0.6: releases, server SDKs, suppression rules [posthog-js @posthog/browser-common remote-config types; posthog-python, posthog-go, posthog-rs] - Release: the first non-empty of `$release`, `release`, `$app_version`, `app_version` (`test-vectors.json` "releases"). - Server SDK `$exception_list` frames carry their own `platform` (`python`, `go`, `rust`, …), `filename`, `abs_path`, `function`, `module`, `lineno`, optional `colno`, `in_app` and, from posthog-python, `pre_context` / `context_line` / `post_context`. They arrive symbolicated; the server never applies source maps to them. - Remote config `errorTracking.suppressionRules` is a list of `{"type": "AND" | "OR", "values": [{"key": "$exception_types" | "$exception_values", "operator": "exact" | "is_not" | "icontains" | "not_icontains" | "regex" | "not_regex", "value": string | string[], "type": "error_tracking_issue_property"}]}`. posthog-js evaluates it before sending an exception; the server applies the same rules at ingest (`test-vectors.json` "suppression_rules"). -
test-vectors.jsonWorked examples for decoding, tokens, timestamps, distinct ids, flag hashing and matching, capture responses person scenarios, sampling, quota_limited answers, source-map decoding and exception fingerprints, and (0.6) index source maps, LLM costs, suppression rules and releases. -
design.mdStack, storage and event-shard decisions, Cloudflare config, deploy and upgrade, data model, ingestion, query semantics, actions, cohorts, SQL, heatmaps, streaming funnels, replay, experiment statistics, surveys, owner API, screens, tests, limits and the Basin cold tier (stream, Iceberg sink, Basin SQL compiler, retention), usage limits, cost estimate, alerts, error tracking and symbolication; 0.6: HTML fallback, public demo, edge usage counters, error tracking 0.6 schema and algorithms, LLM analytics, cold Trends and cold-prune.py. -
tasks.mdBuild order, each step with a check.Tasks
Work in this order. Finish each check before the next task.
- Set up. npm package (Node 22+),
wrangler.example.jsoncand a copy aswrangler.jsoncfromdesign.md,spekkr.inputs.json,src/shared,src/worker,src/ui, the UI build script andscripts/copy-sdk.mjs. Check:npm run devserves a page titledapp_nameand/static/array.jsreturns posthog-js. - Wire library (
src/shared). Body decoding, token rule, timestamp rules, distinct-id rules, UUIDv7, flag hash and property matching, date ranges and buckets. Check:tests/vectors.test.tspasses every vector intest-vectors.jsonexcept the HTTP ones. - Schema and accounts (R1, R2). D1 migration, sign-up policy, sign in/out, sessions, personal API keys, Origin check, projects. Check: the first account can sign up and a second cannot; with
ALLOWED_EMAILSonly listed emails can. - Capture (R3). Legacy and v1 routes, CORS, token lookup, error codes, normalization, queue producer and consumer,
ProjectStore.ingestwith dedupe. Check: thecapture_responsesvectors and an event sent to/i/v0/eis readable from the store after the queue delivers it. - Persons and groups (R6). Merge rules,
$set/$set_once/$unset, illegal ids, personless events, groups. Check: thepersonsscenarios intest-vectors.jsonin the store test. - SDK support (R4, R5).
/flags,/decide, remote config,/staticwith version rewrite, stubs. Check: the flag vectors over HTTP and posthog-js loads with no failed requests (no 404 forsurveys.js). - Queries (R7, R9). Events query with cursor, trends (maths, breakdown, compare), funnels (ordered as the UDF, unordered), retention. Check:
tests/store.test.tsnumbers on the fixed dataset. - Owner API (R8, R10, R11, R12). Persons, insights, dashboards, flags, definitions, property values, stats, project settings, token reset, project delete, cron purge. Check: the HTTP tests.
- Web app (R13). Every screen in
design.md, desktop and 390 px. Check: by hand against the demo project (scripts/demo-setup.mjs). - E2E (R14).
npm run e2ewith the real posthog-js in Chromium and posthog-node; screenshots withscripts/shots.mjs. - Deploy. Follow
design.md("Deploy") withALLOWED_EMAILSset, then point a real site's snippet at it and see events in Events.
0.2 (after 1-11, or on an existing 0.1 build):
- Event shards (R15).
EventShardclass and binding,eventsql.tsprimitives, head registry andingest_log, two-phase trends, funnels, retention and events, legacy table as a pseudo-shard, purge and person deletion across shards. Check:tests/shards.test.ts(day shards, month shards and legacy agree) and every earlier test still passes. - Session replay (R16). R2 binding, D1
0002_replay.sql,/s/handler, queue metadata,session_recordings, snapshots API with field decompression, remote configsessionRecording, retention; then the rrweb-player bundle and the Session replay, person and events screens. Check:tests/replay.test.tswith the recorded fixtures, then a real posthog-js recording plays in the UI. - Experiments (R17).
shared/stats.ts, experiment CRUD creating the multivariate flag, launch and stop, exposure and metrics, results and screens. Check:tests/experiments.test.ts(sampler vs closed form, the 200-person experiment). - Surveys (R18). Survey CRUD,
/api/surveys, remote configsurveys, results and screens. Check: the survey tests with the recorded events, and a survey shown and answered by posthog-js in a browser. - E2E and screenshots.
npm run e2e(now with replay, survey and experiment checks);scripts/demo-02.mjsandscripts/shots-02.mjs. - Upgrade a deployment.
design.md"Upgrading 0.1 → 0.2".
0.3 (after 1-17):
- Actions (R19).
shared/elements.ts(chain parser, selectors),shared/actions.ts, SQL step compiler plus shard JS pre-pass,ActionsNodein trends, funnels, retention and experiments; Actions screens. Check: selector and action tests. - Cohorts (R20).
worker/cohorts.ts, tables, calculation (save, hourly cron, on demand), static input, cohort filters everywhere, cohort flag conditions; Cohort screens. Check: cohort tests. - Streaming funnels and paged surveys (R26). Two-pass funnels in person batches; survey results by cursor. Check: batch size 3 equals one pass.
- SQL (R22).
worker/sqlquery.ts,/sql, HogQLQuery; SQL screen. Check: SQL tests. - Local evaluation and continuity (R23, R24). Definitions endpoints with ETag; hash key overrides from
$anon_distinct_id. Check: continuity test, HTTP 401/304 tests, posthog-node in the e2e. - Survey targeting (R25). Hidden targeting flags, conditions, iterations, partial merge; editor and results. Check: survey tests.
- Replay settings and heatmaps (R21, R26). D1
0003_v03.sql, remote config mapping, paged snapshots, console/network panels, heatmap and scroll APIs and screen. Check: the tests, thenconform03/againstwrangler dev. - E2E and screenshots.
npm run e2e(49 checks);scripts/shots-03.mjs. - Upgrade a deployment. Apply
0003_v03.sql, deploy (adds the hourly cron). Existing data needs no migration.
0.4 (after 1-26):
- Cold tier (R27).
worker/cold.ts(records,compileCold,buildColdQuery,basinSql),prepareSelectshared fromsqlquery.ts,source: "cold"on/sql,cold_tierin/api/config, the source picker on the SQL screen. Check:tests/cold.test.ts. - Basin setup and deploy. Bucket with Basin Catalog, stream (schema in
design.md"Cold tier"), the one token, sink, pipeline, catalog maintenance;COLD_STREAMbinding,BASIN_*vars and theBASIN_TOKENsecret (keep secrets on later uploads). Check: send events, wait about two minutes, and a cold SQL query returns them while an event older than the retention period is excluded. - Ingest-only hostnames (0.4.1, opt-in since 0.4.2).
INGEST_HOSTSandingestHost()inworker/env.ts, host check inworker/index.tsafter the SDK routes (design.md"Hostnames"). Check: the "ingest-only hostnames" HTTP tests (preflight, capture,/flags,/_health, page redirect, 421 on a cookie write; an unlisted host serves the full app).
0.5 (after 1-29):
- Usage limits and sampling (R30). D1
0004_v05.sql;worker/usage.ts(limits,gateEvents, cap state),shared/sample.ts; storeapplyEventCap,usage_daily,syncCapState, replay caps with chunk deletion;quota_limitedon capture and/s/; remote config switches. Check: the caps tests intests/v05.test.ts. - Usage screen and cost estimate (R30).
store.usage(),estimateCost()with list prices, the Usage screen and Settings form. Check: usage tests; e2e "usage page". - Alerts (R30).
alertstable, thresholds, daily spike in the cron,worker/alerts.tswith the optionalEMAILbinding andALERT_EMAIL_FROM; unread badge. Check: alert tests (stub binding). - Error tracking (R28).
worker/errors.ts(exception list, fingerprint), issue tables,recordOccurrence, issues API, screens, remote configautocaptureExceptions. Check: error tests with the REA fixtures (tests/fixtures/v05-errors.json, recorder inresearch.md); e2e exception checks. - Source maps (R29).
shared/sourcemap.ts, symbol set upload/list/ delete, symbolication at ingest,scripts/upload-sourcemaps.mjs. Check: vectors "source_maps", HTTP upload test,scripts/shots-05.mjs. - Upgrade a deployment. Apply
0004_v05.sql(additive; existing projects keep no limits and exception autocapture off), deploy. For email alerts add thesend_emailbinding andALERT_EMAIL_FROMon a sender domain onboarded to Email Sending.
0.6 (after 1-35):
- HTML fallback and public demo (R13, R32).
worker/routing.ts; D10005_v06.sql(projects.public_demo);PUBLIC_DEMOvar,demoApi()andredact()inworker/api.ts; UI banner and write lock. Check: "HTML fallback" and "public demo" intests/v06.test.ts; e2e public-demo checks withPUBLIC_DEMO=1. - Usage counters (R30).
worker/meter.ts,recordEdgeUsage,dropped_past_cap, request types inestimateCost, 10 s project cache, Usage screen. Check: "usage counters and cost" tests; e2e dropped-past-cap. - Error tracking 0.6 (R28, R29). Releases, per-release maps and retroactive symbolication, merge/split with aliases, assignment, grouping and suppression rules (remote config and ingest), server stacks, index maps,
upload-sourcemaps.mjs --release, store upgrade. Check: v06 error tests, vectors "source_map_sections", "suppression_rules", "releases"; e2e error checks. - LLM analytics (R31). AI endpoints in
worker/capture.ts,shared/llm-pricing.ts,eventsql.llmRows,worker/llm.ts, API and screens. Check: "LLM analytics" tests, vectors "llm_costs", e2e LLM checks. - Cold Trends and pruning (R27).
buildColdTrends/runColdTrends, Trends Source picker,scripts/cold-prune.pyand its workflow example. Check: "cold Trends" tests;python3 -m unittest scripts/test_cold_prune.py. - Demo data and screenshots.
scripts/demo-setup.mjs(every screen,public_demo),scripts/shots-06.mjs. Check:shots06/. - Upgrade a deployment. Apply
0005_v06.sql, deploy; setPUBLIC_DEMO=1only on a demo instance.
# Tasks Work in this order. Finish each check before the next task. 1. **Set up.** npm package (Node 22+), `wrangler.example.jsonc` and a copy as `wrangler.jsonc` from `design.md`, `spekkr.inputs.json`, `src/shared`, `src/worker`, `src/ui`, the UI build script and `scripts/copy-sdk.mjs`. Check: `npm run dev` serves a page titled `app_name` and `/static/array.js` returns posthog-js. 2. **Wire library (`src/shared`).** Body decoding, token rule, timestamp rules, distinct-id rules, UUIDv7, flag hash and property matching, date ranges and buckets. Check: `tests/vectors.test.ts` passes every vector in `test-vectors.json` except the HTTP ones. 3. **Schema and accounts (R1, R2).** D1 migration, sign-up policy, sign in/out, sessions, personal API keys, Origin check, projects. Check: the first account can sign up and a second cannot; with `ALLOWED_EMAILS` only listed emails can. 4. **Capture (R3).** Legacy and v1 routes, CORS, token lookup, error codes, normalization, queue producer and consumer, `ProjectStore.ingest` with dedupe. Check: the `capture_responses` vectors and an event sent to `/i/v0/e` is readable from the store after the queue delivers it. 5. **Persons and groups (R6).** Merge rules, `$set`/`$set_once`/`$unset`, illegal ids, personless events, groups. Check: the `persons` scenarios in `test-vectors.json` in the store test. 6. **SDK support (R4, R5).** `/flags`, `/decide`, remote config, `/static` with version rewrite, stubs. Check: the flag vectors over HTTP and posthog-js loads with no failed requests (no 404 for `surveys.js`). 7. **Queries (R7, R9).** Events query with cursor, trends (maths, breakdown, compare), funnels (ordered as the UDF, unordered), retention. Check: `tests/store.test.ts` numbers on the fixed dataset. 8. **Owner API (R8, R10, R11, R12).** Persons, insights, dashboards, flags, definitions, property values, stats, project settings, token reset, project delete, cron purge. Check: the HTTP tests. 9. **Web app (R13).** Every screen in `design.md`, desktop and 390 px. Check: by hand against the demo project (`scripts/demo-setup.mjs`). 10. **E2E (R14).** `npm run e2e` with the real posthog-js in Chromium and posthog-node; screenshots with `scripts/shots.mjs`. 11. **Deploy.** Follow `design.md` ("Deploy") with `ALLOWED_EMAILS` set, then point a real site's snippet at it and see events in Events. 0.2 (after 1-11, or on an existing 0.1 build): 12. **Event shards (R15).** `EventShard` class and binding, `eventsql.ts` primitives, head registry and `ingest_log`, two-phase trends, funnels, retention and events, legacy table as a pseudo-shard, purge and person deletion across shards. Check: `tests/shards.test.ts` (day shards, month shards and legacy agree) and every earlier test still passes. 13. **Session replay (R16).** R2 binding, D1 `0002_replay.sql`, `/s/` handler, queue metadata, `session_recordings`, snapshots API with field decompression, remote config `sessionRecording`, retention; then the rrweb-player bundle and the Session replay, person and events screens. Check: `tests/replay.test.ts` with the recorded fixtures, then a real posthog-js recording plays in the UI. 14. **Experiments (R17).** `shared/stats.ts`, experiment CRUD creating the multivariate flag, launch and stop, exposure and metrics, results and screens. Check: `tests/experiments.test.ts` (sampler vs closed form, the 200-person experiment). 15. **Surveys (R18).** Survey CRUD, `/api/surveys`, remote config `surveys`, results and screens. Check: the survey tests with the recorded events, and a survey shown and answered by posthog-js in a browser. 16. **E2E and screenshots.** `npm run e2e` (now with replay, survey and experiment checks); `scripts/demo-02.mjs` and `scripts/shots-02.mjs`. 17. **Upgrade a deployment.** `design.md` "Upgrading 0.1 → 0.2". 0.3 (after 1-17): 18. **Actions (R19).** `shared/elements.ts` (chain parser, selectors), `shared/actions.ts`, SQL step compiler plus shard JS pre-pass, `ActionsNode` in trends, funnels, retention and experiments; Actions screens. Check: selector and action tests. 19. **Cohorts (R20).** `worker/cohorts.ts`, tables, calculation (save, hourly cron, on demand), static input, cohort filters everywhere, cohort flag conditions; Cohort screens. Check: cohort tests. 20. **Streaming funnels and paged surveys (R26).** Two-pass funnels in person batches; survey results by cursor. Check: batch size 3 equals one pass. 21. **SQL (R22).** `worker/sqlquery.ts`, `/sql`, HogQLQuery; SQL screen. Check: SQL tests. 22. **Local evaluation and continuity (R23, R24).** Definitions endpoints with ETag; hash key overrides from `$anon_distinct_id`. Check: continuity test, HTTP 401/304 tests, posthog-node in the e2e. 23. **Survey targeting (R25).** Hidden targeting flags, conditions, iterations, partial merge; editor and results. Check: survey tests. 24. **Replay settings and heatmaps (R21, R26).** D1 `0003_v03.sql`, remote config mapping, paged snapshots, console/network panels, heatmap and scroll APIs and screen. Check: the tests, then `conform03/` against `wrangler dev`. 25. **E2E and screenshots.** `npm run e2e` (49 checks); `scripts/shots-03.mjs`. 26. **Upgrade a deployment.** Apply `0003_v03.sql`, deploy (adds the hourly cron). Existing data needs no migration. 0.4 (after 1-26): 27. **Cold tier (R27).** `worker/cold.ts` (records, `compileCold`, `buildColdQuery`, `basinSql`), `prepareSelect` shared from `sqlquery.ts`, `source: "cold"` on `/sql`, `cold_tier` in `/api/config`, the source picker on the SQL screen. Check: `tests/cold.test.ts`. 28. **Basin setup and deploy.** Bucket with Basin Catalog, stream (schema in `design.md` "Cold tier"), the one token, sink, pipeline, catalog maintenance; `COLD_STREAM` binding, `BASIN_*` vars and the `BASIN_TOKEN` secret (keep secrets on later uploads). Check: send events, wait about two minutes, and a cold SQL query returns them while an event older than the retention period is excluded. 29. **Ingest-only hostnames (0.4.1, opt-in since 0.4.2).** `INGEST_HOSTS` and `ingestHost()` in `worker/env.ts`, host check in `worker/index.ts` after the SDK routes (`design.md` "Hostnames"). Check: the "ingest-only hostnames" HTTP tests (preflight, capture, `/flags`, `/_health`, page redirect, 421 on a cookie write; an unlisted host serves the full app). 0.5 (after 1-29): 30. **Usage limits and sampling (R30).** D1 `0004_v05.sql`; `worker/usage.ts` (limits, `gateEvents`, cap state), `shared/sample.ts`; store `applyEventCap`, `usage_daily`, `syncCapState`, replay caps with chunk deletion; `quota_limited` on capture and `/s/`; remote config switches. Check: the caps tests in `tests/v05.test.ts`. 31. **Usage screen and cost estimate (R30).** `store.usage()`, `estimateCost()` with list prices, the Usage screen and Settings form. Check: usage tests; e2e "usage page". 32. **Alerts (R30).** `alerts` table, thresholds, daily spike in the cron, `worker/alerts.ts` with the optional `EMAIL` binding and `ALERT_EMAIL_FROM`; unread badge. Check: alert tests (stub binding). 33. **Error tracking (R28).** `worker/errors.ts` (exception list, fingerprint), issue tables, `recordOccurrence`, issues API, screens, remote config `autocaptureExceptions`. Check: error tests with the REA fixtures (`tests/fixtures/v05-errors.json`, recorder in `research.md`); e2e exception checks. 34. **Source maps (R29).** `shared/sourcemap.ts`, symbol set upload/list/ delete, symbolication at ingest, `scripts/upload-sourcemaps.mjs`. Check: vectors "source_maps", HTTP upload test, `scripts/shots-05.mjs`. 35. **Upgrade a deployment.** Apply `0004_v05.sql` (additive; existing projects keep no limits and exception autocapture off), deploy. For email alerts add the `send_email` binding and `ALERT_EMAIL_FROM` on a sender domain onboarded to Email Sending. 0.6 (after 1-35): 36. **HTML fallback and public demo (R13, R32).** `worker/routing.ts`; D1 `0005_v06.sql` (`projects.public_demo`); `PUBLIC_DEMO` var, `demoApi()` and `redact()` in `worker/api.ts`; UI banner and write lock. Check: "HTML fallback" and "public demo" in `tests/v06.test.ts`; e2e public-demo checks with `PUBLIC_DEMO=1`. 37. **Usage counters (R30).** `worker/meter.ts`, `recordEdgeUsage`, `dropped_past_cap`, request types in `estimateCost`, 10 s project cache, Usage screen. Check: "usage counters and cost" tests; e2e dropped-past-cap. 38. **Error tracking 0.6 (R28, R29).** Releases, per-release maps and retroactive symbolication, merge/split with aliases, assignment, grouping and suppression rules (remote config and ingest), server stacks, index maps, `upload-sourcemaps.mjs --release`, store upgrade. Check: v06 error tests, vectors "source_map_sections", "suppression_rules", "releases"; e2e error checks. 39. **LLM analytics (R31).** AI endpoints in `worker/capture.ts`, `shared/llm-pricing.ts`, `eventsql.llmRows`, `worker/llm.ts`, API and screens. Check: "LLM analytics" tests, vectors "llm_costs", e2e LLM checks. 40. **Cold Trends and pruning (R27).** `buildColdTrends`/`runColdTrends`, Trends Source picker, `scripts/cold-prune.py` and its workflow example. Check: "cold Trends" tests; `python3 -m unittest scripts/test_cold_prune.py`. 41. **Demo data and screenshots.** `scripts/demo-setup.mjs` (every screen, `public_demo`), `scripts/shots-06.mjs`. Check: `shots06/`. 42. **Upgrade a deployment.** Apply `0005_v06.sql`, deploy; set `PUBLIC_DEMO=1` only on a demo instance. - Set up. npm package (Node 22+),
-
research.mdHow the protocol was established with REA (0.1-0.3 captures, 0.4 Basin findings, 0.5 exception and quota captures, and conformance runs; 0.6 choices), evidence and obligation ledgers, unknowns, licensing and choices.Research notes
This spec describes an independent product-analytics service that speaks the public wire protocol of PostHog's SDKs, so the published
posthog-js(Apache-2.0 and MIT) andposthog-nodelibraries work against it unchanged. It contains no PostHog code and no PostHog branding. This file records how the protocol was established, the evidence behind each rule, what is still unknown, and the 0.1 and 0.2 choices. It is context, not extra requirements.Method: REA (reverse engineer anything)
REA was installed from its official source only: the npm package
rea-agents6.9.0 (MIT; repositorygithub.com/morluto/rea), with its bundledreverse-engineer-anythingskill. Its method was followed:- Collect evidence, not impressions. Every observation is an evidence record with a content-addressed id (
ev_<sha256>), its parameters, and the tool's own stated limitations and unknowns. - Static analysis of the shipped artifact (
analyze_javascript_applicationon theposthog-js1.450.1 npm package). - Runtime capture of the real SDK in a real browser (
capture_browser_scenario): a scripted page that loads the snippet, captures, identifies, groups, sets properties, aliases, reads a flag, navigates and resets, with request and response bodies recorded. - Read the reference source (the MIT
posthog-fossmirror) for the server-side rules the SDK traffic cannot show: validation, timestamps, person merges, flag hashing, funnel semantics. - Write down obligations (what a server must do), each tied to its evidence, and list what stays unknown.
- Conformance: replay the same browser scenario against the self-built server and require every SDK request to succeed.
What REA contributed
- The static pass over the minified bundle gave an inventory (476 files examined) but little protocol detail: routes and payloads are built dynamically, and REA reported them as unresolved (its limitation list says so). It did confirm the set of lazily loaded extension bundles.
The runtime captures established what the code does not make obvious:
- posthog-js 1.450 sends event batches gzip-compressed with
Content-Type: text/plainand nocompressionquery parameter, so a server must sniff the gzip magic bytes; - it posts to
/e/until remote config loads and then to the config'sanalytics.endpoint(/i/v0/e/); - flags go to
/flags/?v=2&config=truewith a gzipped body; - it fetches
/array/<token>/config.jsand then/static/<version>/surveys.jseven when surveys are off, so every extension bundle must be served (the first conformance run caught four 404s here, fixed before the second); - it drops all events from headless browsers unless
opt_out_useragent_filteris set (no request at all was the evidence); $identifyputs$set/$set_onceat the event's top level;$pageleaveusesfetchwithkeepalive, notsendBeacon.
- posthog-js 1.450 sends event batches gzip-compressed with
- A small recording server (
recorder/) logged posthog-js and posthog-node traffic byte for byte (sdk-wire.recorded.jsonl,node-sdk-wire.recorded.jsonl): posthog-node 6 posts v1 batches to/i/v1/analytics/events(Bearer token, gzip JSON) and treats a{"status":1}answer as unreadable, and calls/flags/?v=4. - The final conformance capture against the self-built server: 16 SDK transactions, all 2xx, none failed.
0.2: session replay, surveys, experiment exposure
The same method was applied to the three new protocols with posthog-js 1.450.1:
Runtime capture against a recorder. A second recording server (
recorder2/) served a page running the unmodified bundles and a remote config withsessionRecordingset andsurveys: true, answered/api/surveyswith an NPS survey (rating, single choice, open) and/flagswith a multivariate flag, and logged every request decoded. The REA scenario typed into a text field and a password field, clicked, read flags, identified, triggered the survey and answered all three questions through the real popover, then navigated (capture_browser_scenario,ev_64cbe291…, summaryev_e4b81384…, 0 failed). What it established (nowprotocol.md9, 11-13):- the recorder is lazy-loaded from
/static/<version>/lazy-recorder.jsand posts to the config'sendpoint(/s/), gzip withContent-Type: text/plainand no query string, a JSON array of$snapshotevents carrying$snapshot_data,$snapshot_bytes,$session_id,$window_id,distinct_idandtoken; - full snapshots and mutations arrive field-compressed (
cv: "2024-10": a field's JSON gzipped and sent as a byte string), which a reader must undo before rrweb can play them; nothing in the request says so exceptcv; - inputs are masked in the browser (the password and the text field arrive as asterisks), and regular events carry
$session_idand$window_id; - surveys are fetched from
/api/surveys/?token=…&_=…, rendered by/static/<version>/surveys.js, and answered withsurvey shownandsurvey sentevents whose properties carry the answers per question id, plus a$survey_submission_id; withoutappearance.whiteLabelthe popover shows a PostHog footer; getFeatureFlagsends$feature_flag_calledonce per flag per page load with the variant in$feature_flag_response. The two recorded/s/bodies and the survey events are the fixtures oftests/replay.test.tsandtests/experiments.test.ts.
- the recorder is lazy-loaded from
- Reference source for names and shapes the traffic cannot show (posthog-js
packages/browser-common/src/types/remote-config.tsandtypes/surveys.ts,packages/core/src/surveys/events.ts,packages/browser/src/extensions/replay/), all in the MIT/Apacheposthog-jsrepository at15aef2ce. - Conformance. The same scenario against the self-built server under
wrangler dev(project with recording on, a launched experiment on a multivariate flag, a launched survey): 22 transactions (config,/flags,/e/,/i/v0/e/,/s/,/api/surveys,array.js,lazy-recorder.js,surveys.js), all 2xx, none failed (ev_db418df4…, summaryev_49b453af…). Afterwards the server held one recording (10 clicks, 4 key presses, 2 chunks; snapshots decompressed, with a full snapshot; the typed password absent), the survey's answers (NPS 100, the chosen option, the text) and one experiment exposure.
0.3: heatmaps, replay capture options, continuity, partial surveys, local evaluation
- Runtime capture against a recorder (
recorder3/, posthog-js 1.450.1 and posthog-node 6.0.3). The recorder's remote config turned on heatmaps, dead clicks, console logs, network timing with headers and bodies, and canvas recording;/api/surveysserved a partial-responses survey with targeting flags, conditions and an iteration;/flagsa multivariate flag. The scenario typed, fetched, logged to the console, drew on a canvas, read flags, scrolled, identified, answered two of three survey questions and navigated. Two runs: onlocalhost(ev_32496edb…, summaryev_8115b374…, 0 failed) and on a non-localhost host so the network recorder runs (ev_4682126f…, summaryev_99fe82a5…, 0 failed). posthog-node with a personal key was recorded fetching definitions. Established (nowprotocol.md14-18):$$heatmapevents and$heatmap_data; the scroll properties on$pageleave; console, network and canvas rrweb events and that posthog-js disables the network recorder on localhost;$anon_distinct_idin the first/flagsafter identify; the partial-response event sequence ($survey_completed: false, cumulative answers,survey abandoned) and the$survey_responded/<id>/ <iteration>person properties; posthog-node's/flags/definitions?token=…&send_cohortswithAuthorization: BearerandIf-None-Match. - Reference source (MIT/Apache only): posthog-js
packages/browser/src/heatmaps.ts,extensions/replay/,extensions/surveys/,browser-common/src/types/remote-config.tsandsurveys.ts; posthog-nodesrc/extensions/feature-flags/;posthog-fossrust/feature-flags(hash key overrides, cohort properties) and public docs for cohorts, actions, heatmaps and the SQL editor. Nothing underee/. - Conformance (
conform03/): the same page and scenario against the self-built server underwrangler dev(project with heatmaps, console, network headers/bodies and canvas on; an experiment flag with continuity; a partial-responses survey with targeting, conditions and 3 iterations): 30 transactions, 0 failed, every step completed (ev_9535baea…, summaryev_112dda4b…). Afterwards the server held 15 heatmap points in 12 cells and one scroll-depth page view, a recording with console and network plugin events and a canvas mutation, one partial survey response merged from twosurvey sentevents (two answers, one abandoned, iteration 1), the hidden targeting and internal flags, and one experiment exposure. - End to end (
npm run e2e, 49 checks) adds posthog-node local evaluation against the instance, actions, cohorts, SQL and the screens.
Evidence ledger
Evidence files are kept with the reference build's working notes (not part of the published spec).
Evidence id Operation What ev_85e84d97…(summaryev_964c577b…)analyze_javascript_application posthog-js 1.450.1 npm package, static ev_fc227d71…(summaryev_6a20c49e…)capture_browser_scenario posthog-js against the recorder (port 8799), first run ev_1bf4f557…(summaryev_5b4d3a36…)capture_browser_scenario same, with identify/group/alias/flag steps; 2 expected failures (recorder has no flag or survey answers) ev_69c9eef3…(summaryev_c588dca9…)capture_browser_scenario posthog-js against the self-built server: 18 transactions, 4 failed ( surveys.js404)ev_d91bc8d5…(summaryev_f5fb4a24…)capture_browser_scenario the same after the fix: 16 transactions, 0 failed ev_64cbe291…(summaryev_e4b81384…)capture_browser_scenario 0.2: posthog-js recorder, surveys and exposure against recorder2(port 8798), 0 failedev_db418df4…(summaryev_49b453af…)capture_browser_scenario 0.2 conformance: the same scenario against the self-built server, 22 transactions, 0 failed recorder2/recorded.jsonlrecorder raw 0.2 requests ( /s/bodies, survey and exposure events)ev_32496edb…(summaryev_8115b374…)capture_browser_scenario 0.3: heatmaps, replay options, continuity, partial survey against recorder3on localhost, 0 failedev_4682126f…(summaryev_99fe82a5…)capture_browser_scenario 0.3: the same on a non-localhost host (network recorder on), 0 failed ev_9535baea…(summaryev_112dda4b…)capture_browser_scenario 0.3 conformance against the self-built server, 30 transactions, 0 failed recorder3/recorded.jsonlrecorder raw 0.3 requests ( $$heatmap, plugin events,/flagswith$anon_distinct_id, partial survey events, posthog-node definitions polling)sdk-wire.recorded.jsonlrecorder raw posthog-js requests (headers, bodies) node-sdk-wire.recorded.jsonlrecorder raw posthog-node 6.0.3 requests Full ids are in the
*.summary.yamland*.evidence.jsonfiles.Obligation ledger
Source paths are in
PostHog/posthog-fossat commit0f73d659(2026-11-18, MIT) andPostHog/posthog-jsat15aef2ce(npm package licenceApache-2.0 AND MIT).Rule ( protocol.mdsection)Source Legacy body shapes, dropped events (2) rust/capture/src/v0_request.rsToken rule (3) rust/capture/src/token.rsError codes (3, 6) rust/capture/src/api.rsDecoding order, gzip sniffing (4) rust/capture/src/payload/decompression.rs; runtime capturesTimestamps, skew, offset, future cutoff (5) rust/common/types/src/timestamp.rsand its testsv1 request and response (6) rust/capture/src/v1/README.md,rust/capture/src/v1/analytics/*; node recordingIllegal distinct ids (7) nodejs/src/common/persons/person-utils.ts,rust/capture/src/v1/analytics/constants.rsMerge rules, identified flag (7) nodejs/src/ingestion/common/persons/PoE.md,person-merge-*.tsFlag hash, rollout, variants (8) rust/feature-flags/src/flags/flag_matching.rs,flag_matching_utils.rs,test_flag_matching.rs(the variant vectors are PostHog's own test values)Property operators (8) rust/feature-flags/src/properties/Flags response shape (8) rust/feature-flags/src/flags/flag_match_reason.rs,api/; runtime capturesRemote config fields (9) runtime captures ( config.js), posthog-jspackages/browser/src/remote-config.tsLazy bundles (9) runtime captures ev_69c9eef3…/ev_d91bc8d5…Recording config fields (9) posthog-js packages/browser-common/src/types/remote-config.ts;ev_64cbe291…Snapshot requests, field compression, masking (11) ev_64cbe291…; posthog-jspackages/browser/src/extensions/replay/Survey fetch, survey shape and events (12) ev_64cbe291…; posthog-jspackages/browser-common/src/types/surveys.ts,packages/core/src/surveys/events.tsExposure event (13) ev_64cbe291…Heatmap events, scroll properties (14) ev_32496edb…; posthog-jspackages/browser/src/heatmaps.tsand its scroll managerLocal evaluation request and response (15) recorder3/recorded.jsonl(posthog-node 6.0.3); posthog-nodesrc/extensions/feature-flags/feature-flags.tsHash key overrides (16) ev_32496edb…($anon_distinct_id);rust/feature-flags/src/flags/flag_matching.rs(hash key override lookup)Survey targeting flags, iterations, partial responses (17) ev_32496edb…; posthog-jsextensions/surveys/,browser-common/src/types/surveys.tsReplay settings, plugin and canvas events (18) ev_4682126f…; posthog-jsbrowser-common/src/types/remote-config.ts,extensions/replay/Cohort and action semantics PostHog public docs (cohorts, actions); rust/feature-flags/src/properties/for cohort property filters; the selector subset is our ownExperiment statistics PostHog's public docs (posthog.com/docs/experiments/statistics-bayesian: flat prior, chance to win, 95% threshold, 50 exposures) for the user-facing rules; conjugate Beta/Gamma/Normal posteriors and Evan Miller's closed form for P(B > A) from standard references; no PostHog code Ordered funnel semantics funnel-udf/src/steps.rsFunnel events limited to the date range products/product_analytics/backend/hogql_queries/funnels/funnel_event_query.pyUnknowns and deliberate differences
- Same-timestamp funnel events. PostHog's funnel query has extra handling for events that share a timestamp; this build takes them in storage order. Results can differ only when two steps' events share a millisecond. Not verified against PostHog.
- Error bodies. PostHog capture answers errors in plain text; this build sends JSON with the same status and a
code. SDKs read only the status. Unknown whether any third-party client parses the text. - Unknown project token is accepted and dropped (as PostHog's capture service does at the edge);
/flagsanswers 401 for it (as PostHog does). - No
regexin insight filters (SQLite has no regex function by default); flags do support it. - Exact matching is case-insensitive in flags (as PostHog's flag service) and case-sensitive in insight filters (as PostHog's queries).
- Local evaluation sends only property-only cohorts; flags with behavioural or static cohorts fall back to
/flags(posthog-node does this itself; observed). PostHog sends more cohort kinds; not needed. /flags?v=4gets the v2-shaped answer; posthog-node 6 accepts it. The full v4 differences were not studied.- Personless events are counted for a person once their distinct id is linked (query-time join); PostHog's exact backfill behaviour was not verified beyond its documentation.
- Geo comes from Cloudflare's request data instead of a MaxMind lookup, so
$geoip_*values follow Cloudflare's naming. - Replay ingestion writes one chunk per request and keeps payloads as sent; PostHog's own replay pipeline (blob storage layout, its
/api/.../snapshotsformat) was not copied and is not needed by the SDK. - Experiments use the first exposure per person and exclude persons seen in two variants; PostHog's exact exposure and metric query rules (and its
ee/code) were not read. Experience continuity follows the MIT flags service's hash key override idea; experiment flags have it on. - Surveys: the feedback widget's server pieces and hosted surveys are not implemented; posthog-js applies display conditions itself.
- SQL is a subset with SQLite semantics (UTC times, case-insensitive
LIKE), not HogQL; queries that need joins are out of scope. - Heatmap unique visitors add per-month distinct counts.
- Cold tier (0.4): enabled on the reference deploy. A token scoped only to the bucket was refused by the sink (code 1012) and needed the catalog and SQL groups on the account (
design.md, "Cold tier"). - Edge counters (0.6) are per-isolate batches flushed through the project store; counts in an isolate evicted before its next flush are lost. Store-side counts stay exact.
- Cold Trends
dau(0.6) counts distinct ids in the cold table, not merged persons (the cold tier has no person table). - Retroactive symbolication (0.6) updates issues and occurrences, not events already in the shards.
- LLM costs (0.6) use the SDK's cost when sent, else a list-price table read from the providers' public pricing pages in November 2026; Anthropic cache writes are priced as input (providers charge more). Not billing data.
- AI endpoints (0.6): the
/i/v0/aimultipart rules (exact part names, Bearer token, gzip) were taken from PostHog's public capture documentation and the MIT capture service's description, not recorded from an SDK in this build. The e2e sends AI events through posthog-node's standard capture (the path@posthog/aiuses through its posthog-node client) plus hand-built batch and multipart requests. - Not studied: web analytics, the data warehouse, the toolbar.
Licensing
- PostHog's
posthog-fossrepository is MIT;posthog-jsis published asApache-2.0 AND MIT. Only those were read; nothing underee/or any other commercially licensed path was opened or used. - The spec states behaviour in its own words; no PostHog source is copied. Test vectors were computed by the reference build and checked against PostHog's MIT test values where they exist.
- The build ships the unmodified
posthog-jsdistfiles and itsLICENSE(asposthog-js.LICENSE.txt) under/static/, as its licences allow. - The replay player is
rrweb-player2.2.2 (npm, MIT, fromgithub.com/rrweb-io/rrweb; licence checked in itspackage.jsonandLICENSE), bundled unmodified into/player.js. - 0.5 error tracking and usage limits were built from: the posthog-js exception autocapture extension and rate limiter (Apache-2.0 / MIT), the posthog-node error-tracking extension (MIT), recorded SDK traffic, the public Source Map Revision 3 specification (tc39.es/ecma426) and Cloudflare's public pricing and Email Sending documentation. PostHog's error tracking backend (Rust cymbal, grouping rules, ee/ billing and quota limiting) was not read; grouping, symbolication, caps and the cost model are our own design.
- The product name, UI and copy are neutral. "PostHog" appears only where the protocol is attributed (here and in
protocol.md).
Choices made for 0.1
- One Durable Object per project instead of Analytics Engine or Basin (
design.md, "Storage"). Basin is the planned path past 10 GB per project. - One team per instance; sign-up limited to the first account or
ALLOWED_EMAILS. - IPs stored by default (as PostHog); per-project "Discard client IP addresses".
- Events kept forever by default (as self-hosted PostHog); a deployer can set
DEFAULT_RETENTION_DAYSfor new projects, and owners change it per project. The daily cron purges when set.
Choices made for 0.2
- Events split into one Durable Object per project per UTC month, persons stay in the head, queries are two-phase (
design.md, "Event shards"). Exact at every age; Basin remains the path for warehouse-scale data. - Session recordings in R2 (one gzip chunk per request), metadata in the head; off per project until enabled; 30 days by default.
- Bayesian experiment statistics with a fixed seed so results are reproducible; 95% probability to be best and 50 exposures per variant before a winner is named.
- Surveys are white-label by default.
Choices made for 0.3
- Actions and cohorts live in the head; selector, href and regex conditions run in JavaScript inside the shards, so no user text becomes SQL. Behavioural cohorts are precomputed (save, hourly, on demand); property-only cohorts are live and go to posthog-node.
- SQL is our own parser over a fixed column map, compiled per shard and merged in the head, with hard limits; no pass-through SQL.
- Funnels stream in person batches instead of loading all rows.
- Heatmap backgrounds are the live page in a sandboxed frame or a recording's first full snapshot; no server-side screenshots.
- Basin cold tier (0.4): stream binding, Iceberg sink, Basin SQL queries from the SQL editor (was: export hook only; needed operator-created Basin and R2 credentials).
Choices made for 0.5
- REA recording (
/workspace/analytics/recorder5-style harness, kept out of the repo): an esbuild-minified demo app with its source map served to real posthog-js 1.450.1 in Chromium against a recorder that logs every request; scenarios: unhandled TypeError, unhandled rejection,captureException, custom$exception_fingerprint, aquota_limitedreply followed by more clicks; plus posthog-node 6.0.3captureException. Saved astests/fixtures/v05-errors.json. Observations: frames innermost last; node frames already carry context lines andin_app; onequota_limitedreply silences that batch key for 60 s. - Caps are enforced exactly in the project store and short-cut at the edge from a D1 cap state, instead of a global counter service: one writer per project already exists, and a lag of up to 30 s past a cap is acceptable. Edge drops were not counted (they are since 0.6).
- Fingerprints ignore line numbers so a rebuild does not split issues; build hashes are removed from filenames for the same reason.
- Symbolication happens at ingest (the stored event is symbolicated); exceptions received before their map was uploaded stay minified.
- Email alerts use Cloudflare Email Sending through a
send_emailbinding (no secret, no extra credential); in-app alerts work without it. - The cost estimate uses list prices and simple per-event assumptions and is labelled an estimate; it is not billing data.
Choices made for 0.6
- Public demo is per project (
projects.public_demo) on an instance that opts in withPUBLIC_DEMO=1, so a demo instance can still hold private projects; visitors get redacted JSON from the same handlers rather than a separate read-only API, which keeps every screen working. - The suppression-rule wire shape and operator semantics were read from posthog-js 1.450.1 as installed (
@posthog/browser-commonremote-config types and the bundled evaluator); rules are enforced again at ingest so older SDKs and server SDKs are covered. - Grouping rules run after
$exception_fingerprintand before the default fingerprint and only affect new exceptions; merge and split use fingerprint aliases instead of rewriting events. - Source maps per release fall back to the release-less map, so 0.5 uploads keep working unchanged.
- Cold Trends compile to one
GROUP BYper series with literal bucket boundaries (CASE WHEN ts < …) instead of date functions, so time zones and DST match the hot engine exactly; physical cold deletion lives in a PyIceberg script because Basin SQL is read-only. - LLM analytics reads
$ai_*events from the existing shards (no new storage), with 19 properties extracted in SQL and full properties only for one trace. - The project cache TTL went from 30 s to 10 s: caps and settings reach every isolate faster for a few more D1 reads.
- Scale and load testing (10M-20M+ events a month) was out of scope for 0.6.
# Research notes This spec describes an independent product-analytics service that speaks the public wire protocol of PostHog's SDKs, so the published `posthog-js` (Apache-2.0 and MIT) and `posthog-node` libraries work against it unchanged. It contains no PostHog code and no PostHog branding. This file records how the protocol was established, the evidence behind each rule, what is still unknown, and the 0.1 and 0.2 choices. It is context, not extra requirements. ## Method: REA (reverse engineer anything) REA was installed from its official source only: the npm package `rea-agents` 6.9.0 (MIT; repository `github.com/morluto/rea`), with its bundled `reverse-engineer-anything` skill. Its method was followed: 1. **Collect evidence, not impressions.** Every observation is an evidence record with a content-addressed id (`ev_<sha256>`), its parameters, and the tool's own stated limitations and unknowns. 2. **Static analysis** of the shipped artifact (`analyze_javascript_application` on the `posthog-js` 1.450.1 npm package). 3. **Runtime capture** of the real SDK in a real browser (`capture_browser_scenario`): a scripted page that loads the snippet, captures, identifies, groups, sets properties, aliases, reads a flag, navigates and resets, with request and response bodies recorded. 4. **Read the reference source** (the MIT `posthog-foss` mirror) for the server-side rules the SDK traffic cannot show: validation, timestamps, person merges, flag hashing, funnel semantics. 5. **Write down obligations** (what a server must do), each tied to its evidence, and list what stays unknown. 6. **Conformance:** replay the same browser scenario against the self-built server and require every SDK request to succeed. ### What REA contributed - The static pass over the minified bundle gave an inventory (476 files examined) but little protocol detail: routes and payloads are built dynamically, and REA reported them as unresolved (its limitation list says so). It did confirm the set of lazily loaded extension bundles. - The runtime captures established what the code does not make obvious: - posthog-js 1.450 sends event batches **gzip-compressed with `Content-Type: text/plain` and no `compression` query parameter**, so a server must sniff the gzip magic bytes; - it posts to `/e/` until remote config loads and then to the config's `analytics.endpoint` (`/i/v0/e/`); - flags go to `/flags/?v=2&config=true` with a gzipped body; - it fetches `/array/<token>/config.js` and then `/static/<version>/surveys.js` **even when surveys are off**, so every extension bundle must be served (the first conformance run caught four 404s here, fixed before the second); - it drops all events from headless browsers unless `opt_out_useragent_filter` is set (no request at all was the evidence); - `$identify` puts `$set`/`$set_once` at the event's top level; - `$pageleave` uses `fetch` with `keepalive`, not `sendBeacon`. - A small recording server (`recorder/`) logged posthog-js and posthog-node traffic byte for byte (`sdk-wire.recorded.jsonl`, `node-sdk-wire.recorded.jsonl`): posthog-node 6 posts **v1** batches to `/i/v1/analytics/events` (Bearer token, gzip JSON) and treats a `{"status":1}` answer as unreadable, and calls `/flags/?v=4`. - The final conformance capture against the self-built server: 16 SDK transactions, all 2xx, none failed. ### 0.2: session replay, surveys, experiment exposure The same method was applied to the three new protocols with posthog-js 1.450.1: 1. **Runtime capture against a recorder.** A second recording server (`recorder2/`) served a page running the unmodified bundles and a remote config with `sessionRecording` set and `surveys: true`, answered `/api/surveys` with an NPS survey (rating, single choice, open) and `/flags` with a multivariate flag, and logged every request decoded. The REA scenario typed into a text field and a password field, clicked, read flags, identified, triggered the survey and answered all three questions through the real popover, then navigated (`capture_browser_scenario`, `ev_64cbe291…`, summary `ev_e4b81384…`, 0 failed). What it established (now `protocol.md` 9, 11-13): - the recorder is lazy-loaded from `/static/<version>/lazy-recorder.js` and posts to the config's `endpoint` (`/s/`), gzip with `Content-Type: text/plain` and no query string, a JSON array of `$snapshot` events carrying `$snapshot_data`, `$snapshot_bytes`, `$session_id`, `$window_id`, `distinct_id` and `token`; - full snapshots and mutations arrive **field-compressed** (`cv: "2024-10"`: a field's JSON gzipped and sent as a byte string), which a reader must undo before rrweb can play them; nothing in the request says so except `cv`; - inputs are masked in the browser (the password and the text field arrive as asterisks), and regular events carry `$session_id` and `$window_id`; - surveys are fetched from `/api/surveys/?token=…&_=…`, rendered by `/static/<version>/surveys.js`, and answered with `survey shown` and `survey sent` events whose properties carry the answers per question id, plus a `$survey_submission_id`; without `appearance.whiteLabel` the popover shows a PostHog footer; - `getFeatureFlag` sends `$feature_flag_called` once per flag per page load with the variant in `$feature_flag_response`. The two recorded `/s/` bodies and the survey events are the fixtures of `tests/replay.test.ts` and `tests/experiments.test.ts`. 2. **Reference source** for names and shapes the traffic cannot show (posthog-js `packages/browser-common/src/types/remote-config.ts` and `types/surveys.ts`, `packages/core/src/surveys/events.ts`, `packages/browser/src/extensions/replay/`), all in the MIT/Apache `posthog-js` repository at `15aef2ce`. 3. **Conformance.** The same scenario against the self-built server under `wrangler dev` (project with recording on, a launched experiment on a multivariate flag, a launched survey): 22 transactions (config, `/flags`, `/e/`, `/i/v0/e/`, `/s/`, `/api/surveys`, `array.js`, `lazy-recorder.js`, `surveys.js`), all 2xx, none failed (`ev_db418df4…`, summary `ev_49b453af…`). Afterwards the server held one recording (10 clicks, 4 key presses, 2 chunks; snapshots decompressed, with a full snapshot; the typed password absent), the survey's answers (NPS 100, the chosen option, the text) and one experiment exposure. ### 0.3: heatmaps, replay capture options, continuity, partial surveys, local evaluation 1. **Runtime capture against a recorder** (`recorder3/`, posthog-js 1.450.1 and posthog-node 6.0.3). The recorder's remote config turned on heatmaps, dead clicks, console logs, network timing with headers and bodies, and canvas recording; `/api/surveys` served a partial-responses survey with targeting flags, conditions and an iteration; `/flags` a multivariate flag. The scenario typed, fetched, logged to the console, drew on a canvas, read flags, scrolled, identified, answered two of three survey questions and navigated. Two runs: on `localhost` (`ev_32496edb…`, summary `ev_8115b374…`, 0 failed) and on a non-localhost host so the network recorder runs (`ev_4682126f…`, summary `ev_99fe82a5…`, 0 failed). posthog-node with a personal key was recorded fetching definitions. Established (now `protocol.md` 14-18): `$$heatmap` events and `$heatmap_data`; the scroll properties on `$pageleave`; console, network and canvas rrweb events and that posthog-js disables the network recorder on localhost; `$anon_distinct_id` in the first `/flags` after identify; the partial-response event sequence (`$survey_completed: false`, cumulative answers, `survey abandoned`) and the `$survey_responded/<id>/ <iteration>` person properties; posthog-node's `/flags/definitions?token=…&send_cohorts` with `Authorization: Bearer` and `If-None-Match`. 2. **Reference source** (MIT/Apache only): posthog-js `packages/browser/src/heatmaps.ts`, `extensions/replay/`, `extensions/surveys/`, `browser-common/src/types/remote-config.ts` and `surveys.ts`; posthog-node `src/extensions/feature-flags/`; `posthog-foss` `rust/feature-flags` (hash key overrides, cohort properties) and public docs for cohorts, actions, heatmaps and the SQL editor. Nothing under `ee/`. 3. **Conformance** (`conform03/`): the same page and scenario against the self-built server under `wrangler dev` (project with heatmaps, console, network headers/bodies and canvas on; an experiment flag with continuity; a partial-responses survey with targeting, conditions and 3 iterations): 30 transactions, 0 failed, every step completed (`ev_9535baea…`, summary `ev_112dda4b…`). Afterwards the server held 15 heatmap points in 12 cells and one scroll-depth page view, a recording with console and network plugin events and a canvas mutation, one partial survey response merged from two `survey sent` events (two answers, one abandoned, iteration 1), the hidden targeting and internal flags, and one experiment exposure. 4. **End to end** (`npm run e2e`, 49 checks) adds posthog-node local evaluation against the instance, actions, cohorts, SQL and the screens. ## Evidence ledger Evidence files are kept with the reference build's working notes (not part of the published spec). | Evidence id | Operation | What | |---|---|---| | `ev_85e84d97…` (summary `ev_964c577b…`) | analyze_javascript_application | posthog-js 1.450.1 npm package, static | | `ev_fc227d71…` (summary `ev_6a20c49e…`) | capture_browser_scenario | posthog-js against the recorder (port 8799), first run | | `ev_1bf4f557…` (summary `ev_5b4d3a36…`) | capture_browser_scenario | same, with identify/group/alias/flag steps; 2 expected failures (recorder has no flag or survey answers) | | `ev_69c9eef3…` (summary `ev_c588dca9…`) | capture_browser_scenario | posthog-js against the self-built server: 18 transactions, 4 failed (`surveys.js` 404) | | `ev_d91bc8d5…` (summary `ev_f5fb4a24…`) | capture_browser_scenario | the same after the fix: 16 transactions, 0 failed | | `ev_64cbe291…` (summary `ev_e4b81384…`) | capture_browser_scenario | 0.2: posthog-js recorder, surveys and exposure against `recorder2` (port 8798), 0 failed | | `ev_db418df4…` (summary `ev_49b453af…`) | capture_browser_scenario | 0.2 conformance: the same scenario against the self-built server, 22 transactions, 0 failed | | `recorder2/recorded.jsonl` | recorder | raw 0.2 requests (`/s/` bodies, survey and exposure events) | | `ev_32496edb…` (summary `ev_8115b374…`) | capture_browser_scenario | 0.3: heatmaps, replay options, continuity, partial survey against `recorder3` on localhost, 0 failed | | `ev_4682126f…` (summary `ev_99fe82a5…`) | capture_browser_scenario | 0.3: the same on a non-localhost host (network recorder on), 0 failed | | `ev_9535baea…` (summary `ev_112dda4b…`) | capture_browser_scenario | 0.3 conformance against the self-built server, 30 transactions, 0 failed | | `recorder3/recorded.jsonl` | recorder | raw 0.3 requests (`$$heatmap`, plugin events, `/flags` with `$anon_distinct_id`, partial survey events, posthog-node definitions polling) | | `sdk-wire.recorded.jsonl` | recorder | raw posthog-js requests (headers, bodies) | | `node-sdk-wire.recorded.jsonl` | recorder | raw posthog-node 6.0.3 requests | Full ids are in the `*.summary.yaml` and `*.evidence.json` files. ## Obligation ledger Source paths are in `PostHog/posthog-foss` at commit `0f73d659` (2026-11-18, MIT) and `PostHog/posthog-js` at `15aef2ce` (npm package licence `Apache-2.0 AND MIT`). | Rule (`protocol.md` section) | Source | |---|---| | Legacy body shapes, dropped events (2) | `rust/capture/src/v0_request.rs` | | Token rule (3) | `rust/capture/src/token.rs` | | Error codes (3, 6) | `rust/capture/src/api.rs` | | Decoding order, gzip sniffing (4) | `rust/capture/src/payload/decompression.rs`; runtime captures | | Timestamps, skew, offset, future cutoff (5) | `rust/common/types/src/timestamp.rs` and its tests | | v1 request and response (6) | `rust/capture/src/v1/README.md`, `rust/capture/src/v1/analytics/*`; node recording | | Illegal distinct ids (7) | `nodejs/src/common/persons/person-utils.ts`, `rust/capture/src/v1/analytics/constants.rs` | | Merge rules, identified flag (7) | `nodejs/src/ingestion/common/persons/PoE.md`, `person-merge-*.ts` | | Flag hash, rollout, variants (8) | `rust/feature-flags/src/flags/flag_matching.rs`, `flag_matching_utils.rs`, `test_flag_matching.rs` (the variant vectors are PostHog's own test values) | | Property operators (8) | `rust/feature-flags/src/properties/` | | Flags response shape (8) | `rust/feature-flags/src/flags/flag_match_reason.rs`, `api/`; runtime captures | | Remote config fields (9) | runtime captures (`config.js`), posthog-js `packages/browser/src/remote-config.ts` | | Lazy bundles (9) | runtime captures `ev_69c9eef3…` / `ev_d91bc8d5…` | | Recording config fields (9) | posthog-js `packages/browser-common/src/types/remote-config.ts`; `ev_64cbe291…` | | Snapshot requests, field compression, masking (11) | `ev_64cbe291…`; posthog-js `packages/browser/src/extensions/replay/` | | Survey fetch, survey shape and events (12) | `ev_64cbe291…`; posthog-js `packages/browser-common/src/types/surveys.ts`, `packages/core/src/surveys/events.ts` | | Exposure event (13) | `ev_64cbe291…` | | Heatmap events, scroll properties (14) | `ev_32496edb…`; posthog-js `packages/browser/src/heatmaps.ts` and its scroll manager | | Local evaluation request and response (15) | `recorder3/recorded.jsonl` (posthog-node 6.0.3); posthog-node `src/extensions/feature-flags/feature-flags.ts` | | Hash key overrides (16) | `ev_32496edb…` (`$anon_distinct_id`); `rust/feature-flags/src/flags/flag_matching.rs` (hash key override lookup) | | Survey targeting flags, iterations, partial responses (17) | `ev_32496edb…`; posthog-js `extensions/surveys/`, `browser-common/src/types/surveys.ts` | | Replay settings, plugin and canvas events (18) | `ev_4682126f…`; posthog-js `browser-common/src/types/remote-config.ts`, `extensions/replay/` | | Cohort and action semantics | PostHog public docs (cohorts, actions); `rust/feature-flags/src/properties/` for cohort property filters; the selector subset is our own | | Experiment statistics | PostHog's public docs (posthog.com/docs/experiments/statistics-bayesian: flat prior, chance to win, 95% threshold, 50 exposures) for the user-facing rules; conjugate Beta/Gamma/Normal posteriors and Evan Miller's closed form for P(B > A) from standard references; no PostHog code | | Ordered funnel semantics | `funnel-udf/src/steps.rs` | | Funnel events limited to the date range | `products/product_analytics/backend/hogql_queries/funnels/funnel_event_query.py` | ## Unknowns and deliberate differences - **Same-timestamp funnel events.** PostHog's funnel query has extra handling for events that share a timestamp; this build takes them in storage order. Results can differ only when two steps' events share a millisecond. Not verified against PostHog. - **Error bodies.** PostHog capture answers errors in plain text; this build sends JSON with the same status and a `code`. SDKs read only the status. Unknown whether any third-party client parses the text. - **Unknown project token** is accepted and dropped (as PostHog's capture service does at the edge); `/flags` answers 401 for it (as PostHog does). - **No `regex` in insight filters** (SQLite has no regex function by default); flags do support it. - **Exact matching** is case-insensitive in flags (as PostHog's flag service) and case-sensitive in insight filters (as PostHog's queries). - **Local evaluation** sends only property-only cohorts; flags with behavioural or static cohorts fall back to `/flags` (posthog-node does this itself; observed). PostHog sends more cohort kinds; not needed. - **`/flags?v=4`** gets the v2-shaped answer; posthog-node 6 accepts it. The full v4 differences were not studied. - **Personless events** are counted for a person once their distinct id is linked (query-time join); PostHog's exact backfill behaviour was not verified beyond its documentation. - **Geo** comes from Cloudflare's request data instead of a MaxMind lookup, so `$geoip_*` values follow Cloudflare's naming. - **Replay ingestion** writes one chunk per request and keeps payloads as sent; PostHog's own replay pipeline (blob storage layout, its `/api/.../snapshots` format) was not copied and is not needed by the SDK. - **Experiments** use the first exposure per person and exclude persons seen in two variants; PostHog's exact exposure and metric query rules (and its `ee/` code) were not read. Experience continuity follows the MIT flags service's hash key override idea; experiment flags have it on. - **Surveys:** the feedback widget's server pieces and hosted surveys are not implemented; posthog-js applies display conditions itself. - **SQL** is a subset with SQLite semantics (UTC times, case-insensitive `LIKE`), not HogQL; queries that need joins are out of scope. - **Heatmap unique visitors** add per-month distinct counts. - **Cold tier** (0.4): enabled on the reference deploy. A token scoped only to the bucket was refused by the sink (code 1012) and needed the catalog and SQL groups on the account (`design.md`, "Cold tier"). - **Edge counters** (0.6) are per-isolate batches flushed through the project store; counts in an isolate evicted before its next flush are lost. Store-side counts stay exact. - **Cold Trends `dau`** (0.6) counts distinct ids in the cold table, not merged persons (the cold tier has no person table). - **Retroactive symbolication** (0.6) updates issues and occurrences, not events already in the shards. - **LLM costs** (0.6) use the SDK's cost when sent, else a list-price table read from the providers' public pricing pages in November 2026; Anthropic cache writes are priced as input (providers charge more). Not billing data. - **AI endpoints** (0.6): the `/i/v0/ai` multipart rules (exact part names, Bearer token, gzip) were taken from PostHog's public capture documentation and the MIT capture service's description, not recorded from an SDK in this build. The e2e sends AI events through posthog-node's standard capture (the path `@posthog/ai` uses through its posthog-node client) plus hand-built batch and multipart requests. - Not studied: web analytics, the data warehouse, the toolbar. ## Licensing - PostHog's `posthog-foss` repository is MIT; `posthog-js` is published as `Apache-2.0 AND MIT`. Only those were read; nothing under `ee/` or any other commercially licensed path was opened or used. - The spec states behaviour in its own words; no PostHog source is copied. Test vectors were computed by the reference build and checked against PostHog's MIT test values where they exist. - The build ships the unmodified `posthog-js` `dist` files and its `LICENSE` (as `posthog-js.LICENSE.txt`) under `/static/`, as its licences allow. - The replay player is `rrweb-player` 2.2.2 (npm, MIT, from `github.com/rrweb-io/rrweb`; licence checked in its `package.json` and `LICENSE`), bundled unmodified into `/player.js`. - 0.5 error tracking and usage limits were built from: the posthog-js exception autocapture extension and rate limiter (Apache-2.0 / MIT), the posthog-node error-tracking extension (MIT), recorded SDK traffic, the public Source Map Revision 3 specification (tc39.es/ecma426) and Cloudflare's public pricing and Email Sending documentation. PostHog's error tracking backend (Rust cymbal, grouping rules, ee/ billing and quota limiting) was not read; grouping, symbolication, caps and the cost model are our own design. - The product name, UI and copy are neutral. "PostHog" appears only where the protocol is attributed (here and in `protocol.md`). ## Choices made for 0.1 - One Durable Object per project instead of Analytics Engine or Basin (`design.md`, "Storage"). Basin is the planned path past 10 GB per project. - One team per instance; sign-up limited to the first account or `ALLOWED_EMAILS`. - IPs stored by default (as PostHog); per-project "Discard client IP addresses". - Events kept forever by default (as self-hosted PostHog); a deployer can set `DEFAULT_RETENTION_DAYS` for new projects, and owners change it per project. The daily cron purges when set. ## Choices made for 0.2 - Events split into one Durable Object per project per UTC month, persons stay in the head, queries are two-phase (`design.md`, "Event shards"). Exact at every age; Basin remains the path for warehouse-scale data. - Session recordings in R2 (one gzip chunk per request), metadata in the head; off per project until enabled; 30 days by default. - Bayesian experiment statistics with a fixed seed so results are reproducible; 95% probability to be best and 50 exposures per variant before a winner is named. - Surveys are white-label by default. ## Choices made for 0.3 - Actions and cohorts live in the head; selector, href and regex conditions run in JavaScript inside the shards, so no user text becomes SQL. Behavioural cohorts are precomputed (save, hourly, on demand); property-only cohorts are live and go to posthog-node. - SQL is our own parser over a fixed column map, compiled per shard and merged in the head, with hard limits; no pass-through SQL. - Funnels stream in person batches instead of loading all rows. - Heatmap backgrounds are the live page in a sandboxed frame or a recording's first full snapshot; no server-side screenshots. - Basin cold tier (0.4): stream binding, Iceberg sink, Basin SQL queries from the SQL editor (was: export hook only; needed operator-created Basin and R2 credentials). ## Choices made for 0.5 - REA recording (`/workspace/analytics/recorder5`-style harness, kept out of the repo): an esbuild-minified demo app with its source map served to real posthog-js 1.450.1 in Chromium against a recorder that logs every request; scenarios: unhandled TypeError, unhandled rejection, `captureException`, custom `$exception_fingerprint`, a `quota_limited` reply followed by more clicks; plus posthog-node 6.0.3 `captureException`. Saved as `tests/fixtures/v05-errors.json`. Observations: frames innermost last; node frames already carry context lines and `in_app`; one `quota_limited` reply silences that batch key for 60 s. - Caps are enforced exactly in the project store and short-cut at the edge from a D1 cap state, instead of a global counter service: one writer per project already exists, and a lag of up to 30 s past a cap is acceptable. Edge drops were not counted (they are since 0.6). - Fingerprints ignore line numbers so a rebuild does not split issues; build hashes are removed from filenames for the same reason. - Symbolication happens at ingest (the stored event is symbolicated); exceptions received before their map was uploaded stay minified. - Email alerts use Cloudflare Email Sending through a `send_email` binding (no secret, no extra credential); in-app alerts work without it. - The cost estimate uses list prices and simple per-event assumptions and is labelled an estimate; it is not billing data. ## Choices made for 0.6 - Public demo is per project (`projects.public_demo`) on an instance that opts in with `PUBLIC_DEMO=1`, so a demo instance can still hold private projects; visitors get redacted JSON from the same handlers rather than a separate read-only API, which keeps every screen working. - The suppression-rule wire shape and operator semantics were read from posthog-js 1.450.1 as installed (`@posthog/browser-common` remote-config types and the bundled evaluator); rules are enforced again at ingest so older SDKs and server SDKs are covered. - Grouping rules run after `$exception_fingerprint` and before the default fingerprint and only affect new exceptions; merge and split use fingerprint aliases instead of rewriting events. - Source maps per release fall back to the release-less map, so 0.5 uploads keep working unchanged. - Cold Trends compile to one `GROUP BY` per series with literal bucket boundaries (`CASE WHEN ts < …`) instead of date functions, so time zones and DST match the hot engine exactly; physical cold deletion lives in a PyIceberg script because Basin SQL is read-only. - LLM analytics reads `$ai_*` events from the existing shards (no new storage), with 19 properties extracted in SQL and full properties only for one trace. - The project cache TTL went from 30 s to 10 s: caps and settings reach every isolate faster for a few more D1 reads. - Scale and load testing (10M-20M+ events a month) was out of scope for 0.6. - Collect evidence, not impressions. Every observation is an evidence record with a content-addressed id (
Is it for me?
- Kind
- web app
- Platforms
- web, backend
- Tools
- TypeScript, Cloudflare Workers, Cloudflare Durable Objects (SQLite), Cloudflare D1, Cloudflare Queues, Cloudflare R2, Cloudflare Basin (Pipelines, Catalog, SQL), Wrangler, lz-string, posthog-js, rrweb-player, esbuild, Vitest, Playwright, posthog-node, Cloudflare Email Sending (optional), PyIceberg (optional cold-tier pruning)
What does it need?
- Agent permissions
- shell, network
- Commands it runs
npm installnpx wrangler d1 migrations apply DB --localnpx wrangler devnpm testnpm run e2enpx playwright install chromium
What changed?
- 0.6.020 Nov 2026LLM analytics: $ai_generation/$ai_trace/$ai_span/$ai_embedding events from the PostHog AI SDKs (standard capture plus /i/v0/ai/batch, multipart /i/v0/ai and /i/v1/ai/events) with an overview (generations, tokens, cost from the SDK or a built-in price table, p50/p95 latency, errors, models), a filtered traces list and a trace page with a span tree and latency waterfall. Error tracking gains releases ($release/$app_version) with per-release source maps and retroactive symbolication on upload, merge and split, assignment, custom grouping rules, suppression rules sent to posthog-js remote config and enforced at ingest, Python/Go/Rust stacks and index (sectioned) source maps. Trends can read the Basin cold tier (Hot/Cold source), and scripts/cold-prune.py physically prunes cold rows past retention with PyIceberg. Usage counts flags, config and API requests and events/requests dropped past a cap, and the cost estimate leads with list price. Optional PUBLIC_DEMO=1 read-only demo mode with redaction; deep links serve the app. Upgrade: apply D1 migration 0005_v06.sql (projects.public_demo); Durable Objects upgrade themselves.
- 0.5.019 Nov 2026Error tracking: $exception events from posthog-js and posthog-node are grouped into issues by fingerprint (type plus top in-app frames, or $exception_fingerprint), with status, first/last seen, occurrences, affected users, stack traces symbolicated from uploaded source maps (API plus upload script), person and replay links, new-issue and regression alerts, and exception autocapture served in remote config. Usage: per-project monthly caps on events, recordings and replay bytes with drop or sample past the cap and quota_limited answers so posthog-js pauses; autocapture and replay sampling defaults for new projects; a Usage page with events per day by type, replay volume and an estimated Cloudflare cost; alerts at 50/80/100% of a cap and on daily spikes, in the app and optionally by Cloudflare Email Sending. D1 migration 0004_v05.sql.
- 0.4.219 Nov 2026Ingest-only hostnames are now opt-in: every hostname serves the full app unless listed in the new optional INGEST_HOSTS var, which keeps the 0.4.1 behaviour (SDK endpoints, /static and /_health only; pages redirect to SITE_URL). design.md recommends one instance per measured site on its own subdomain.
- 0.4.119 Nov 2026A Worker can serve extra hostnames, such as a first-party ingest subdomain on the measured site: SDK endpoints, /static and /_health answer on every host, while dashboard pages redirect to SITE_URL and other owner API writes there get 421, so sign-in and cookies stay on one origin. No config change.
- 0.4.019 Nov 2026Optional cold tier on Cloudflare Basin: every ingested event is also streamed through a Basin Pipelines Worker binding into an Apache Iceberg table in R2 (Basin Catalog), and the SQL editor gains a Cold source that compiles the same query language to Basin SQL with the project and retention filters always applied. Off unless the COLD_STREAM binding, BASIN_* vars and BASIN_TOKEN secret are set; upgrades a 0.3 instance in place (no D1 migration).
- 0.3.018 Nov 2026Actions (CSS selector, text, href, URL matchers), static and dynamic cohorts usable as filters in insights, flags, surveys and experiments, click/rage/dead-click/mouse-move heatmaps and scroll depth over the live page or a recording, a read-only SQL editor across shards, posthog-node local evaluation, experience continuity, survey targeting/iterations/partial responses, replay sampling/triggers/network/console/canvas settings with paged playback, and streaming funnels and surveys without row limits. Upgrades a 0.2 instance in place (D1 0003, hourly cron).
- 0.2.018 Nov 2026Session replay (posthog-js recorder, R2 chunks, rrweb-player), experiments (multivariate flags, Bayesian results), surveys (posthog-js /api/surveys, results) and time-sharded event storage past 10 GB per project with exact counts. Upgrades a 0.1 instance in place.
- 0.1.118 Nov 2026posthog-js is credited with its actual licence (Apache-2.0 and MIT). The e2e flag check waits for the flags reload after identify, so it also passes against a deployed instance over the network.
- 0.1.018 Nov 2026First version.
For agents: start.md says how to build this spec, spec.json holds everything on this page, and bundle.tar.gz holds its files.