Data sources

Bharat Tracker publishes nothing it cannot attribute. Each of the 767 series on this site carries a list of source ids resolving into data/sources.json, where a source is a specific document — a named press release, a yearbook chapter, a parliamentary answer — with a URL and the date it was read, not a vague gesture at an institution.

Data reaches the site by two separate routes that never mix:

The measured route. Curated series in data/series/ and the World Bank connector in scripts/etl/connectors/worldbank.ts. These become charts. Every value is either published verbatim by its source or computed by a labelled transform.

The reported route. RSS from ministries and news desks, through scripts/etl/connectors/ingest.ts. These become map pins and tracker headlines. They never become chart data. A headline that a plant was announced is evidence that a number may have moved, not the number.

Coverage at a glance

Series carry one of three frequencies: annual (717), fiscal-year (22, India's April–March year, never silently converted to calendar years) and point-in-time (28, a stock count as of a stated date rather than a flow).

Key providers

World Bank — World Development Indicators

The backbone of every cross-country comparison. 690 indicators, fetched from 2001 to the present for India and five comparators — China, Vietnam, Brazil, Indonesia and the United States — chosen because each answers a different question: China as the manufacturing benchmark India explicitly measures itself against, Vietnam as the small-economy counterexample, Brazil and Indonesia as fellow large developing federations, the US as the frontier.

The connector rewrites every series in full on each run and never appends. The World Bank revises past years; an append-only store would freeze a stale revision and quietly disagree with its own source.

Press Information Bureau and the ministries

The primary record for Indian government activity — production and export totals, highway kilometres, airport counts, budget allocations, port cargo. PIB's national feed carries every ministry's releases, so the sector split happens in the classifier rather than in the source list.

PIB, PMO, MEA and Business Standard answer HTTP 403 to any user-agent identifying as a bot, including for their own public RSS. All sixteen PIB ministry feeds were rejected this way on the first live run, so scripts/etl/lib/http.ts retries a 403 once presenting as a browser. That is the only way to read a feed published for the public to read.

SIPRI

Where India publishes nothing, or publishes something that cannot be compared internationally.

SIPRI figures are estimates, and the schema knows it. A think-tank provenance graded high confidence is rejected outright by scripts/lib/validate-series.ts and by a CHECK constraint in the database. Warhead counts in particular are external inferences — India publishes no number at all — and are graded low confidence with an explicit note. Treat them as order-of-magnitude.

South Asia Terrorism Portal

The spine of the internal-security series. SATP publishes year-by-year fatality tables for left-wing extremism and for terrorism as plain HTML — no API, no download — so the connector reads the table.

It is a compilation, not a register. SATP reads the same press reporting and parliamentary answers everyone else does, and disagrees with MHA figures by a few per cent in most years and by more on contested incidents. Nothing sourced from it is graded above medium confidence.

The connector will not publish a fragment. A year is emitted only if all three of its columns parsed; a table yielding fewer than ten years is treated as a layout change rather than a short history, and the previous data is kept. Deaths are not a series where a silent truncation is survivable.

Natural Earth

The gazetteer behind every map pin. npm run geo:build fetches ne_10m_populated_places_simple, filters to adm0name === "India", and merges a curated overlay of project sites that a populated-places dataset does not carry — Sriharikota, Vizhinjam, Dholera, Jewar, Vadhavan, Bhogapuram, Chandipur and others — plus an alias map for the renamed cities.

Result: 211 places across 34 states and union territories, written to data/geo/places.json. Public domain, no attribution required. The file is generated; hand-editing it is a bug.

State boundaries and corridor geometry live in data/geo/india-states.topo.json and data/geo/corridors.json. Corridor lines join real geocoded endpoint cities in straight segments — they are schematic, not surveyed alignments, and the map says so. The opening years are the load-bearing fact.

Think tanks and analysis

The news and events pipeline

109 active feeds declared in lib/sources.ts and nowhere else. No ingest code knows about any particular publisher; adding one is a one-line change. They fall into four groups.

kind on the source decides the label, so trust is a property of the publisher rather than a hardcoded list inside the parser. Reputable and primary are different things: The Hindu is marked reported, not verified, because it is reporting someone else's announcement.

Every sector, from more than one newsroom

Each feed declares the sectors it is expected to carry, and npm run sources:verify fails if a sector is down to fewer than three independent working publishers. Three is the floor because two is a coin flip: when one desk goes dark — and they do — a sector with two sources becomes a sector with one, and nobody notices until the map has a hole in it.

"Independent" counts newsrooms, not feeds. Ten Economic Times desks are one newsroom's editorial judgement ten times over, and they all go down together, so they count once.

npm run test:ingest asserts the declared floor without touching the network; npm run sources:verify proves the feeds actually answer, and writes data/live/source-health.json with the status of every one of them.

Keyword discovery

A publisher's desk only carries what that desk wrote. A plant commissioning covered by one regional paper reaches the map through a keyword search or not at all — which is the difference between eleven states on the map and most of them.

So 31 of the feeds are sector phrase searches, built at run time from SECTOR_KEYWORDS rather than pasted in as URLs. Widening a sector's reach means adding a phrase. Each search is a plain Google News RSS URL — no key, no quota, no account, which is the only reason keyword discovery is affordable here.

Phrases are quoted so they match intact: a bare port matches Portugal, and a bare corridor matches wildlife corridors. Every search asks for the last two days only, matching the map's 2-day mode.

Attribution survives the round trip. Aggregator items carry <source url="…">The Hindu</source>, so the event is credited to the publisher, never to the aggregator, and the " - The Hindu" suffix is stripped from the headline. When a body fetch follows the aggregator link to the publisher, the resolved URL replaces the redirect, so most pins cite the article directly.

When a publisher's own feed stops working

Fourteen declared feeds are disabled, each keeping its URL and the reason the probe gave. Financial Express answers 403 to every user-agent; ThePrint, Construction World and Raksha Anirveda serve an interstitial instead of the feed; Saur Energy and Entrackr 404; ISRO has no stable feed at its published path; MEA blocks; DD News timed out on both probes; and Swarajya, ET Defence and ET Startups parse as feeds but have returned zero items on every run.

Dropping those would quietly remove desks this tracker was asked to follow, so seven of them are reached through a site-scoped search instead — the one mechanism here proven to answer, and one that still credits the publisher, because the aggregator names it on every item. It is a fallback, not a preference: a publisher's own feed is better and gets restored the moment it works again.

From headline to pin

Ingest runs in two passes. Pass one reads every feed, capped at 40 items per desk and 25 per search so no single high-volume feed decides what the run looks at. Pass two fetches article bodies, but only for candidates the headline alone cannot resolve — plus every aggregator link, because following it is the only way to cite the publisher. The budget is 120 requests on the full pipeline and 60 on the half-hourly refresh.

A story becomes a map pin only if it survives, in order:

1. Sector rules — ordered, first match wins: space → defence → trade deals → exports → pipelines → ports → roads & airports → energy → startups → PSU & MSME → manufacturing → infrastructure. A body-only match needs the term twice in the body and once in the first 600 characters, so a passing mention cannot set the sector.
2. Action gate — something must have been done: inaugurated, commissioned, sanctioned, awarded, ordered, signed, test-fired, inducted, delivered, flagged off. Commentary, analysis and speeches are not events.
3. Incident filter — fires, crashes, collapses, arrests, protests, raids and outages are dropped. A tracker of what is being built is not a tracker of what went wrong.
4. Legal filter — court rulings, tribunals, pleas, petitions and appeals are dropped. A Supreme Court judgment about a power company is not an energy project.
5. Geo-location — the gazetteer must find a place in the title or the lede. It returns null rather than guessing, and an unplaceable story is kept off the map rather than pinned to a state capital.
6. Deduplication — three outlets reporting one port commissioning is one pin. Near- duplicates are collapsed on shared place, shared date, an overlap coefficient of ≥0.5 over title tokens and at least three shared tokens; verified wins over reported, then the earliest report.

Every run prints its funnel, and the losses are large by design:

ingest funnel: 784 items -> 273 candidates -> 24 events
               (lost: 178 no sector, 41 not an action, 30 no place)

Stored events are re-checked against the current rules on every merge, so tightening a rule retroactively removes pins it should never have created. Events older than 730 days are dropped.

Official X handles

lib/sources.ts lists 17 official handles — the PM, Home, Defence, Road Transport, Commerce, Railways, Ports, Civil Aviation and Renewable Energy ministers and ministries, plus ISRO, DRDO and PIB. The connector is dormant: X has no free read tier, so it activates only when X_BEARER_TOKEN is set and otherwise reports itself inactive and the pipeline carries on. The PIB and PMO feeds carry the same announcements from the primary source.

Handles are configuration. No post content is ever committed to this repository — it is fetched at run time like any other feed.

The two constructed indices

Every other number on this site reports something somebody else published. The Tonality Score (−100 to +100) and the Action Index (−1.6 to +1.6) do not — they are constructed here, they carry a judgement in their names, and they are graded derived and low confidence for that reason.

Tonality is a proxy for the state's posture: negative reads accommodative, positive reads security-first. Five dimensions, each bounded to ±20 — initiative, civilian protection, containment, attrition, escalation dominance. Action is a performance measure: four components, each bounded to ±0.4 and each measured against the series' own average, with neutralisations and arrests-plus-surrenders pushing up and civilian deaths and incident volume pushing down.

Nothing in either is hand-scored. The obvious way to build a willpower index is to read the year's policy documents and code doctrine and framing by judgement; that tracks the stated posture more directly and is unfalsifiable, unreproducible and impossible for a reader to check. So posture is inferred from what the state did, not from what it said, and every dimension is computed from published fatality counts you can recompute yourself.

The cost of that choice, and five other limits, are published in full on /methodology and enumerated in LIMITS in lib/security-index.ts. The first one matters most: a government that talks tough while operations stall scores low, and one that talks softly while operations succeed scores high. The name says willpower; the arithmetic reads behaviour.

The fatality counts underneath stand on their own. A reader can reject both indices entirely and still use every death count on the page.

AI, research and the digital base

"Defence innovation" and "AI development" are claims, and claims have to be measurable to be worth making. Ten World Bank indicators carry this section: R&D spending as a share of GDP, researchers and technicians per million people, resident and non-resident patent filings, scientific and technical journal articles, charges for the use of intellectual property in both directions, and ICT goods and service exports.

The pair to read together is IP receipts against IP payments — the honest test of whether research becomes property, rather than whether papers get written. The second is researchers per million, where the constraint on any AI or defence-technology ambition shows up as people before it shows up as money.

How sources are graded

Tier — how close to the record

Current register: 52 sources — 17 tier 1, 20 tier 2, 15 tier 3.

Provenance — who produced the number

official (28 series) · multilateral (83) · think-tank (3) · press (9) · derived. derived is reserved for figures this site computes from cited underlying series, and it is always labelled on the chart.

Confidence — how much weight it bears

high (709 series) · medium (40) · low (18).

Four rules are enforced mechanically, in scripts/lib/validate-series.ts and again as CHECK constraints in supabase/migrations/0001_init.sql:

• A series with no sources is rejected.
• confidence: "low" without a note explaining the uncertainty is rejected.
• think-tank provenance graded high is rejected — estimates are not records.
• A series backed only by tier-3 press and graded above low raises a warning.

The database constraints use cardinality(), not array_length(). array_length() returns NULL on an empty array and a CHECK passes on NULL, which meant the constraints accepted exactly the rows they existed to reject. The Postgres-in-WASM tests caught it; npm run test:schema keeps it caught.

Freshness

HTTP responses are cached on disk for six hours (fifteen minutes for feeds), so a development re-run does not hammer public APIs. Every request is retried with exponential backoff and jitter under a hard timeout, and failures are returned rather than thrown — one dead publisher must not take down a run.

A run in which a connector fails is marked partial: the previous committed data stays in place and the site keeps serving last-known-good values rather than blanking a chart. Output is written only after validation passes, so a malformed upstream response cannot land in the repo.

Known limitations

These are properties of the underlying data, not bugs to be fixed in code. /methodology on the site carries the full list.

• Nominal rupees. Indian financial series are stored as published. A large share of the apparent growth in defence exports, production and budget is inflation and currency depreciation, not real expansion. Comparisons across years need deflating; the site does not do it for you, and says so on every affected chart.
• Fiscal versus calendar years. Indian releases run April–March, the World Bank runs calendar years. The two are never silently reconciled — frequency records which is which.
• Definitional drift. Airport counts vary between releases depending on what counts as operational. National highway length grows partly by re-designating existing state roads, so "kilometres added" and "kilometres built" are different quantities that press coverage routinely conflates.
• Estimates presented as data. Nuclear warhead counts and some arms-transfer volumes are external inferences. Graded low confidence, order-of-magnitude only.
• Schematic map geometry. Corridor lines are straight segments between real geocoded endpoints, not surveyed alignments.
• Feeds are a partial window. RSS carries recent items only, so data/events.json accumulates forward from when the pipeline started rather than reaching back. Twenty hand-verified seed events carry the map's earlier history; they are marked with a seed- id prefix and can be retired as ingested history accumulates.
• Recall is deliberately traded for precision. Roughly 180 items per run fail the sector match and are dropped. Two attempts at widening the rules produced false pins — a PM CARES story filed as Tamil Nadu defence, a High Court marriage ruling filed as defence, "fire breaks out in old ATC building" filed as an airport opening — so the gates stay tight. A missing pin is a gap; a wrong pin is a false claim.

Licence and attribution

Underlying data is public-domain government output, CC BY 4.0 multilateral data, or published research credited to its authors. Every figure on the site links to its publisher. Natural Earth is public domain. Bharat Tracker is not affiliated with any government body, political party or contractor, and receives nothing from any of them.

Full register: data/sources.json, rendered on the site at /sources.