The index

Dataprovider.com continuously crawls the publicly observable web. Each month the crawler visits domains across 50+ country extensions and generic TLDs, rendering pages as a browser does and extracting 200+ structured variables per domain: technology fingerprints (CMS, payment providers, analytics, frameworks), hosting and network data (IP, autonomous system, CDN, name servers), content classification (topics, industry, language), and commercial signals (e-commerce features, contact data, brand mentions).

Reports on this hub use a monthly snapshot of this index. The measurement date - the crawl month - is stated on every report, separately from the publication date.

Primary hostname determination

A single website is often reachable on several hostnames (example.com, www.example.com, shop.example.com). To avoid double-counting, every production query on this hub filters on primaryhostname = true: the one hostname per domain that we determine to be the canonical website, based on redirects, canonical tags, and content comparison. This yields 390.1M primary hostnames - the denominator behind every “% of all websites” figure we publish.

Detection

Technology detection is based on observable front-end evidence: fingerprints in HTML source, generator tags, script URLs, plugins, rendered hostnames, and HTTP headers. This is transparent and reproducible, but it has a known bias that we state on every report: technologies that are visible on the page (wallets, checkout buttons, CMS asset paths) are counted more completely than technologies that operate invisibly behind a server (backend payment processors, headless CMS installations).

How to read our percentages

Every share we publish names its denominator. “Share of sites with a detectable CMS” uses the detected population; “share of all primary hostnames” uses the full 390.1M index. Sources that measure only the top 1M or top 10M sites by traffic will report different - usually higher - shares for enterprise-heavy technologies, because parked domains and small sites are absent from their samples. Neither number is wrong; they answer different questions. When citing any figure from this hub, please state the denominator alongside the percentage.

Known limitations

  • Front-end detection undercounts invisible backend technologies (e.g. Stripe, Adyen, headless CMS).
  • Reverse proxies such as Cloudflare mask origin hosting; hosting figures attribute proxied sites to the proxy unless stated otherwise.
  • Content classification depends on crawlable text; image-heavy or login-gated websites are underrepresented in topic data.
  • Third-party, registry, or proprietary sources may hold information we do not observe. Figures are estimates derived from our dataset, not exhaustive counts.

Refresh cycle and data access

Reports are refreshed monthly. Each report offers its full aggregate dataset as JSON (a dated snapshot plus a stable -latest alias) and a PDF edition. Public files contain aggregates only - never per-domain data. Time-series data for publicly-traded platforms is available exclusively through Dataprovider.com. Reuse is governed by our terms.