Culture

Business websites

.com

Brand protection

Online stores

Hosting

Data privacy

Policy-making

Registrars

Investing & reporting

Internet Basics Series

How to

People

Cybersecurity

Web trends

Tech news

Security

COVID-19

Technology detection

domains

Software

WordPress

TLDs

GDPR

Dataprovider.com

Who controls the internet’s infrastructure?

A handful of tech giants—Google, Apple, Meta, Amazon, and Microsoft—wield monopolistic control over key aspects of the web, from search engines and social media to cloud computing and app marketplaces. Their influence dictates how billions of people access information, communicate, and conduct business online. However, beyond these five companies, other critical parts of the internet are also dominated by just one or a few powerful players, creating a digital ecosystem where choice might often be more illusion than reality. Let’s dive into the data. 

We’ve explored the corporations behind internet infrastructure in previous blog posts. Web hosting, for instance, is largely concentrated in the hands of just three companies. The vast majority of websites are built using WordPress, and the backbone of the internet—such as domain registration and SSL certification—is overseen by just a few key companies.

When it comes to some fundamental online services, just a select few lead the way as well. As seen in Figure 1, WooCommerce, Shopify, and Wix eCommerce account for two-thirds (70.4%) of the shopping cart system market. Microsoft 365 holds a 57.7% share of online email marketing services, while Calendly manages 63.6% of online appointment scheduling. Regarding live chat solutions, Meta’s Facebook Messenger and WhatsApp Business lead with 61.3% of the market, and Google Tag Manager, Analytics and Call Conversion Tracking power nearly half (46.6%) of all web analytics tools.

Other key sectors of the internet are similarly concentrated. The largest affiliate program on the web is Amazon Associates with a ​​53.9% market share. Google reigns supreme in both online advertising and mapping services, with its ad network comprising Google Publisher Tag, AdSense, Ads, AdSense Custom Search, and Marketing Platform dominating the industry with 69.2% and Google Maps holding a 74.2% share of map technologies. Meanwhile, YouTube is the undisputed leader in video playback, accounting for two thirds (66.7%) of all embedded video players on the web.

Even crucial behind-the-scenes elements of the internet remain far from decentralized. 46.6% of SSL certificates are issued by Let’s Encrypt, while GoDaddy is the largest domain registrar with a 27.1% share.

The concentration of power discussed here raises serious concerns. Tech oligopolies not only collect vast amounts of user data, but also shape what information people see, often with little transparency or oversight. Critics argue this threatens privacy, innovation, and even democracy itself. Supporters counter that these companies offer valuable, widely used services and that users are free to seek alternatives. However, with such overwhelming market dominance, true competition remains stifled, leaving consumers with limited choices.

Governments and regulators worldwide are now wrestling with how to curb this growing power. Potential solutions include breaking up major companies, restricting mergers and acquisitions, enforcing interoperability between platforms, implementing stricter data usage rules, and demanding more algorithmic transparency. Whether these efforts succeed remains to be seen, but one thing is clear: the internet, once envisioned as a decentralized and open platform, is now largely controlled by a select few corporations.

In a world increasingly dominated by centralized or concentrated internet services, a fascinating anomaly persists: the control of core internet security rests, surprisingly, with seven individuals, scattered globally. Each holds a unique key; each key a component of a system that (theoretically) allows for the shutdown and reboot of the entire internet. While the details of the keys' precise functionality and security measures remain rather opaque, this system demonstrates a remnant of decentralized control. That said, it entails other risks, like being potentially vulnerable to circumstances like the unavailability or compromise of keyholders. 

This seemingly retro approach offers a counterpoint to the consolidation of power within the digital realm, and highlights a tension between security and the potential for both exceptional control and catastrophic failure in this model. This system’s reliance on human trust and geographical dispersion stands in stark contrast to the ever-increasing concentration of power within the tech industry. 

Further, blockchain and related technological developments are built on the idea of decentralization. At its core, a blockchain distributes control among more than one participant and thus transfers decision-making authority from one to many. Tech privacy activism is growing in the digital sphere, and one group has even launched an experimental data-leak warning system based on the principles of a gas leak: the Smell of Data device (seen above) alerts users if digital insecurity is present or data is leaking.

Of course, it’s hard to compete against the interests, cashflow, and (political) influence that big tech has on the web and our modern world. But this does not mean we have no options: decentralization remains a core tenet of the internet, and people are still fighting for its value in the face of challenge. Our own choices come down to ensuring our privacy settings are up-to-date, knowing where our information goes, and supporting small and growing (digital) companies over the big one, where possible. Importantly, policy is a key way to shape the behavior of big companies: according to the NYT, 

Successes in these fields are a quiet testament to the enduring possibility of decentralized power and a vital reminder that even in our hyper-connected world, some critical functions can still rely on a balanced distribution of authority, far removed from the monolithic structures of modern tech giants.

Tech

Our data shows that just a few companies control key aspects of the web, from communications to email to appointment scheduling. While this concentration raises serious concerns about privacy and innovation, we also take a look at the efforts to bring core internet values like decentralization back to the forefront. Who really owns the internet, and how can we reclaim it? Read on to learn more.

The overlooked monopolies of the internet: who really owns the web?

Loura Kruger-Zwart

It's not just people roaming the web anymore, and we’ve known about the new kids on the block for a while: robots. A company or organization’s online robot—also called a bot, crawler, scraper, or spider—is tasked with exploring the internet for specific reasons, like indexing to rank websites or collect data. For most websites, being open to these bots and being indexed by a search engine, for example, is an excellent way to optimize and rank higher in search results. But not every website wants each of its webpages accessed or indexed by a bot—what then? The robots.txt file is this modern problem’s modern solution. 

Essentially, robots.txt is a text file that a website administrator adds to their site to tell bots and crawlers where they can or cannot go, and what they should or shouldn’t index. It's the digital bridge between human and machine, one that helps maintain a symbiotic relationship between the two online.

The bots that traverse the web and collect information are looking to index your site and its webpages, ready to be shown to searchers (in the case of Googlebot) or broken down into technical details, as the Dataprovider.com spider does. SEO is improved when search engine bots can quickly and accurately identify the pages of a website. 

This is where a robots.txt file comes in: it indicates how crawlers can index the web by setting rules and directing to sitemaps. The website owner explicitly allows or disallows (particular) robots to access or index the site, and outlines which bots can access what parts of the site. The owner can direct bots to the unique sitemap, which usually outlines the relevant and appropriate pages to be indexed, and excludes private, irrelevant, or administrative parts of the site. In contrast to LLMs.txt, which gives additional information and specific details, robots.txt is intended to be a more ‘objective’ and straightforward resource for our robotic co-surfers.

However, all of this hinges on an important agreement: adherence to the robots exclusion protocol. First developed by Dutch engineer Martijn Koster in 1994 and used across the web since then, Google sought to upgrade the protocol to an official industry standard in 2019. In 2022, Koster and Google wrote the Proposed Standard together, and it was accepted by the Internet Engineering Task Force (IEFT). As with other kinds of .txt files, this one can also be added to a website root as a markdown file. Website administrators can include incredibly specific instructions to bots, as long or short as needed. Find the original guidelines here and more modern recommendations here.

Robots.txt is the practical outcome of the robots exclusion protocol, but, of course, it is still up to crawlers, bots, and the organization behind them to adhere to the rules. Safe to say that most do—there is a benefit for sites to be indexed, but there is also a benefit for crawlers to have access to these sites and their relevant webpages. Crawlers would rather not be denied access, and so, they adhere to the rules. For example, Wikipedia’s robots.txt file is very explicit about its various instructions: it includes small written explanations (for a human reader’s benefit, no doubt) about which bots are in the ‘naughty corner’ for not following the rules. There are too many bots out there to have specific rules for each, but having some indication on a website about the expectations for robots seems integral to the internet today. Ultimately, however, the protocol remains advisory—a polite suggestion—despite using terms like allow and disallow. 

Curious to see some examples of robots.txt on major websites? Take a look at the notable files at Nike, YouTube, Reddit, LinkedIn, and Wikipedia, or add /robots.txt after any domain name to see that website's bot rules.

At Dataprovider.com, we closely monitor robots.txt files as part of our web crawling process. Since our spider scans the web on a monthly basis, the robots.txt file is our first checkpoint to determine whether we are permitted to access a website: if the file explicitly disallows crawling, we respect those directives and do not access the site. This is part of our privacy by design protocol. While we have always recorded whether a robots.txt file is present, we did not systematically track which bots were specifically allowed or disallowed until recently. This has revealed some impressive numbers: in March 2025, our crawler detected mentions of 112,327 unique user agents specified across all robots.txt files. While some of those mentions may be typos, or for bots that are no longer ‘in service’, the total number gives an indication of just how important it is to provide structure and instructions.

As the web becomes increasingly crowded with more LLM crawlers and other automated agents, understanding who’s out there and what they’re allowed to do is increasingly relevant. Robots.txt provides an interesting glimpse into how website owners manage bot traffic—who they welcome and who they prefer to keep out.

In Figure 1, we break down the share of websites based on whether they have a robots.txt file and whether they allow or disallow crawling. The data reveals that 79% of websites include a robots.txt file, indicating that a significant majority of website owners take bot management seriously.

Next, we examined the bots most frequently mentioned in robots.txt files. One key observation is that among the 70 million robots.txt file analyzed, 24% explicitly allow all bots by using a wildcard directive (*). When looking at specific mentions, various Google-owned bots dominate the list, with one major exception: AhrefsBot. This SEO-focused crawler appears in 4% of robots.txt files, making it the most frequently mentioned bot overall (see Figure 2).

To take the analysis a step further, we focused on bots that are explicitly disallowed from crawling—meaning they are entirely blocked from accessing any part of the website. In Figure 3, we highlight the most frequently disallowed user agents. Among the top bots are PetalBot, Nutch, and MJ12bot.

PetalBot is operated by Petal Search, a search engine developed by Huawei, and is used to index web pages for Huawei’s search ecosystem. Nutch, on the other hand, is an open-source web crawler widely used for research purposes, custom search engines, and large-scale web scraping projects. MJ12bot is the crawler for Majestic, a search engine and SEO tool specializing in backlink analysis and website rankings.

None of these crawlers are inherently malicious; their impact largely depends on how they are configured and used. While concerns about PetalBot stem from Huawei’s ties to the Chinese government (given past allegations of surveillance and data privacy issues), there is no direct evidence that the bot engages in malicious activity. Wikipedia’s robots.txt specifically calls out MJ12bot, saying “Observed spamming large amounts of [...] and ignoring 429 ratelimit responses”, despite MJ12bot’s website saying it obeys robots.txt. 

Creating a robots.txt file on your website isn’t complicated and has plenty of benefits, so why not? Depending on the nature and content of a site, it may be useful to indicate clearly which URLs the crawler can access on your site, to indicate the (ir)relevance of certain pages, or to manage and avoid unwanted visitors. Without it, crawlers can and will access all parts of a site. Website owners put different degrees of detail into their robots.txt instructions: Reddit, for example, disallows all crawlers with two lines of text, while LinkedIn’s robots.txt is over 4,000 lines of specification and rules.

As discussed in our recent article, the rise of generative artificial intelligence (genAI) has introduced another visitor to the digital world. Large Language Models (LLMs) also gather all kinds of information from the web; including LLMs.txt on a website specifies which relevant details the bot should share if asked via ChatGPT, Claude, or another service. However, since this development is still rather recent, many website owners opt to include LLMs instructions in their robots.txt files. 

The internet is an ever-evolving place, as we know, and keeping up with the changes is complicated. Access, security, privacy, and responsibility must be considered when new developments arise, and protocols like robots.txt are one way to bridge the gap between humans and our machines.

Robots surf the web alongside us, but have a different set of rules for how to traverse the digital world. Whether they’re indexing pages for a search engine, tracking changes for SEO, or collecting information for a structured database, bots need structure and instruction. How do we ensure they’re doing the right thing? Read on to explore the wonderful world of robots.txt.

The digital bridge between human and machine? How robots.txt works

Olle Vastbinder

To understand how useful WHOIS is, it's good to first take a look at its origins. First, it's good to know WHOIS isn't short for anything, it is literally the question 'Who is'. WHOIS can be traced back to the beginnings of the internet when it was still called ARPANET. At that time, the network was almost solely used by research institutes and universities. Despite its relatively small size, a directory listing all owners was useful to get contact details fast and quick.

But as the internet kept growing, it was becoming more important that everyone was providing their data similarly. Only by ensuring everyone was supplying their information in the same way, it was possible to make it accessible to everyone. In the early 1980s, the WHOIS protocol was standardized. With just a simple query, everyone could now look up domains, people, and other information for any domain. Check the list below for the most common fields.

What information can you typically find in a WHOIS record?

The contact information for the domain owner

The registrar and its contact information

Creation, update and expiration dates for a domain

The technical and admin contacts for a domain

To make any data useful, it is essential that it's organized following a standard, or protocol. That's why WHOIS is a protocol. A protocol makes sure everyone is using the same names for the same fields, so things don't get mixed up. A good example of the need for a protocol for WHOIS data is for when you want to know the date of registration for a domain. There are several dates mentioned for different aspects of a domain, such as registration, last change of ownership or when it expires. A protocol helps to prevent all those dates from getting mixed up and ensures you get the date for the information you require.

While the WHOIS protocol dictates how the information is shared, it can differ how much information there is available for a domain. Despite it being compulsory for every website owner to supply the information to their registrar, many fields are considered optional or are just outright ignored. Take a look at the list with all possible data fields in a WHOIS here. 

At Dataprovider.com, we currently get the data for the 12 most important WHOIS fields. But as we stated above, that doesn't mean each website will have entered their data in all those fields. When we dive into our data, we see that out of just over 134 million available websites, 117 million are using one or more of the 12 WHOIS fields we detect. That means 17 million websites do not have any information in their fields or are using names for their fields that do not follow the WHOIS protocol. Below, you can see the 12 data fields we track and the percentage of websites that gave an available response that are using that field. 

Website owners are still quite free to enter any information they want in their WHOIS and leaves room for website owners to be creative. They can even add their own fields or use slightly different names to describe a certain field. This way, you can end up with thousands of variations for the same field. As this can cause unclarity about the ownership of domains and who is responsible for them, the call for a new protocol has been going for some years now. That's why - in due time - WHOIS will be replaced by a standardized and machine-readable JSON format protocol called RDAP. More on that later in this blog.

Whenever someone registers a domain, the registrars are required to provide identifying and contact information to be used in WHOIS. Not only is the registrar responsible for getting this information, but also for keeping it up-to-date. 

Despite the variety of WHOIS fields website owners can use, most websites only offer their basic contact information in their WHOIS. The key elements for most are the creation, expiration and last change dates, along with contact information like country, email address, name, and organization.

There are numerous cases where WHOIS data can play an important role. The reasons for requiring the information can differ. Maybe you want to buy a domain for your business, or you're just curious about who is behind a certain website. But there are also some more legal reasons for using WHOIS data. 

For example, companies battling trademark infringements require the contact information to know where to send cease and desist or takedown notices. Or law enforcement wanting to take down websites with illegal content need the registrar information to close down domains. We've listed the main reasons why people use WHOIS data below.

Some general use cases of why and how WHOIS data is used:

Tracking availability of domain names
See if a domain is actively used and maintained. Inactive domains or domains that might be nearing expiration can be interesting to acquire or keep track of. 

Finding trademark and brand infringements 
Identify and take down domains with trademark infringements or stolen content. The contact information is key to determine where to send any communication. 

Discover domain or typo squatting
Checking WHOIS records can help uncover these “typo squatters” or fraudulent domain owners attempting to capitalize on other brands. The registration information may point to someone with no obvious connection to the name, or the domain may have just been recently registered despite its resemblance to an already prominent site.

Law enforcement
For law enforcement, WHOIS is essential in getting the right information to take down websites with illegal content and track down owners and registrars to do so.

Identify domain trends
See and interpret trends in a registrant’s domain portfolio or changes in contact information over time. Great to spot trends.

Due diligence 
WHOIS is an essential resource for online research to assess the reliability of a domain and its contents. It plays a part in due diligence investigations.

Detect and prevent spam, malware, and phishing
Uncovering details about registrants promoting illegal or dangerous content on their domains allows hosts to disable access or report the offenders to the proper authorities. Scrutinizing registration patterns across multiple dubious domains at once can also reveal wider spam or malware networks seeking to evade detection.

Public contact data gives a sense of trust
If you can easily find ownership information for most domains. That can give a sense of trust, no matter if you are a casual user, checking news or doing online shopping, or a professional tracking down business opportunities. 

While the need for clear WHOIS data is apparent, it does pose some problems when it comes to privacy. Where WHOIS wants as much contact information as possible, privacy regulations can sometimes conflict with that.

Privacy regulations can prohibit the public listing of some or all contact details. In these cases, no or strongly limited contact details are listed. Although this information is left out of the public WHOIS record, the relevant contact details are kept by the Registry and Registrar. Only people with legitimate interest can get the details by contacting either the Registry or Registrar. In contrary to the above, some countries have made it mandatory to show all contact details in a WHOIS.

As WHOIS can be easily accessed by anyone, it's also available to spammers, scammers, and others out to misuse data. That's why not everyone wants their data listed in WHOIS. In these cases, privacy protection services can be used, although they are not allowed for all TLDs. These privacy services fill the necessary WHOIS data with unique but untraceable contact details, concealing the real identity of the owner. In the meanwhile, the privacy service itself does have your contact details and can forward any communication to you. These services are sometimes part of the registrars package, but for a fee, you can also enlist a separate WHOIS Privacy Service. 

But there are more ways people use to avoid their private data ending up in the public domain. When we check our data for on what people are using in their WHOIS name field, a whopping 53 million websites (40% of the total) are not listing a real name, but are showing some kind of privacy disclaimer. They use terms like 'redacted for privacy', 'hidden' 'not disclosed' and even 'Ano Nymous' in their WHOIS name field to prevent having to give their real name. 

Despite its benefits, the WHOIS protocol will not last forever. New (privacy) regulations, technologies, and protocols will make the existing protocol obsolete. For a few years already, people have tried creating more modern substitutes, but all had their downsides and never made it. But with RDAP, a very likely successor is on its way. 

RDAP is an acronym for Registration Data Access Protocol. And just like WHOIS, its goal is to create a standard method in which domain ownership can be determined. But that's where the comparison ends. RDAP lists the data in a fixed, standardized and machine-readable JSON format. Using a fixed format, all fields for all users are the same, ensuring uniformity. The adaption of RDAP will probably become compulsory as of 2025 according to ICANN. That gives registrars time to adapt their systems to work and comply with RDAP.

RDAP will most likely mean the end of WHOIS, but it doesn't have to be, as both systems can exist together. But as that means that the data must be entered in both systems, it seems likely that the use of WHOIS will gradually die out. 

WHOIS is the best way to find out more information on who owns a domain. With a WHOIS query, you can get all sorts of ownership information on a website. These can be quite short, from just a name, e-mail and country of residence, to quite extensive information listing all details for an organization. The information in WHOIS can be used just to contact an owner. But it also has its uses in brand protection, combating spam and fraud and much more. Keep reading and learn all there is to know about WHOIS.

WHOIS in depth - What data fields are website owners using?

Veronika Vilgis

How Google Analytics became the go-to tool

Google Analytics launched in 2005 and offered free statistics and insights into website traffic and user behavior. It quickly became the most used analytics tool on the web, taking over 50% of the market by 2007. There were several factors why Google Analytics managed to dominate the market in just a few years:

﻿Free and user-friendly: For small businesses with limited resources, Google Analytics provided an accessible way to gain insights into website traffic and user behavior.

Google Ads: Tracking the performance of Google Ads campaigns through Google Analytics made the service even more appealing. Marketers could see how ads translated into site visits and conversions.

Innovation: Google continually rolled out new features, reports, and integrations to provide more robust insights. Features like goal tracking, ecommerce tracking, and mobile app analytics were introduced over time.

Big data: Google Analytics leveraged Google's data infrastructure to offer large sample sizes, fast processing, and features like custom dimensions and metrics. 

﻿Decision-making: Google Analytics popularized a culture of data-informed digital marketing. Marketers were able to make decisions based on actual user behavior rather than hunches.

As you may have noticed, Google Analytics caters mostly to the needs of the marketers. The individual users whose data is used don't have a say about their data. That's where the GDPR comes in, putting the privacy and rights of the individual user at its core. But the GDPR clashes with Google's interests, resulting in quite some conflicts between Google and Europe and some of its individual countries.

Core conflicts between Google Analytics and GDPR compliance

To better understand the conflict, it's important to first see how the GDPR works and where it clashes with Google Analytics. There are three main points of conflict where the GDPR and Google Analytics clash:

Data Collection and Consent: Imagine someone walks into a store. GDPR basically says you can't just take their picture without asking for permission. Similarly, sites now need to ask visitors if they agree to gather some data about their visit.

Data Retention: The GDPR requires that personal data be kept for “no longer than is necessary.” Let's say you keep a diary of who visits your house. GDPR advises not to keep notes about a guest from 5 years ago, as that's no longer relevant. The same goes for web data; some experts even recommend not holding on to it for more than 2 years.

Data Transfer: Google Analytics data is stored in the U.S. (and other countries). But the GDPR prohibits transferring EU citizens' personal data to countries with inadequate data protection laws. It’s like if someone from Europe gave you a letter to keep safe, GDPR wants to ensure you don’t just store it anywhere else.

Under the GDPR, websites and apps must obtain explicit consent from users before collecting and processing their personal data. All three of the main points above must be clear to the website visitor. Failure to inform the visitor – or not adhering to other rules set in the GDPR – can result in hefty fines, running into the hundreds of millions.

Why Google Analytics currently isn't compliant

Achieving compliance with the GDPR while using Google Analytics requires carefully evaluating the terms of service and privacy policies. Among other things, you need to limit data collection and retention, properly obtain user consent, and avoid data transfers outside the EU. With some adjustments, Google Analytics can operate legally, but compliance ultimately depends on how website owners choose to implement and use the service.

Some critics argue that Google Analytics inherently collects more data than is necessary to fulfil its purpose. Additionally, users cannot opt out of data collection if they wish to use a website. This has led to numerous legal challenges for Google in Europe. 

Data protection authorities ban Google Analytics 

Some European countries have banned Google Analytics altogether to avoid potential GDPR violations. The bans began after a landmark ruling by Austria's data protection authority in late 2022, which was later followed by similar court decisions in other countries, including Italy and France. The rulings emphasized that Google Analytics did not comply with GDPR's requirements, as some or all data was transferred and stored in the U.S. At the time of writing, the U.S. isn't considered to be a safe country, as its privacy regulations aren't comparable to the more strict ones in Europe. Storing data there is therefore considered unlawful. 

(Non)compliance with these bans is difficult to identify and even more difficult to enforce. Although in July 2023, the Swedish authorities have issued the first fines for several major companies not complying with the ban. At Dataprovider.com, we scan the web on a monthly basis and gather diverse information from publicly accessible websites. So we can see if the ban in those countries has had any effect on the use of Google Analytics. We focused on Italian, French and Austrian websites as the ban was put in place there over a year ago. (After the ban in those countries, Google Analytics was banned in various other European countries too. But less than a year has passed since those bans, so we left them out of the equation for now.)

In total, we tracked 653,321 Italian, 900,876 French and 167,963 Austrian websites that had Google Analytics or Google Tag Manager installed before the announced ban. For Austria, we looked at data from January 2022. For France, we selected data from December 2021 and for Italy May 2022. We then checked if these websites made any changes to their website analytics tool by March 2023. What percentage of these websites removed Google Analytics or switched to another analytics provider? Might there even be websites that added Google Analytics after the ban?

In Italy, 82% of websites did not remove Google Analytics. In France, the percentage is slightly lower but still quite high, with 74% of websites remaining with Google to track their analytics. The highest percentage of websites changing to another provider can be seen in Austria: there, 34% removed Google Analytics, leaving 66% that stayed with Google. 

If website owners removed Google Analytics, where did they move to? Both in France and Italy, the vast majority adopted Adobe Analytics software. In contrast, website owners in Austria opted for Clicky and Statcounter to keep track of website analytics. 

Obviously, in response to the ban, Google has made changes in an attempt to become GDPR compliant. Website owners now must obtain proper consent from users before enabling Google Analytics tracking. Google also allows disabling of certain features like user-level data and offers IP anonymization and data retention controls.

But additional steps may be needed for full GDPR compliance. Some websites use consent management platforms to clearly explain how data is used and obtain verifiable consent from users. Privacy policies must transparently disclose the use of Google Analytics data collection and must allow users to opt out.

The future of Google Analytics in Europe remains uncertain. While Google has made efforts to assist websites in following GDPR guidelines, some critics argue their analytics tool will never be fully compliant. Ongoing legal challenges and bans seem likely, as privacy regulators closely monitor how personal data is collected and used across the digital landscape. The banning of Google Analytics may not be a solution that works for most website owners, especially when the ban is not being reinforced everywhere. 

Our data shows clearly that the majority of website owners in Austria, Italy and France did not change their analytics tool after the ban was imposed in their countries. Nevertheless, the Google Analytics ban is an opportunity for other companies that provide GDPR complaint web analysis solutions to gain an advantage in a Google dominated market. The decision for which analytics tool to use remains with website owners in Europe, who face a difficult decision in balancing user privacy concerns with business needs. With ambiguous regulations open to interpretation, no easy or risk-free solutions currently exist. Constant re-evaluation of privacy policies and tools will be necessary to keep up with this evolving data protection landscape.

Google Analytics has been the go-to tool for website owners looking to track traffic and user behavior since it was launched in 2005. But concerns about privacy and the European General Data Protection Regulation (GDPR) have put website analytics under intense scrutiny and even led to outright bans of Google Analytics in some countries. Thanks to GDPR, individuals have more control over their personal data. Besides that, companies and organizations are fully responsible for safeguarding their clients' personal information. In some European countries, it was deemed that using Google Analytics went against GDPR regulations, and its usage was banned.  In this blog, we're taking a look at how website owners are responding to those bans.

How GDPR-driven bans affect the use of Google Analytics in Europe

Mathijs Baas

The Dark Web is the internet's hidden alleyways, tucked away from the bustling streets of the World Wide Web that most of us are familiar with. Unlike the websites you visit every day, the Dark Web is inaccessible via standard web browsers or indexed by traditional search engines, such as Google.

The internet most people know about can be broadly divided into three distinct segments or levels: the Surface Web, the Deep Web, and the Dark Web. The Surface Web and Deep Web account for 94% of all the information on the internet, while the Dark Web is estimated to account for around 6%.

The Surface Web, also known as Clearnet or the visible web, is the portion of the internet that is readily accessible to the general public and is indexed by traditional search engines. When you search for a recipe, read news articles, or visit social media platforms, you are navigating the Surface Web. 

The Deep Web is a much bigger part of the internet, but standard search engines do not index it. This includes databases, private academic libraries, member-only websites, and personal email accounts. You are accessing the Deep Web when you log into your online banking account to check your balance or view transactions. While the term 'Deep Web' might sound mysterious, it’s largely unnoticeable, encompassing content that’s not intended for access without the appropriate authorization or credentials.

Lastly, there's the Dark Web, a small and deliberately hidden part of the internet that is not accessible through regular browsers. It requires special software, such as Tor or I2P, to access it. The Dark Web is notorious for its illegal activities. Although it’s true that illicit activities are prevalent on the Dark Web, it is also a platform where political dissidents can engage in communication in countries with oppressive regimes, or where whistleblowers can divulge information without fear of reprisal. 

The roots of the Dark Web can be traced back to the 1990s, when the U.S. Naval Research Laboratory initiated a project called 'onion routing', a technique for anonymous communication over a computer network. This project ultimately led to the creation of Tor in the mid-2000s. 

The primary objective of Tor was to safeguard the online communications of U.S. intelligence. However, once it was released to the public, the broader community began to recognize its potential for ensuring privacy and resisting censorship. As internet use increased, people started to worry about privacy, surveillance and censorship, which led to Tor becoming more popular. 

By the 2010s, with the emergence of marketplaces like Silk Road, the Dark Web started to gain notoriety as a hub for illicit activities. Although these marketplaces brought attention and controversy to the Dark Web, they also ignited the broader debate regarding internet freedom, privacy, and the challenges of digital governance.

Until you see how Tor works, the onion in the Tor name might seem like an odd choice. The onion represents the concept of working in layers. All traffic in Tor is relayed and encrypted three times as it travels over the Tor network. The data is passed through several of the thousands of Tor relays. There’s no single relay that knows where the data originally came from or where it is going. You can compare it to passing a secret note through a group of people, with each person adding a new layer of code to keep the message hidden.

With Tor, internet traffic is routed and encrypted multiple times, which makes it difficult to track activities or locations. Despite Tor's association with accessing the hidden parts of the internet, such as the Dark Web, it can also be used for regular browsing.

Privacy on the Dark Web: The good & the bad

The concept of privacy is a fundamental characteristic of the Dark Web, and it is one of the primary reasons why individuals and organizations may choose to operate there. The Dark Web is distinguished from other parts of the internet by its unique combination of secrecy, anonymity and association with both legal and illegal activities. The illegal aspect of the Dark Web often prevails when people are referring to it. The origin for that negative sentiment can partly be found in the marketplaces on the Dark Web. 

Marketplaces are places that facilitate trade and communication in a shielded environment. Users can buy, sell and exchange goods and services with a heightened sense of anonymity. Common illegal goods and services on offer include, for example, illicit drugs, stolen data, malware and hacking services, firearms and counterfeit identity documents or money. In 2022, you could buy valid US credit card information for as little as $17 or a New York driver's license for around $70.

Some marketplaces have gained notoriety, the most notable of which being AlphaBay and Silk Road. Silk Road, which debuted in 2011, was a pioneer in the underground digital bazaar scene, offering a diverse range of products, ranging from illicit drugs to counterfeit currency. Its user-friendly interface and reliance on Bitcoin as an anonymous payment method made it a massive hit among Dark Web users. However, its prominence also made it a prime target for law enforcement, leading to its eventual closure in 2013. 

AlphaBay quickly filled the void left behind by Silk Road and expanded the range of illicit activity. These marketplaces, while providing a glimpse into the Dark Web's hidden economy, also highlight the seemingly endless cat-and-mouse game between cybercriminals and global law enforcement agencies. AlphaBay was shut down in 2017 by law enforcement. 
In 2021 it was relaunched. But at the beginning of 2023, the owner seemingly disappeared. Due to the way security was set up, the site was basically rendered unusable and it died a slow death. That Illicit activity is lucrative is not a surprise. A prime example of that is Hydra. In 2022, that marketplace was shut down by German and U.S. authorities. At that time, it was the highest-grossing Dark Web marketplace, with an estimated revenue of €1.23 billion.

The Dark Web is rife with illicit activities and potential dangers, but it also serves as a sanctuary for whistleblowers and those evading political repression. Nevertheless, its reputation is primarily shaped by the trade in illegal goods, services and stolen data. It's important to recognize the dual nature of the Dark Web and remain vigilant about personal and organizational cybersecurity measures. To safeguard against data leaks and the potential dissemination of sensitive information on the Dark Web, businesses should adopt comprehensive cybersecurity strategies. 

You probably have heard of the term “Dark Web” before. And although the Dark Web may seem like something out of a mystery novel, it is real and is a big part of the internet. Let's look at what it is, how it works, and how it differs from conventional web.

What is the Dark Web and how does it work?

Privacy laws have a significant impact on society. People are better protected against organizations misusing their personal information. And on the other hand, it has an impact on businesses and governments who are limited in how they work, and it comes with costs as they have to spend a lot of money on legal advice and build privacy-friendly systems.

Since the implementation of GDPR, many other countries have followed suit and implemented their own privacy laws to protect the privacy rights of their citizens. 

Here are a few examples of other countries that followed suit.

China: China's Cybersecurity Law (CSL) is a comprehensive privacy law that regulates the collection, use and storage of personal data by both domestic and foreign companies operating in China.

Canada: The Personal Information Protection and Electronic Documents Act (PIPEDA) is a federal privacy law that governs the collection, use and disclosure of personal information by private sector organizations.

Japan: The Act on the Protection of Personal Information (APPI) is a comprehensive privacy law that regulates the handling of personal information by both public and private entities.

South Korea: The Personal Information Protection Act (PIPA) governs the collection, use and disclosure of personal information by public and private entities.

Brazil: The Brazilian General Data Protection Law (LGPD) governs the collection, use and processing of personal data by both public and private entities.

Companies and governments often communicate about these laws on their websites. In some countries, it is even mandatory to include a privacy policy on the website. Let’s have a look at some European countries and how many websites mention the GDPR or a national equivalent. We focus specifically on business and e-commerce websites as these more frequently collect personal identifiable information (PII) than, for example, a personal website or a blog. Figure 1 shows the percentage of business websites per country that mention a data protection law on their website. The country with the highest percentage of business websites that mention data protection laws is Germany. Here, 41% of business websites reference either the GDPR or the German Datenschutz-Grundverordnung (DSGVO). Austria follows suit after Germany with 37%, Spain comes third with 31% of business websites mentioning data protection. In our sample of thirteen countries, the Netherlands, Finland and Poland have the lowest share: less than 10% of business websites reference data protection regulation, despite all three countries implementing their own legislation.

According to the German data protection law, everyone who collects, transmits, uses or processes personal data through their websites must publish a data protection declaration. This would suggest that approximately 59% of German business and e-commerce sites don’t handle personal information, and therefore, don’t see a need to publish a privacy policy. We dug a little deeper into our data and looked at German business and e-commerce sites that have a feature on their website, such as “login”, “subscribe”, “book”, “register”, or have a shopping cart system and payment providers. These are all features that suggest PPI is being collected. Among these 610,162 websites, 86% have a data protection section on their website, mention data protection or the DSGVO. This indicates that around 14% of business websites that collect personal information may not comply with the law. Having said that, the absence of a published privacy policy on the website doesn't necessarily mean that any PII collected is treated irresponsibly.

Let’s have a look across the pond. The United States don’t have a federal law that designates countrywide rules regarding privacy policies. However, some states have regulations in place. The California Online Privacy Protection Act (CalOPPA) introduced in 2003 requires all commercial websites that collect PII of California residents to post a clearly visible privacy policy that complies with the regulatory requirements. CalOPPA applies as long as the website is accessible by California residents — in other words any website accessible from California, even if it’s not located there. The Act also has a section on rules regarding children. If products or services target children, the Children’s Online Privacy Protection Act (COPPA) comes into play. If a website is directed to, or collects PII from children under the age of 13, parental consent must be sought. Failing to comply with COPPA can be expensive, as was shown by the 5.7 million settlement paid by Musica.ly (now known as TikTok) in 2019 in response to violations.

In Figure 2, we show a map of all websites that mention the California Consumer Privacy Act. As you can see, many websites outside of California mention the Act on their website, likely because they are doing business with California residents. But let’s have a look at the proportion of websites in California that mention the CCPA. Only around 4% of websites mention the actual Act. In contrast, the number of Californian business sites that have a privacy policy is much larger (35%). However, overall the percentage is still surprisingly low. 

Companies that don’t publish privacy statements or legal disclaimers about how they protect their clients PII risk hefty fines and the loss of trust from consumers. The largest fine to date for GDPR violations was issued to Amazon. Inc for €746 million, not for failing to publish a privacy policy but for not being transparent about the collection of information from consumers. Nowadays, there are plenty of free sources where you can find prefabricated Privacy Statements. As a website owner, you can modify the statement  to comply with all the countries’ legislation that you are planning to do business with. Obviously, if you do have a statement, you should also ensure that you really do comply with what you state and protect your clients’ data.

Public Sector

The General Data Protection Regulation (GDPR) is a privacy law that has had a significant impact on the way businesses around the world handle personal data. The GDPR sets strict guidelines for the collection, processing and storage of personal data of EU citizens, and it applies to any organization that handles such data, regardless of their location.

What is the state of privacy laws on business websites?

Gijs Barends

Attackers use what is called an Inline Frame (iFrame) that loads another HTML page within the original website. That iFrame generally imitates the original website, making the user think they are still operating on the original website. The root of this problem is a function that allows a website to load content and scripts from another website. 

Configuring X-Frame-options prevents abuse

A website owner can prevent this by configuring the X-Frame-Options in the HTTP header of their website. The X-Frame-Options HTTP header can be used to indicate whether a browser should be allowed to render a page in a <frame>, <iframe>, <embed> or <object>. This solution can prevent clickjacking attacks by ensuring that foreign content isn’t embedded into the website.

This doesn't mean websites can’t embed content from other websites, you just need to specify who can load content on your website in the header: for example, you can allow specific content from Instagram.

Implementing X-Frame-Options isn’t hard. Yet, when we look at our data, we see that only a tiny proportion of websites globally have implemented an X-Frame-Option in their HTTP header. Out of the more than 120 million websites scanned for this research, only 8% (9.5 million) are protected against clickjacking.

This is staggeringly low. Given the ever-increasing threat of cyberattacks, preventing clickjacking from happening by implementing an X-Frame-Option is a simple way to make the web just a bit safer for everyone.

Clickjacking tricks a user into performing an action they don’t intend to carry out such as clicking on a disguised webpage element. As a result, the user may enter a fake but similar looking website, have his personal information stolen through fake text boxes on top of the real ones or unintentionally spread malware, worms or scams.

Follow us

Questions?