Culture

Business websites

.com

Brand protection

Online stores

Hosting

Data privacy

Policy-making

Registrars

Investing & reporting

Internet Basics Series

How to

People

Cybersecurity

Web trends

Tech news

Security

COVID-19

Technology detection

domains

Software

WordPress

TLDs

GDPR

Loura Kruger-Zwart

There are all kinds of behind-the-scenes things going on online, including ‘secret messages’ for different kinds of website visitors. If you’ve had a look around on our blog before, you’ll know all about the humans.txt file message for crediting the humans behind a website’s creation, ads.txt for advertising protocols, and security.txt as the first stop for cybersecurity-related queries. But since the launch of a certain AI helper in November 2022, there’s been a new digital visitor on the online block: Large Language Models (LLMs) are collecting all kinds of information from the internet—about websites, companies, products and services, tips, tricks, and everything in between. But are these LLM-collected pages of information the right details from each website? How do they know which webpages have the most relevant information? 

It makes sense that website creators should be able to communicate with LLMs to make sure they’re retrieving high-quality answers, should future visitors first come in contact with their products, services, or advice via ChatGPT or another GenAI colleague. Introducing the newest website ‘secret message’: LLMs.txt.

As LLMs and crawlers traverse the web and collect information, they’re looking at the first pages of your site — they don’t know if your ‘about us’ page is more important than that obscure blog you wrote five years ago, and it certainly isn’t going to check every single page on your site either. Just as robots.txt indicates how crawlers can index by setting rules and directing to sitemaps, the point of LLMs.txt is to help an LLM find the best information about your website in the most direct way. This could be a list of prioritized pages with relevant details about your site, links to documentation, or a collection of quick points or resources. See a comprehensive example here, on an AI observability platform’s website. 

As with other kinds of .txt files, this one can also be added to your website root as a markdown file. According to llmstxt.org, it should include your website’s title and description, and perhaps some important details about your business’s products or services, if applicable. Next come the links to documentation, core information, or anything relevant you want to be sure an LLM could answer or share if asked. Keeping structured information on hand and easy to access in LLMs.txt files benefits LLMs. This allows them to easily access specific details, like software library features, corporate structures, legal information, personal CV details, product categories, or educational course offerings. Jeremy Howard of llmstxt.org writes: 

Creating a LLMs.txt on your website is not difficult and could bring many benefits, so is there any reason not to have one? On the one hand, this protocol is still very new and more or less still a proposal. Most LLMs likely aren’t scanning for it yet, which means the text might not even be seen. Not even Google has a page for LLMs.txt—but that doesn’t mean LLMs aren’t being considered. For example, Google currently sets rules for tools like ChatGPT, AnthropicAI, PerplexityBot, and Claude in its robots.txt file, as does the Guardian here. Most websites might start this way, including instructions for LLMs on a different page, rather than creating a whole new file. On the other hand, why not get ahead of the curve and do both? 

Keen to see which websites have already embraced LLMs online? Take a look at the growing list of examples here in the LLMs.txt directory.

Tech

The way we use the internet is changing: Large Language Models like ChatGPT or Claude can direct human users to websites without a search engine in the middle. But how do LLMs know what site information is actually relevant to the prompt? LLMs.txt seems to be the answer—a way to specify exactly what information LLMs and crawlers should gather and ultimately share if someone asks. Read on to find out more.

Optimizing websites for the future of search? LLMs.txt explained

Tim Kreutz

In the 2021/2022 academic year, 40 percent of students who enrolled in their first year at a Dutch university, came from abroad. To grow its highly educated workforce and attract workers with specialist skill sets from outside the European Union, the Dutch government offers the Highly Skilled Migrant Program. 

Highly Skilled Migrant program offers visas

Through this program, companies can function as visa sponsors for migrants, offering them a Dutch residence permit with an accelerated application procedure. The Dutch Statistic Netherlands Bureau CBS reports that at the end of 2021, the highly skilled migrant scheme was the most used migration program by the 87 thousand labor migrants from outside the EU/EFTA zone currently working in the Netherlands (figure 1).

The Public Register Recognized Sponsors is a list of organizations that offer Dutch visa sponsorship under the Highly Skilled Migrant program. The list currently contains 10,087 company names and their Chamber of Commerce registration number (KVK-nummer in this Dutch example), but lacks further information (see Figure 2). 

This lack of information makes it hard for Non-EU citizens applying for a job in the Netherlands to find relevant positions. Information such as website, branch, location of (main) office and contact details is currently missing from the register, but is vital for jobseekers in their job search. 

Dataprovider.com's web data is an out-of-the-box solution for enriching open data, such as this public register. To give an example of how our web data can be used, we enriched this public register of visa sponsors with information that is relevant for applicants from outside the EU to find positions that are suitable to their interest and field of expertise. 

When you have only snippets of information, such as a company name or a Business Registry Number, we can match it to our database of websites and supplement it with relevant context for more than 200 data points.

In the case of the Highly Skilled Migrants program, we want to add relevant data to all visa sponsors, such as a website address, location information, (region, city, and address), branch information (Standard Industrial Classification or SIC) and extracted keywords. Without any refining, we directly matched roughly 33% of the company names in the register. The hardest fields to find were company address and email address (25% and 24% respectively), and the easiest were hostname and SIC-code (36% and 34%) (see figure 3).

The preliminary results indicate that some company names are more challenging to match to our database. To improve the number of matches, we clean up the company names in the register. We remove the mention of the legal form in company names (the Dutch BV is comparable to an LLC and N.V. is similar to an Inc). Matching the names without these abbreviations may provide better results. As an extra step, we remove common abbreviations from the names in the register and attempt to match these alternative versions of the names. The improved results are listed in Figure 3. 

The enriched dataset contains 2,225 rows with detailed company information. Download this PDF with all the information we found and added. Using this file, it's easy to find, for example, legal companies in Amsterdam, navigate to their website and look for vacancies. Alternatively, you can query the keywords field and seek companies working with, for example, ChatGPT, and contact them directly at the extracted email address.

Have a dataset you would like to enrich? Contact us at info@dataprovider.com, and we will set you up with a free demo.

Business Information

The Netherlands is a knowledge-based economy. With eight Dutch universities ranked in the top 100 of the Times World University Rating in 2023, it doesn't come as a surprise that Dutch universities are popular among students from other countries. 

Enriching the Dutch public register of visa sponsors with detailed company information

Veronika Vilgis

We already showed you how data centers power the web. But if they are not connected, there is no web. That's where the submarine cables come into play. All your digital information is passing through these undersea cables, travelling hundreds of thousands of kilometers in mere fractions of a second. This infrastructure forms the backbone of our digital lives, but for most people it remains largely unseen and unconsidered. 

Submarine cables are connecting the world

Without this extensive cable network, there wouldn't be a functioning global internet. Undersea fiber optic cables crisscross the seabeds, connecting coastal data centers to ensure fast data transfer around the world. 

Globally, there are currently 485 undersea cable systems in service and another 70 planned for construction (Figure 1). Multiple cables often follow similar routes, providing redundancy in case of cable faults or breaks. The longest cable to date is called the SEA-ME-WE 3. It connects, as the acronym suggests, South-East Asia, the Middle East and Western Europe, and stretches 24,233 miles (39,000 kilometers). 

But even that impressive length will soon be surpassed with the completion of the 2Africa project in 2024. The 27,961 mile (45,000 kilometer) long 2Africa cable will connect 33 countries on 3 continents; Africa, Europe and Asia, linking over three billion people.

Until recently, telecommunications companies were laying down most of the cables. But over the past decade, big tech giants have started taking more control. Google alone privately owns six cable systems and has invested in an additional 19 subsea cables. Amazon, Meta, and Microsoft have invested in their fair share of cable networks too. Planning, building and deploying a new cable system is an enormously complex and expensive undertaking, costing up to US$ 500 million and can take several years. 

Before a cable can be laid, the sea floor needs to be mapped to determine the best possible route. That means avoiding strong currents, volcanic activity, and extreme changes in elevation. While deep down in the ocean protection is generally high, when cables approach shore additional safeguarding may be needed, such as concrete trenches. 

Once a route is decided upon, cable laying vessels are sent out to sea. The loading of the cables alone can take months, as was the case for the Marea cable. That 4,104 mile (6,605 kilometers) long, just over 10 million pounds (4.6 million kilograms) heavy transatlantic cable now connects Virginia Beach in the United States with Sopelana in Spain.

Undersea cables can experience faults and disruptions

Cable faults are more common than you might expect. They can happen for multiple reasons, but the most common culprits by far are fishing vessels and ships. They're responsible for two-thirds of all cable faults, with their anchors and nets dragging across the ocean floor tearing at and ripping up cables. Fish bites, including shark attacks, have occurred in the past. But while mentionable, they are so rare that they don’t really pose a threat to global connectivity. 

Forces of nature have disrupted cables before and will likely keep on doing so. In 2012, hurricane Sandy damaged and disrupted several key exchanges on the US East Coast, where undersea cables link North America and Europe. In 2006, a magnitude-7.0 earthquake off the southwest coast of Taiwan cut eight submarine cables, causing internet outages in several countries in the region. More recently, in 2022, the only international sea cable connecting the island of Tonga was cut off due to the undersea eruption of the Hunga Tonga Hunga Ha'apai volcano. 

Espionage and privacy hidden under the sea

Underwater internet cables have also become strategic targets for espionage and warfare. Historically, countries have tapped into these cables to intercept and record sensitive information. During the Cold War, in Operation Ivy Bells, the US placed listening devices on Soviet undersea communication lines, gathering valuable intelligence about Soviet activity. The operation was compromised in the early 1980s due to a betrayal from within the American intelligence community. But you don’t have to go back as far as the 1980s to find cases of cable interception. 

In 2013, The Guardian revealed that both the British Government Communications Headquarters (GCHQ) and the US National Security Authority (NSA) accessed and processed vast quantities of communications tapping 200 fiber optic cables. The data gathered included recordings of phone calls, email messages, social media entries and browsing histories of individual internet users. Although there has been no recent report of this kind of tapping, you can be sure that with increasing geopolitical tensions, someone is probably listening in somewhere.

In times of war, sabotaging or cutting undersea cables can cause significant communication disruptions. This dates back as far as World War I, when the UK severed Germany's undersea telegraph cables. In recent years, concerns about the vulnerability of undersea cables and the potential for espionage have grown. Russian submarines have been observed operating near crucial undersea cables. While no open-source reporting has confirmed any tampering or tapping, their proximity has sparked unease among Western defense officials about Russia's capability to disrupt or monitor the data flowing through these cables. An interruption in these cables could have dramatic impacts on global financial markets, communications, and internet access. 

For most internet users, undersea cables are out of sight and out of mind. Yet, these unseen conduits form the backbone of global connectivity and are a crucial part of the infrastructure that allows everyone around the world to stay in touch and share information online instantly. The next time you send an email or stream content from servers on another continent, spare a thought for the network of submarine cables that enable your digital experience.

While most of us now largely experience the internet through Wi-Fi and mobile phone data plans, those systems eventually link up with physical cables that carry the information across continents and oceans. Today we’re taking a deep dive to discover the physical infrastructure of the internet, hidden beneath the sea. 

From the ocean floor to your screen: submarine cables and the internet

Olle Vastbinder

A humans.txt is nothing more than a simple text file listing the people working for your organization. It’s a nice way to give them the credits they deserve. Often the file is focused on the people involved in making the website, but sometimes everyone in the company is listed. The choice for which people and which departments to include is entirely yours.

No matter what you put in the humans.txt, make sure people agree to be included. Not everyone wants to share their data that freely, so be sure to make it optional. If you don't want to put in a lot of names or keep the file up to date, you can always choose to only name higher management or founders and list the rest under departments or locations.

Humans.txt is still small, but generating traction 

Our data tells us that midway through 2023, just under 200.000 websites have a humans.txt on their website. That’s not a very impressive number. But there are some very big names among the companies that do have it. Google, Medium, Wired and Netflix are just a few of the well-known companies sporting a humans.txt. With big (tech) embracing the humans.txt, it might be just a matter of time before it is considered best practice to use one for all websites. 

When you take a more in depth look at the numbers in Figure 1 above, it stands out that businesses account for almost half (48%) of all websites using a humans.txt. They are probably eager to show the people involved in making and maintaining their websites. And it won't be a surprise that content related websites account for over a quarter (27%) of all websites with a humans.txt. The reason for that is that content websites are more focussed on recognizing the people creating and making the content for their websites. 

When it comes to creating the file, you can add any information  you want. Keep it simple and create a list of names of people working at your company. Or make it more elaborate and add  their job titles, the departments they work at or even links to LinkedIn profiles. Personalcompany’s email addresses, phone numbers, work locations and much more are also great to add.

You can also include other companies whose products you are using, and maybe even some individuals from those companies. It’s your list, and you can add anything you want.

Setting up your own humans.txt is easy. Just open a text editor and make sure you save the end result as a .txt file. Write a small intro and put in the information you want to share. We’ve listed the main things you can put in your personal humans.txt below.

Names and job descriptions or roles
List the people involved in the website's creation and management. Don't forget to mention their roles and responsibilities to give it a personal touch. If your company is a bit bigger, you can just mention the departments or even create a list for each department in the humans.txt.


Contact Information
You can put everything you want in the humans.txt, such as contact details like email addresses, phone numbers or their social media. Be sure you check with everyone if they are ok with sharing all that info. When in doubt, don’t put in contact information.


Acknowledge external partners
Odds are you are using third-party resources, tools, or maybe even open-source projects. Mentioning them shows you appreciate what they do. Spread the love!


Links
Links won’t be clickable in the humans.txt, but you can put them in. Add your companies LinkedIn or maybe a corporate website where people can see who is working at your business. And if you’re always looking for new staff, be sure to add your vacancies site.

Although a text file with names and departments isn’t the place where you would expect some fun, you can play around with the design. Check out the humans.txt we set up for Dataprovider.com. With some ASCII art, even your company name can look nice in a text file. And don’t worry about designing it yourself, there are plenty of tools out there that generate a good-looking file for you.

Once you're happy with the end result, upload the file to the root directory of your website. That’s the place where the well known robots.txt and ads.txt are located. Where those two files both serve a critical task, the humans.txt is just an extra.

Even though it doesn’t offer any mentionable benefits at this time – although some people claim it may very slightly benefit your SEO efforts – it is nice to just acknowledge the people who are working for you. It may also become important to add your own humans.txt to show your website is created by real people instead of it being generated by AI. The big search engines have not yet indicated they are tracking that, but showing your content is original can certainly help.

With the rising popularity of AI, you would almost forget that people are still doing most of the work in creating and maintaining a website. Programming, developing, designing, writing, creating, at the basis of all that there still are actual humans doing the work. To acknowledge the work those people put in, it is becoming increasingly popular to add a humans.txt file to a website. Although the concept of the humans.txt file has been around for some time, it is becoming increasingly popular to add this simple file to a website. But what is a humans.txt file, should you be using it, and how can you create one yourself? We’re covering all these questions in this blog. 

Why you should add a humans.txt file to your website – and how to make one

For an algorithm to learn what information is important and how to allocate a website to a certain category, we first need to train the algorithm with a list of websites that are already correctly classified. In order to create those lists, we employ several people whose task is to scan through thousands of websites each year to help train our classifiers. 

This is very important work as it helps us to ensure and continuously evaluate data quality. There are different types of annotations the team makes, sometimes it is a yes or no decision, for example if a website is an online store. Other types of annotations are more elaborate, for example, determining the name of a company and extracting address information. In the video below you can see how our annotators go through different websites to extract information and record information on whether the site is an online store. To facilitate this process, we developed a custom tool that aids in the recording of information.

When we build a new classifier we start off with a relatively short list of websites for each annotator. Then we determine how much the annotators agree with each other. This is done by our data analytics team that calculates an agreement score, called Cohen’s kappa coefficient to measure inter-rater reliability. In Figure 1 below you can see how well annotators agree on website classification. Overall annotators 1-3 are performing similarly, annotator 4 might find this task a bit more tricky. For some websites it is relatively easy to determine the type, such as a forum and what is called a placeholder-parking. But for other types of placeholders (e.g., hosting and developer), nearly all annotators show some level of uncertainty. In such cases we first discuss the reasons within the team for any uncertainties, then review our definitions and concepts and repeat the process until we reach a high level of agreement. 

Once the team reaches a high level of agreement, we move on to the next round of annotations. Usually, during this second stage an annotator goes through a longer list of randomly selected websites. We sample those on the same day as our spider crawls the web so we ensure information used for existing classifications and the one our annotators see is the same. Each annotator also regularly goes over the lists their peers have already annotated to ensure continued agreement and detect any mistakes that may have occurred.

One frequent challenge our data analytics team faces is to determine just how many websites need to be annotated to ensure the classifier is trained well but at the same time keep the number of annotations as low as possible to reduce workload.

We calculate various metrics to achieve exactly that. In Figure 2 you can see the evaluation (F1, Precision, Recall) of the detection of a field and its error versus the number of annotations. With bootstrapping we can calculate the estimate metric (graph on the left) and the error of this estimate (graph on the right). We can see that after approx. 500 annotations, the estimates are stabilising while the error maintains over 0.05. We stop annotating at 650 annotations because the error is finally below 0.05.

As you can see, annotating websites is a challenging and time-consuming process but is absolutely crucial when it comes to structuring data, to ensure our data is accurate and adheres to the highest quality standards. Last year our annotators visited tens of thousands of websites, delivering data for classifiers, data surveys and field quality evaluations. They are one of the few people in the world that have been to the strangest and weirdest corners of the web. If they find a hidden gem they’ll share it with the rest of the company and because this has been a lot of fun, we'll soon be sharing some with you in our upcoming LinkedIn posts.

As you hopefully know, at Dataprovider.com we index the entire internet on a monthly basis and structure the data. In order to meaningfully structure data we frequently need to classify data, detect new data and assess data quality. For example, in order to determine to which country a website belongs, you generally need to take various sources of information into account such as the language used on the website, whether there is an address or where the website is hosted.

Data quality: a glimpse into our annotation team

Mathijs Baas

Asset management is the practice of identifying, classifying and protecting a company's physical and digital assets. This includes buildings, computers and sensitive data, among other things. By implementing effective asset management policies and procedures, you can minimize risks of loss or damage to your assets, so you can ensure that they’re being used securely.

One important aspect of asset management is asset discovery. This is the process of identifying and cataloging your company's assets so that you have an accurate inventory of what you have and where it is. This information can then be used to inform decision-making and identify potential security risks.

It’s also important to make sure that external parties aren’t using your assets. If someone gets access to an unprotected asset, they could use it for nefarious purposes. A forgotten asset can result in a breach of a network. Assets that are vulnerable to hacking and security breaches can be anything that’s hooked up to the internet. For example, these can be websites, forgotten subdomains or machines within a domain.

To mitigate these risks, it’s important to know what you have. Especially within larger organizations, it can be hard to keep track of what is connected to the internet. 

An example of a potentially dangerous asset is a website about a company event that runs on outdated software: you’ve forgotten that it exists but it’s still up and running. It could also be a team website whose domain wasn’t registered through the right channels and years later it still exists without the network administrator’s knowledge. 

Luckily, at Dataprovider.com we’re able to identify these assets and keep track of what’s connected to the internet within a domain. 

Let’s take a look at an example: the domain kosherwine.com. We identify 20 domains and 26 subdomains that are connected to kosherwine.com. The owners of kosherwine.com also own, for example, kosh.com and challahconnection.com. The domains are linked to 21 separate servers and 17 unique IP addresses. The majority receive little to no traffic, which could indicate that these subdomains/domains are not being maintained very well, were simply registered at some point but have remained dormant or a combination of the two.

With this data, it’s possible to create a complete overview of all connected domains and subdomains and the security risks they may carry. As a system administrator, you can create a dashboard with multiple data points that can help you assess the security of the domain by monitoring:

Ownership: You can see any related domains. 

Subdomains: This indicates the total number of subdomains associated with the main domain. 

IP addresses: You can see the IP addresses that are associated with the domain.

Traffic index: This shows the amount of traffic to the domain.

Using the example of kosherwein.com, we show how you can use web data to get an overview of the assets associated with your domain. Such information is crucial to protect your business from security threats because you can only create a secure business if you know what you have to protect.

Are you keeping track of your company's assets? If not, you could be putting your business at risk.

Follow us

Questions?