Why one User Agent is not enough for web services and bots

Web services and bots play an increasingly important role in the modern internet. They help provide information, enhance interaction, and improve comfort levels for users. However, for effective operation and data protection, a simple User Agent is not enough, which is why using DNS names for IP addresses is necessary.

A User Agent is a string that your browser sends to the server with each request for a web page. However, it can be easily spoofed. Malicious actors and malicious bots can easily change the User Agent to appear as a regular user. This can allow them to bypass protection mechanisms such as request limits or IP address blocking.

Why DNS Name is important

If your company develops web services or maintains useful bots, the best solution would be to use your own User Agent and DNS name within your domain. Without using a DNS name for IP addresses, distinguishing a hacker from a bot can become a challenging task. Hackers often use various methods to forge the User Agent to mask their true identity or intentions. They can create their own User Agent to make it look like a legitimate service or bot. Therefore, without accurate information about the DNS name, distinguishing these hackers from legitimate services may be impossible.

Using a DNS name can also help prevent "phishing" attacks, where malicious actors try to create the impression that they represent a legitimate service or bot. Thanks to the DNS name, it is possible to verify that the requested IP address matches the expected domain name, which helps protect users from such attacks. In the case of such phishing attacks, the reputation of the service may be damaged because users will see in their website logs your service's User Agent performing malicious requests. This may lead to a loss of reputation.

If a website uses a firewall to filter malicious traffic and only allows access for trusted services based on User Agents, there is a vulnerability in the defense, and the firewall may be useless in the case of attacks with forged UAs. Additionally, the firewall may block access to the website for web services or bots due to the nature of their requests to the website, and it is necessary to somehow add them to the firewall exceptions.

The service should provide either an up-to-date list of its IP addresses publicly, so that it can be regularly updated in the firewall, or use its own DNS Name. If your service does not have static IP addresses, using a DNS Name is simply essential.

How to check if the service uses DNS Name

For example, you can check the Google bot. Google provides instructions on how to identify if the bot visiting your site actually belongs to Google. To do this, perform a reverse IP resolution, and you will see that the IP has a DNS name in the domain google.com, and it is impossible to forge. 

If your website uses a web service and you want to ensure that requests to the site are indeed coming from the web service's IP addresses, request from their support to add a DNS Name to the IP address for accurate identification.

For example, by executing the command "nslookup 66.249.64.48" in the console, we will see that this IP address has the DNS Name crawl-66-249-64-48.googlebot.com, and now we are sure that this IP belongs to the Google search bot. Similarly, we check "nslookup 17.58.101.179" and receive the response 17-58-101-179.applebot.apple.com, and we are confident that the address belongs to the Apple bot.

Ultimately, the use of custom DNS names for the IP addresses of web services and bots becomes an important mechanism for effective protection and identification. It allows distinguishing hackers from legitimate services and bots and also prevents possible attacks and threats associated with forged User Agents.