Link Rot: When Content Disappears from the Web

When browsing websites, seemingly strange things can happen. Out of nowhere, a video of the moon landing may appear, accompanied by the phrase “Houston, we have a problem” and the instruction “Return to base”¹.

Or visitors may suddenly find themselves in a subway station, surrounded by advertising posters reading “Lost” and “Missing” and a display board urging them to “Please proceed to main level.” To complete the scene, a train pulls into the station with its wheels screeching². There is no question that websites like these are creative. But the underlying issue is anything but amusing.

The idea is to prevent website visitors from becoming frustrated and leaving when they try to access an invalid page, for example because they mistyped an address in their browser or clicked a link that no longer works. Normally, they would encounter a message containing a “404” notice along with wording such as “Post not found.” “404 Not Found” is an HTTP status code returned by a web server when the requested resource cannot be found³. For visitors, this is monotonous and frustrating, and many simply leave the website. To prevent this, some companies use creative error pages instead. Some entertain their visitors and redirect them to the homepage or another page on the website. It is an alternative way of dealing with the error, but it does not address the underlying problem.

An Unresolved Problem

A typing error is rarely the reason why a requested page cannot be accessed. More often, the cause is an unresolved problem inherent to hypertext. It has several names: dead links, broken links, link rot, and link decay. There are numerous reasons why they occur. A domain may no longer exist, may have been registered by someone else and currently contain no content, or its web server may no longer be reachable. Dead links also frequently appear when the content of a website is reorganized or when its content management system is replaced, something that happens regularly even at major companies and institutions. For example, after the Swiss Canton of Basel redesigned its website, existing links to the website of the Grand Council no longer led anywhere⁴.

In cases like these, articles are often moved to another directory, renamed, or deleted. For example, suppose a text was previously located in the “Current News” category under the path

[www.companyxy.com/](http://www.companyxy.com/)Current-News

and is then moved to a new category called “News.” In many systems, the corresponding part of the URL automatically changes to

[www.companyxy.com/](http://www.companyxy.com/)News

If the original link had already been included in emails, press releases, newsletters, blog posts, or similar publications, or had been indexed by search engines, readers would now be clicking a link that no longer leads to the intended destination. The same problem can occur when a headline is changed later, for example to correct an error, because URLs are often generated automatically from the page title. Thus,

[www.companyxy.com/News/](http://www.companyxy.com/News/)Our-new-release-is-outt

may become

[www.companyxy.com/News/](http://www.companyxy.com/News/)Our-new-release-is-out

once the spelling mistake in the headline has been corrected.

The spelling error may be gone, but the URL has changed, and the old link can no longer automatically lead to the original article.

Cases like these are common worldwide. Harvard University found⁵ that half of all links appearing in documents issued by the Supreme Court of the United States no longer lead to the original information. Social media is also affected. Computer scientists at Old Dominion University in the United States examined more than 11,000 links contained in tweets relating to six major events between 2009 and 2012. They found⁶ that after one year, 11 percent of the referenced resources had disappeared, and after two and a half years, the average had risen to 27 percent. Even Internet news articles ranked in positions one through three by Google are eventually forgotten by the Internet: 23 percent disappear within the first year, according to an ongoing experiment by Dr. Anja Johanning and Dr. Lutz Goertz⁷.

A Troubling Development

It has been known for a long time that link rot causes problems, and the nature of those problems is nothing new either. In 1997, around 60 percent of users considered link rot one of the biggest problems when using the Web, ten percent more than just one year earlier⁸. A survey conducted in 1998⁹ also stated that link rot irritates users. Even more seriously, however, it warned that link rot contributes to the disintegration of the Web itself. There is a danger that the Web could cease to function as a universal, interconnected hypertext system and instead fragment into a collection of isolated islands of information. Anything that reduces the distribution and usefulness of links between websites can therefore be regarded as a direct attack on one of the fundamental principles of the Web.

The fact that link rot continues despite this is concerning. Closely connected to this development is not only the question of what value a published link really has if there is a strong likelihood that sooner or later it will stop working. It also raises the question of how much value a document itself loses when it contains broken links and important references can no longer be accessed. What does this mean in a world where an enormous number of online articles contain links?

Former Reuters writer Felix Salmon offered one answer. He observed¹⁰: “We have moved into a world of streams, where flow matters more than stock and where the half-life of every piece of content has never been shorter. Nor has it ever been easier to delete enormous quantities of content all at once.” Salmon fears that the entire Web is essentially turning into a kind of slow-motion short-message service, where content exists for some unknown period before it dies and disappears forever.

Salmon refers to blogging pioneer Anil Dash, who wrote about social networks characterized by streams¹¹: “We lost key features that we used to rely on, and worse, we abandoned values that used to be fundamental to the world of the Web. The great achievement of today’s social networks is that they have brought hundreds of millions of new participants into these networks, and they have certainly made a small number of people rich. But they have not shown the Web itself the respect and stewardship it deserves as the medium that made their success possible. Instead, they have diminished the possibilities of the Web for an entire generation of users who do not realize how much more innovative and meaningful their experience could be.”

An Invisible Solution

At least one solution is well known: every URL that has ever been published on the Internet should remain available permanently. URL stands for Uniform Resource Locator and generally refers to an Internet address or a specific path to a resource. It indicates where something is located and how it can be reached. A URL is a subset of a Uniform Resource Identifier, or URI, which provides a standardized identifier for a resource. The inventor of the World Wide Web, Tim Berners-Lee, summarized the principle in a single sentence in 1998: “Cool URIs don’t change”¹². Yet implementing this principle is not as simple as it sounds. For most Internet users, the issue remains invisible. Tim Berners-Lee identified a lack of foresight as one reason for the enormous number of dead links.

The underlying problem is that very few people critically consider every action they take when maintaining content online. Most act with the best intentions. Business owners, webmasters, and editors try to improve their websites by restructuring them or deleting outdated content. In doing so, they often fail to consider that they may be destroying existing links, or they simply do not realize how far-reaching the consequences can be.

Serious Consequences

Dead links invariably damage a company’s image¹³. LEGO appears to have experienced problems with dead links as well, as demonstrated by the following post:

“Once again, a new building instruction includes a link to a website that doesn’t exist… It’s infuriating! For years, people here – and not just me – have repeatedly pointed out dead links, but LEGO simply doesn’t respond. LEGO, you’re incompetent. Shut down your Internet presence if you’re too stupid to maintain it and go play with your colorful plastic bricks instead of constantly misleading your customers with this garbage.”¹⁴

A review of an app from Daimler AG published by CHIP Online is worded more politely, but its judgment is just as damaging:

“If you want to learn more about Mercedes-Benz in the brand section of the iOS app, you are taken only to a dead link. Embarrassing for a company as large as Daimler.”¹⁵

In addition to damaging a company’s image, dead links can also have negative effects on search engine rankings. Links provide search engines with information about the credibility of a page, which is why search engines do not “like” broken links. According to Matt Cutts, former head of Google’s Webspam team, the best-case scenario is that faulty links simply receive no trust¹⁶. More serious consequences are possible if the number of broken links becomes excessive, including potential penalties against the website¹⁷.

Other Internet users may also take advantage of dead links for their own purposes. The relevant terms are “creative link building” and “broken link building.” The principle is that other website operators, perhaps those seeking to generate advertising revenue, identify high-quality and thematically relevant links that currently point to error pages and attempt to redirect that link value toward their own content. Online guides describe two primary approaches: keyword research and targeted research on selected domains¹⁸.

With the keyword-research approach, target keywords are identified first. In addition to the subject matter, factors such as geographic relevance and the target audience may also be considered. Google, Bing, or specialized tools are then used to find pages containing relevant content, often referred to as “relevant resources.” Their external links are extracted, after which the links producing 404 errors are identified. Among these, the most attractive candidates are usually those with the largest number of incoming links. Alternatively, software can be used to identify 404 pages directly. Link-analysis tools and Excel can also help identify “usable” dead links, including when conducting targeted research on a specific domain.

Regardless of the approach, the subsequent process is similar. A web archive is used to retrieve the content that previously existed at the dead URL. Based on that archived material, content on the researcher’s own website is adapted or recreated and uploaded, ideally under a similar name. Finally, the webmasters of websites that still link to the dead resource are informed that their link no longer works and are told that comparable content is available at the newly supplied URL. This is generally considered the most difficult part of the process. However: “Many webmasters appreciate being informed about a newly created page because it allows them to continue offering useful information to their users. Someone who previously linked to an external page is often willing to link to a ‘replacement page’ as well. There is no guarantee of receiving a link, but the chances are not bad,” explains SEO specialist Marcus Obermeier¹⁹.

There is little evidence of guilt about using someone else’s broken links for personal benefit. “The objective is not to steal content, but to restore value that has been lost,” Tino Schade explains in a blog post²⁰. Broken link building is even regarded as one of the most effective link-building tactics^21,22. Ultimately, however, it is highly questionable whether this practice serves the interests of the company that originally created the dead link.

Desert scraping goes even further. It is a black-hat SEO technique involving the reactivation of expired or abandoned domains in order to improve the performance of another website²³.

The conclusion is simple: companies are well advised to keep their links alive whenever possible.

What Is Black-Hat SEO?

Hacker culture traditionally distinguishes between three types of hackers. White hats are considered the good ones. Through their hacking activities, they identify and disclose security vulnerabilities, often become politically involved, and seek to share their knowledge with others. Their hacker ethics date back to the 1960s, and their principles include statements such as “All information should be free” and “You can create art and beauty on a computer.”

Grey hats take a less restrictive view. They may consider damage to other systems or violations of the law acceptable if they believe their actions serve a greater social purpose. Rather than seeing their actions as criminal, they may justify the resulting collateral damage as “constructive destruction.” Black hats, by contrast, are often despised within hacker communities. The stereotypical digital bank robber who hacks systems purely for personal gain or simply to act out their anger is the classic “bad hacker.” Their only justification is a distorted sense of entitlement: “We are the elite. Everyone else has no business being here!”

Solutions

Tim Berners-Lee’s recommendations¹² provide one of the most useful starting points. They describe how links can be designed with long-term stability in mind. “Foresight” is the key concept. It implies an awareness that a problem exists in the first place. Everyone in a company who works with its website must remember that existing content may already have been linked to elsewhere, completely outside the company’s control, including on social networks and in search engines.

This awareness creates the foundation for understanding that renaming a published file because its title is no longer appealing, moving it because it was stored in the wrong directory, or deleting it simply because it is outdated cannot happen without consequences. Conversely, it also means recognizing that many external links included in one’s own content may change or disappear over time. With this understanding, the path toward permanent and stable URIs becomes much clearer.

In principle, there is no reason to change URIs. Simple alternatives exist to moving, renaming, and deleting resources, as Tim Berners-Lee has explained. Server-side controls, for example, can be used to create a flexible relationship between the URI of an object and the physical location of the file representing it. Files therefore do not have to be moved, and their URIs do not have to change simply because the underlying technical mechanisms change. Companies can also include information about a document’s distribution, creation date, and ideally its expiration date within its metadata. This makes it easier to maintain an overview of what is current, even when managing large volumes of information.

The father of the Web sees the central task as designing permanent URIs. Because a URI changes when the information embedded within it changes, creating a stable URI means deliberately excluding information that may later become obsolete. The creation date can be a useful starting point because it does not change. Other information may eventually create problems. An author’s name, for example, may change between versions. Topics and their organizational structures can also change over time. The same applies to server names. Anyone who considers these factors while designing URIs can prevent dead links before they occur and avoid substantial work later.

If dead links already exist, they should be identified. This applies to broken internal and external links on a company’s own website, faulty links from other websites that are supposed to lead to the company’s website – known as backlinks – and non-functioning links contained in newsletters and other publications. Tools such as Google Webmaster Tools, Dead Link Checker, SEOmoz Open Site Explorer, and Majestic SEO can assist with this process. Reports received through Twitter, Facebook, or email can also be useful.

When broken backlinks are discovered, companies can contact the operators of the affected websites and ask them to correct the links. On self-service portals, companies may be able to make corrections themselves. If the original link cannot be corrected because, for example, it was posted on Twitter, distributed through a newsletter, or points to an article that has since been moved or renamed, a redirect can be created instead: a so-called 301 redirect. When implemented correctly, it takes the visitor – including search engine crawlers – directly to the new URI. This ensures that the content can still be found²⁴. During a website relaunch, it also helps transfer a substantial portion of the existing link popularity to the new domain²⁵.

Everyone Shares Responsibility

Several initiatives support the preservation of links. One of the most important is the nonprofit Internet Archive at archive.org. It was founded in 1996 with the goal of building a library of the Internet. Its Wayback Machine allows users to view archived websites from different points in time. Selected pages are periodically accessed and archived through the Alexa server service. By October 2012, this collection had reached a size of 10 petabytes²⁶. Since the previous year, it had also been powering a type of time machine that can transport 404 pages into the past with the help of a script whenever an archived copy of the missing page exists in the Wayback Machine. When a copy is available, the 404 page is provided with a link to the archived version, allowing users to access the lost content²⁷.

The on-demand web archiving system WebCite archives websites upon request. It is aimed specifically at authors, editors, and publishers of academic papers and books. Memento seeks to make access to the Web of the past as easy as access to the current Web. To improve link stability, the Memento team at Los Alamos National Laboratory and Old Dominion University has also begun working with colleagues at Harvard University on the so-called “404 No More Project”²⁸. Using a special attribute²⁹ added to the HTML code, the project is intended to make previous versions of linked content accessible³⁰. The Perma.cc service also helps create stable links. At the time described here, it was in beta and operated under the direction of the Harvard Law School Library. When a user creates a Perma.cc link, the service archives a copy of the content and generates a permanent link to that archived page. The link can then be used in publications and, once confirmed, remains fixed³¹.

What all these initiatives have in common is their commitment to preserving digital content and preventing a Digital Dark Age – an era in which historical electronic documents become difficult or impossible to access. At the same time, they face a complex problem: deciding what should actually be preserved from the Internet. This question has also confronted the German National Library in Frankfurt, which has a mandate to collect electronic publications but has had to abandon the goal of archiving everything³². No single organization can archive the entire Internet, and even collectively the task remains extremely difficult. The volume of digital data continues to grow exponentially. According to EMC’s Digital Universe study, the amount of data in Western Europe doubles approximately every two and a half years, while technology is changing faster than ever. Long-term archiving projects, however, are only gradually developing. As a result, enormous numbers of websites still fall through the gaps between these initiatives³³.

Responsibility for preserving links therefore ultimately lies with everyone. Tim Berners-Lee assigned this responsibility to webmasters. But corporate decision-makers, editors, and everyone else who publishes content on the Web also have a role to play – unless their intention is to create little more than temporary messages that exist for a while before disappearing forever: content destined for the digital scrapheap.

Sources

1. blueleaf, bit.ly/1xsLpOm
2. Daniel Karcher, “bluedaniel.com”, bit.ly/1jcC03K
3. Wikipedia, “Toter Link”, bit.ly/1xsLKk5
4. TagesWoche, “Basels vergessene Links”, bit.ly/1mstfn6
5. Social Science Electronic Publishing, “Perma: Scoping and Addressing the Problem of Link and Reference Rot in Legal Citations”, bit.ly/1rYfTHF
6. Hany M. SalahEldeen and Michael L. Nelson, Old Dominion University, “Losing My Revolution – How Many Resources Shared on Social Media Have Been Lost?”, bit.ly/1idpFfo
7. Dr. Anja Johanning, Dr. Lutz Goertz, “Das Internet vergisst doch!”, bit.ly/1hHBYGg
8. Georgia Tech Collage of Computing, “Problems Using the Web”, b.gatech.edu/1p9BURH
9. Jakob Nielsen, “Fighting Linkrot”, bit.ly/TvhJAo
10. Felix Salmon, “The spread of link rot”, reut.rs/Tvi3PM
11. Anil Dash, “The Web We Lost”, bit.ly/1kaPj5Q
12. Tim Berners-Lee, “Cool URIs don’t change”, bit.ly/1jlu5WA
13. Marius Däpp, “404 Fehler bei Lexmark”, bit.ly/1kWUjd0
14. Hümpfch, “Warum ist LEGO bloß so unfähig?”, bit.ly/1ids0XI
15. CHIP online, “App-Experte Nicole Hery-Moßmann sagt”, bit.ly/1hHEjB0
16. Matt Cutts, “Free links to your site”, bit.ly/TvknpQ
17. Abhik Biswas, “Tracking 404 Error Pages”, bit.ly/1o2uMXV
18. SEO Südwest, “Broken Link Building – effektiver Linkaufbau”, bit.ly/1idt8dW \
19. AKM3 Team, “44 kreative Linkbuilding-Ideen made in Germany”, Tipp 25, bit.ly/1hHFe4s
20. trafficmaxx, “Linkbuilding Tipp: Broken Links finden – und auf die eigene Seite leiten”, bit.ly/Umx3A7
21. Popularity Reference GmbH, “PR Wochenrückblick KW29 – Die Highlights der Woche”, bit.ly/1oKDqGm
22. The Moz Blog, “Link Building Survey 2013 – The Results”, bit.ly/UmxlHo
23. Blueverse, “Desert Scraping: How it can help your online business”, bit.ly/1nsU7Xo
24. Ralf Zosel, “Cool URIs don’t change”, bit.ly/1p9HAvc
25. isiCore GmbH, “SEO beim Website Relaunch – Wie Sie die Katastrophe verhindern”, bit.ly/1ozfUOn
26. Wikipedia, “Internet Archive”, bit.ly/1msEmfM
27. Eric Herrmann, “Waybackmachine-Fallback statt 404-Error”, bit.ly/1k9CBTe
28. memento, “404-No-More”, bit.ly/TvqSsG
29. Ryan Westphal, Herbert Van de Sompel, Michael L. Nelson, “The mset Attribute”, bit.ly/1kRHaGM
30. Patrick Howell O’Neill, “This project aims to make ‘404 not found’ pages a thing of the past”, bit.ly/1msFdgC
31. perma.cc, “About Perma”, bit.ly/1k9DY4r
32. Frankfurter Allgemeine Zeitung, Thomas Thiel, “Am Boulevard der toten Links”, bit.ly/1p9K6Bx
33. Derek Mead, “When Will the Internet Defeat Link Rot? “, bit.ly/1d7Y0ZF