Are you working in a firm that has gone through a merger, or been rebranded through a takeover? Have you had occasion to look on the web and try to locate the old firm by name or by its URL? In most cases if you do this, you will no longer find the websites which once displayed the proud information of the firm, its partners and its achievements.
Why has this happened? In most instances when a firm is merged or subsumed with another, or becomes part of a new network, it is thought to be bad for client relations if clients can find online references to the former incarnation of your firm. A decision is taken that the old site URL will redirect to the new firm, and effectively the old site is wiped from virtual memory.
The web has numerous articles which track these mergers, and there is also an archive maintained by Altman Weil which covers many jurisdictions. Of course, whilst the websites disappear, information about the existence of predecessor firms is not removed – there are often references in Wikipedia, and other historical references can be found. But if you are looking for the actual website identity that the firm had, the logo it used, the partners there at the time of the merger/demise, such a search via a search engine for the old websites yields nothing, and, as mentioned, the old URL may well just redirect to the new identity.
Aha, you say – what about the Wayback Machine? Surely this marvellous Internet Archive will still host the site for posterity? Well, here’s the rub.
To search the Internet Archive you either need to know the URL (web address) of the site you want, or there is a search box in the top bar, marked with a magnifying glass. You will find that some sites do not come up with a text search, but they do with the URL search. You do need to search both ways. (Many of us can’t remember the URL of the old firm, so a problem arises).
If you armed with this information, you may then find that the site which SHOULD be there, one that you know USED TO BE there, is not accessible. And this is where the marketers and webmasters in firms themselves have been at work. You receive this message:
Page cannot be crawled or displayed due to robots.txt.
And this is the explanation from the Wayback Machine:
Siteowners might have also requested that their sites be excluded from the Wayback Machine. When this has occurred, you will see a “blocked site error” message. When a site is excluded because of robots.txt you will see a “robots.txt query exclusion error” message.
So this is a deliberate action by website owners who do not want their superseded sites available for posterity.
This is symptomatic of much that happens on the internet these days. Losing a firm’s online presence may seem a small thing until you are the one seeking the information. Similar things happen in every area of life that has had a presence on the internet, whether a restaurant or a sports club or a government department. History is being lost every single day. No matter how much organised trawling for preservation is done by national libraries, the Internet Archive or any other body with the capacity and funding to undertake this mammoth task, there will be a huge hole in records from about 1995 onwards. Documents such as government reports, investigations, party political research, statistical charts, parliamentary papers, prepared by or for government, non-government, ngos, businesses or quangos may at some stage have been published online, and later disappeared, as pages were revised, governments changed, CEOs changed. Stuff that used to once be housed physically somewhere is now in some virtual graveyard of missing files.
The precision and regimentation of organised document retention, established in the past by institutions such as the civil service, has broken down, before a web equivalent has had time to evolve. Perhaps in the next decade or so some more systematic and agreed approach to retaining the websites of deposed governments, defunct law firms, bankrupt businesses, will prevail. Or perhaps we will be condemned to living in the constant present, and George Orwell’s chilling scenario in 1984 will have come to pass.