Please Feed the Spider

The program that runs around the Internet reading every single page and throwing it into Google’s* giant database is GoogleBot (Yahoo!’s is called Slurp). GoogleBot is your best friend, your worst enemy, your teddy bear and your mommy all rolled into one. GoogleBot is a very, very clever piece of software, but it’s not magical. Here is what GoogleBot does:

1. It reads the text on your page and looks for “important” words and phrases
2. It reads the links on your page and sees what pages you’re linking to
3. It reads the links on the rest of the Internet and looks for pages that link to you
4. It then calculates how relevant your page is to the words in it, based on the words on the pages that link to it, and how relevant they are based on other sites, and so on

Key take-home: it’s all about keywords and links. It is all about text. Attractive design and a witty site slogan and pictures of bikini models holding your product count for naught. As I mentioned in my last post, Google is in effect a disabled user using only the most basic of assistive technologies:

1. It cannot see your images
2. It will not execute JavaScript. Not any. (Real disabled users can often do JavaScript using better software these days)
3. It’s not reading every bit of text on your page. It’s looking for the important words. And it’s in an almighty hurry.
4. It does not follow links that do not look like web pages.
5. It does not magically work out what your site is about. You need to make it obvious.

Already, some of the key things you need to do for SEO are obvious, in order of importance:

• Link text is important. Every time I see a link saying “click here” in 2007 it makes me want to weep. Link text is, above anything else, how Google decides what the page you’re linking to is about, and by working out what you’re linking to is how it works out what your page is about. Outgoing links aren’t silly, they’re essential.
• All your information must be accessible with JavaScript disabled, because that’s how Google sees the page.
• All your links must go to real information. “#” links are ignored, as are “javascript:..”. And the information on those pages must be relevant to the link text, obviously: don’t just link back to your home page.
• Any images must be described as text somewhere on the page, either within the ALT attribute, or some other technique**. Google ignores images (even image search is based on the text nearby).

What’s an Important Word?

It’s important to know what Google considers an “important” word. Google is more than a little secretive about this, but Google has its own guidelines for site design and professional, non-evil SEO people have their own search accessibility guidelines. My own, very subjective impression from several years of experience, is that the most important words on your page are:

1. The link text going to your page. Nothing you can do about this but be a very good website, and hope people link to you. You can do your bit by linking to other people with sensible keywords, of course, and hoping they link back — but trading links explicitly is something GoogleBot is designed to detect. And it’s been spotting fakers a lot longer than you’ve been faking, so I don’t recommend trying to fake it.
2. The page title. Don’t repeat your site name and slogan endlessly: say what this page in particular is about. Put keywords in there! It’s also what users see on the search results page, too, so make sure it makes sense to human beings.
3. The meta description tag. This is an odd one: Google doesn’t pay too much attention to it in calculating relevance, but at a certain level of relevance, this is what it puts as the text under your page title in the search results, where it suddenly becomes very important to users who are about to click your link. So it’s important that this text is descriptive, useful, and short — something under 100 words. And again, load up on keywords. Repeat yourself, phrasing the same thing several different ways.
4. H1 and H2 tags. H3 is dicey and everything beyond that is meaningless, but H1s in particular are super important, but only because they are rare. If everything on your page is a goddamn H1 obviously Google is going to ignore you. Use 1 or 2 H1s, and less than 10 H2s.
5. ALT attributes on images. This is way down the list, so if you have really important text in your images, it’s best to use the technique I outlined in the third footnote so that it turns up in an H1 or H2.

Order is important, or, Don’t use Tables

Another aspect of your page that is extremely important to Google is source code order: literally, the order things appear in your source. Things that appear early on are likely to be more important than things that appear later. That seems obvious, right? But now look at your code: you’ve got the head, full of juicy meta data, and then you’ve got 5k of navigational elements, sidebar text, various other cruft, just placed first because you were using a left-floated column and so it was easier to put it there. This is killing you.

What’s much worse is when your source code order physically separates content that is semantically related: for instance, your headline is at the top of your page, then you have 5k of navigational cruft, then you have your content. Google will either fail to realise that your headline is describing your content, and thus not link the words, or worse, it will decide that your page doesn’t actually have any content on it relating to your headline, and you’re trying to spam it. Danger, Will Robinson!

And of course the number one offender from this perspective is using tables for layouts. If you care about web development, you’re probably aware that tables have serious issues with flexible, attractive layouts. However, that’s usually not a good enough reason to take to your boss: after all, it doesn’t bother her that your job is hard. However, tell her that using tables is causing an 80% drop in traffic to your site (as I explained in the last post) and suddenly you have an easy, obvious business case for reworking the layout of your code.

Tables put data into grid layout. If your data is in columns — and it frequently is, this means you often end up with a site code layout that looks like this:

Site logo	Article headline
List of nav links	Article body

To Google this reads like:

• Site logo
• Article headline
• List
• Of
• Nav
• Links
• Article body

So you can see why Google might get confused. So examine your code, and put things in the order of importance: you can use CSS to move stuff around on the page later. Coincidentally, source code order is also the order in which screen readers will read out your page to a blind user. So once again there’s a useful coincidence of making your site accessibile when you make it search accessible.

Of course — and I would have thought this was obvious, but I get questions about it that indicate to the contrary — you can use tables when the data is tabular. Don’t try to mark up your spreadsheet data using a series of stacked lists. Tables have real semantic meaning, but it has been diluted almost beyond help by their consistent misuse.

There is more I could tell you about SEO — the various hazily-defined statistical rules about how many links on a page is too many, optimal keyword density, and more, but these advanced techniques are icing on the cake, and the cake is made of search accessibility. It doesn’t matter what your keyword density is if Google can’t even get to your pages. So get out there and make the case for accessibility. And when the traffic is rolling in and your boss is giving you your huge bonus, you can get a tiny little extra bit of joy from knowing your site is also accessible to disabled users.

* When I say Google, obviously I mean Yahoo!, Ask and all the other major search engines as well. They all work the same way. If Google didn’t want me to use their name to mean all search engines, they shouldn’t have made it a verb.

** For important text like headlines, it’s often better to put the text into the page directly in a semantically-meaningful element (like H1, H2, etc), make the text transparent, and then put the nicely-styled image in as a background image. This makes no difference to what your users see but it makes the words look a lot more “important” to Google.

Accessibility: because it’s the Right Thing™?

The dirty secret of accessibility, swept under the rug by many an evangelist, is that the cost of making your site accessible is relatively high: in my experience, something like 20% additional dev time on a new project, although experienced developers can bring this down, and the cost decreases dramatically for incremental updates once the project is up and running. But a 20% margin is definitely non-trivial. And if you’ve not been thinking in terms of accessibility from the start, this pricetag rises sharply: retrofitting accessibility often involves fundamentally reworking the architecture of a web page*. You’ll be looking at spending something like 50% of the time you spent originally developing the site on the retrofit. Ouch.

The other dirty secret of accessibility is that the number of disabled users is relatively low. Not tiny, but I often hear figures like “60% of Americans are disabled”, and while this is true, it’s disingenuous because that figure can include people like amputees or paraplegics who can use the web with no problems whatsoever. The truth is that somewhere between 10% and a maximum of 20% of your users will have trouble using your site without assistive technologies. This makes it a very close call, when starting a new project — serving 80% of your possible users doesn’t seem ideal, but is an acceptable loss to get it out of the door 20% faster, right? You can build the accessibility in later!

Except you can’t. After launch, you’ve got an inaccessible site and you’re facing a 50% dev time bill to retrofit that acessibility in: another 3 weeks on what was a 6-week project, just to get 20% more users? That makes no business sense: much better to build another project, and get another 80% of users in the door quickly.
This is the kind of unavoidable math that has made the web inaccessible today. And that’s the harsh truth: building in accessibility for disabled users does not make business sense. It’s still a good idea, a noble idea, but it’s not a financially sound one. This is true in the real world, too, which is why legislation was necessary to force everybody to put accessible toilets and wheelchairs in everywhere.

Accessibility: because you could get sued?

Of course, legislators have (eventually) worked out this problem, and as such there is already web accessibility legislation in place in many countries that makes it illegal to produce an inaccessible website. Problem solved! It’s the law! We have to do it! Right?

In an ideal world, yes. In the real world, the law is only patchily enforced. Only a few very large, very high-profile sites have been sued so far (plus some government sites). You can always fly under the radar, hope nobody notices, and not build in accessibility until they sue you. It’s a good gamble to make to avoid increasing the cost of your site by 50%, right? Again, the math defeats us.

But this is all very unsatisfying. You, the clever, compassionate, standards-compliant modern web developer, feel that this cold logic is intrinsically, morally wrong. So you make the case for accessibility: you try to inflate disabled user numbers (counterproductive; it will make your manager trust you less) and deflate the amount of time it will take to make it accessible (an even worse idea; now you’re missing deadlines because of “that damn accessibility stuff”, making your manager hate the whole idea).

So here’s how you, as a developer, can stay true to your noble impulses to build an elegant, accessible website: stop calling it accessibility.

SEO: Open up, Google is coming!

Search Engine Optimization, or SEO, is the hot shit right now. Google is the Internet for a lot of people, and if Google can’t find it, then it doesn’t exist. Nobody goes deep-diving on a site to try and dig up information anymore. Either they type in their search terms and your site comes up with exactly what they need on that page, or they will never click your link. Sites these days get 50-90% of their traffic from search engines**, and the overwhelming majority of that is deep links to pages within the site.

So it’s absolutely imperative that search engines be able to access your site, and this isn’t just keywords on your home page: Google must be able to get at every single page of the site, every nook and cranny, and see every little bit of information. A site that can’t be indexed is throwing away up to 90% of its audience. In other words, this traffic is lost by sites that are not search-accessible. And there’s an interesting word in that phrase.

Search Accessibility: because you’d be an idiot not to

Here’s the final dirty little secret of this situation: Google is a disabled user. Or more accurately, Google has all the same limitations of somebody using assistive technologies:

• It doesn’t look at pictures
• It does not execute any Javascript
• It isn’t reading every bit of text on your page; it is looking for the important bits

Suddenly, the equation changes: at least 55% of your users need your site to be accessible, and possibly over 90% do. Only 10-20% of them need it to be accessible all the time, but that doesn’t matter, because up to 90% of your users will never even visit your site if it isn’t search accessible. This isn’t out of solidarity, or legislation. They simply won’t find it. Search accessibility is not an optional component, to be bolted on after the main launch. Chances are, if you haven’t got your search accessibility right, there will never be a second launch, because your site will fail.

How can I further underline the importance of search accessibility to a web-based business? Let’s turn the numbers around: you can more than double traffic to your website by making it search accessible. Does that sound like something you could take to your manager as a business case? Keep in mind, 50% traffic from search engines is an absolute minimum. If you’re getting 90% of your traffic from Google, then making yourself search accessible will result in a tenfold increase in traffic. Those sorts of numbers are why SEO is now big business, with a whole industry built around paying consultants to tell you how to get it right. That industry wouldn’t exist if they weren’t getting results.

But you don’t need to pay somebody. Once you’ve got the big, obvious business case out of the way, and swallowed the bitter pill that doing things properly will take 20% longer, search accessibility is super-easy. For my own personal how-to for search accessibility, see my next post.

* For example, if you’ve put a lot of business logic into JavaScript to enable Ajax goodness, making it accessible often means moving this logic to the server-side, which means reimplementing in a different language entirely, which is terribly expensive. You can write Ajax accessibly, so that business logic is always on the server and Ajax is merely a bridge, but you have to be thinking about it from the start. And as we’ve already established, you didn’t do that.

** This figure is affected by the type of site, and the levels of traffic to that site. So your blog might get half its traffic from regular readers, but on an e-commerce site the figure is going to be 90%.