Please Feed the Spider

The program that runs around the Internet reading every single page and throwing it into Google’s* giant database is GoogleBot (Yahoo!’s is called Slurp). GoogleBot is your best friend, your worst enemy, your teddy bear and your mommy all rolled into one. GoogleBot is a very, very clever piece of software, but it’s not magical. Here is what GoogleBot does:

1. It reads the text on your page and looks for “important” words and phrases
2. It reads the links on your page and sees what pages you’re linking to
3. It reads the links on the rest of the Internet and looks for pages that link to you
4. It then calculates how relevant your page is to the words in it, based on the words on the pages that link to it, and how relevant they are based on other sites, and so on

Key take-home: it’s all about keywords and links. It is all about text. Attractive design and a witty site slogan and pictures of bikini models holding your product count for naught. As I mentioned in my last post, Google is in effect a disabled user using only the most basic of assistive technologies:

1. It cannot see your images
2. It will not execute JavaScript. Not any. (Real disabled users can often do JavaScript using better software these days)
3. It’s not reading every bit of text on your page. It’s looking for the important words. And it’s in an almighty hurry.
4. It does not follow links that do not look like web pages.
5. It does not magically work out what your site is about. You need to make it obvious.

Already, some of the key things you need to do for SEO are obvious, in order of importance:

• Link text is important. Every time I see a link saying “click here” in 2007 it makes me want to weep. Link text is, above anything else, how Google decides what the page you’re linking to is about, and by working out what you’re linking to is how it works out what your page is about. Outgoing links aren’t silly, they’re essential.
• All your information must be accessible with JavaScript disabled, because that’s how Google sees the page.
• All your links must go to real information. “#” links are ignored, as are “javascript:..”. And the information on those pages must be relevant to the link text, obviously: don’t just link back to your home page.
• Any images must be described as text somewhere on the page, either within the ALT attribute, or some other technique**. Google ignores images (even image search is based on the text nearby).

What’s an Important Word?

It’s important to know what Google considers an “important” word. Google is more than a little secretive about this, but Google has its own guidelines for site design and professional, non-evil SEO people have their own search accessibility guidelines. My own, very subjective impression from several years of experience, is that the most important words on your page are:

1. The link text going to your page. Nothing you can do about this but be a very good website, and hope people link to you. You can do your bit by linking to other people with sensible keywords, of course, and hoping they link back — but trading links explicitly is something GoogleBot is designed to detect. And it’s been spotting fakers a lot longer than you’ve been faking, so I don’t recommend trying to fake it.
2. The page title. Don’t repeat your site name and slogan endlessly: say what this page in particular is about. Put keywords in there! It’s also what users see on the search results page, too, so make sure it makes sense to human beings.
3. The meta description tag. This is an odd one: Google doesn’t pay too much attention to it in calculating relevance, but at a certain level of relevance, this is what it puts as the text under your page title in the search results, where it suddenly becomes very important to users who are about to click your link. So it’s important that this text is descriptive, useful, and short — something under 100 words. And again, load up on keywords. Repeat yourself, phrasing the same thing several different ways.
4. H1 and H2 tags. H3 is dicey and everything beyond that is meaningless, but H1s in particular are super important, but only because they are rare. If everything on your page is a goddamn H1 obviously Google is going to ignore you. Use 1 or 2 H1s, and less than 10 H2s.
5. ALT attributes on images. This is way down the list, so if you have really important text in your images, it’s best to use the technique I outlined in the third footnote so that it turns up in an H1 or H2.

Order is important, or, Don’t use Tables

Another aspect of your page that is extremely important to Google is source code order: literally, the order things appear in your source. Things that appear early on are likely to be more important than things that appear later. That seems obvious, right? But now look at your code: you’ve got the head, full of juicy meta data, and then you’ve got 5k of navigational elements, sidebar text, various other cruft, just placed first because you were using a left-floated column and so it was easier to put it there. This is killing you.

What’s much worse is when your source code order physically separates content that is semantically related: for instance, your headline is at the top of your page, then you have 5k of navigational cruft, then you have your content. Google will either fail to realise that your headline is describing your content, and thus not link the words, or worse, it will decide that your page doesn’t actually have any content on it relating to your headline, and you’re trying to spam it. Danger, Will Robinson!

And of course the number one offender from this perspective is using tables for layouts. If you care about web development, you’re probably aware that tables have serious issues with flexible, attractive layouts. However, that’s usually not a good enough reason to take to your boss: after all, it doesn’t bother her that your job is hard. However, tell her that using tables is causing an 80% drop in traffic to your site (as I explained in the last post) and suddenly you have an easy, obvious business case for reworking the layout of your code.

Tables put data into grid layout. If your data is in columns — and it frequently is, this means you often end up with a site code layout that looks like this:

Site logo	Article headline
List of nav links	Article body

To Google this reads like:

• Site logo
• Article headline
• List
• Of
• Nav
• Links
• Article body

So you can see why Google might get confused. So examine your code, and put things in the order of importance: you can use CSS to move stuff around on the page later. Coincidentally, source code order is also the order in which screen readers will read out your page to a blind user. So once again there’s a useful coincidence of making your site accessibile when you make it search accessible.

Of course — and I would have thought this was obvious, but I get questions about it that indicate to the contrary — you can use tables when the data is tabular. Don’t try to mark up your spreadsheet data using a series of stacked lists. Tables have real semantic meaning, but it has been diluted almost beyond help by their consistent misuse.

There is more I could tell you about SEO — the various hazily-defined statistical rules about how many links on a page is too many, optimal keyword density, and more, but these advanced techniques are icing on the cake, and the cake is made of search accessibility. It doesn’t matter what your keyword density is if Google can’t even get to your pages. So get out there and make the case for accessibility. And when the traffic is rolling in and your boss is giving you your huge bonus, you can get a tiny little extra bit of joy from knowing your site is also accessible to disabled users.

* When I say Google, obviously I mean Yahoo!, Ask and all the other major search engines as well. They all work the same way. If Google didn’t want me to use their name to mean all search engines, they shouldn’t have made it a verb.

** For important text like headlines, it’s often better to put the text into the page directly in a semantically-meaningful element (like H1, H2, etc), make the text transparent, and then put the nicely-styled image in as a background image. This makes no difference to what your users see but it makes the words look a lot more “important” to Google.

Accessibility: because it’s the Right Thing™?

The dirty secret of accessibility, swept under the rug by many an evangelist, is that the cost of making your site accessible is relatively high: in my experience, something like 20% additional dev time on a new project, although experienced developers can bring this down, and the cost decreases dramatically for incremental updates once the project is up and running. But a 20% margin is definitely non-trivial. And if you’ve not been thinking in terms of accessibility from the start, this pricetag rises sharply: retrofitting accessibility often involves fundamentally reworking the architecture of a web page*. You’ll be looking at spending something like 50% of the time you spent originally developing the site on the retrofit. Ouch.

The other dirty secret of accessibility is that the number of disabled users is relatively low. Not tiny, but I often hear figures like “60% of Americans are disabled”, and while this is true, it’s disingenuous because that figure can include people like amputees or paraplegics who can use the web with no problems whatsoever. The truth is that somewhere between 10% and a maximum of 20% of your users will have trouble using your site without assistive technologies. This makes it a very close call, when starting a new project — serving 80% of your possible users doesn’t seem ideal, but is an acceptable loss to get it out of the door 20% faster, right? You can build the accessibility in later!

Except you can’t. After launch, you’ve got an inaccessible site and you’re facing a 50% dev time bill to retrofit that acessibility in: another 3 weeks on what was a 6-week project, just to get 20% more users? That makes no business sense: much better to build another project, and get another 80% of users in the door quickly.
This is the kind of unavoidable math that has made the web inaccessible today. And that’s the harsh truth: building in accessibility for disabled users does not make business sense. It’s still a good idea, a noble idea, but it’s not a financially sound one. This is true in the real world, too, which is why legislation was necessary to force everybody to put accessible toilets and wheelchairs in everywhere.

Accessibility: because you could get sued?

Of course, legislators have (eventually) worked out this problem, and as such there is already web accessibility legislation in place in many countries that makes it illegal to produce an inaccessible website. Problem solved! It’s the law! We have to do it! Right?

In an ideal world, yes. In the real world, the law is only patchily enforced. Only a few very large, very high-profile sites have been sued so far (plus some government sites). You can always fly under the radar, hope nobody notices, and not build in accessibility until they sue you. It’s a good gamble to make to avoid increasing the cost of your site by 50%, right? Again, the math defeats us.

But this is all very unsatisfying. You, the clever, compassionate, standards-compliant modern web developer, feel that this cold logic is intrinsically, morally wrong. So you make the case for accessibility: you try to inflate disabled user numbers (counterproductive; it will make your manager trust you less) and deflate the amount of time it will take to make it accessible (an even worse idea; now you’re missing deadlines because of “that damn accessibility stuff”, making your manager hate the whole idea).

So here’s how you, as a developer, can stay true to your noble impulses to build an elegant, accessible website: stop calling it accessibility.

SEO: Open up, Google is coming!

Search Engine Optimization, or SEO, is the hot shit right now. Google is the Internet for a lot of people, and if Google can’t find it, then it doesn’t exist. Nobody goes deep-diving on a site to try and dig up information anymore. Either they type in their search terms and your site comes up with exactly what they need on that page, or they will never click your link. Sites these days get 50-90% of their traffic from search engines**, and the overwhelming majority of that is deep links to pages within the site.

So it’s absolutely imperative that search engines be able to access your site, and this isn’t just keywords on your home page: Google must be able to get at every single page of the site, every nook and cranny, and see every little bit of information. A site that can’t be indexed is throwing away up to 90% of its audience. In other words, this traffic is lost by sites that are not search-accessible. And there’s an interesting word in that phrase.

Search Accessibility: because you’d be an idiot not to

Here’s the final dirty little secret of this situation: Google is a disabled user. Or more accurately, Google has all the same limitations of somebody using assistive technologies:

• It doesn’t look at pictures
• It does not execute any Javascript
• It isn’t reading every bit of text on your page; it is looking for the important bits

Suddenly, the equation changes: at least 55% of your users need your site to be accessible, and possibly over 90% do. Only 10-20% of them need it to be accessible all the time, but that doesn’t matter, because up to 90% of your users will never even visit your site if it isn’t search accessible. This isn’t out of solidarity, or legislation. They simply won’t find it. Search accessibility is not an optional component, to be bolted on after the main launch. Chances are, if you haven’t got your search accessibility right, there will never be a second launch, because your site will fail.

How can I further underline the importance of search accessibility to a web-based business? Let’s turn the numbers around: you can more than double traffic to your website by making it search accessible. Does that sound like something you could take to your manager as a business case? Keep in mind, 50% traffic from search engines is an absolute minimum. If you’re getting 90% of your traffic from Google, then making yourself search accessible will result in a tenfold increase in traffic. Those sorts of numbers are why SEO is now big business, with a whole industry built around paying consultants to tell you how to get it right. That industry wouldn’t exist if they weren’t getting results.

But you don’t need to pay somebody. Once you’ve got the big, obvious business case out of the way, and swallowed the bitter pill that doing things properly will take 20% longer, search accessibility is super-easy. For my own personal how-to for search accessibility, see my next post.

* For example, if you’ve put a lot of business logic into JavaScript to enable Ajax goodness, making it accessible often means moving this logic to the server-side, which means reimplementing in a different language entirely, which is terribly expensive. You can write Ajax accessibly, so that business logic is always on the server and Ajax is merely a bridge, but you have to be thinking about it from the start. And as we’ve already established, you didn’t do that.

** This figure is affected by the type of site, and the levels of traffic to that site. So your blog might get half its traffic from regular readers, but on an e-commerce site the figure is going to be 90%.

1. Persistence, persistence, persistence.
   1. Persistent presence: nobody keeps a web page open all day. No, really. Obviously, one use-case Widgets already have heavily covered is information that changes a lot: stock prices, the weather. And with the advent of the Dock in 4, you don’t even have to worry that your widget is going to take up too much space: your users can hide it and bring it back up whenever they feel like it, and you can build a mini-display for the dock to alert them when something interesting might be happening.
   2. Persistent use: are you building a messaging application, and you’ve considered building a little popup window that users keep open all the time to keep track of their messages? Then you were already considering building a Widget, just on the wrong platform.
   3. Persistent data: does your application play with a lot of data that it’s tedious to pull in from a network? Is it possible your users would want to use their data offline? Would a client-side cache improve the performance of your app? Widgets have access to the filesystem and their own, local SQL database. If that doesn’t make you giddy with possibilities, you’re not the developer I thought you were.
2. Desktop interaction
   Upload a file… click browse… blah. If you want to interact with your user’s files, web pages suck. Widgets can accept drag-and-dropped information, send data to and from the clipboard, and any number of capabilities that the browser won’t give you without significant effort and a couple of security dialogs. JavaScript freed from the chains of the browser is fun.
3. Performance
   Web applications are an amazing, wonderful hack: a medium that was designed to display static pages of information has been twisted into an application development platform. But despite all the joy that brings, it remains fundamentally a hack. Some applications are just too rich and too complicated to be made in a web context, or worse, when made in a web context their performance makes them unusable. Konfabulator on the other hand was built to make applications from the start, and it shows.

Desktop 2.0

Note what I said just now: applications, not toys or gimmicks. Widgets are not a web replacement. They’re not web 3.0. They are a completely new breed of desktop application: a term we somebody threw out in the office as a joke was Desktop 2.0, but that’s really what we’re talking about. Persistent, performant, web-connected, and a damn sight easier to build than Java or C++ or any other desktop-application development language you care to name.

Widgets are the natural result of the explosion in the number of developers with web-centric skills since the late 90s. Any development environment that attracts large numbers of developers produces a push to increase the capabilities of that environment into other areas: this is why there is command-line PHP, GUIs written in Python and web apps written in VBScript. So it was inevitable that JavaScript would migrate to the desktop, and taking a bunch of other web-like technologies like XmlHttpRequest and the DOM with it is good and important too, because Widgets aren’t just a different language for writing applications, they’re a new way of writing applications.

The thing about Widgets that endears them to me as a development platform is that they get software development the right way around: build the interface first, and hook the functionality into that. This is such a simple idea, and yet there’s no desktop development language that works this way: yes, there are visual ediitors for C++ and Java, but they are just working with a set toolkit: the code is actually central, and if you want to do something out of the ordinary with the interface, you have to go back to the code. Because the truth is that for the vast majority of applications, the processing that’s going on is minor, and all the value is in the interface: presenting the right information at the right time, in the right way. Widgets let you concentrate on that.

Not all roses

Of course the problems. JavaScript is a lot faster in Konfabulator than it is in any browser, but JavaScript is still an interpreted language, and that has performance implications. There are other problems, too: the documentation (a personal bugbear and current project of mine) still has some way to go before it’s up to the standards of developer.mozilla and PHP.net. You are still insulated from the operating system to a certain extent. And despite our best efforts, there will inevitably be some bugs.

Widgets are not the be all and end all. They do not replace desktop applications, nor do they replace web applications. Instead they complement both, and produce a useful middle ground, where a web dev like you can create something that looks and works like a desktop application.

Frameworks — like Ruby on Rails, and a raft of emerging PHP on Rails MVC frameworks, of which my favourite is CodeIgniter — are just the next generation of this principle. Where once we marvelled at how easy PHP makes it to query a database (only 5 lines of code!), now we can marvel at how easy a framework makes it (after you’ve set up your framework, only one line of code!).

The “I don’t need all that” trap

A common issue experienced coders run into when they look at frameworks is that they will say “this framework is too heavy, I don’t need any of that stuff”. This is particularly likely if what they’re building is supposed to be an experiment or a “prototype”. Being too heavy is of course a perfectly valid criticism of some frameworks, depending on the nature of your web application. But to use the same excuse to brush off frameworks in general is dangerous, and the “prototype” excuse even more so.

You don’t need the overhead of a framework to build your single-page blog software: often, doing it from scratch would be shorter and possibly even faster. This is, in fact, a problem with the overly simple demos the frameworks often promote via screencasts to demonstrate their capabilities. What you need a framework for is when your application becomes more than a single page, and when there’s more than one developer working on it.

This is why the “prototype” argument is also a false one: when’s the last time you actually threw away a prototype? You build the base, it works, so you tell everyone in the office about it. So you add a few extra features, tidy things up, eventually things snowball and the whole thing goes into production. And by the time you do that, you have to start maintaining it, and you should have used a framework.

The reason you need frameworks is because there’s no such thing as a small application. There’s just baby applications, which like all babies are small, simple and cute, and old applications, which are bigger, uglier, and frequently stink.

The real benefit of a framework is not in the screencast

There are two main branches of benefits of using frameworks:

1. First-write functionality: by not reinventing the wheel, you develop faster
2. Re-write functionality: having code that fits a standard pattern eases debugging, maintainability, and portability of code from one developer’s brain to the next.

The first benefit is the one the frameworks advertise in their screencasts, and often the area where they concentrate their further development efforts, building up complex AJAX and other functionality*. But it’s my strongly-held conviction that the second benefit is by far the greater: however, it’s achieved pretty much as soon as the framework is created, so it’s difficult for the developers working on the framework to remained excited about it. Fundamentally, significantly reduced maintenance cycles aren’t sexy, but they are useful as all hell.

Maintainability remains an enormous stumbling block in web development. It’s easy to write a monolothic, procedural script that handles all your data capture, validation, processing and output. Once you keep your internal model of that script in your head, it’s even relatively easy to maintain. The problem turns up six months later: you’ve forgotten how you wrote the script, or worse, you’ve had to pass the code on to another developer, and they can’t make head or tail of it.

A framework gives you built-in breakpoints for effective debugging: if the return statement in your model code is returning accurate data, you know for sure the problem is presentational, and vice versa. The nature of framework URLs makes tracing execution whole orders easier: you know automatically which function is being called in which controller, just from the URL. As your project begins to grow past the capabilities of a single developer, these features become essential, since your team members will be working with code they didn’t write. Frameworks by their nature provide your team-mates with a lot more information about function X than your average function name.

Speaking of which, some will say: what about naming conventions? Surely a framework is just a really elaborate naming convention, involving whole directories and files rather than just function names?

Not really. It’s a naming convention, but more importantly a coding convention: we don’t just specify what the function does, we specify where whole classes of related functions go, and the nature in which they interact. This provides those debugging benefits I mentioned earlier, and MVC also provides a structure that scales easily to applications with dozens of models and hundreds of views: modern web applications.

Maturity, at last

Frameworks provide something that has never before existed in the web development field: a convention that exists in more than one company. The more we as an industry use frameworks, the great the network effect: it means code and debugging can work effectively across companies, that new hires will be able to quickly understand the operation of your software and get productive faster.

Frameworks are a sign of a new maturity in the field of web development, a side-effect of the shift from writing “pages” to “sites” to “applications”. And it’s about time.

* I have yet to see a framework build AJAX that meets a high standard of web development. It doesn’t count as separating behaviour from content if your view has to make a bunch of ajax-specific calls and you end up with a bunch of inline onclick handlers.
** Unless you write consistent, comprehensive and up-to-the-minute documentation, of course. But nobody ever has. No, not even that guy.

I’m writing the web dev stuff because that’s what I was and what I still do sometimes, and I’m writing the Widget dev stuff because that’s what I am now. So the web stuff will likely be relatively advanced, and the Widget stuff relatively simple. In particular I’m hoping to write a lot of the Widget stuff from the perspective of a complete beginner, before that feeling wears off for me.

I’m aiming for one post a week. Let’s see how we go, shall we?