An evaluation of the Law Commission of Ontario will begin this fall. (Since it has been full operating – working on projects – only since February 2008, this may seem premature or early. Early it is, premature it is not because it is tied to the funding renewal/renewal of mandate process that will begin in spring or thereabouts next year.) There are many questions about evaluation (who should do it? who should be part of the process? and so on), but I am bringing one to this group: whether anyone has had experience with evaluating an organization's use of technology.
Of course, one can count "hits" on the website. Are there other ways to assess the effectiveness of the technological processes used by the organization? What about the alignment of the function of the organization with technological processes? the appropriateness of the communication methods the organizaton employees? the appropriateness of the "sophistication" of the technology given the organization's target audience(s)? Does comparing with the websites and other technologies used by similar organizations elsewhere matter? Are any of these even reasonable/feasible/useful questions to ask and are they likely to attract useful responses (and from whom?). Who should do the evaluation? Should it be part of the overall evaluation? Are evaluation experts likely to understand measures of technology?
Should we just stay clear of the whole thing?
Patricia, it seems to me that what you are describing is an information audit. There are lots of people out there who can come into your organization and find out how technology is used in your current business processes. The real value for money comes in the frank assessment of the processes – this will come from interviews with staff, direct observation (where possible). It might be interesting as well to see how a given process would work if it was totally manual, and compare that against the existing process with technological support.
As far as the "hits on the website" exercise goes, that sounds more like a usability assessment to me. This usually takes the form of expert evaluation of the design and layout of the site, as well as interviews/surveys/observation of users of the site. You can hire outsiders to do this work, but it's not hard to come up with a decent set of questions yourself.
I'm currently working on a project to redesign my library's website. The first thing I did was to invite some folks from my user community to come into the library to run some sample searches I'd created. I peeked over their shoulders and took notes as they recited to me the reasons for the choices they were making. It was interesting to see the spots where they got stuck – totally unexpected! What I learned will allow us to improve the layout of the site and has prompted us to review the language and clarity of the site's content. As we progress through the redesign, I'll be inviting different groups to come in to give us feedback on our ideas. It does add time to the process, but I think we'll wind up with a site that better anticipates how people use our information, and what they expect to see.
With websites the tendency is to evaluate number of "hits" or visits. However, inside an organization that measurement is not meaningful: just because a document or file is used only once does not mean it has not been valuable. It could have been the key file to work worth a million dollars to your organization.
So, other measures should be used. For example, how is the workflow: are people repeating the same work more than once? Are there bottlenecks slowing down the work? Are there pieces of the work that could be automated to speed things up?
Also, if people are looking for files, documents, or information, are they succeeding in finding? If you have a search engine running inside the organization, you can check to see where people are searching and not retrieving results. What are the terms they are using? Is there a way to tweak your metadata/indexing terms so that people can find what they are looking for? That is also something to ask if you are interviewing them during an information audit as Wendy suggests.
There are lots of different ways to evaluate technology inside the organization. It also helps to evaluate when you first implement something new to set an early benchmark.
I agree with Wendy that this sounds like a case for calling a business process analysis company. Of course, I may be biased, seeing as my husband works for one…