Transcript of the video "Cost, Effectiveness and Cost-effectiveness"
Video length: 00:27:00
[Canadian Heritage signature]
[Canadian Conservation Institute signature]
[Text on screen:
Canadian Conservation Institute (CCI)
Presents an excerpt from:
"Risk management and risk-based decision making for museum, gallery, archive and historic house collections"
Speaker: Stefan Michalski, CCI
Session: "Cost, effectiveness, and cost-effectiveness"
This advanced professional development workshop was held March 15-18, 2016 at CCI in Ottawa.]
Stefan Michalski (CCI): "So I'm going to speak about cost, effectiveness, and cost-effectiveness.
It's one of those phrases where the pauses and the commas are important to distinguish, these are three different concepts and tools, obviously related."
[Photograph of exterior view of Glanmore House]
Stefan Michalski: "I'm going to speak mostly from examples from the first place that we did a survey on, that Irene has mentioned already, Glanmore and you heard the curator Rona Rustige, speaking about it.
But then I will also consider all of the data we've collected so far from several different surveys.
When we first went to Glanmore, honestly Irene and I were totally pumped about risk assessment, per se."
[Photograph of interior view of Glanmore House, showing Julie Stevenson seated on stairs]
Stefan Michalski: "Julie who was our education officer at the time, was with us to help on communication issues and so on.
At one point we're sitting down at a meeting and she said so what's going to be in the report?
We said it's going to be all these risk assessments and graphs and analysis and so on and she looked a little disappointed and said umm, are you going to tell them what to do after you tell them what all their problems are?
We thought, well yes, probably we should do that and actually we realized that in the past when we did preventive conservation reports we knew we were heading towards recommendations and then we got excited about risk assessment there was a tendency to stop at the assessment point and hand over to the client.
Then say, now you know what your priorities are you can figure out what to do about them.
That was at this point where we realized, we knew we had thought about developing tools that would connect to the assessment that would do options and options analysis, we certainly taught it in the last two days of a three-week course but now we got more serious about it.
So I want to show you some of the results that came out of actually adding to all of the risk assessment reports, a set of options which were analyzed and they were analyzed for their effectiveness and they were analyzed for their costs and they were analyzed for the cost-effectiveness.
You've seen the five-stage cycle for risk management were jumping straight ahead to step 5, which is to treat risks.
I put in initials here, preventive conservation, because very much treat risks stage is what we would conventionally thought of as preventive conservation.
The difference is that you do analysis and assessment to find out really what you should be doing and your priority for preventive conservation.
But when you get into the treat risks stage, it is making suggestions and recommendations which we traditionally would have said these are my preventive conservation recommendations.
Within step 5 then there's identify the risk treatment options.
Quantify those options and then evaluate them.
These are the terminology of the risk analysis literature and the ISO standard.
Evaluation, identification and then plan and implement selected options.
So that's where you do hand over to the client, to the museum they would do that and you've seen the people we've been dealing with him more than able to understand and take that further step.
I'm going to be looking at the evaluate options, 5.3.
Within that we're going to look at evaluating it by its effectiveness, by its costs, by its cost-effectiveness and the question is which one should you use, A, B, C or D?"
[Text on screen: a) By its effectiveness, b) By its costs, c) By its cost-effectiveness and d) By all of the above]
Stefan Michalski: "Because, I'll give you the punchline now, those three different things don't always agree on which option is the sweetest, the best, the optimal, it can be in conflict.
Which is not always what we expected at the beginning.
Of course the answer is, you should consider all of the above.
What is the effectiveness of an option?
It's for our purposes, it's the risk reduction, how much is the risk reduced?
That's the untreated risk, subtract the residual risk after treatment, so it's the difference in risk, how much of you lowered it.
Because it's important to understand with risk assessment, you might make it very small so you think of it as pretty much zero, gone, perfectly solved, it's never like that.
You are reducing risk you're not making it all go away.
Cost-effectiveness on the other hand is the traditional, the business definition is the effectiveness divided by the cost and since for us the effectiveness is how much will reduce the risk, it is the reduction of risk divided by the cost per year to do that.
Another way of expressing it, is it's the prevention of loss in value per year divided by the cost per year and you saw in Zé Luis' talk and also in Rob's talk some vertical axis showing a fractional savings of the collection per year.
So, I am going to look at the general relation of these three different parameters. I'm going to plot first various graphs, always on the same horizontal axis which is risk and on this risk axis I'm showing loss in value per year, fractionally.
So, if you're over on number one that means that the risk is so bad that will probably destroy the entire collection in one year.
That's the upper limit, clearly we wouldn't have clients if anybody had this risk.
But it has occurred, I have been approached by people dealing with war zones where they make a decision that could literally change whether or not they're museum exists in a year.
So it's not a completely, unfeasible proposition.
It's simply not part of normal museum life.
But it's the far end of the scale and then the scales get smaller here these are logarithmic scale.
You would have noticed on some of Rob's vertical scales there was a decimal and then 01 and decimal 001 and he spoke about orders of magnitude.
I will be speaking about the same kinds of scale, these are orders of magnitude.
Each of these steps is 10 times less, 10 times less, and this is 10 to the minus 5, so one part in a hundred thousand.
So one part in a hundred thousands of our collection we saved this year and so on.
Now on the vertical one here I'm going to plot the reduction of the risk.
If we had a perfect risk reduction, all of our options solved all of the risks, we would have this 45 degree line.
Risks that are scaled one, we lose everything in here, it's gone.
The risks where we lose 10 parts in a million, 10 to the minus 5, perfectly solved.
Now I'm going to show you the options that we developed for the Glanmore House.
No I'm not, I'm going to show you a bit more of an animation of what I'm talking about here.
A 10 to the minus 10 risk, a 10 to the minus 5 risk, something here. If we solve that risk with an option, I'm going to show you the option up there.
If we solve that risk with the option I'll show the option exactly on that deadline.
If more realistically we have a fairly high option but we can't reduce it perfectly but we can do a pretty good reduction job it will be slightly below this line.
Now I'm going to show you the real data.
These are 68 options that were developed for 37 risks in this particular historic house museum.
You can see one here which is the fire risk, top one, it's not perfect but it's close to, it's a big substantial reduction, this was an option that really didn't do very much less than ten percent reduction.
Each of these steps as a log unit.
Down here is something that really didn't do anything at all, in fact this one actually generated more risks we didn't have negative numbers for the scale, but this was a bad idea option [audience laughter].
But completely plausible and reasonable if you didn't stop and work out the numbers; kind of thing we would have suggested 10 years ago.
Here are the cost for those options; and that's the scale, it's also a logarithmic scale, so it jumps quickly from a dollar a year to a hundred thousand dollars a year.
Here is when I divide the cost; divide the reduction by the cost I get these numbers, so this is the cost-effectiveness.
This scale is for those who want to work out the numbers, but the important thing is that the higher we are up here, the better the cost-effectiveness and I'll elaborate a bit more on this.
What I want to point out is that, if the option costs were proportional to how much risk we're reducing, so in other words, if you had an option that was a thousand times more important than another one, in terms of reducing risk, you would think it makes sense to spend a thousand times as much money.
When you actually look at the cost of those options they're not on the same slope of line, they're here, it's actually almost flat.
Which is the bad news, it means that, this cost-effectiveness line, is in fact sloped, if the cost of reducing each risk, each option we looked at from acid-free tissue up to fire reduction.
If the cost was proportional to what we're actually achieving, this line of cost-effectiveness should be flat, should be horizontal, it's really far from flat.
That's the whole point of my talk in case I speed up a little bit.
Options that reduce large risk tend to have a better cost-effectiveness and I'm just referring this in a business sense which is the economies of scale.
If you ever wondered whether fixing the whole building for a million bucks was better than picking away at the edges with acid-free tissue, not only in terms of effectiveness, but cost-effectiveness, bang per buck, is better, because this line is not horizontal.
I'm going to look at another place, this was Eldon House.
This is on the right, I'm showing Eldon House characteristics and on the left for reference, slightly dimmed, is Glanmore, fundamentally the same shape curve, same stuff going on.
Slightly different here, this is interesting, I haven't told Irene about this yet.
But what this says to me is these risks and options that Irene was looking at which cost a lot, but which are small options because they're on the left side of this graph.
We were looking conscientiously, this is a plausible option, people have talked about this in preventive conservation and recommended this.
You can see that Irene is subconsciously, or consciously, no longer bothering with those goofy things and not putting in things that have high cost at the low end, so in fact this region is less populated.
And what you see up here is, this region above, which had lots of really, poor cost-effectiveness on little tiny risks, there's only one of them hanging around at Eldon House.
No point even entering that one in the database, I know those are goofy suggestions.
But they'll come back again when we go to a different kind of museum.
This is the Art Gallery."
[Photograph of interior view of the Art Gallery, showing both adult and children visitors viewing some paintings]
Stefan Michalski: "Now this is the Art Gallery in comparison.
You can see that we've populated again with high-priced options which address fairly small risks.
Now partly that's because this whole operation is ten times bigger.
I think what happened is, you can be in one room and the mechanism it says, that's probably a goofy thing to look at it's not going to be cost-effective, at all.
You're in one room of a museum that's got 10 or 20 rooms compared to the historic house that had two.
That switches back on, that seems plausible is going to fix quite a bit of stuff, I think, in this room.
But when you scale it to the whole place, you can see that what's happened is these are coming back and these poor cost-effectiveness options have repopulating.
What's interesting about this place as well is that this is a well-controlled museum already they don't have the high really high risk issues, like fire and roof collapse, which we saw in the earlier one.
They don't even begin until 10 to the minus 4, so one part in ten thousand.
You can see it's clipped off because this is a well operated, funded museum; so already we backed off on the bigger risks."
[Photograph of an employee inside an archival holdings room of the Saskatchewan Archives]
Stefan Michalski: "Now we're going to look at the Saskatchewan Archives numbers and again a lot of scatter.
But the fundamental slopes of the green line and the grey line of cost-effectiveness, unchanged on these four very different kinds of operations and scales.
When a well experienced person discussing options with staff with long experience propose different options, because you may well ask, you could just seed this four things with really irresponsible option development, that you know are going to have appalling cost-effectiveness, it wasn't like that.
I didn't actually start crunching these numbers until most of these assessments were over, in terms of putting on a pattern.
We certainly were generating the cost-effectiveness on a table, but until you step back and look at them all on a graph, you can kind of convince yourself their still reasonable.
But this pattern just kept coming over and over.
Now I'm going to see whether we can make a master graph of this, can we add, can we plot al the four institutions data on the same graph.
Of this one I'm looking at the scale has changed.
Because the museum is ten times bigger, or 100 times bigger, actually the scale of the options have gone up almost two units by the yellow arrow.
So it's a hundred times more expensive on average each of the options we looked at.
Then the cost-effectiveness comes down a couple of order of magnitude.
But that's because a hundred percent of the Saskatchewan Archives is objectively, somewhat more valuable, within the grand global scheme of things, than a hundred percent of Glanmore.
Now I'm going to see if we can scale those so that each persons a hundred percent is meaningfully related.
I have done that, so I'm going to show you the results first, so that's a 111 risks for which 244 options were developed.
You can see on average Irene was developing two options per risk.
The general pattern is very well established.
The prices of options are all over the map, literally.
Almost horizontal in the end, by the time you actually do something even if it's not very effective, it can cost as much per year.
Now some of these are really high, so there's things that cost a lot, things that cost very little.
What I did to do this is that one of the questions that came in from the audience was; what about insurance value?
I'm glad whoever asked that, asked that, because I can segue way into it.
The Art Gallery had done and as you heard her, the director of operations, finance, their very current and up-to-date.
In fact, the year that we approached them, they said we've just finished doing evaluations of all of our works and pieces.
We know the market value of the entire collection, are you interested in that?
Absolutely, we're really interested, not that we want to promote the crass, horrible, capitalist notion that value has a dollar sign in front of it.
But if somebody comes along with a pilot project where they say, oh by the way in case market value would be interesting to look at, we've got it.
They've got Emily Carr's and they've got archival documents for the city of Oshawa, so a nice range.
So their total value we knew was roughly $40 million, market value.
So what I did for these numbers, would say, roughly, the historic house museums are order magnitude smaller, they are about 10 times smaller.
The value the house plus all the property and selling it, they might get $1 million, they might get $4 million, that's not important, we all know they're going to get more than $100,000 and they're not going to get $10 million, so it's ballpark.
On the other hand what's the value of a large provincial archives?
I borrowed here from ideas that Rob's Natural History Museum did years ago with Gerald Fitzgerald, to look at, Gerry Fitzgerald, to look at an argument for Treasury Board.
They costed what it would be to actually replace, you know, what is the labour and maintenance costs embedded in a huge collection.
I think it's not unreasonable to say that an archive with tens of millions of records, if you said, what would it cost to go back and find each of those catalogs and label and put them on the same place again, not to mention how much we've spent.
Anyway order of magnitude ten times bigger, so that makes these clouds of cost-effectiveness things, line up.
So now that we've got a dollar value here we can go and do something really inappropriate and dirty.
Which is put dollar to dollar, so, if I've scaled these value of these collections and I know that this black line used to be percent of collection saved.
What about dollar of object value saved, per dollar spent, per year?
Well, that dotted line is the line where we break even, above that line every options of pure business, go for it decision.
If you can find somebody with the funding.
But if they're saying is this really a business case?
Absolutely, if somebody in real estate could have some of these things, where you spend ten cents to save a dollar a year.
Which these some of these things are one-and-a-half orders of magnitude higher, we are talking about saving a dollar of collection by spending ten cents per year.
Let's put something that maybe is a little scary, we actually have options where you're spending a hundred thousand dollars to save a dollars' worth of collection.
These are gigantic differences because this is a log scale, each one of these lines; 10 times, 10 times, 10 times.
That's the evidence I'm going to leave you with, which is we can in our field actually come up with pure business case preventive conservation or risk treatment.
We have to think about how do we recommend other things?
For example this area of options is a straight business case because it's cost-effective.
This area of options are, because of effectiveness these are the biggest risks being reduced for each of these collections.
These are the no-brainers, you've got cost-effectiveness and how big the risk is that it's addressing, supporting those options.
These ones you say, they're not the biggest risk but it's good value for money, on the other hand you wouldn't want to be spending money here, knowing that this may be less value for money, but this is your biggest risk.
For example, high-risk and medium cost examples, good cost-effectiveness. Monitor moisture content of the joists to reduce probability collapse."
[Photographs of a roof region of a building and the recording of the moisture levels]
Stefan Michalski: "That that was the risk here, that was the cost, that was its effectiveness, do it.
High risk and low cost so good cost-effectiveness. Improve exterior door hardware to reduce probability of theft."
[Photograph of a door with missing or damaged locking hardware]
Stefan Michalski: "It was a middle-of-the-road risk, it was a super low cost and it' is terrific, cost-effective. Low risk and high cost so very poor cost-effectiveness.
Provide tour guides to control visitors who may abrade picture frames."
[Photograph of hanging pictures in a hallway that is not cordoned off]
Stefan Michalski: "By the time you put in minimum wage costs, we didn't say, oh let's just assume you can find a slave or volunteer and we won't put a dollar cost down.
We said that's irresponsibility, we want to cost what it cost him to do stuff.
You asked the staff to do something more than they're already doing and they're all busy, then you have to say, okay.
That one gives you a risk down here, it's not very big, it has a cost up there, which is thousands, I mean it's basically a part-time salary for one person and you have a cost-effective which was the lowest one on the map.
This is low risk and low cost, so you'd think you could get away with it, but the risk is so low, cost-effectiveness is still poor.
Improve filtration on air-handling system to reduce pollutants, dust, on objects."
[Photograph of room that is full of many different items]
Stefan Michalski: "In other words, spend some money to buy filters more frequently and better ones for the furnace.
Here's the risk, pretty low-end, here's the cost and here's the costs effectiveness.
You can't get good cost-effectiveness at the low end of risks, you might wonder what's our bottom line here, a dollar a year, we've even costed things here and said okay, just thinking about it will cost you a dollar, you thought about it for 5 minutes bang, that's a dollar, it's not like a cent.
You could make it ten dollars it wouldn't change the argument.
A lot of these are things, we are like, oh it'll take them two to three hours to work it through once a year and it still didn't make necessarily sense.
If somebody spends a day out of the year and there's only three staff, it better be a useful day and some of these options didn't make it.
This is one where it was a tricky decision because of the effectiveness.
Very high risk and medium cost-effectiveness."
[Photographs of before and after the fire that occurred at the Miner's Museum]
Stefan Michalski: "Install a fire suppression system; so you don't have that, Miner's Museum in Glace Bay, decades ago.
That was the risk, it was the biggest one for Glanmore.
There was the cost, even amortizing, that was amortized.
You may wonder, how do we do per year, for something which is a capital cost?
That's where what we call a time horizon for the risk, that management comes in.
What is the time period over which you're trying to act responsibly as a management team?
Governments tend to use five, six, seven years.
Heritage agencies and people like that, might go for at least 30, 100.
It doesn't make my sense to amortize costs over a hundred years, even if you were preserving them a hundred years.
So we use 30 years, which is pretty generous to spread the cost out, but to save it.
This was a cost per year, given the fact that you would pay for it over 30 years and there it is, we certainly recommend it, it was our top recommendation.
But not because it had fantastic, the cost-effectiveness was good enough we thought.
A little bit less than a business case, but not so much less, it was it was only like a factor of 4 or 5.
So what do we do next?
We would like to make the tool work better, in terms of the tools we use which generated those graphs.
Effectiveness of one option that reduces several risks.
Comes up realistically, often, and we'd like to do it better.
Cost-effectiveness of all options together.
So you can look at a suite and calculate the cost-effectiveness.
These are not a big deal these are just getting the arithmetic right and making sure the tool allows you to enter the numbers without making conceptual errors.
The other thing I'd really like to do and it's part of sustainability and in term part of CCI's strategic plan and with a slight shift in political climate in our country, it might become more important, as well.
One could also just do carbon footprint effectiveness.
One could build in borrowed from carbon footprint calculators.
What is the carbon footprint of each option?
So you can find out how much preservation can you do for the least effect on the planet.
Thank you very much."