Iyfjky

Question

I recently completed a migration from one server to another of a service I manage that affects over 4,000 users, and it all went off without any interruption or degradation in functionality. The few issues that did arise (one software bug in a tool I wrote that affected a team of 11 people, for which I had a workaround and fix same-day) were impossible to foresee, as my small team and I cannot test for every scenario; we can only be reactive in certain situations.

My manager considers the migration "a failure" (his words) because the team we directly support - those 11 people that were affected for a few hours by the bug, which again, was rapidly fixed and had an immediate workaround - had to use a workaround at all; in his words, "a step backwards."

I feel like he's missing the big picture, and I was pretty insulted by his characterization of this huge project that involved multiple teams and went smoothly for over 3,989 people. I made that clear in our one-on-one after the migration, but I feel like his "striving for perfection" (again, his words) leads to unrealistic expectations. How do I give constructive feedback to him about my thoughts on why the project was a success?

Did you communicate to the 11 people that there might be a problem, or at least let them know a migration was happening before you started? I find that communication tends to help people prepare mentally for any possible issues and tends to make them more accepting of problems that do occur. — Sep 29 '14 at 2:22
If the team you directly support is the 11 member team that had the issue, then the other 3989 are really technically irrelevant. If the team you support had an outage, then your migration caused a 100% effective outage to your stakeholders. — Sep 30 '14 at 12:44

score 20 · Answer 1 · 2014-09-30 07:27:12Z

I see this from a different perspective: the failure here was a collective failure to agree on what constituted "success" for the project before implementation - all stakeholders in the project need to have agreed on a set of objective criteria which could be used to define whether the project was a success or not. A potential set of these for your project could be something like:

The system will be partially usable for all users at all times

No more than X% of users will suffer partial degradation in functionality (whatever you may say in your first paragraph, 11 users having to use a workaround is an "interruption or degradation in functionality", although possibly a small one).

Any partial degradation in functionality will be remedied within Y hours.

Note that these aren't predictions of what anyone thinks is actually going to happen, but an indication of the impact of the disruption the project causes to the business as a whole, and how much of that can happen before the disruption caused by the project outweighs the benefit. Note that these criteria are as much to help you as they are to help the rest of the business - first of all by helping you to understand what the business needs from your project, and secondly by giving you leverage if you need extra resources to meet the agreed criteria ("Oh. You want X to happen. In that case, I'll need a couple of guys from team Y available to flurble the thingy-ma-whotsit").

In real-world scenarios, it is REALLY hard to predict how many users might be impacted by a change, let alone how much time it will take to resolve issues. Much better to just indicate in advance that an upgrade is being performed on a particular day/time, that there might be problems, and give some type of easy expedited way to contact tech support in the event of a problem. — Sep 29 '14 at 11:59
It does not have to be precise, but if you do not have realistic expectations, you are 100% sure to disappoint. — Sep 29 '14 at 12:11
@teego1967: I think Philip's point is that you need to define 'success' as something other that 'everyone is 100% happy and nothing at all happened' when discussing something of this magnitude. That management was unaware that, in a large scale migration, some problems are expected(not desired obviously) is an issue that could be solved with prior communication. — Sep 29 '14 at 14:52
Yes, I basically agree with Philips answer. The important point is prior communication about a change, and expectations. Just not the part about being able to predict "X%" users and a fix within "Y" hours. — Sep 29 '14 at 15:41
@teego1967 it's not so much about predicting how many users might be impacted or how long it might take to fix - it's about agreeing on what level of impact is acceptable and what level is going to cause genuine problems for the business. — Sep 30 '14 at 4:23

score 7 · Answer 2 · 2014-09-29 12:31:59Z

How do I give constructive feedback to him about my thoughts on why
the project was a success?

You need to schedule and conduct a longer discussion with your manager about expectations, this project, and future projects.

Basically, you don't yet have a meeting of the minds as to what constitutes success and what constitutes failure for this project. So that it doesn't happen again in future projects, you need to strive for a deeper understanding of your manager's point of view.

Rather than trying to convince your manager, start off asking for understanding.

Try to understand what constitutes project failure in the general case from his point of view, and why. There may be something about the bug experienced by those 11 people that was critical in his mind, and you need to understand that fact so you can work to prevent it in the future.

Ask him about future projects. Ask if it is reasonable to be prepared with rapid workarounds in the future or not. If it's not reasonable, ask for suggestions on how you could prevent it next time.

In my company, we have hundreds of clients. When we release a major project, business interruption for any of them - even for just a few hours - could be very significant. At my company, for some projects, it's quite possible that management would view the loss of a few hours for a few key clients as "complete failure", even though the other hundreds were unaffected. But I would always know that before the release, and plan accordingly.

It's possible that your manager is unrealistic. But you would want to know that before a release event and know what it would take to gain confidence in the release enough so that you could proceed. You might need to get his formal approval before release, you might need a significantly longer User Acceptance Test period, you might need to work closer with the departments that might be affected by a release, etc.

Trying to get your manager's thoughts on "how to do better next time" may buy you more political capital than trying to tell your manager that he "isn't seeing the forest for the trees" or trying to tell him that "really, the project was a success" when he has already reached a conclusion.

And often involving your manager in the "go/no go" decision, can get you where you really want to be.

I like your approach, but in this case, when a manager proclaims they expect perfection, is there much room for discussion either before or after the project? — user8365, Oct 27 '14 at 16:24

user27390 111 · Answer 3 · 2014-09-29 16:38:09Z

A strategy would be to do it during the night hours or on a weekend and have one or two of the users around for testing. If it is a 24/7 service, that doesn't help of course.

Did you test everything on an extra server before? There's not much more you can do otherwise. Unplug old, plug in new and fix whatever problem shows up.

Even if you look at "perfectionist" companies like Apple, every second of their releases is a failure or has bugs, etc. You cannot achieve 100%, even with unlimited resources.

Talk to him, what exactly should have been done different, and if he wants to upgrade the next migrations with more manpower or longer testing. I would mention that it is bad for morale if the team thinks their hard work is not appreciated, not that it hit me personally.

user8365 · Answer 4 · 2014-10-27 16:39:18Z

You need to consider what your boss's boss considers as well. If they expect him to be perfect, there isn't much you can discuss unless you want to educate upper-management on how to see the forest for the trees.

If this is purely your boss's interpretation, hopefully, he has the same level of expectations for himself. Does he recognize the fact that he failed as well?

Other than these two possible exceptions, my gut tells me you're trying to argue with an idiot. They say great people are very demanding and unreasonable. Taking this too far tells me you manager could be replaced by 5 year old brat who cries when he doesn't get everything he wants.

Instead of trying to convince him to take the constraints into consideration and realize there's no way you could be perfect, use this to your future benefit. If he wants perfection, more resources are required. More time for testing and creating perfect testing environments will be needed. When perfection is the only acceptable outcome, there are not consideration for costs. If he can not obtain 100% of all the resources your next project may need, then he'll be the failure.

score -2 · Answer 5 · 2014-09-30 09:48:23Z

I can probably sympathize with you, as I hate perfectionists. In terms of uselessness, I rank them right up - or should I say, down there with the workaholics. Being a perfectionist and having good coping skills are mutually exclusive. I would lodge a complaint with the manager's manager and with HR, and let the manager "strive for perfection" with his boss. Since your manager was conceited enough to call the migration a "failure", I would make him eat his words. Everyone of them.

For what it's worth, your migration went more smoothly, by the way, than most of mine. I can't remember a single instance where any of my migrations went off without a hitch. No matter how well prepared I am - and I am scared enough to be usually very well prepared, something always happens because reality gets a vote and Mr. Murphy from Murphy's Law has a voice. Migrations are rarely my idea, as I am all too aware that I could get ambushed by the unexpected. Usually, the idea of migration either comes from the higher ups, or it comes from a senior colleague who is being an eager beaver and who is not around when the shooting starts.

I don't know where your manager comes from, but he is not in the same business and he does not come from the same side of the house as you and I. I would lodge a complaint with the higher ups and HR, knowing that the result could be all-out war with the manager. I would use my filing of a complaint as an opportunity to strengthen my relationship with some of the higher ups and keep my lines of communication with them open, in case the manager gets the idea of badmouthing me.

Caveat: the migration could be legitimately viewed by the manager as a failure if the team of 11 that you support is responsible in turn for supporting a customer whose SLA (Service Level Agreement) requires a say 99.999% uptime. Hopefully, the SLA provides adequate coverage for scheduled downtime due to maintenance to allow enough time to resolve surprise glitches. If not, the company is in trouble with the customer. In this case, the migration would be a technical success but also a management failure - yes, something can be both a technical success and a management failure and you have to be sensitive to that. Needless to say, managements are not happy about management failures. I will accept criticism that a migration imperiled an SLA. I will not accept criticism based on somebody's standard of perfectionism.

teego1967 questions the need to go to war over the disagreement, and @JoeStrazere stresses the importance of trying to see the issue from the manager's point of view. I am going to adjust my answer - anger is not the wisest councillor - and state that you should make sure to understand why the manager views the issue the way he does before lodging the complaint. My reaction of anger was based on his statement that "he is striving for perfection", which set me off like a rocket. However, if his viewpoint is based on a legitimate business consideration, then I will accept his criticism as well founded.

Always remember, though, that the game of office politics is like the Game of Thrones: you win, or you die. — Sep 29 '14 at 7:01
Slightly less aggressive would be to tell the manager himself or herself that you feel insulted by his interpretation of the results as a failure, and ask him or her whether they reported it to their manager as a failure. (And if not, there's the obvious question "why not"). — Sep 29 '14 at 11:21
@gnasher729 You've it right - I would definitely feel that this manager had insulted me. I doubt that the manager would have reported the migration as a failure, because he would have to own the alleged failure. Plus, he would have to submit to an inquiry as to why he would think of the migration as a failure, with me insisting that it was not. — Sep 29 '14 at 11:36
Filing a complaint leading to possible "all-out-war" with your manager is a high-risk, low-return strategy for what amounts to a mis-communication of expectations. Seriously , come on ! — Sep 29 '14 at 12:10

score -2 · Answer 6 · 2014-10-27 21:20:22Z

It sounds that for the time, and money involved you delivered with some outstanding results.

It's a matter of how much polish in the final output can be achieved with the time and money available. As an analogy, a simple mirror to put on your bathroom costs a few dollars per square foot, the finished product will look smooth and polished but if you look from the side you will see some visible distortions, now a half decent mirror, using the same materials, but with a higher polish, suitable for a decent telescope is going to cost a few thousand per square foot, and has a distortion that is not visible to the human eye but may be visible when magnifying objects. Now look at the Hubble mirror - we are now talking millions per square foot, and is flat to a few measurable atoms.

My point is; Eliminating defects and engineering to perfections is a diminishing returns exercise.

It sounds like your skills were up to the task, you had a defect that affected 0.275% of users - this is outstanding. I would consider any disruption under 2-5% (and less then 24 hours) a huge success for the effort. Realistically, for anyone to get a defect rate lower would require significant investment in QA - were talking a lot of edge cases being tested here, I don't see that it could be done for less than 6-12 months of man time - literally testing every feature for every single user account. I doubt your manager would have been willing to pay this to break the four-nines (99.99% good enough) barrier.

I would talk to you manage about why he categorized the outcome as a failure, and what would he have considered a success. Work with him to highlight how you could have achieved that success but point out how much it would have cost to achieve that level of success, and ask if he would actually invest such extensive testing in the future.

While the above may or may not be true (you don't actually know what migration the OP was doing and therefore whether his failure rate was acceptable or not), you don't seem to have actually made any suggestions as to how the original poster should handle the situation he's in. Could you expand your answer to include that? — Oct 27 '14 at 13:41

score 20 · Answer 7 · 2014-09-30 07:27:12Z

I see this from a different perspective: the failure here was a collective failure to agree on what constituted "success" for the project before implementation - all stakeholders in the project need to have agreed on a set of objective criteria which could be used to define whether the project was a success or not. A potential set of these for your project could be something like:

The system will be partially usable for all users at all times

No more than X% of users will suffer partial degradation in functionality (whatever you may say in your first paragraph, 11 users having to use a workaround is an "interruption or degradation in functionality", although possibly a small one).

Any partial degradation in functionality will be remedied within Y hours.

Note that these aren't predictions of what anyone thinks is actually going to happen, but an indication of the impact of the disruption the project causes to the business as a whole, and how much of that can happen before the disruption caused by the project outweighs the benefit. Note that these criteria are as much to help you as they are to help the rest of the business - first of all by helping you to understand what the business needs from your project, and secondly by giving you leverage if you need extra resources to meet the agreed criteria ("Oh. You want X to happen. In that case, I'll need a couple of guys from team Y available to flurble the thingy-ma-whotsit").

In real-world scenarios, it is REALLY hard to predict how many users might be impacted by a change, let alone how much time it will take to resolve issues. Much better to just indicate in advance that an upgrade is being performed on a particular day/time, that there might be problems, and give some type of easy expedited way to contact tech support in the event of a problem. — Sep 29 '14 at 11:59
It does not have to be precise, but if you do not have realistic expectations, you are 100% sure to disappoint. — Sep 29 '14 at 12:11
@teego1967: I think Philip's point is that you need to define 'success' as something other that 'everyone is 100% happy and nothing at all happened' when discussing something of this magnitude. That management was unaware that, in a large scale migration, some problems are expected(not desired obviously) is an issue that could be solved with prior communication. — Sep 29 '14 at 14:52
Yes, I basically agree with Philips answer. The important point is prior communication about a change, and expectations. Just not the part about being able to predict "X%" users and a fix within "Y" hours. — Sep 29 '14 at 15:41
@teego1967 it's not so much about predicting how many users might be impacted or how long it might take to fix - it's about agreeing on what level of impact is acceptable and what level is going to cause genuine problems for the business. — Sep 30 '14 at 4:23

score 7 · Answer 8 · 2014-09-29 12:31:59Z