Iyfjky

Question

As a premise, I am the only person in the company who knows how to use a new Technology, and there are only a handful people in the whole country who use it.
The Technology is in a very early phase, with minimal documentation and almost zero support from the company that created and maintains it. The only reason our company uses it is because we want to be a pioneer and leader in the industry of the Technology users. And the nature of our work means that we must "test-as-we-go", i.e. there is no way to test this technology except by using it in direct action with the business' core operations.

Working with this Technology makes up ~10% of my overall role and I work directly with the CEO on it (though he is not my actual line manager). There are many times where the Technology doesn't work as expected. In one serious case it just stopped working altogether during a business critical time, with no explanation from the providing company, and while everyone at the providing company was on holiday.

With a furious and ranting CEO next to me, I was trying every possible action, hack or trick to make it work again but to no avail, while also holding off on the remaining 90% of my regular work. He didn't insult me directly, but the way he spoke he was insinuating that I was the one who screwed up.
Even worse, we managed to get hold of one of the few experts out there (who has five years experience, vs me just one) who repeated one of the solutions I tried following my same steps and all of a sudden it started working again.
This triggered many questions by him as to how this was possible, how come I had not been able, etc. My best guess is that it was just a random downtime, at the wrong time.

I am generally very attentive and think of every possibility, but there is a huge degree of unreliability and unexpected changes like the incident described above.

How can I make this clear to the CEO and anyone in the company who tries to point fingers at me or uses pressure words to try and squeeze out work from me?

P.S.: I cannot reveal more than what I described here about the exact nature of the technology, for security and legal reasons.

Do I understand it right that the technology at least in part consists of software running on servers that are not under control of your company and possibly not even under contract by you? — Jul 19 '14 at 10:12
what does "uses pressure words to try and squeeze out work" mean is he trying to get you to do unpaid overtime? — Jul 19 '14 at 12:16
Now that the software has been restored to operational status and everyone is again happy/content... mention to your line manager that due to the stress from the outage, you are considering looking elsewhere. Then do so. Either you'll find another job and be happier, or the management of your current company (incl the CEO) will realize that you are needed and make you feel wanted. If they don't, go back to the other option. (It sounds like I'm being flippant, but I am not.) BT/DT 10 years ago. — Jul 20 '14 at 0:51

O. Jones 13.6k24070 · Answer 1 · 2014-07-19 12:56:17Z

It's unproven tech without a reliable support infrastructure. The term "bleeding edge" was invented to describe such things.

You said the "company uses it is because we want to be a pioneer and leader in the industry of the Technology users." Presumably this CEO has set the company's direction in this area, and feels personally responsible for this decision. That would explain his anxiety.

If he is in fact "furious and ranting" when there's an unscheduled outage, his behavior is inconsistent with his desire to position his company on the bleeding edge. If his behavior drives away the people who are figuring this out for him, he'll be bleeding but not on the edge.

This probably feels like a huge risk to you, but I suggest you have a personal conversation with the CEO directly about this. Ask him for some time and say something like this. "During that downtime of the Zumbifram system last Tuesday, you know I was doing my best to bring it back online. We all know Zumbifram is very new and we're on the bleeding edge, and still the business depends on it. I know it took a while to bring it back, and I'm sorry about that. May I respectfully ask you to be more patient with me in, heaven forbid, future emergency situations? You know I'm not a slacker and I'm doing my best to become expert with this."

The point of this conversation is (1) to acknowledge the pressure the CEO is under when Zumbifram isn't working, and (2) to ask him not to make your job harder by directing all that pressure at you.

You might also talk to your direct supervisor and try to get more time and resources to work on this. Maybe you can get a test-lab version of Zumbifram on which to experiment. Maybe you can get more time to work on it.

You might consider developing a runbook -- a collection of checklists and procedures -- to use in future emergency situations. This can include contact information for external experts as well as things to do yourselves.

score 1 · Answer 2 · 2014-07-19 11:22:59Z

Ah, the joy of working with experimental technology. I am currently also involved in such a project. The tool we are using is all new and shiny, but bugs lurk around every corner, nobody in the world knows how it works and the vendors support is everything but enthusiastic. So we have to solve most problems ourself. It's a challenging and interesting task, but I am very glad that the management isn't exerting too much pressure on us.

When working with untested technology, the management (as well as any involved customers) must be made aware that the technology is unreliable, so that they don't make any decisions which rely on the technology being more mature than it actually is. It should not be used for anything business-critical, because downtimes and other problems are to be expected. When things must not go wrong, always lobby for using a tried and true technology instead. You also should be very pessimistic when committing to any deadlines for projects which involve the new technology.

atk 2,26411420 · Answer 3 · 2014-07-19 13:46:07Z

It doesn't matter whether the technology is established or experimental, transient issues are the worst necause it takes a while (and luck) to track them down due to the 'now you see'em, now you don't' nature.

The next time you run into serious trouble, try to get your hands on someone who is more senior to you ASAP, after doing all the troubleshooting you can immediately think of. You may know at least as much as the senior, but you could use another pair of eyes who sees the situation differently. The difference is that if the senior gets stumped, the CEO is far less likely to point the finger at you.

The expert succeded where you failed. You should have pointed out that you had tried the very same steps earlier and you had failed. In fact, you might as well ask the expert why the expert succeeded where you failed, using the same procedure. If the expert is stumped as to what happened, send a memo to the CEO that the expert is stumped, too.

Establish your professional credibility not only through your own actions, but with your interactions with the seniors and the experts. The closer your relations with them, the less ground the CEO has for blaming you. If the seniors and the experts think highly of you, the CEO is likely to take their opinions into account in the CEO's evaluations of you.

Unfortunately, you are not in a particularly good professional situation since the only time you have any visibility to the CEO is when the app is misbehaving.

I'll note that the CEO (and management in general) does not care why the system went down and how it went down - that's your job (and mine) to figure out, the management cares only that the system is back up. And the steps that you take for getting the systems back up may be very different from the steps you take to troubleshoot the system.

In the bad old days, we used to reboot the Windows Server machines and then pore through the logs to figure out what the hell happened. Of course, when you reboot the machines, some of the info we need to help troubleshoot may be gone, taking the troubleshooting process with it.

When the systems go down, your top priority is not to troubleshoot them but to get them back up. If that means restarting the app and the servers, so be it. In the special case where the technology is inadequately tried and documented, you cannot always afford the "we troubleshoot so that we can fix it" scenario. Because gathering the facts for effective troubleshooting can take forever. When an unconscious patient is brought into the hospital and the hospital has no medical history to work with, the top priority is to keep the patient alive, and figuring out what happened to the patient has to wait for later.

To summarize: do NOT hold bringing the systems back up hostage to the troubleshooting process and your (incomplete - not your fault) understanding of the systems. If you can't figure out what happened, then the reason must be something that you haven't thought of. And that something is a transient. The systems were in an unknown state when they crashed. Restarting the systems usually brings them back into a known state. If they don't restart and you can't figure out why, you are in serious trouble and it's time to acknowledge that you are in serious trouble and mobilize every senior and expert resource available to your firm.

This doesn't appear to actually answer the question.
â€“Â CGCampbell
Jul 20 '14 at 3:02 — Jul 20 '14 at 3:02

score 0 · Answer 4 · 2014-07-20 19:41:35Z

I can only go by what you are saying. It seemed that on the occasion described, your boss was stressed out and put the blame on you in an unfair way. If your boss had posted here what to do, correct answer would have been to calm down, don't pass your stress on to your employees because it is not helping the situation in any way, and worst case you lose a valuable employee for no good reason. Since you have posted and not your boss, pointing these things out to your boss is likely not beneficial to you. Writing an email to your boss explaining calmly what is going on and where the problem comes from (especially that it's not due to you) is likely more helpful.

Now a few software-development tips: Add logging to your software, so when the server you are communicating with responds in a way that you don't expect, it is logged in a visible way. There is a good chance that when you tried to make the software work, there would be a log showing that your code sent X to the server and the server unexpectedly responded with Z, and when the "expert" tried, the code sent the exact same X but now the server responded with the correct reply Y. Because the server was broken in some way outside your control and got fixed in some way outside your control.

There is another possibility: That the "expert" didn't do exactly what you did, but something just slightly different. And that the difference is something that shouldn't make a difference, but does make a difference. Logs could be able to show the difference. Let's say you did three things A, B and C, and the order of these things shouldn't make any difference. But it does due to a bug on the server. Doing them in the order A, C, B is the only way that works. The expert has been trained in five years unconsciously to always use order A, C, B. He would never do A, B, C in that order. He wouldn't be able to explain why not, because he doesn't know consciously why he avoids that order. Unconsciously he knows it doesn't work. Logging gives you a chance to find that kind of problem.

O. Jones 13.6k24070 · Answer 5 · 2014-07-19 12:56:17Z

It's unproven tech without a reliable support infrastructure. The term "bleeding edge" was invented to describe such things.

You said the "company uses it is because we want to be a pioneer and leader in the industry of the Technology users." Presumably this CEO has set the company's direction in this area, and feels personally responsible for this decision. That would explain his anxiety.

If he is in fact "furious and ranting" when there's an unscheduled outage, his behavior is inconsistent with his desire to position his company on the bleeding edge. If his behavior drives away the people who are figuring this out for him, he'll be bleeding but not on the edge.

This probably feels like a huge risk to you, but I suggest you have a personal conversation with the CEO directly about this. Ask him for some time and say something like this. "During that downtime of the Zumbifram system last Tuesday, you know I was doing my best to bring it back online. We all know Zumbifram is very new and we're on the bleeding edge, and still the business depends on it. I know it took a while to bring it back, and I'm sorry about that. May I respectfully ask you to be more patient with me in, heaven forbid, future emergency situations? You know I'm not a slacker and I'm doing my best to become expert with this."

The point of this conversation is (1) to acknowledge the pressure the CEO is under when Zumbifram isn't working, and (2) to ask him not to make your job harder by directing all that pressure at you.

You might also talk to your direct supervisor and try to get more time and resources to work on this. Maybe you can get a test-lab version of Zumbifram on which to experiment. Maybe you can get more time to work on it.

You might consider developing a runbook -- a collection of checklists and procedures -- to use in future emergency situations. This can include contact information for external experts as well as things to do yourselves.

score 1 · Answer 6 · 2014-07-19 11:22:59Z

Ah, the joy of working with experimental technology. I am currently also involved in such a project. The tool we are using is all new and shiny, but bugs lurk around every corner, nobody in the world knows how it works and the vendors support is everything but enthusiastic. So we have to solve most problems ourself. It's a challenging and interesting task, but I am very glad that the management isn't exerting too much pressure on us.

When working with untested technology, the management (as well as any involved customers) must be made aware that the technology is unreliable, so that they don't make any decisions which rely on the technology being more mature than it actually is. It should not be used for anything business-critical, because downtimes and other problems are to be expected. When things must not go wrong, always lobby for using a tried and true technology instead. You also should be very pessimistic when committing to any deadlines for projects which involve the new technology.

atk 2,26411420 · Answer 7 · 2014-07-19 13:46:07Z