A malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.
I’m sending this to my boss to remind him why monitoring disk space is vital.
When I was working in that old job we had one particular fiasco that legit stresses me to remember. We have this account, no one knows what it does, but the password has never been rotated, it’s not vaulted, etc. There’s 5 apps that share the DB. I contact all the app owners, no response.
I wait a week and escalate the their bosses. No response. I send emails every single day to everyone including all the dev teams. Not one “lemme check on that” or anything. Our policy was to wait 90 freaking days for a non single user account. I’m getting yelled at to get this ticket closed when the day comes.
I go in, lock the account, change the password, and kill all DB sessions. Within 15 minutes I’m paged for a priority one incident because a trading app is down, causing the whole floor to be out and they’re losing millions every minute.
I tell them what I did and forwarded emails to everyone. The executive director is screaming at me, telling me I’m gonna be fired soon and I better fix it right now.
Sure, I can unlock the account and even force the password back to the old version. What’s that? No one knows what the old password was? Nothing I can do. Fortunately my executive director was awesome and stepped in to take the call. Overall they were down for an hour and a half. I looked at the incident later and they claimed $100 million in losses. The app owners wanted me fired. They got the uno reverse though and lost their jobs over it.
Our system wasn’t quite as critical, thankfully, but the app owners failing to respond to “Hey, by the way, your service account for your data base is gonna be closed” is just gross negligence. My condolences that you had to take the brunt of their scrambling to cover their asses.
For all the complaints I may have about certain processes and keeping certain stakeholders in the loop about changing the SQL Views they depend on, at least I acknowledge that plenty of people did heed the announcement and make the switch. It’s just that the “Oops, that mail must have drowned in my pile of IDGAF what our sysadmins are writing about again. Can’t you just give me the new password again, pretty please?” are far more visible.
When I was working in that old job we had one particular fiasco that legit stresses me to remember. We have this account, no one knows what it does, but the password has never been rotated, it’s not vaulted, etc. There’s 5 apps that share the DB. I contact all the app owners, no response.
I wait a week and escalate the their bosses. No response. I send emails every single day to everyone including all the dev teams. Not one “lemme check on that” or anything. Our policy was to wait 90 freaking days for a non single user account. I’m getting yelled at to get this ticket closed when the day comes.
I go in, lock the account, change the password, and kill all DB sessions. Within 15 minutes I’m paged for a priority one incident because a trading app is down, causing the whole floor to be out and they’re losing millions every minute.
I tell them what I did and forwarded emails to everyone. The executive director is screaming at me, telling me I’m gonna be fired soon and I better fix it right now.
Sure, I can unlock the account and even force the password back to the old version. What’s that? No one knows what the old password was? Nothing I can do. Fortunately my executive director was awesome and stepped in to take the call. Overall they were down for an hour and a half. I looked at the incident later and they claimed $100 million in losses. The app owners wanted me fired. They got the uno reverse though and lost their jobs over it.
Fuck that job lol.
Our system wasn’t quite as critical, thankfully, but the app owners failing to respond to “Hey, by the way, your service account for your data base is gonna be closed” is just gross negligence. My condolences that you had to take the brunt of their scrambling to cover their asses.
For all the complaints I may have about certain processes and keeping certain stakeholders in the loop about changing the SQL Views they depend on, at least I acknowledge that plenty of people did heed the announcement and make the switch. It’s just that the “Oops, that mail must have drowned in my pile of IDGAF what our sysadmins are writing about again. Can’t you just give me the new password again, pretty please?” are far more visible.