keith wrote: ↑Sat Dec 25, 2021 6:13 pm
bill_g wrote: ↑Sat Dec 25, 2021 7:57 am
noblepa wrote: ↑Sat Dec 25, 2021 1:27 am
If the parts inventory I mentioned was somehow lost, the parts themselves would still be there and could be counted. If the files I was dealing with were lost and could not be recovered, that money would effectively disappear. So would the bank, for that matter.
No pressure there. (gulp)
An Australian study in the mid 80's came to the conclusion that if a bank's entire computer system failed for no more than three days the bank itself would cease to exist. I never bought into that doomsday scenario 100%, but it did generate a lot interest in funding proper backup systems at the board level, and it was a lot easier to get stuff done for a while.
We almost had that happen once. It was in mid-december of 1973, I think.
Banks are required by law to produce a transaction journal and a trial balance every day.
Our DDA posting program had checks and balances six ways from sunday, in order to stop anything from getting out of balance. Unfortunately, it turns out that the balancing totals for the deposits and withdrawals for a single day could not handle any number larger than $999,999,999.99. (For you older techies out there, that is a six byte packed decimal number)
This was back when Christmas Club accounts were still a thing. People deposited a little bit every payday throughout the year, and withdrew it all in December to buy presents.
One day, a lot of people cleaned out their Christmas accounts. Add that to the normal traffic and the withdrawals for that day exceeded $1B. The computer blithely threw away the leading 1, which, of course threw the system way out of balance. It should be the case that yesterdays-balance + todays-deposits - todays-withdrawals = todays-balance. But when you through away $1B of withdrawals, that equation isn't going to work. As we debugged the problem, we discovered that all would be well if BOTH deposits and withdrawals for a single day exceeded $1B, but on this fateful day, Deposits were less than $1B and withdrawals were more than $1B. At the time, our DDA total deposits usually ran about $700-800M.
Our system was set up so that if, after running the posting program, anything was out of balance, it would refuse to run any of the follow-on programs, mostly reports, including the aforementioned transaction journal and trial balance.
This put the entire bank in a pretty pickle. By law, the bank wasn't supposed to open its doors if those two reports were not produced. If the bank (National City Bank of Cleveland, now part of PNC) had not opened its doors, it would have been seen as a bank failure. A failure of a bank of that size would have had major ramifications nationally, even though the bank was actually still very much solvent.
While we worked diligently to correct the problem, the bank opened as usual. I never found out if the higher-ups in the bank simply chose to ignore the law, or if they quietly contacted the authorities, explained the situation and got approval to open anyway.
I was only involved with the posting program, but I was relatively inexperienced, so, given the importance of the situation, others fixed the problem. I don't know if we were able to make our usual deposit with the Federal Reserve or not. That was a different set of programs.
Investigation into this issue showed that the tape pool used for these generations showed wear and tear as they had never been replaced since the beginning or time. Also determined that for this kind of critical data a better quality tape product with slightly higher price tag would overall be more cost effective.
