The Digital Detective, Wall Street part 3

I’m still astounded Fortune 500 companies and government facilities not merely allowed, but invited me, a 19-to-20-something freelance me to play with their very expensive computers. I mean work, not play, yeah, work is definitely the word. Reputation is everything. And okay, I have authority issues. So I’m told.

Striking off on my own meant no security blanket, no 401K, no pension, no profit-sharing. It meant scary months when I wondered if the phone would ring with a client and months when I wondered if the previous client was going to pay or not. That’s a concern– some companies withheld payment until they once again needed help. Sometimes managers wouldn’t like what I reported. My type of work– designing systems software– was specialized, so occasionally famine struck.

During one drought, camels were toppling over, birds fell from the sky, and my bank account appeared a distant mirage. Finally a call came in before the telephone company could cut me off. It was Wall Street again, a mutual funds house we’ll call MuFu. Loretta was their CIO, Chief Information Officer.

100 Maiden Lane
NYC © Emporis

“Darling, are you available?”

“Personal or pleasure?”

“Are you saying personal isn’t pleasure?”

“You’re married.”

“Was, Darling, was.”

“Loretta, I’m sorry.”

“Don’t be, I’m not.”

She lied. I could almost hear the sounds of tears leaking from her eyes. She was a nice lady who’d come up through the ranks.

“Loretta, what’s happening?”

“If you’re available, I need help.”

“Please don’t let it be application programming.” Even if it was, I desperately needed the work.

“Well… Did you hear we’re undergoing a conversion from Cobol to C?”

“You and every other firm with fresh university graduates.”

My professors, Paul Abrahams and Malcolm Harrison, were language experts. Abrahams was chairman of ACM’s SIGPlan and would eventually be elected president of the US’s professional organization, the Association for Computing Machinery. They received early releases of Unix and with it the C language. For my part, C was co-respondent in a love-hate relationship. It constituted a step up from assembler language, but I wanted more.

She said, “I know you’ll be simply shocked, but we’re experiencing crashes. We can’t cut over until we nail the problem. Nobody around here can read machine code. I know it’s not your thing, but nobody knows Cobol either.”

In the following, I’ve tried to trim back technical detail to make it more accessible and I apologize where I failed to restrain it. The gist should suffice.

---

Next day I took the Staten Island Ferry to lower Manhattan, where I strolled up Pearl Street and turned onto Maiden Lane. The mutual funds house took up a few floors of an older building, although the interior was done in chrome movie set futurism.

The glass room remained there running their big iron computer. Off to one side was a new server chamber covered in curved, blue plexiglass. Very spaceshipish.

Loretta blended 10% boss and 90% Cub Scout Den Mother, which made her a popular manager among the guys. She called in her lead analyst and chief programmer, Richard and Robert. The latter radiated lethal hostility.

“Leigh’s here to shoot that bug that’s killing us.”

“We don’t need help,” Robert said. “He’ll just waste our time.”

Loretta said evenly, “You’ve had months and it’s still not identified. Please give Leigh all the help he needs. He’ll likely work after hours to have the computer to himself.”

After Loretta departed, Robert said, “I know who you are. You used to be hot shit.”

“I’ve never heard it put so charmingly. Listen, I’m not here to take your job. I’m not here to threaten you. I’d like to get the job done and move on. Show me what’s going on.”

As predicted, the program started and died with an out-of-address exception– the program was trying to access memory that wasn’t there.

I asked for listings and a ‘dump’, formerly called a core dump, a snapshot of memory when the system died. The address of the failing instruction allowed me to identify the location of the link map, an org chart of routines that made up the program. Sure enough, the instruction was trying to reference a location out of bounds of its memory.

I took the program source listing home with me and spent a couple of days studying it. It was ghastly, a compilation of everything wrong with bad programming and especially in C. It contained few meaningful variable names and relied on tricks found in the back of magazines. Once in a while I’d see variables like Principle or Interest, but for the most part, the program was labeled with terse IDs such as LB, X1 and X2. This was going to take a while.

The company had no documentation other than a few layouts from the analyst. When I called in to ask a question, Robert stiff-armed me. I arranged my first slot for Friday evening with time over the weekend.

I began with small cleanup and immediately hit snags. I’d noticed a widely separated pair of instructions that read something like:

hash_cnt = sizeOf(Clientable);
      :
cust_cnt = abs(hash_cnt);

Wait. What was the point of the absolute value? C’s sizeof() returned the number of items in an array. It should never be negative. You could have five apples on a shelf or none, but you couldn’t have minus five.

As part of the cleanup, I commented out (disabled) the superfluous absolute value function. Robert dropped down as I compiled and prepared to test. I typed RUN and the program blew up. What the hell? Robert appeared to sneer, looking all too pleased.

He said, “That section was written by that old guy, John. We fired him because didn’t know crap, so no surprise it’s hosed up.”

I knew who he was talking about, a short, pudgy bear in his late 40s with Einstein hair. I’d never been introduced, but I’d heard him on a conference panel. John was no dummy, no matter what Robert said.

Robert smugly departed. I stepped through the instructions, one by one, studying the gestalt, the large and small. My head-smack arrived on Sunday. Curious why sizeof() would return a negative value, I traced how hash_cnt was used. As I stepped through the instructions, I saw it descend into a function called MFburnish().

I couldn’t find source code for MFburnish(). No one could. Without source, it would be very difficult to determine what happened inside it.

I went back to the variable Clientable passed to sizeof(). The array was loaded from a file, Clientable. Both consisted of binary customer numbers. I spotted something odd.

C is peculiar in that it uses null (binary zero) to mark the end of arrays and ordinary file streams. This file had two nulls, one about the seven-eights mark and another at the absolute end.

At first, I thought the file had shrunk and the marker moved down while remaining in the same space. But when I looked at the file, it had the same defect… or feature.

As some point, I looked at the link map to check upon another routine and for the first time noticed what I should have spotted earlier. There amid C Library functions of isalpha(), isdigit(), islower(), isupper(); was sizeOf().

Double head-smack. First, C’s authors claim sizeof() is a unary operator like +n and -n. To me, sizeof() looks and acts like a function and nothing like a unary operator. But by their definition, it shouldn’t show up in a link map with real functions. On closer inspection, the program read not sizeof() but sizeOf(). Another annoyance of C is that it’s case sensitive, meaning sizeof and sizeOf and SizeOf and even SIZEOF are not the same thing. This kind of nonsense wouldn’t have been possible with their old Cobol system.

The deception seemed awfully abstruse, even by C standards.

The Clientable contained account numbers of a sizeable fraction of clients. Why some customers and not others would take me a while to discover. Unlike sizeof(), the ginned-up sizeOf() showed the actual record position within the full file expressed as a negative number, hence the abs() function.

Someone had written deliberately misleading code. But why?

Money, of course. Moving backwards, I began to look at the code with a different eye. And there it was… not merely the expected interest calculation, but the conversion from binary to decimal, another Cobol to C difference. I suspected one of the company’s programmers had pulled off the oldest thefts in computerdom– siphoning off money by shaving points when rounding numbers.

This wasn't the problem Loretta had asked me to solve. Robert had directed me to the wrong program, which turned out to be a stroke of luck. Loretta had invited me to track down a program bug, but I suspected I had unearthed traces of virtual villainy.

Next week: The Confrontation

Following are Cobol versus C notes for the technical minded. Feel free to skip to next week.

Cobol versus C for Commercial Programming

Cobol (Common Business-Oriented Language) has a storied history, its development led by famed Dr Grace Hopper in 1960. The language was particularly well suited for banking and commerce.

Unusual among computer languages, Cobol doesn’t require one to be among computer cognoscenti to comprehend. A well-written program might be understood by almost anyone. Compare these rough amortization examples, the first in Cobol, the other in C. Which is easier to understand?

Cobol:

Divide CashFlow by (rate+1)**years giving PresentValue.

C:

pv = cf / (++r)^t;

This is a mild example, but C contains numerous pitfalls and pratfalls compared to Cobol. Cobol ideally would conduct calculations in decimal. C always used binary. Consider the lowly nickel. In Cobol, it could be stored as 0.05, that is, 5¢ exactly. Using C, the value is stored in hexadecimal as 0.0CCCCCCCCCCC… ad infinitum, or converted back to decimal as 0.049999999999…

It’s ‘good enough’ for all but the most extreme financial calculations, but it’s not precise. Over time and many calculations, trailing digits eventually erode.

Next, the C programmer encounters the problem of printing values, converted automatically in Cobol. Before doing anything, the old tried-and-true rounding has to add 0.001 (or 0.0001 if calculating mils), trying to remember not to accidentally store that value. C’s printf() can facilitate this.

While Cobol can ignore leading zeros, C requires extra processing. It’s not unusual for tables and other data to contain leading zeroes, but these present a major problem for C. When it outputs leading zeroes and tries to read the same data, it wants to convert them to octal (base 8), which is virtually unused these days. In other words, C may not be able to read the same data it wrote. It requires something like an atoi() function to strip leading zeros to ‘decimalize’ the data.

It seems ridiculous that technology should be dependent upon fads and fashion, but some application programmers opposed Cobol because it was wordy and not… sexy. Some programmers took pride in writing ‘obfuscated C’, especially where pointer arithmetic could manipulate memory without limitations. At least one C textbook urged programmers to avoid subscripting and use direct addressing to access arrays.

Don’t misunderstand, Cobol can’t replace C. Cobol was designed for applications, not systems software. But contrariwise, C is a poor substitute for purpose-built languages. C was never meant to be the universal programming language.