One of the coolest projects I got to work on was at Alcan’s Oswego Works facility. This is a blue-collar aluminum casting and rolling plant, now operated by Novelis. The casting part of the operation melted down used beverage cans and other cast-off aluminum to create ingot (large, rectangular blocks of aluminum weighing 15,000 to 30,000 pounds), which is then used as input to the rolling process.
These ingots were heated and inserted into pizza rollers the size of buses, squishing it back-and-forth into a long thin strip, and then coiled up on a spool the size of a mid-range sedan. The end result is a coil we sent to Budweiser and Coors for their beer canning operations. It was a large scale industrial process that was fascinating to watch, involving temperatures, pressures, volumes you don’t usually get to see in day-to-day life. Four of the six cans in every six-pack sold in the U.S. were rolled at that plant.
It took a ton of computers and electronics to run the Hotmill, a machine 3 football-fields long. It also took 90 people, stretched across 3 shifts, as the Hotmill ran 24 hours a day. Downtime (measured at $40,000/hour) was frequent – not uncommon for a machine this large and complex.
In reality, it was many machines and many computers. The low level Siemens MMCs (essentially microcontrollers) drove a lot of the electronics, and there were about 100 of those. An assortment of PLCs drove more of the electronics, and above all those were the Unix and VMS computers. These computers collected real-time data, and also issued the setup instructions to the lower level MMCs and PLCs to control the overall process. These were the computers I worked on.
I did two major projects for the Hotmill. The first one was so successful management decided they’d give us a try on the second. But the first project was definitely more interesting – the Hot Mill Sprays project.
As aluminum moves through the pizza rollers of the hotmill, coolant is sprayed on it to keep the heat in check (squishing metal – or anything – usually releases heat). The coolant is sprayed through 384 nozzles with enough force to throw a man 50 feet. In the pulpit, where plant operators control the process, an ancient computer terminal controlled the sprays system, talking to an equally antiquated computer. Maintenance contract costs were soaring for this hardware, and parts were almost impossible to find. Management knew that thing was going to eat their lunch one day, so they gave our automation team the assignment of replacing it.
The new system would consist of a shiny new Allen Bradley PLC to control the spray nozzles in real-time (100 times per second per nozzle?), an OpenVMS Alpha cluster (two boxes, for redundancy) to perform real-time high-level control, and a stock Gateway PC in the pulpit with a touch screen. In 1997, touch screens were uncommon, and PC’s were running Windows NT. We built the user interface on NT in Visual Basic. Yes, Visual Basic. I did most of the OpenVMS code in C, and a hired-gun did the PLC code. We made a great team (personalities clicked), and we knocked it out of the park.
But commissioning a new system in a 7/24 manufacturing plant where downtime costs $40,000/hour is not for the faint of heart. We were scheduled to install our system at 2am one morning, when the mill would be down for an hour for other reasons. We had been care to build our system such that failing back to the existing system was simple and quick. If ours crashed, we’d have the mill back in operation using the old system in about 30 minutes. Or so we thought.
I’m not sure how to describe this, but in this plant (and I’m sure, many others), there was a love/hate relationship between the workers and the management. Management blamed the workers for everything, and the workers went out of their way sometimes to show management they were screwing up.
So this night at 2am, our team gathered in the pulpit along with the mill operators, and we set up the PC that would be their touch interface to the new Sprays system. We asked that the existing interface be nudged over a few inches to make room for the PC, but the operator flexed a few too many Hotmill muscles and managed to rip the connecting wires off the back of it, rendering it dead. Laughing, he yanked the entire thing out, tossed it in the corner and turned to us with a ‘hope your new system works’. So no more backup plan. If the new system failed, we’ve have downtime, and get to play yet another round of forty-thousand-dollars-per-hour downtime blame-game. But the new system ran flawlessly (at least for a couple of days), and that old terminal was never brought back to life.
The only teething problem this system had was an Oracle database table that recorded log messages hit it’s maximum size and crashed the system. Simple to fix. And apparently, this was one the first uses of a relational database in a semi-real-time capacity in the metals/automation business. I was asked to speak about this at an annual industry gathering of metal producers, but that never happened for completely unrelated reasons.
Once we were confident the new Sprays system would stay up, we decided to go for beers or breakfast, but it was 7am. This Alcan plant was located in the outskirts of Oswego, and further down the road were three nuclear plants. This meant there around-the-clock truck traffic in and out of all four facilities, giving rise to a truck stop around the corner. The owner of the truck stop was used to people working around the clock, so he gave us the keys to the bar, and asked we keep our empties up on the bar so we could tally them up when we left. You don’t find 7am hospitality like that very often!
Lesson: Sometimes it’s ok to drink beer at 7am.