The probability of winning the lottery is slightly greater if you buy a ticket.

Application Problems

The environment is appalling, dealing with the massive pressures found 5 kilometres underwater is only a part of it. Construction work in places like the North Sea and the Gulf of Mexico go on all year round. Weather conditions on the surface are often bad and even subsea, the visibility is often measured in a few inches. Surface operations in the splash zone can see a five ton machine thrown around like a toy. Currents can exceed 3 knots and operators (Pilots) may need to weave their way around steel and rope downlines from the vessels, scaffold poles and debris lying on the seabed. Opening an electronics pod in a snow blizzard is always fun. Operators work with voltages up to 5kv or higher, signals measured in mV, digital and analogue electronics, computers and software, hydraulics, mechanics, fibre optics, acoustics, electromagnetics, X-ray equipment and of course the ever present sea.

Massive structures are lowered onto the seabed by crane barges and positioned by free flying Remotely Operated Vehicles (ROV's). With special tools, these machines will then pick up the ends of pipelines and align them to valves on the structures, inserting gaskets, nuts and bolts before tightening them to preset torques. They operate valves and other machinery using manipulators controlled from the surface with a delicacy that has to be seen to be believed. Ten minutes later they may have to ram a 20 ton pile into position so the structure can be pinned into the ground. Piling hammers create shockwaves that will quickly destroy poorly designed electronics and hardware.

In days gone by, much of this work was carried out by divers but in today’s world the risks involved are unacceptable and even with the most advanced diving techniques, commercial diving rarely goes below 400 metres. Most modern ROV's are capable of 3000 metres and new systems are often rated for 5000 metres and over. Why? Because in the hunt for new oil reserves, that’s where we have to go.

The solution of course, is to keep it simple. Complex equipment will not be reliable in this environment. However, keeping it simple is often not that easy, the straightforward task of putting a nut on a bolt can only be achieved when you know exactly where the nut is, where the bolt is, where the holes are and you can align all the pieces perfectly with bad visibility and a current that wants to drag you away. Every job has to be planned carefully before any structure or tool enters the water.

The industry is highly competitive, there are many companies out there creating innovative tools and products and the race is to get your product to market first whilst producing the most reliable equipment possible. Successes are largely forgotten but screw-ups will be remembered in the industry for years, so even the smallest of jobs must work perfectly first time. The oil industry is particularly conscious of price and delivery. Companies like ROV.NET have built up a niche market where complete control systems can be designed, built and delivered in just a few weeks. Most of these tools are unique so development costs cannot be amortised over a large production run. Even where a product is being built with a production run of over 30 units over a 3 or 4 year period and a unit cost of up to $3,000,000 per system, every unit will be adapted to the clients own specific needs, no two units are identical.

The upshot is that development costs and lead time will make or break a project. Locking a dozen highly skilled programmers into a room for 6 months won’t cut it. We need to develop the base algorithms and HMI within a few days and have the completed software ready in time for the hardware appearing a few weeks later.

There are other issues. There is a history in this business of projects that quickly become unsupportable within a few short years. Hardware is the classic headache where chips, boards and busses become obsolete.

Typically, engineers, programmers and even project managers may change companies 3 or 4 times in a ten year period. Long term support for unique projects is difficult when the fundamental historical knowledge base from a company has dissipated. Yet the tooling systems can have a life well in excess of ten years. Clients need assurance that critical hardware and support will still be around long after the manufacturer, engineers and software programmers have disappeared.

Rate of technical change in our industry is as high as any other. Systems built today must be able to support new tools and technologies that will come into use over the next ten years. Clients need to be able to expand systems to cater for job based technical demands that they cant even begin to imagine. Force feed back rate manipulators, dynamic positioning, high bandwidth acoustic imaging, remote diagnostics and control through satellite links, compatibility with equipment as yet undesigned. No operator wants to be ruled out of contracts due to compatibility issues on a system barely a few years old.

Of course, the working environment also includes commercial, political and safety/QA concerns too. Equipment has to be not only fit for purpose and compliant with all current and most proposed regulations or guidelines -- but it also has to look good. Our clients have to sell their services to people who are very aware of current trends in technology. They expect output compatible with Microsoft Office products, email and web browsers. They are well aware of the difference between an analogue video tape and a digital DVD disk in terms of usability and ease of processing. When the job you do is hidden by several kilometres of murky water, the product your client gets is printed reports and survey information including video recordings. Their opinion is often swayed by the quality of these deliverables.

Technical support offshore is critical. Systems used must be easy to diagnose, HMI's need to be able to display different languages. Systems must hold enough spares to cover most eventualities without breaking the bank through over stocking. Every operator has stories of $200,000 a day vessels on downtime due to the lack of some $10 potting compound or a $100 connector.

Application Solution

So designers of equipment for underwater use have to be very careful in their selection of the appropriate technology. There is no right and wrong way of designing systems, all the systems working out there are good, the bad ones quickly disappear. But some are very definitely more suited to the environment than others. The solutions we aim for are not meant to be clever, they are meant to be simple. Right from the start we try to use standard protocols and busses with the emphasis on compatibility and longevity. The obvious example here is PC's. No engineer will claim that PC technology is the best of designs but due to the massive user base, you can guarantee that the technology will be around long after many industrial bus systems have disappeared. The other advantage is that next year’s hardware will be faster and cheaper than this year’s. God bless Moore’s law.

We often use PC/104 PC’s. The PC/104 implementation of PC's does have drawbacks. Dismantling a PC/104 stack needs a lot of care and I’ve never liked ribbon cables in noisy environments. On the plus side, PC/104 stacks are extremely rugged. The cost of the boards is very low, so supplying complete built and tested stacks as spares is commercially viable. The physical size allows smaller diameter housings to be used which are more suited to the high pressures of deep water. Every electronics engineer out there even in third world countries knows and understands PC technology. There are hundreds of PC/104 manufacturers -- many with international stockists. We use a lot of Diamond Systems products. The quality and range is superb. When problems arise we get to talk to the designers and solutions are provided quickly. As with all companies, products change over time creating compatibility issues but we can create new drivers in a couple of hours with simple installation instructions. Our software is normally supplied on Flash disks so a driver can be emailed out and dropped straight onto the networked system by the operators. Or on satellite linked systems, we can do it ourselves from here in the UK.

SoftPLC allows us to quickly create very compact control systems using ladder programming. The product has a range of communications options including RS232/RS485 and Ethernet and a communications link can be set up to reliably transfer any amount of data with a single instruction. Obviously this type of programming has limitations but SoftPLC allow us to easily tie in new instructions written in C to cater for time critical situations or unusual algorithms. ROV.NET has a library of modules both for hardware interfaces and applications such as notch filters, saving and reading set-up files and interfaces to intelligent sensors. Most of these routines were written purely to optimise existing SoftPLC instructions. It is an extremely versatile language. This alone is not enough to justify using this product but it has two other superb qualities. Because it's simple, it can be read and understood by other programmers. Trying to get to grips with someone else's “C” Code is usually a nightmare but a well written Ladder program can be very quickly figured out and it helps that the programs themselves are modular and very small in size.

The biggest advantage of all may be unexpected. SoftPLC programs can be monitored and reprogrammed in real time. The software provides a window into the code so that final development and bug finding can be carried out live whilst the equipment is being operated. The reduction in development time and thus cost from this simple technique is incredible. On our systems, the software is loaded into the newly built hardware in the morning and by lunchtime we are debugging the drivers and by supper we are usually crossing the T’s and dotting the I’s. Then the client comes along and says “Yes that’s good, but can we do this and make that do this” so our programmer will sit down at his desk and implement live changes to a system whilst in-water function testing is going on.

I sat offshore on a vessel in the middle of the Atlantic last year watching a software engineer having major problems with a heave compensation system. He would watch the equipment go into violent oscillation. Sit and read his program for a while, make a change to it, compile it, blow an EPROM, walk across the deck and install the EPROM to the controller and then go back and switch everything on again, wait for the system to load and carry out another test. Having the right tool for the job makes life a lot easier.

The software is not hardware dependent so it can be ported to any system that can be made to run DOS, only the drivers need changing. We have used it on VME and STD legacy equipment without any problems. Most new designs require compact solutions. A three inch PC/104 cube with 32 AI, 20 AO 64 DIO, 6 RS232/485 ports, Ethernet, USB and a couple of quadrature inputs fits into a very small housing and costs relatively little.

Everything we do underwater needs to be isolated so there are additional boards and modules required for this but again where possible we use off the shelf equipment such as Burr Brown or Opto22. This makes sourcing of spares easy even in the more far flung reaches of the world. We do build proprietary equipment but only where the equivalent does not already exist or is not fit for purpose.

Every solution will have drawbacks. SoftPLC uses a hardware based dongle which although perfectly reliable, is difficult to mount and looks out of place. But this is a small niggle for a product that brings years of development and utter reliability. For low volume and unique products in a highly competitive market, where future changes and upgrades will be a way of life, this solution is hard to beat.

On the surface ship, a control panel of switches and joysticks connected directly to a SoftPLC communicate with a network of 1 or more subsea SoftPLC controllers up to 5 kilometres away through fibre optic links bringing video, telemetry, Ethernet and analogue or digital sonar data back to the operators. Fast, efficient and highly robust. In over 5 years of using the product we have not had a single critical failure that wasn’t caused by water ingress and our clients appreciate the simplicity and ease of use.

So that’s the base control system. The design is simple, spares easily obtained and a projected life of well over the ten years we are looking for. Should an interface card become obsolete, operators have dozens of choices for alternatives and the driver module can be rewritten by any competent programmer.

What of the graphical HMI. We’ve already touched on the control panels but operator displays are critical. Not just for monitoring conditions and navigation but for diagnostics and fault handling/backup systems. The standard solution is once again a team of highly skilled programmers locked away developing screens from Visual Basic or C++. One-off projects though cannot justify this expense nor the potential for costly mistakes. Modern SCADA systems have developed into incredibly powerful packages with bolt on applications for alarms, logging, diagnostics, communications, language conversion, fault tolerant data channels and a host of extras such as Web browser interfaces and DCOM/OPC and DDE communications. Whilst the unit cost for these packages is high, remember that 10’s of thousands of hours have gone into their creation cutting the end developer's time from months to a few days. With the use of a good OPC server such as Kepware’s Kepserver, programming becomes a mixture of desktop publishing and point and click visual programming.

There are many good SCADA systems out there. The system of choice for us is Iconics Genesis. Is it the perfect solution? No not completely. It’s a very large package with capabilities that we barely touch on for our projects and the soft license system can be a pain when someone decides to defrag the disk. But again it’s utterly reliable and scaleable from the smallest project to the biggest.

Our systems are built with a common procedure every time. First we get the specification of what the client requires. Then we try to work out what the client actually needs. We create the structure and do a first pass at the system with mock up screens which are submitted for approval. We then get the client in, sit him down with the programmer and modify the screens and systems to their preference. What colour, how big, what position. None of this affects the underlying design but the client buys into the product through helping to develop an interface that suits his purposes. Once he’s gone we then tie the dynamic functions into the SoftPLC’s. This operation simply links a readout or switch to an input or output on the SoftPLC through a drop down list of I/O created originally in Excel. Gains and offsets can be handled by any of the software packages, we use the OPC server for simplicity and to ensure that other clients on the network get the same data.

The software is then loaded to the newly built hardware, debugged and tested. Then it gets interesting.

It’s now we find out what can go wrong and why that clever piece of programming isn’t worth a damn. So we send a programmer and an electronics engineer out with it for commissioning. Whilst the operator is working the system, the engineers are sitting in the corner monitoring and adjusting the system to tune it and make it work in the real world. With feedback back from the end users, Language translations supplied by the people who actually use it and diagnostics designed to cope with the reality of the situation. (Just what is the Chinese for “Nut Loader carousel deploy”?)

All our systems are open source. We train the operators not only to use the system but how to modify it to add additional lockouts and visual interfaces. If operators don’t like your system, it’s the kiss of death and by letting them change the interface to suit their needs without letting them into the core, it very quickly becomes their system.

We take all that knowledge back with us so the next system is even better and when the client needs an upgrade, we’re there to help. A recent trip to Angola for a system designed three years ago found major changes to the interface and a warm welcome from the operators. Their 20 ton tooling systems have over 1500 I/O and over 30 pages of touch screen controls alone. We upgraded the SCADA to the most recent version giving them increased capability and reliability. What was most impressive was the extent to which they had changed the system, tying in visual basic routines to the SCADA software to provide extended functions.

The use of Commercial Off the Shelf (COTS) software and hardware provides ROV.NET with the lowest cost fastest development time solutions available. Our clients are comfortable with the product. It looks and is at the leading edge of development but has the reliability of established designs. They can modify it themselves or bring us or anyone else in to do it for them. They get longevity and easy sourcing, simplicity with extreme ruggedness and a future upgrade path. The use of Windows technology for the graphics is not to everyone’s liking but remember that the graphics are not integral to the control. Should the top side graphics fail, the controller is unaffected. We still have cameras and sonar for feedback. Should the top side controller fail, the graphics system can talk directly to the subsea pods to allow an emergency recovery. (Did I mention how versatile these solutions are?)

Are these systems as reliable as a purpose built system? Yes, in our opinion they are more reliable. The systems we design are by nature intricate. The COTS products we use have taken decades to develop with 100’s of man years of engineering. They are used in control solutions worldwide and have been for years. If they have a bug, you better believe that someone found it. That hand crafted system cannot be proved reliable until it’s been properly punished for a few years.

We want to overlay real time graphics onto a video picture from a satellite hook up at three in the morning from a rig in the south China seas, “No problem”. And this, for a small company like ROV.NET, is critical. My clients want solutions yesterday and knowing I can draw on the resources of companies like Diamond Systems, SoftPLC, Kepware and Iconics lets us sleep at night. Our suppliers are not unique, there are many good companies out there who go that extra distance, with engineers who give a damn, who want your business and want to help you succeed. (But they aren’t the “big” suppliers.) There are many rubbish products out there too. Famous names who couldn’t give a toss whether your product works or not unless you’re going to buy unit quantities of 1000 or more and even then only let you talk to an engineer via email or through a witless salesman.

Does our solution have drawbacks? Well yes, of course it does, it cuts into our profits for one. These products are not cheap solutions. They are also not proprietary solutions. Our clients can take the product and run with it themselves or go to any other company for support. Although our clients are keen on the idea, being open with your technology can be a risky business.

It may be possible to design systems that are smaller or cheaper or more reliable but you can only get one of those options at the expense of at least two of the others. The use of COTS technology for our industry gives the best all round reliability, performance and price.

. . .