OT Cyber Lessons from the Production Floor: What Client Installations Taught Us About Water, Data Centers, and Smart Buildings

Most of what is written about OT cybersecurity is written from a desk; Threat reports, regulatory analyses, reference architectures. We publish that kind of material ourselves, and it matters. But there is a second body of knowledge that is rarely showcased: what actually happens when an engineer arrives on site with a laptop, a bag of patch cables, Rubyk-OT appliances and a two hour maintenance window.

We have installed hardware-enforced segmentation in drinking water and wastewater facilities, in enterprise data centers, and in building management systems across commercial and institutional properties. From the street, these environments could not look less alike. Underneath, they fail in remarkably similar ways.

What follows is a set of lessons we have learnt through leg work, sweat and difficult conversations with clients.

Lesson One: The Asset Inventory Is a Hypothesis, Not a Specification

Every project begins with a list. The client provides a spreadsheet of PLCs, RTUs, controllers, and HMIs, and that list is almost always incomplete.

In water and wastewater, the gap is geographic. Treatment plants are well documented, because people work in them daily and regulators inspect them. Remote pump stations, reservoirs, chlorination points, and lift stations are documented far less well, and in most systems they outnumber the plants by an order of magnitude. These are the sites with a cellular modem in a roadside cabinet and nobody on shift.

In building management, the gap is historical. A twenty year old property has been through several controls contractors, each of whom added devices to the same flat network and documented them in their own format, if at all.

We now treat the client's inventory as a hypothesis to be tested. Passive discovery on the live network, before any design work is committed, has become non-negotiable for us. The devices nobody remembers installing are reliably the ones running default credentials and firmware from the previous decade.

Lesson Two: The Binding Constraint Is Physical, Not Digital

Whilst security architecture may be decided in a conference room, it can then become constrained by a steel cabinet.

We have arrived at sites where the correct design was obvious and the installation was nearly impossible: no spare DIN rail, no available 24V supply, and a door that would not close over anything thicker than a paperback.

This shaped our hardware more than any threat model did. Rubyk-OT is a compact, hardened, in-line appliance precisely because the realistic alternative in a pump station cabinet is nothing at all. A solution that requires a new enclosure, a new power circuit, and an electrician turns a two week rollout into a two quarter capital project, and capital projects are where OT security initiatives go to die.

Photographs of the inside of the cabinet are worth more than the network diagram.

Lesson Three: The Integrator Holds the Keys, and Nobody Wrote That Down

In building management and enterprise data center environments especially, the controls integrator typically retains a permanent remote access path for maintenance and diagnostics. That path was established years ago, is often tied to a personal or shared credential, and rarely appears in the client's own network documentation. It exists because it is genuinely useful. It is also, functionally, a third party route into equipment that governs cooling, power, and access control.

We have written before about the ownership gap between IT teams protecting servers and facilities teams managing buildings. The integrator sits squarely in that gap, accountable to neither.

Our practice now is to ask three questions early and in writing: who has remote access today, through what path, and what happens operationally if we terminate it. The third question is the one that changes the design. Removing vendor access without providing a replacement mechanism does not improve security. It produces an outage, an angry phone call, and a client who has learned to distrust the security program.

Lesson Four: Much of What Is Called Remote Access Is Actually Remote Visibility

This is one of the most useful lessons we have learned, and it comes mostly from water.

When a utility says it needs a connection between the OT network and the IT corporate network, the stated requirement is access. Probe it, and the real requirement is almost always data moving in one direction: flow and pressure readings to a historian, compliance figures to a regulatory report, tank levels to a dashboard, alarms to an on-call engineer's phone. Nobody in the corporate environment needs to write a setpoint.

Once that is established, the problem changes shape. A bidirectional firewall rule is a policy, and policies can be misconfigured, bypassed, or quietly widened over years of change requests. A unidirectional gateway is a physical property of the equipment. Rubyk-UDR (a one way diode) was developed for exactly these cases: telemetry leaves the process network, nothing routes back in, and the segmentation claim rests on hardware rather than on the discipline of whoever last edited the rule set.

The same pattern recurs in enterprise data centers, where telemetry from chillers, PDUs, and UPS systems needs to reach monitoring and business intelligence platforms that have no legitimate reason to command that equipment.

Lesson Five: The Wireless You Did Not Approve Is Already Installed

In most sites we have worked in, we have found wireless where the documented architecture says there is none.

Commissioning engineers bring access points to avoid a cable run. Contractors add a temporary link during a retrofit and it becomes permanent. Vendors ship equipment with wireless enabled by default and nobody disables it. In multi-tenant and institutional properties, the pattern is close to universal.

Prohibiting wireless in a policy document does not remove it from the real life network. What works is providing a controlled, segmented, monitored wireless path that is more convenient than the unsanctioned one. Field engineers and third party technicians get the access their task requires, scoped and logged, and the incentive to improvise disappears.

Security controls that make legitimate work harder are not durable in real life. They get worked around, quietly, by competent people trying to finish a job.

What Changed in How We Deliver Client Installations

Over the years, these lessons moved us away from treating OT security as a software layer added to a network, and toward something narrower and more physical: a small, hardened device placed in line with the asset, enforcing its boundary in hardware, installed inside a normal maintenance window without redesigning the plant.

If you are scoping an OT security program for a water system, an enterprise data center, or a building portfolio, three questions are worth answering before you evaluate any product: what is actually on this network, what will physically fit, and who can already reach it. Answering those honestly will shape your specification more than any feature comparison will.

About RubyComm

RubyComm delivers tailored operational technology cybersecurity solutions designed specifically for the unique challenges of industrial and critical-infrastructure environments faced by organizations of all sizes. Unlike one-size-fits-all security products, RubyComm addresses the operational constraints, legacy system realities, and integration complexities that conventional off-the-shelf solutions often cannot adequately handle. Our approach maintains operational efficiency and business continuity while providing robust protection against sophisticated OT-specific threats.