HoloGraphic: bringing the control room to the machines

Title frame of the storyboard: the word HoloGraphic over a pencil sketch of an industrial boiler — The storyboard, 2017: the idea, told before it was built.

In 2017, during a four-month study at the Swedish Institute of Computer Science (SICS) in Västerås (today RISE), I explored what a mixed-reality headset like Microsoft HoloLens could bring to the people who work next to the machines in factories.

The information stays in the control room

In an industrial plant, everything you need to know about a machine lives on the screens of the control room: the temperature, the pressure, the alarms. However, the operators’ work happens out on the floor, next to the different machines. Operators and technicians in the field cannot carry that information with them, and when they can see it in the control room, they are far away and out of the context where things happen. As a result, they lose very valuable information, such as ambient sounds, the look of the machines and other contextual cues they can sense when they are standing in front of them.

Sketch of an operator in a high-visibility jacket sitting in front of the screens of a control room — The operator sits in the control room.

Blueprint of a plant with the control room on one side and four machines: main boiler, iron tank, miller motor and boiler — The information is in the control room, not with the equipment.

The pre-study had four goals:

Bring the information found in the control room to the plant, where the equipment is.
Let operators see the digital information of each process on top of the real equipment.
Improve safety with reminders based on location, so people know which safety measures to take where they are.
Find out how ready HoloLens technology really was for industry.

A walk through the plant

The storyboard illustrates possible uses to build, based on earlier interviews with operators, and follows one operator on their usual round through the plant. First, the headset goes on, under the safety helmet (although, in reality, the headset and the helmet were not compatible).

Two sketches of the same operator: first putting on the HoloLens, then putting the safety helmet on top — The operator wears the HoloLens, with the helmet on top.

Then the operator leaves the control room and walks to the first machine, the iron tank. As they get close, a virtual layer of information appears on top of it, and they can interact with it with gestures to see more details.

Blueprint with footsteps going from the control room to the iron tank — The operator goes to the iron tank.

Sketch of the operator from behind, walking towards a large industrial boiler — The operator walks towards it.

Sketch of the operator touching floating panels of charts and an operator note in front of the machine — The operator interacts with a virtual layer of information.

The round continues to the miller motor, where the information changes to the one that belongs to that machine. Finally, the operator goes back to the control room.

Blueprint with footsteps going from the iron tank to the miller motor — The operator goes to the miller motor.

Sketch of the operator in front of a motor, reading floating panels with charts — The operator sees information about the miller motor.

Blueprint with the whole route of footsteps back to the control room — The operator goes back to the control room.

Sketch of the operator sitting again in the control room — And sits in the control room again.

Two phases: edit and use

The scenario has two phases. In the edit phase, someone enters an edit mode and places the virtual objects on top of the real equipment. In the use phase, people walk around the factory floor and see the virtual information blend with the real equipment, updated in real time from the servers of the control room.

The prototype

The prototype showed information for machines such as the miller motor and the iron tank. When the user came within three metres of one of them, its information appeared on top of it as text, 2D graphics, 3D models and heat maps. Animations simulated the connection to the servers of the control room, so the data could be seen updating, and the interface was designed to adapt to the environment and stay easy to read.

A voice command opened the edit mode. With gestures, the user could then move each piece of information to where they wanted it, and the new place was saved for the next time. Safety reminders appeared depending on where the user was.

What the factory taught me

From the first visit to the factory, I knew conditions were not ideal: it was too noisy for voice commands, the gloves were too big for gestures, and the headset took up so much space that the helmet did not fit. I built the prototype anyway, to test it with operators and see how far it could go.

The HoloLens’ spatial mapping turned out to be the most interesting part. It let the application react to where the operator was in the factory: a reminder of the special equipment needed when someone entered a restricted area, or an evacuation route, showing where other operators were, in case of an alarm.

Gestures and voice commands were appreciated too. With a phone or a computer, you cannot carry out maintenance with your hands while holding the device.

Dissemination

I presented the concept to several manufacturing companies in Sweden, to gather feedback and new ideas. For example, I adapted the prototype for another company to explore how to visualise radiation (for training and learning). The question I brought to every meeting was the same: which use cases could make a real difference to the safety, efficiency and motivation of industrial workers?

I also presented these concepts and prototypes as a guest lecturer for interaction design students, with activities to help them reflect on designing mixed-reality applications.