Robots & Risk

Emerging Risk Report 2019 Technology Taking control Robots and risk

2 Lloyd’s of London disclaimer Key contacts This report has been co-produced by Lloyd's and Trevor Maynard University of Surrey staff for general information Head of Innovation purposes only. While care has been taken in gathering [email protected] the data and preparing the report Lloyd's and University For general enquiries about this report and Lloyd’s of Surrey staff does not make any representations or work on innovation, please contact warranties as to its accuracy or completeness and [email protected] expressly excludes to the maximum extent permitted by law all those that might otherwise be implied. About the authors Lloyd's and University of Surrey staff accepts no At the time of writing this report Roger Maull was a responsibility or liability for any loss or damage of any Professor of Management Systems in Surrey Business nature occasioned to any person as a result of acting or School. He was a Director of Surrey's Centre for the refraining from acting as a result of, or in reliance on, any Digital Economy (CoDE). He has held over £15m of UK statement, fact, figure or expression of opinion or belief research income and published over 130 publications in contained in this report. This report does not constitute leading journals and conferences. He re-joined the advice of any kind. University of Exeter on October 2018 as Academic © Lloyd’s 2019 Director of the Initiative for the Digital Economy at Exeter All rights reserved (INDEX) based in London. About Lloyd’s Steve Brewer is the founder and director of Infoculture, a startup offering support and facilitation for organisations, Lloyd's is the world's specialist insurance and teams and leaders in digital transformation. Steve has reinsurance market. Under our globally trusted name, we been involved in communication, community engagement act as the market's custodian. Backed by diverse global and project management for a number of years in UK capital and excellent financial ratings, Lloyd's works with and EU research-related projects. a global network to grow the insured world –building resilience of local communities and strengthening global Dr Wendy Maull was trained as an Engineer at economic growth. Cambridge University, Cranfield Institute of Technology and Université de Technologie de Compiègne and With expertise earned over centuries, Lloyd's is the worked for the Ministry of Defence before teaching at foundation of the insurance industry and the future of it. Bristol Polytechnic and Plymouth University. Her PhD Led by expert underwriters and brokers who cover more studied the way engineering students understand mathematics. She worked as a Research Fellow at the than 200 territories, the Lloyd’s market develops the University of Exeter before retiring. essential, complex and critical insurance needed to underwrite human progress. We would also like to acknowledge the support of the About University of Surrey research teams from two UKRI research awards. Digitally Enhanced Advanced Services (DEAS) EP/R044937/1 The University of Surrey was established on 9 and Technology Driven Change and Next Generation September 1966 with the grant of its Royal Charter, but Insurance Value Chains (TECHNGI) ES/S010416/1. its roots go back to a late 19th-century concern to provide greater access to further and higher education for the poorer inhabitants of London. The University of Surrey is a global community of ideas and people, dedicated to life-changing education and research. With a beautiful and vibrant campus, we provide exceptional teaching and practical learning to inspire and empower our students for personal and professional success. Through our world-class research and innovation, we deliver transformational impact on society and shape future digital economy through agile collaboration and partnership with businesses, governments and communities. Taking control: robots and risk

3 Acknowledgements The following people were consulted or commented on earlier drafts of the report, attended workshops and 1:1 Lloyd’s project team meetings, and provided their time and support to the  Dr Trevor Maynard, Innovation project; we would like to thank them all for their  Dr Keith Smith, Innovation invaluable contributions:  Anna Bordon, Innovation Insurance industry interviews and consultation  Kristian Jones, Innovation  Adam Seager, Argo  Linda Miller, Marketing and Communications  Michael Repper, Beazley  Kieran Quigley, Marketing and Communications  Stephen Wares, MS Amlin  Flemmich Webb, Speech and Studies  Ed Mitchell, MS Amlin  Elinor Kemp, Hiscox University of Surrey project team and area of expertise  Tim Allen, RenRe − Attila Emecz, Director of Strategic Partnerships − Yin Lim, Editor and Writer, Yin F Lim Editorial  Josef Schindler, W/R/B Services  Martin Twells, W/R/B Lloyd’s Market Association  May Ahmed, W/R/B  Tony Elwood, Senior Executive, Underwriting  Peta Kilian, Senior Executive, Market Operations and Further thanks go to the following for their expertise, Innovation feedback and assistance with the study:  David Branson, Associate Professor of Dynamics and Control and Director of the Nottingham Advanced Robotics Laboratory in the Faculty of Engineering, University of Nottingham  Mike Chantler, Professor of Computer Science, Texture Lab and the Strategic Futures Lab, Heriot- Watt University  Manuel Giuliani is Professor in Embedded Cognitive AI for Robotics at the Bristol Robotics Laboratory, University of the West of England  Jeremy Hadall, Intelligent Automation group, Manufacturing Technology Centre  James Law, Director of Operations, Sheffield Robotics, University of Sheffield  Oliver Lemon, Director of the Interaction Lab in MACS, Heriot-Watt University  Simon Pearson, Professor of Agri-Food Technology, University of Lincoln  Robert Bogue, consultant and cobot safety expert − David Plans, CEO, BioBeats Taking control: robots and risk

4 Contents Executive summary ............................................................................................................................................................... 5 1. Background ..................................................................................................................................................................... 11 Definition .......................................................................................................................................................................... 11 History ............................................................................................................................................................................. 12 Value chain ...................................................................................................................................................................... 12 Adoption .......................................................................................................................................................................... 13 2. Data and statistics ........................................................................................................................................................... 17 Growth ............................................................................................................................................................................. 17 Robot producers .............................................................................................................................................................. 19 Sectors ............................................................................................................................................................................ 21 3. Laws and regulations ...................................................................................................................................................... 23 Safety .............................................................................................................................................................................. 23 Standards ........................................................................................................................................................................ 24 Ethics in robot design ...................................................................................................................................................... 24 4. Wider impacts ................................................................................................................................................................. 27 Perceptions and acceptance ........................................................................................................................................... 27 Societal implications ........................................................................................................................................................ 27 Robot ethics..................................................................................................................................................................... 28 Jobs and skills ................................................................................................................................................................. 29 Robots and other digital technologies ............................................................................................................................. 31 5. A study of four sectors .................................................................................................................................................... 34 Industrial .......................................................................................................................................................................... 34 Agriculture ....................................................................................................................................................................... 37 Healthcare ....................................................................................................................................................................... 39 Retail ............................................................................................................................................................................... 42 6. Conclusions .................................................................................................................................................................... 45 References .......................................................................................................................................................................... 46 Appendix A: Glossary of terms - Robots ............................................................................................................................. 52 Appendix B: Safety standards ............................................................................................................................................ 53 Taking control: robots and risk

Executive summary 5 Executive summary This report focuses on the rapidly emerging development The logistics and healthcare sectors are also adopting of cobots - devices that help humans by extending their cobots, particularly in countries with labour shortages and physical capabilities - and its implications for the ageing populations. This has resulted in large facilities insurance industry. Cobots are a fast-developing processing up to 200,000 orders per day with only four segment of the robotics market and are becoming human workers present (LeVine, 2018) and hospitals increasingly popular as they are cheaper, smaller and adopting robots for a variety of jobs including surgery, smarter than regular robots (Gurman, 2018). As a result, cleaning and rehabilitation. Recent figures suggest that they are increasingly moving out of factories and being since 1990, robot prices have halved whilst labour costs used in sectors such as agriculture, healthcare and retail have more than doubled. In short, the economic case for where they interact with humans or help them to do jobs robot and cobot adoption is becoming increasingly that are dirty, dangerous, repetitive and difficult. As the compelling. focus of this report excludes software bots, we define Since cobots are a relatively new and emerging robot as ‘a machine situated in the world that senses, technology it is hard to predict how quickly they will be thinks and acts’ (Bekey, 2011). adopted. However, it is highly likely that cobots will play a The cobots market is growing fast significant role in transforming many industries, sectors and regions across the world in the next few years. Whilst cobots currently account for only 3% of the total Measuring the impacts on society robotics market, this figure is expected to reach 34% by 2025 (Smith, 2018) with the total value of sales set to Robot use has several implications for society, including reach US$9-12 billion by then (Murphy, 2017). The top determining the responsibilities and rights of the cobot exporters include Japan, Germany, Italy and machines, and where liability lies between owners, France (Trade Map, 2018), while 75% of imports are designers, programmers and other collaborators. made by China, the Republic of Korea, Japan, the United Introducing cobots into a public environment is much States and Germany (IFR, 2018c). more than a technological challenge. Possibly the biggest At the moment, manufacturing remains the principal limit on the use of cobots is their compatibility with health market for cobots. However, there are also clear and safety regulations and public attitudes, although emerging markets for industrial cobots (CB Insights, recent research points to increasing acceptance as 2018) in new set of industries (IFR, 2018b) where jobs people become more accustomed to seeing robots in are: use. − Dirty (e.g. construction and demolition: 1,100 While there is consensus that robots are already units sold in 2018); displacing jobs and will continue to do so, robots, − Dangerous (e.g. defence: 12,000 units bought in particularly cobots, rarely replace workers; they replace 2017); tasks. They often help workers through decision-making, or physical handling rather than replacing them. − Difficult (e.g. surgery: US$1.9 million worth sold However, as with any other tool, on aggregate robots will in 2018); and impact employment. − Repetitive (e.g. farming: 7,200 units sold in The shift towards automation will also create new jobs, 2018). as predicted by PwC’s 2018 UK Economic Outlook. Taking control: robots and risk

Executive summary 6 Taking safety into account Reacting to the new risk landscape A total of 38 robot-related accidents was reported to the Widespread cobot use will create new risks, change US Occupational Safety and Health Administration existing risks and reduce others. To develop a more (United States Department of Labor, 2019) in the 33 compelling picture of the cobots landscape, the report years between 1984 and 2017. Twenty-seven of those looks at four sectors (industrial environments, agriculture, led to the death of a worker (Nichols, 2017). In healthcare and retail) in which the use of cobots is comparison, the total number of workplace fatalities in currently constrained by concerns about the risks the US in 2013 alone was 4,585 (Bureau of Labor highlighted below. By helping insureds identify the risks Statistics, 2015). In Germany, severe industrial accidents and by offering ways to mitigate them, insurance could (i.e. those resulting in fatality or loss of limbs) are very help increase and speed up cobot adoption. rare, ranging from three to 15 annually between 2005 and 2012 (DGUV, 2015). Murashov et al (2016) pointed Reducing the risks out that there are few reports detailing accidents Robots can make a huge difference to how companies involving industrial robots, and that such incidents are operate. They can prevent people from having to work in rare worldwide. hazardous environments and inaccessible places. They The reliability of robots and cobots depends greatly on increase productivity because they can operate their design application and where they are used. Use of continuously and reduce human errors in warehouses. cobots typically involves multiple parties that include not The use of robotics in surgery has been found to shorten just the organisation where the cobot is installed, but also lengths of hospital stay, decrease complication rates and its installation team, systems integrator, allow surgeons to perform finer tasks. Robotics in consultants/advisors and maintenance team, possibly agriculture could potentially reduce environmental telecommunications and cloud service providers as well impact. Robotic devices executing precision tasks and operating either alone or in clusters can be less as the cobot’s designer and manufacturer. While most damaging than combine harvesters with their significant cobots are not necessarily considered dangerous, given weight and load-bearing footprint. The gains are not their relatively low payload, it is nonetheless important limited to productivity. Lower waste is also a major that firms ensure there is oversight from a health and environmental gain. In manufacturing and logistics, safety human expert and that their cobots operate in robots that work over a 24-hour period can enable fully compliance with international standards for robot safety automated environment with no human presence on-site standards. The design, manufacture and operation of and products to be built in small units that enable robots and cobots fall within the scope of several layers localised manufacture. In agriculture, cobots picking at of ISO Standards and Technical Specifications. night reduces the need to cool the produce. In short, Even though cobots are not necessarily considered cobots play an important role in meeting the objectives of dangerous, one of the challenges for the insurance the triple bottom line (people, planet, profit). industry is that “there are simply not enough cobots in the The report finds: market to get accident statistics” as stated by Interviewee #1 (see page 23). Given the low numbers of robot-related − The widespread adoption of autonomous mobile accidents and the fact that only around 3% of installed robots in retail and agriculture is likely to reduce robots were cobots in 2015, it is not surprising the data on cobot-specific accidents is not yet available. employees’ accidents and failures caused by fatigue. A dedicated risk assessment will be crucial, as are − The use of robots in dangerous environments, additional measures to reduce risk based on real-world such as nuclear decommissioning, mining, space performance data (Platbrood & Görnemann, 2018). and construction, can help improve safety. − For insurers, increasing adoption of robots in dangerous environments would reduce number of employee injury claims by automating processes. Taking control: robots and risk

Executive summary 7 Changing risks − Unscrupulous manufacturers might insert − The risk profile of employers’ liability and public unethical behaviours into the robots’ code. liability could change as liability could be pushed − Robots that have user-adjustable ethics settings back onto the robot product (e.g. choice between maximising length of life or manufacturer/designer. quality of life) may have their settings somehow − There is the potential for large-scale insurance set outside an “ethical envelope”. losses resulting from business interruption in − There is likely to be an ongoing struggle around supply chains that use cobots that could need the ownership of data between the intelligence replacing and redesign due to a cyber failure or functions, the cobot manufacturer, the internet of faults. things (IoT) provider, the product manufacturer − Cobots weighing less than 100kg are more and, in some cases, the consumer. Whilst cobots vulnerable to natural catastrophe events, which in an industrial environment collect large creates the potential for risk aggregation and amounts of factory and supply chain data, those therefore higher losses (e.g. 20 agricultural interacting with humans in the home or in a retail robots are more vulnerable to windstorm damage environment gather highly-sensitive personal compared to a 20-tonne tractor). data. Security breaches in these domains could lead to large losses and slow cobot adoption. The ensuing property damage and business interruption losses covered could be large. Artificial intelligence and robotics − Cobots require vast data storage facilities which could be vulnerable to cyber-attacks. Artificial Intelligence (AI) is a core technology in the future − Robot use in the healthcare sector can development of robotics. Indeed, all interviewees in the complicate liability. Medical clinicians using report said that the future of robotics is dependent on the robots without the necessary training, or future of AI. Developments in AI and their potential incorrectly operating may amount to medical impacts on the insurance industry are discussed in malpractice; a robot may be defective and further detail in Lloyd’s report Taking control: artificial covered under product liability policies. The intelligence and insurance. difficulty lies when robots are not fully Much of research in AI, such as deep learning for vision, autonomous and there is not a consensus speech recognition in interface design and the transition whether the clinician is negligent, the robot is from supervised to unsupervised learning, are central to defective or both. the future application of cobots. Challenges around New risks designing an interface to minimise safety concerns where speech may be misheard or misunderstood and could − Faulty cobots have the potential to cause lead to potentially life-threatening situations. A second damage to property (e.g. a moving robot might problem is making the interface engaging and entertaining so that people will continue to interact with drive into a supplier’s vehicle) and to other the device. Future developments in robotics are closely workers or people around as they start working in linked to those in AI. increasingly autonomous ways. − In healthcare and homecare, there are risks Conversational AI has immediate application where associated with working directly on people. robots are already fulfilling informational rather than Interviewee #7 (see page 24) notes that safety physical needs. For example, in healthcare there is tests with cobots tend to be done with adult male research into and development of text and speech subjects, which indicates that they may not conversational bots for mental health therapy and for necessarily be safe for environments with more health information provision. There is also research on vulnerable persons such as older people or physical robots in hospitals and care homes which can children. guide patients to locations or encourage them to do − Interviewee #2 points to a future for agriculture rehabilitation exercises. The use of cobots for healthcare where farms and fields will be worked by robots in the home puts the emphasis on their interaction with and drones together with tractors, but with high humans so developments in technologies such as levels of artificial intelligence. This scenario conversational AI will be central to their adoption. introduces new risks, e.g. the consequential Questions around new intellectual property (IP) losses to crops from hacking or design faults, ownership will arise from AI-enabled cobots in and increased losses from theft of valuable unsupervised learning within a factory or home. technology. Taking control: robots and risk

Executive summary 8 Does the IP belong to the firm, the owner, the systems Business opportunities for insurers integrator or even the robot itself? There are also security and privacy concerns for firms whose IP is in Growth manufacturing where the theft of the IP and ideas by disgruntled employees with good understanding of the − With an estimated compound annual growth rate systems can be a risk. of about 60%, cobots represent a substantial new Cyber-risks, including hacking and data theft from emerging market that offers considerable systems as well as when devices are communicating with opportunities insurers to provide products and each other are another concern. Processes could be services to cobot developers and adopters. hacked or systemic defect introduced, and factories and − These markets are likely to be international, but workspaces could be held to ransom. For example, cyber as the pressure for onshoring and responsive criminals could threaten to shut down robots on a farm manufacturing grows, opportunities in Western during peak picking season and threaten leaving the economies are also expected to grow rapidly. crops to rot. − The development of Robot as a Service business There is also the potential for deliberate unethical models will also expand opportunities with SMEs, which may previously have been priced out of the “training” of a cobot, although it would be the robot market. responsibility of the robot installer/systems integrator to minimise the likelihood of this occurring. Insurance products − Increasing adoption of cobots in environments that work closely with humans will expand the need for insurance products including: product liability, product recall, cyber, property, (contingent) business interruption and medical malpractice, all of which could be marketed as comprehensive insurance solutions for the robotics sector. Partnerships − There is an opportunity for the insurance industry to work directly with manufacturers to identify the risks associated with cobots deployment. This may well help to address health and safety concerns and therefore speed up adoption. Data − Data from cobots enables a much greater understanding of risk and offers opportunities for improved risks and pricing models. There are also opportunities to collaborate with clients to share risks relevant data to create better products. For example, in “precision farming”, sensor data from fields could be combined with external climate and weather data to allow developers to develop algorithms that help the farmer make best use of their land this in turn might allow insurers to create bespoke and more accurately-priced crop insurance. Taking control: robots and risk

Executive summary 9 Conclusions − The market for cobots is a fast-growing sector in the global economy and presents potential for specialty insurers. The large traditional robot – challenging to design, make and maintain – is being replaced with cobots that have voice recognition, are linked to industrial IoT, can be set up in half a day, use open-sourced code and can optimise around libraries of algorithms with learning capabilities and in many cases, are more mobile and agile. − The adoption of cobots is opening up a new world of commercial possibility for developers, suppliers, users and insurers, at the same time as creating new risks, some of which may be unknown today. Other risks will reduce and change. This will necessitate innovation in both existing and new lines of business. − Safety is the main concern and number one barrier to adoption, followed by issues of trust and acceptance (which are closely related to safety). These will require research into safety technologies, but also engagement and upskilling of stakeholders as lack of knowledge is the main barrier to uptake of automation, followed by the bespoke nature of products and concerns about the length of time to get a return on investment. This also relates to the issue about who has responsibility – it is not just safety technology that is required, but knowledge and training of users. − The report shows how the adoption of cobots is currently constrained by safety, security, liability and physical risks. By helping insureds identify the risks and by setting out ways to mitigate them, insurance could help increase and speed up cobot adoption. − The report suggests robotics designers and manufacturers, systems integrators and users should work with the insurance industry to mitigate and transfer the risks associated with robotics more fully. By leading in this space, the insurance sector will acquire the necessary knowledge to provide insureds with guidance on cobots best practices, thereby shaping the ecosystem in which they operate and the product offering. − To conclude, predicting how rapidly robot and AI technology will be adopted and implemented will occur is difficult, but it is highly likely cobots will play a significant role in transforming many industries, sectors and regions across the world in the next 5 years and beyond. Insurers can facilitate this growth by working with sectors to develop the products and services they need. Taking control: robots and risk

Background Emerging Risk Report 2019 Technology

1. Background 11 1. Background The widespread adoption and use of robotics have Typically, programming a cobot is also easier; this is given rise to emerging opportunities and risks, which we sometimes done through 3D visualisation or by moving will examine and discuss in this report. Broadly, there a robot arm through waypoints for faster set-up. a are three categories of robots (although it is important The disadvantages of cobots are that they typically to note that many would not include software bots and handle a payload of around 20kg only (although Fanuc RPA as robots): recently launched a 35kg cobot (Corbin, 2016); they 1. The traditional industrial robot that typically have limited reach; and their operating speeds are operates in a caged environment, working with considerably lower than traditional robots. This however high payloads and/or speeds. It is relatively has not affected their popularity. Collaborative robots well-understood in terms of its strengths, are increasingly moving out of factories and applied in opportunities, research challenges and growth sectors such as agriculture, health and welfare, retail potential. and entertainment where they are in contact with both 2. Cobots or collaborative/cooperative robots trained and lay operators. The leading robot and cobot which typically interact closely with humans and experts we interviewed for this report expect speed and b have much lower payloads . They can be fixed payload to increase each year as technology continues or mobile. to develop. 3. Software bots have few if any physical characteristics, but can respond to a variety of Definition inputs such as automated queries. This category more generally includes developments in Robotic Process Automation (RPA). Software Carnegie Mellon University’s Robotics Institute defines constitute a rapidly expanding market which a robot as ‘a machine that senses, thinks and acts’. has its own challenges, principally around the Robots therefore include: development of Artificial Intelligence (AI) (Gurman, 2018). − Sensors, which provides vision and/or force sensing; This report focuses on the rapidly-emerging − ‘Thinking’; they process information either development of cobots; devices that help humans by locally or remotely and make judgements; and amplifying their cognitive processes, interacting with − Acting; the application of a robot that includes both customers and employees and extending human welding, assembling and now, interacting with a physical capabilities (Wilson and Daugherty, 2018). A customer. fast-developing segment of the robotics market, cobots are becoming increasingly popular as they are cheaper, As the focus of this report excludes software bots, we smaller and smarter (Gurman, 2018). Unlike traditional add a sense of physicality to our definition of a robot: robots, they do not need to operate in a protective cage. They are cheaper to maintain and can often carry out a ‘a machine situated in the world that senses, thinks and wider range of tasks. acts’ (Bekey, 2011). a b A Glossary of Terms for robots used in this report can be found in There are inconsistencies in the use of this term. Interviewee #9 Appendix B stated that in his experience, ‘cobot’ is used to refer to the small industrial robot arms marketed for collaborative use, and ‘collaborative robot’ used for more general robots designed to work with/alongside people. Taking control: robots and risk

1. Background 12 An important debate is whether all the functions of In the period following the widespread adoption of ‘lean’ ‘sense, think, act’ must be in a single box. This seems across industry, sales of industrial robots remained unduly restrictive in today’s world of cloud computing, relatively static at around 100,000 units per annum. unlike the early days of robots where vision/tactile However, since 2010 when ISIXSIGMA reported that systems were frequently found in separate devices. 70% of companies were using lean principles (Woods, Today’s concept of a robot is also becoming 2010), there has been a substantial increase in increasingly blurred, overlapping strongly with many worldwide sales of industrial robots; 387,000 units similar definitions of digital technology as we head recorded in 2017 (Statista, 2018). towards recognising anything with a microprocessor as a robot. Value chain The perception or ‘robotness’ may also be context- dependant. For example, a comparatively simple ‘pick The following framework is useful for understanding the and place’ device in a factory is called a robot whilst a value chain of robotics, which comprises of three main sophisticated washing machine in the home is not. elements: Devices that society frequently label as ‘robot’ tend to 1. Brains. The intelligence that is the machine have their sensors and actuators outward-facing, while learning, natural language processing and inward-facing devices like ovens and fridges manipulate artificial intelligence aspects of the robot. As a controlled internal environment and are therefore categorised as appliances. ‘intelligence’ providers build a platform of data they will be able to increase a robot’s Despite these challenges, we consider the addition of understanding of its environment. This gives the term ‘machine situated in the world’ to the standard rise to opportunities to build a platform that hosts data from multiple robots, learns from ‘sense, think and act’ definition as a good basis for this analysis. Much of this report considers devices that many examples and provides ‘intelligence’ as a service. This is discussed further in Lloyd’s exhibit increasingly sophisticated examples of ‘sense, report Taking control: AI and insurance. think and act’. 2. Manufacture and assembly. Robot History manufacturers are classified into industrial robots, medical robots, consumer robots and d drones . Robots were first used in car manufacturing by General 3. Components. Includes specialist component Motors in the early 1960s, but it was not until the 1980s manufacturers, many of whom have been when widespread application and research saw them engaged in the robotics industry for long being employed for processes such as welding, paint periods providing basic mechanisms such as spraying, assemblies and inspection. By then, robots motors gears and actuators. There are also were also used in electronics through pick and place specialist software providers who enable users machines for printed circuit boards, as the application of to tailor their robots to specific environments. robots became an essential part of Computer Integrated Finally, there is the long supply chain of basic Manufacturing (CIM) and the factory of the future. component parts such as semiconductors. However, the business case for robotics implementation On the supply side, the advent and adoption of digital was often problematic, as humans were cheaper and technology has been a major enabler in the growth of more agile. In car manufacturing for instance, cobots. This includes the prevalence of computer chips productivity from the application of the Toyota supporting machine learning and deep learning, 3D Production System (TPS) was found to be much higher cameras and drones, which along with other sensors enable the capture of large volumes of data. All of this than from automation, and the mantra became ‘lean is supported by the scalability of cloud-based computing first then automate’. Closely associated with the resources. Taken together, they contribute to a huge techniques of lean production, TPS includes a range of upsurge in possibilities that are now being recognised process improvement techniques aimed at reducing by technology developers as well as entrepreneurs and waste and focusing on the social and procedural venture capitalists (VCs). aspects of the process flow rather than the technologyc. c TPS was widely researched by a team at Massachusetts Institute of TPS techniques led to lower defects, higher productivity and higher Technology (MIT) which found that engaging employees through levels of worker engagement (Adler, 1993). process improvement led to much higher productivity than technology d adoption: an echo of previous extensive socio-technical systems This report does not cover drones, which are discussed in Lloyd’s research. MIT’s research resulted in the book The Machine That report Drones take flight: Key issues for insurance Changed The World (1990) which provided substantial evidence that Taking control: robots and risk

1. Background 13 Adoption Box 2: New business models Technology adoption is often non-linear and tends to Across industry, new business models are emerging follow an S-curve. Five key factors determine the that focus on the provision of an asset ‘as a service’. adoption of new products (Rogers, 2010): Probably the most famous example is with large equipment suppliers such as Rolls-Royce who offer 1. The relative advantage of the new technology ‘Power-by-the-Hour’ to airlines, such that the airline over previous technologies. The greater the no longer needs to buy the engine outright but pays a advantage, the faster the adoption; usage fee for every mile flown. This shift to ‘outcome- 2. Compatibility with existing values and practices; based contracting’ is also occurring in robotics where 3. Simplicity and ease of use; the user firm can contract for a Robot as a Service 4. Trialability; and (RaaS). 5. Observable results. RaaS offers the robot user many advantages, e.g. no Each of these factors is important when considering the capital outlay, so it is OPEX (operating expense) not speed of cobot adoption. Possibly the biggest limitation CAPEX (capital expenditure). It also provides more for cobot adoption however, is its compatibility with predictable monthly expenses,costs and lower health and safety regulations. We will discuss this maintenance and repair costs. For the robot provider, further in Section 3 on Laws and Regulations. it reduces the sales barrier and provides recurrent revenue. However, there are some challenges that Despite the widespread application of the Rogers model arise e.g. around negotiating priorities where a cobot (Rogers, 2010), a smooth linear growth trajectory for is shared across facilities or setting expectations cobots is unlikely. For example, as sales increase, around conditions of use and any damage that may production unit price fall and accessories such as ensue. grippers become cheaper, software and ancillary services become more available. Predicting when this Cobots from Universal Robots can now be hired for inflection point will occur is notoriously difficult (Robotics as little as £65 per day (Bots.co.uk, 2018), a cost Business Review, 2017; Clements, 2018) but despite very close to the 2019 UK’s National Living Wage. As these caveats around timings and trajectories, it is cobot business models develop, we can expect to highly likely that cobots will play a significant role in see, for example, pricing models based on hourly transforming many industries, sectors and regions usage or even the productivity gains achieved by the across the world. This will result in societal and robot. technological changes that will provide the insurance industry with a similar range of opportunities and challenges. The speed of adoption is likely to differ across sectors. For example, its use in manufacturing could pick up when unit prices fall, changing the economic trade-offs, but in healthcare, adoption is likely to be limited by the need to train professional support workers and in the home, through the development of appropriate interfaces. From the demand side, robotics technology is no longer limited to industrial use, but is being implemented in many areas from factories and hospital theatres to vineyards, care homes and theme parks. We are also beginning to see more devices in the home, such as vacuum robots that can now be integrated with Artificial Intelligence (AI) interfaces like Amazon’s Alexa (Song, 2018). The growing number of devices available for the domestic environment will have ramifications for the insurance market as well as wider societal impacts. Looking longer term for example, mini gardening robots could enable personal food production at home which could help reduce food waste and purchasing. This in turn has implications for employment and tax revenue of the traditional food value chain. Taking control: robots and risk

1. Background 14 Economics of robotics adoption In his analysis of the economics of cobots, Schmidt (2018) outlined a number of main drivers for cobot There is no denying that robots can make a huge integration: improvement, innovation ergonomics, new difference to how firms operate. They protect people assembly processes, quality improvement and from operating in hazardous environments (noise, monotony reduction. All these factors alter the temperature, chemicals, etc) and inaccessible places. traditional cost-benefit analysis of technology adoption, They also offer substantial productivity gains by prompting Schmidt to develop an economic comparison operating continuously and in warehouses through of alternative production systems. reductions in shrinkage (damage, theft, admin error, Figure 2 shows that the range of economic usefulness etc). for Human-Robot Collaboration (HRC) is quite narrow. Due to their ability to carry out repetitive tasks At low production volumes, manual labour is used, and accurately, robots allow for much lower defect levels in at high volumes full automation is economic. In the manufacturing products. As part of an integrated space between these extremes, traditional robots are system in a factory, the home, the farm or the economical at slightly lower volumes than full warehouse, robots can warn other devices of variations automation, and HRC has taken a place at slightly and therefore allow them to adapt accordingly. As part higher volumes than manual labour. There are of of a fully-integrated network system they can also course many products and industries that fit this provide data for optimising logistics and resource description including, for example, most small batch allocation. Recent figures suggest that since 1990, production in the food industry (according to robot prices have halved whilst labour costs have more Interviewee #2). than doubled (see Figure 1). In short, the economic Figure 2: Cobot integration and economic comparison case for robot adoption is becoming increasingly of alternative production systems compelling. The gains are not limited to productivity. Lower waste is also a major environmental gain. In manufacturing and logistics, robots that work over a 24-hour period can enable a fully automated and unmanned environment and products to be built in small units that enable localised manufacture. In agriculture, cobots picking at night reduces the need to cool the produce. In short, cobots play an important role in meeting the objectives of the triple bottom line (people, planet, profit). Figure 1: Robot prices have fallen in comparison with labour costs Source: Schmidt, 2018 This model has implications across all four of the sectors included in the report. In industrial, agricultural and retail contexts, the model indicates the economics of replacing labour. It may indeed underestimate the effect by not emphasising the benefits of 24-hour working, the increasingly low costs and sophistication of devices as well as the benefits for process flow and quality control. Indeed, as cobot use increases, production of cobots acquires scale economies, unit Note: Index of average robot prices and labour compensation in costs of sale and installation will decrease, giving rise to manufacturing in United States, 1990=100% further opportunities for their adoption. Source: Tilley, 2017 Taking control: robots and risk

1. Background 15 Modified framework – focus on process types This provides a useful framework to consider multiple Schmidt’s adoption model however has some types of robots and their risk implications. For those potentially confounding effects. For example, it does not highly-repetitive contexts requiring very limited human adequately reflect the increased personalisation of interaction, robots are being widely applied. Those robots that have a high degree of autonomy are products and services. Digital technology is driving the designed to work in stranger/alien environments and emergence of a ‘market of one’, where consumers are are heavily dependent on sensors, control systems, AI demanding increased product personalisation and and software design. service environments, higher levels of personalised care. These demands drive variety into product and The challenge facing widespread firm adoption of service delivery and require robots to operate on a cobots is the classic Innovator’s Dilemma; the greater range of tasks. This is one of the main drivers of incumbent has few incentives to change its business the ‘S’ curve. model from its existing economies of scale. The speed of adoption will therefore be closely aligned to how The lower the variety and greater the repeatability in the persuasive their economic arguments are. task, the more applicable is the traditional robot and its implementation will depend on a classic cost-benefit analysis around processing times, volumes, costs of acquisition/contract terms (see Figure 2). In environments where it is important or necessary to interact closely with a human, the traditional heavy/fast robot is unusable. Many of these contexts have much higher potential variety. This focus on the nature of the task rather than the sector enables us to develop a modified adoption framework that is applicable across sectors. This modified framework has four process types: − Repetitive. Standard processes which are carried out frequently, are highly predictable, consistent and usually efficient. These are tasks where traditional caged robots are useful. At very high volumes, dedicated automation is used. − Repeaters. Processes which are still predictable but less frequent in occurrence. The type of robot applied will depend on the cost- benefit analysis. − Strangers. Processes which are highly customised, rarely occurring and often requiring a high level of specialised resource. These are likely to be environments where cobots are used if the variety does not overwhelm the analytic capability (e.g. in vision systems). − Aliens. Things that have not been seen before and will therefore be outside the robot/cobot’s area of expertise. Robots/cobots are unlikely to be used in these contexts until much more sophisticated data analytics and learning have been developed. Taking control: robots and risk

Data and statistics Emerging Risk Report 2019 Technology

2. Data and statistics 17 2. Data and statistics Growth Figure 4: Estimated worldwide operational stock of industrial robots 2016-2017 and forecast for 2018 through 2021 The main adopters of robotics technology have been industrial firms; in manufacturing, extraction industries 4000 and industrial service applications. 98% of the 253,748 robots delivered in 2015 were traditional robot systems 3500 (Murphy, 2017). These are typically in caged environments working with heavy payloads at fast 3000 speeds, costing around US$100,000 each including the physical device and software. The International its2500 Federation of Robotics (IFR) has charted a sharply- un2000 rising curve for industrial robot sales in the decade to 0 2017 (see Figure 3) while Loup Ventures estimates that '001500 this market will grow by 11.8% annually to a value of over US$33 billion by 2025 (Murphy, 2017). 1000 Figure 4 shows the projected growth in operational 500 stock of industrial robots, estimated at 16% per year. Figure 3: Estimated worldwide annual shipments of 0 industrial robots by regions 300 Source: IFR World Robotics, 2018 250 200 tsi un 150 '000100 50 0 Asia/Australasia Europe America Source: IFR World Robotics, 2018 Taking control: robots and risk

2. Data and statistics 18 Cobots forecast to rise sharply in the next five years (see Figure 6). A substantial acceleration in sales will occur in cobots. Figure 6: Service robots for personal/domestic use. Unit Loup Ventures estimates that the numbers shipped will sales 2016 and 2017, forecast 2018 and 2019-2021 increase from 8,950 in 2016 to 434,404 in 2025 at a compound annual growth rate (CAGR) of 61.2%; a total 12 value of around US$9 billion (Murphy, 2017). Other reports predict similar growth; MarketsAndMarkets.com 10 (2018) estimates a market worth US$12 billion by 2025 and a growth rate of 50% CAGR. BIS Research predicts the market’s value to be US$5.5 billion by 2023 its8 with a CAGR of 64% while Jürgen von Hollen, president un of Universal Robots, expects market growth of between on6 e i 50-70% over the next five years (Demaitre, 2018) . ill Whilst cobots currently account for only 3% of the total m 4 robotics market, this figure is expected to reach 34% by 2025 (Smith, 2018). 2 Service robots 0 The IFR predicts that the market for service robots will 2016 2017 *2018 *2019-2021 grow by a 21% CAGR over the next three years (see Household robots Entertainment and leisure robots Figure 5). Figure 5: Service robots for professional use. Unit sales Source: IFR World Robotics, 2018 2016 and 2017, forecast 2018 and 2019-2021 45 40 35 tsi30 un 25 20 '00015 10 5 0 2016 2017 *2018 *2019-2021 Source: IFR World Robotics, 2018 The IFR also examines robots for personal/domestic use separately, as their unit value is generally only a fraction of many types of service robots for professional use, particularly medical robots (IFR, 2018a). Even so, the use of household and entertainment robots is e The difference in estimates are due to varying baseline figures, economic assumptions and methodologies employed. Taking control: robots and risk

2. Data and statistics 19 Robot producers Who are they? The world’s three largest industrial manufacturers of industrial robots are: 1. Fanuc with a strong automotive installed base (totalling 400,000); 2. Yaskawa which has a business built on its expertise with servo motors (installed base of 360,000); 3. ABB which is integrating robotics within its industrial Internet of Things (IoT) solution ability (installed base of 300,000). As stated previously, the immediate future for industrial robot sales looks buoyant (Francis, 2018a) with established manufacturers expanding their range and many start-ups emerging with new technologies and materials, such as Grabit and its electroadhesion devices capable of lifting all kinds of different objects. ABB, Fanuc, Yaskawa and Kawasaki are considered the leading manufacturers of cobots, with Asia Pacific as the fastest-growing region, according to Inkwood Research (2018). Yaskawa, Fanuc and ABB are also developing smaller profile, lighter-weight cobots, but emerging players in this space include Omron, Universal Robots and Robotiq, whose lighter, cheaper and more easily programmable devices are fundamentally changing the market. We are arguably at an inflection point for cobots (Clements, 2018; Robotics Business Review, 2017) and the sector is moving into a new phase; from being a market with heavy barriers to entry including expensive research and development (R & D) and marketing costs and expensive programming (Market Research Engine, 2017) to one that is rapidly opening with many new suppliers and technologies. The large traditional robot – challenging to design, make and maintain – is being replaced with cobots that have voice recognition, are linked to industrial IoT, can be set up in half a day, use open-sourced code and can optimise around libraries of algorithms with learning capabilities and in many cases, are more mobile and agile (as noted by Interviewee #1). Taking control: robots and risk

2. Data and statistics 20 Where are producers located? In 2017, worldwide sales from the export of industrial robots by country totalled US$6 billion (Workman, 2018), with the top 11 countries contributing 86% of the world’s exports in robots (see Figure 7). The 11 countries each exported over US$2 billion during 2017 (figures compounded from Trade Map, 2018). However, the world production of robots is larger than this, as within-country sales are not accounted for. Figure 8 indicates that North America and Europe have a predominance of smaller manufacturers of service robots while Asia is dominated by a few larger companies. Figure 7: Largest exporters of robots in 2017 Rest of world, 14.10% Taiwan, Sweden, 2.50% $816.6 2.80% $153.7 million 166.6 million Japan, 36.60% Austria, 3% US$2.2 billion $184.1 million Denmark, 3.10% $185.3 million S. Korea, 3.30% $198.7 million China, 3.40% USA, 5% $207.9 million $303.3 million France, 5.50% Germany, 14.20% $332.8 million Italy, 6.50% $858.5 million $392.5 million Source: Trade Map, 2018 Figure 8: Origins of over 700 robot companies 29 133 307 250 Asia North America Europe Others Source: IFR World Robotics, 2018 Taking control: robots and risk

2. Data and statistics 21 Sectors Agriculture is also seeing considerable interest in robots following John Deere’s acquisition of the ‘see and spray’ robotic start-up Blue River Technology for For the present and immediate future, robots are mainly US$305 million. Cobots are being used in the RASberry used in manufacturing (IFR, 2018b) particularly in project at the University of Lincoln, where human industries like automotive and electronics where jobs strawberry pickers are supported by mobile robots are: acting as transporters (Duckett et al, 2018). Finally, the − dirty (e.g. metals and machinery: 44,500 units growing application of cobots in the health and social sold in 2017); care sector is expected to be driven by significant − dangerous (e.g. plastics and chemicals: about demand for labour as the UK is forecast to be short of 20,000 units sold in 2017|); around 400,000 care workers by 2026 (Matthews-King, − difficult (e.g. electronics; 121,300 units in 2017); 2018). and − repetitive (e.g. automotive industry: over 130,000 units sold in 2017). Typically, these jobs would be handled by large industrial traditional, caged robots. All aspects of manufacturing willremain the principal market for cobots, but there is evidence of growth in service robots for home and domestic use (Jacobs and Virk, 2014).There are also clear emerging markets for industrial cobots (CB Insights, 2018)that contribute to lowering labour costs and enabling strategies such as the reshoring of manufacturing, in the same type of contexts but a new set of industries (IFR, 2018b): − dirty (e.g. construction and demolition: 1,100 units sold in 2018); − dangerous (e.g. defence: 12,000 units bought in 2017); − difficult (e.g. surgery: US$1.9 million worth sold in 2018); and − repetitive (e.g. farming: 7,200 units sold in 2018). Internationally, 75% of total robot sales go to five countries: China, the Republic of Korea, Japan, the United States and Germany (IFR, 2018c). There are also extensive and growing robotics markets in logistics and warehouses, particularly in countries that experience labour shortages. For example, in the US, Amazon’s Jobs Day in August 2017 saw only 20,000 applications for 50,000 job openings (Morris, 2017). Major start-ups in this robotics space include Kiva (acquired by Amazon in 2016), Seegrid, Clearpath and Fetch. Cobots are also being used by DHL within its life sciences division for picking (i-SCOOP, 2017). In high-volume contexts there is evidence that jobs will be displaced; Chinese e-commerce giant JD.com has a 100,000 sq. ft. facility in Shanghai processing up to 200,000 orders per day but only four human workers (LeVine, 2018). This raises the issue of changing property risks with concentration of high-value equipment and limited human supervision. Taking control: robots and risk

Laws and regulations Emerging Risk Report 2019 Technology

3. Laws and regulations 23 3. Laws and regulations Attitudes on regulating industrial automation have been worker (Nichols, 2017). In comparison, the total number found to vary from country to country. For instance, of workplace fatalities in the US in 2013 alone was worker safety and job security are important 4,585 (Bureau of Labor Statistics, 2015). In Germany, considerations in framing industrial regulations and severe industrial accidents (i.e. those resulting in fatality robot law in the US, while in China where industrial or loss of limbs) are very rare, ranging from three to 15 automation is a mean of growing its economy, there is annually between 2005 and 2012 (DGUV, 2015). an added focus on patents from robot laws (Prakash, Murashov et al (2016) pointed out that there are few 2017). South Korea is developing a Robot Ethics reports detailing accidents involving industrial robots, Charter, a code of conduct established for people and that such incidents are rare worldwide. involved in the development, manufacture and use of In the medical field, a 2016 US study by Alemzadeh et intelligent robots to prevent harmful or adverse effects al (2016) found that 144 people had died during or after that may arise, such as the destruction of social order, robot-assisted surgery in the US between 2000 and and to ensure intelligent robots contribute to enhancing 2013. With over 1.75 million robotic-assisted the quality of human life (Statutes of the Republic of procedures performed over this period, the number of Korea, 2019). The EU report Guidelines on Regulating deaths per robotic procedure is very small and has in Robots (2014) stated that stringent product-safety rules fact decreased as the number of robotic procedures has should not stifle innovation. gone up. In the UK, health inspectors have been found to be The reliability of robots depends greatly on their design more risk averse than their EU counterparts to the risks application and use environment. Modern Fanuc of installing cobots in UK factories, often insisting that industrial robots are said to have a Mean Time between all robots should be fully guarded (Interviewees #7 and Failures (MBTF) of between 80,000 to 100,000 hours #9). The Health and Safety Executive (HSE) magazine (Motion Controls Robotics, 2019). In comparison, ST cites a number of people who have gone through the Robotics (2018) quotes a MTBF of 15,000 hours for its process of putting a cobot into their factory and then R12 cobot arm in a workshop situation. A 2004 being told by a health and safety inspector: ‘That’s not reliability analysis of mobile robots in a hostile safe, stop.’ (Warburton, 2017). When the appropriate environment found an average MTBF of 24 hours, an regulations have been followed at all stages of design improvement from the eight hours reported in 2002 and implementation, with robust mitigation procedures (Carlson et al, 2004). Mobile robots are lighter-built, with and training in place, then it can be shown that the less mechanical redundancy than static industrial cobot will probably not be considered ‘dangerous’ robots, which means they have lower reliability. The (Warburton, 2017). mobile robots tested were however not operating in Safety optimised conditions in a factory with low-variety tasks and loads. Instead, they were simulating military operations in urban terrain and urban search-and- In 2015, a technician died in an accident with a robot at rescue operations, hence in varied and hostile a Volkswagen plant in Germany, sparking a flurry of environments, and with a high variety of loadings. discussion about robot safety (Financial Times, 2015). Given the low numbers of robot-related accidents and A total of 38 robot-related accidents was reported to the the fact that only around 3% of the installed robot base US Occupational Safety and Health Administration were cobots in 2015, it is hardly surprising that data on (United States Department of Labor, 2019) in the 33 cobot-specific accidents is not yet available. As years between 1984 – when the first human was killed Interviewee #1 stated: “There are simply not enough by a robot at Ford’s Flat Rock plant in Michigan – and cobots in the market to get accident statistics”. 2017. Twenty-seven of those led to the death of a Taking control: robots and risk

3. Laws and regulations 24 Usage of cobots typically involves multiple parties that construction, installation and use in three categories: include not just the organisation where the cobot is mobile service, physical assistant robots and person installed but also its installation team, systems carrier robots (BSI, 2014). Its publication in 2014 was integrator, consultants/advisors and maintenance team, welcomed by the CLAWAR (Climbing and Walking possibly telecommunications and cloud service Robots Association) for providing much-needed clarity. providers as well as the cobot’s designer and Previously, whenever there was an accident involving a manufacturer. When considering risks of cobots coming new robot product, its manufacturer could easily be in contact with so many humans, the question that sued for potentially large damages as the manufacturer arises is, can cobots damage humans? Interviewee #7 would have faced great difficulty in proving that all has noted that safety tests with cobots tend to be done necessary steps had been taken to ensure that a new with adult male subjects, which indicates that they may robot was ‘safe’ and therefore not at fault (CLAWAR, not necessarily be safe for environments with more 2014). vulnerable persons such as older people or children. Although the regulations for determining whether a While most cobots are not necessarily considered device qualifies as a medical device in markets such as dangerous, given their relatively low payload, it is the EU, USA, Canada, Brazil, Australia and Japan are nonetheless important that firms ensure there is broadly comparable, regulators can exercise a oversight from a health and safety human expert and significant amount of discretion in assessing whether a that their cobots operate in compliance with given system or device meets those criteria. Such international standards for robot safety standards (see variations however make it difficult to state with Section 3 on Laws and Regulations). A dedicated risk certainty whether a specific robotic device used in a assessment is crucial, as are additional measures to medical application will be classified as a medical reduce risk based on experience (Platbrood & device, and is therefore subject to regulatory oversight, Görnemann, 2018). review or clearance (UL, 2017). This then affects the As home robots and RaaS becomes increasingly insurance status of any given device, which may vary popular, even the ownership of the device becomes from country to country. problematic. Interviewee #9 noted that in RaaS, the See Appendix B for further details about ISO standards. service provider will harbour the bulk of the responsibility, subject to users following established guidelines and manufacturers providing equipment that Ethics in robot design comply with standards and ratings. Methods for recording and analysing incidents to help identify what It is clear that health and safety considerations should caused them are currently in development. be implicit in the design of a robot, especially those that Standards may come into contact with humans and more particularly, lay people. With the use of a professional service robot, some shortcomings may be mitigated by The design, manufacture and operation of robots and its operator receiving adequate training. However, cobots fall within the scope of several layers of ISO where the operator is a lay person physically interacting Standards and Technical Specifications. Cooperative with a personal service robot, safety is a major and robots are specifically addressed in ISO 10218-1:2011, primary concern (Röhrbein et al, 2013), as ‘naïve users’ which provides for four modes of safe working. do not have a good understanding of how a robot moves (Rodrigues et al, 2016). Published in 2016, ISO/TS 15066 only applies to cobots in industrial environments, although its principles are The Engineering and Physical Sciences Research relevant in other sectors. This Technical Specification Council pointed out that Asimov’s laws of robotics are aims to provide a comprehensive risk assessment guide inappropriate because they insist that robots behave in of all the motions, interactions and operations a robot certain ways, as if they are people (EPSRC, 2010). In should perform in environments where humans are real life however, it is the people who design and use present. For instance, its pressure and force limit the robots who must be the actual subjects of any law. specifications can help prevent injury if there is incidental contact between a human worker and a robot Hence, the Council has developed five “rules” for the (Robotiq, 2018). For example, in the case of a designers, builders and users of robots: packaging application where a robot’s points of human 1. Restricting the design of robots as weapons contact are already limited to workers occasionally only for national security purposes; supplying one bin and occasionally removing the other. 2. Robot design and operation should conform to Personal care robots are governed by the ISO existing law, including privacy; 13482:2014, which provides guidance on safe design, 3. Robots should be designed to be safe and secure; Taking control: robots and risk

3. Laws and regulations 25 4. The machine nature of robots should be transparent, and illusion of emotions not used to exploit vulnerable users; and 5. It should be possible to find the person responsible for a given robot. Set out in 2010, these rules were intended as a living document and a basis for discussion and debate. Research however shows that it is hard to stop people from bonding with service robots and attributing volition to them (Knight, 2014). Also, it is not clear whether rule no. 5 applies to the Responsible Person of the Machinery Directive (HSE, 2011) or a person directly responsible at a given moment. The UK’s British Standards Institute (BSI) standard BS 8611:2016 provides guidelines for the identification of potential ethical harms (BSI, 2016a) as well as for the safe design and protective measures of industrial, personal care and medical robots (BSI, 2016b). The standard recognised that potential ethical hazards arise from the growing number of robots and autonomous systems being used in everyday life, highlighting that ethical hazards have a broader implication than physical hazards. However, claims of compliance with BS 8611:2016 cannot be made, as it is written as guidance and recommendations rather than a specification or code of practice. Hence it is important that different ethical harms and remedial considerations are duly considered. Taking control: robots and risk

Wider impacts Emerging Risk Report 2019 Technology

4. Wider impacts 27 4. Wider impacts Perceptions and acceptance Societal implications A 2012 study of public attitudes to robots by The application of robots opens up a number of f Eurobarometer (2012) found that 70% of Europeans implications on society, including the responsibilities had a positive view of robots. A majority agreed that and rights of the robots and the lines of responsibilities “robots are necessary as they can do jobs that are too between owners, designers, programmers and other hard or too dangerous for people” (88%) and that collaborators. This in turn raises issues around the “robots are a good thing for society because they help distribution of value between the supply chain partners people” (76%). They were however unhappy about and the potential for robot ownership to be located in having robots in their homes caring for children, elderly tax havens free from the human complexities of or the disabled; 60% of respondents thought they domicile and residency (Ahmed, 2017). should be banned from such care activities. They also Researchers have pointed out that tax policies currently reported concern around their application in education encourage automation – even in cases where human (34%) and health care (27%). workers are more efficient – because it enables firms to However, we should recognise that most research into avoid the ongoing costs of employment such as wages public attitudes on robots suffers from a methodological and medical insurance contributions (Abbott & bias in that they are often using hypothetical examples. Bogenschneider, 2018). As robots displace workers, tax For example, Eurobarometer also reported that only authorities are now considering their response. 12% of EU citizens have used or are using a robot and Recently the European Parliament voted down a their opinions may be exacerbated by perceived job proposed robot tax while South Korea has taken the displacement. Other research showed that 68% of opposite approach by limiting tax incentives for respondents have a positive or approving attitude when businesses applying automation. The implications for g they have been exposed to a robot as opposed to only government policy are clearly at a very early stage. 18% in hypothetical examples (Savela et al, 2017). Unsurprisingly, these results suggest that when people have very little actual experience with robots, they are more likely to have negative attitudes; this may change as more people become exposed to robots in their daily lives. This is not untypical for innovations generally. In short, as a technology is perceived to be more useful and is easier to use, intention to use as well as usage behaviour rises (Savela et al, 2017). f g Through a TNS Opinion and Social network poll conducted in 2012 Abbott and Bogenschneider’s 2017 paper Should Robots Pay through 26,571 face-to-face interviews Taxes? Tax Policy in the Age of Automation provides more detailed consideration of this issue. Taking control: robots and risk

4. Wider impacts 28 Box 3: Can robots replace the human Robot ethics touch? ElliQ can respond to voice, gaze and touch, and Moor (2009) proposed that a robot may be four kinds of suggest personalised activities at the right time to ethical agents: keep her companions sharp, active and engaged. 1. Ethical impact agents whose actions have But ElliQ is not human; she is an AI-driven social ethical consequences whether intended or not; companion robot designed to help the elderly who 2. Implicit ethical agents which have ethical live alone keep connected with their family, friends considerations built into (i.e. implicit in) their and the world around them (Elliq, 2019). Her design (e.g. safety and security considerations); creator, San Francisco-based Intuition Robotics, 3. Explicit ethical agents which can identify and describes ElliQ as ‘the sidekick for happier ageing’. process ethical information about a variety of situations and make sensitive determinations According to human-robot interaction researcher about what should be done; and Danielle Ishak, Intuition Robotics’ research has 4. Full ethical agents which make ethical found that ElliQ’s human beta testers tend to form judgements about a wide variety of situations. an attachment with the robot and are also more likely to open up to the robot, telling her when Robots and cobots can be generally classed as implicit ethical agents, in that they are specified and designed they’re depressed or lonely because they don’t feel they will be judged (Bloomberg, 2018). Despite this to operate safely in a given environment. It is however, however, these is only so much ElliQ can do to possible that some cobots may soon be intended as cheer those who are severely depressed through explicit ethical agents (though there is some argument loneliness and isolation; no source of AI robot will about the distinction between ‘ethical’ and ‘safe’, see for ever be able to replace human companionship or example Sharkey (2017)). care, Ishak adds. Winfield (2018)identified three risks associated with robots intended as explicit ethical agents: 1. Unscrupulous manufacturers might insert some unethical behaviours into the robots. 2. Robots that have user-adjustable ethics settings (e.g. choice between maximising length of life or quality of life) may have their settings somehow set outside an ‘ethical envelope’, 3. The rules may be vulnerable to malicious hacking. Winfield concluded that even with strong encryption, there is always a risk of hacking, so the responsibility for ethical behaviour must always lie with human beings. Cave et al (2019) explored the risks beyond safety considerations and surmised that unless they can be properly managed, it is unwise to develop explicit ethical machines. Taking control: robots and risk

4. Wider impacts 29 Robots as explicit ethical actors Jobs and skills There is no shortage of debate around the prospect of The impact of automation – specifically, robots – on robots becoming explicit ethical agents, with such a jobs is a complex debate with many different possibility often expressed in the application of the perspectives as well as statistics. In 2015, the Bank of ‘trolley problem’ with self-drive cars (Bonnefon et al, England predicted that 15 million jobs would be taken 2016). Put simply, the problem asks whether it is better over by increasingly sophisticated robots. According to to let an out-of-control trolley kill five people, or actively McKinsey researchers, advanced robotics and AI could switch the tracks so it will kill one person. In practice, potentially automate 50% of work activities while in a the answer is almost always to “slam on the brakes” highly-detailed analysis of tasks, Frey and Osborne rather than swerve into anything (Hern, 2016). (2017) considered 47% of total US employment to be at Johansson and Nilsson (2016) point out there is little risk. A 2017 study by Nesta (co-authored by Osborne) coverage of the trolley problem in driver instruction estimated that 20% of the workforce are in jobs with handbooks; instead, the focus is on avoiding accidents shrinking demand but 10% are in occupations where by constantly planning for surprising events. They suggest the design of robot AI should do likewise. demand will rise (Bakhshi, 2017). PwC’s UK Economic Outlook predicted that although 20% of jobs would Roboticist Rodney Brooks (2019) called the problem “a disappear due to AI, robotics and similar technologies made-up question that will have no practical impact on over the next 20 years, a correspondingly similar any automobile or person for the foreseeable future. amount of jobs would be created, with the health sector Just as these questions never come up for human seeing the greatest number of new additions (PwC, drivers they won’t come up for self-driving cars.” He 2018). compared the problem with Asimov’s laws and the Despite these different headline numbers, there is a Turing test as thought experiments which have little or strong consensus that robots are already displacing no impact on the way on the way robots are actually jobs and will continue to do so. Analysing the effect of designed. industrial robot use between 1990 and 2007 on the US The 2017 European Parliament report also included a labour market, Acemoglu and Restrepo (2017) suggestion to grant self-learning robots “electronic estimated that one robot per thousand workers reduces wages by between 0.25-0.5% and employment by personhood” status, enabling them to be insured and 360,000-670,000 jobs. More importantly, between 1.8 held liable if they caused damage to people or property. million and 3.5 million jobs will be lost should robot The report stated however, that “Asimov's Laws must stock quadruple by 2025. Although these figures are be regarded as being directed at the designers, based on some major assumptions and the effects may producers and operators of robots, including robots be slow, the researchers also pointed out that this may assigned with built-in autonomy and self-learning, since rapidly accelerate as robots deployed exceed the those laws cannot be converted into machine code” inflection point. (European Parliament, 2017). It is however not a straightforward transition from robots The European Commission strategy did not adopt the taking over tasks to job decreases. It is important to proposal of legal personhood for AI or robots, but it recognise that robots, particularly cobots, rarely replace commits to ensure an appropriate ethical and legal workers. They replace tasks, sometimes by removing framework since “artificial intelligence may raise new the need for employees but often augmenting workers ethical and legal questions, related to liability or through decision-making, or physical handling. Even in potentially biased decision-making” (European simple retail environments such fast-food chains, where Commission, 2018). the ordering transition and some aspects of the cooking can be automated, consumers may continue to expect and enjoy the human interaction and the implied safety that staff presence offers. Introducing robots in a public environment is much more than a technological challenge. There is a need to involve user experience and industrial design specialists in the process. For example, Café X in San Francisco developed a public-facing robot barista programmed to display some human gestures including tilting the cup to present the beverage and then placing the cup with a flourish, whilst completing the task in less than 15 seconds. The designers determined that Taking control: robots and risk

4. Wider impacts 30 choreographing the movement of the six-axis industrial Ethics in robot employment robotic arm was more endearing to human consumers than adding facial features or superfluous speech But when and how is it ethical to use a robot? Today, (Budds, 2018). “our robots do the dirty, dull, and dangerous tasks The shift towards automation may displace jobs, but it people might not want to do” (Yakowicz, 2016), as we will also create new ones, as predicted by PwC’s 2018 mentioned earlier in this report. Not everyone however UK Economic Outlook. Leading organisations are believes that the key motivation for creating robots is to increasingly recognising that technologies such as AI eliminate the need for people to perform unattractive and robotics are most effective when they complement, jobs. There is a contrasting perspective that robots not replace, humans, notes Deloitte. Respondents to should be deployed in occupations that require its Global Human Capital Trends survey foresee vigilance, responsibility and consistency, and that they tremendous future demand for human skills such as should or could occupy any traditional human h complex problem-solving, cognitive abilities, social occupation (Takayama et al, 2008). skills, and process skills, but companies are struggling An article in The Register (Out-Law.com, 2016) pointed to recruit and develop these human skills of the future out the following legal and ethical issues related to (Deloitte, 2018). robot employment: Reskilling the workforce around the technology should be an obvious priority. However, while there appears to − the difficulty in apportioning risk and liability for be government investment for doctoral studies in areas a failure when a robot has hardware, software, including AI, robotics automation and safety, process telecommunications that all contribute to its design and cybersecurity, there seems to be a huge use; gap in skills development between the levels of − data protection laws that cover information apprenticeships and undergraduate degrees, according captured about employees; to the experts we interviewed. Therefore, a machine − health and safety issues; operator who is made redundant would have little − displacement of employees by robots leading to recourse to obtain new skills for robotics and AI. unemployment and the social and economic problems arising. Robots have ethical effects, positive and negative, on the people they displace, on their co-workers, and on the people they serve. A 2017 report for the European Parliament noted particularly the effects of robot care in the context of an aging population, stating that despite its many benefits to older people and people with disabilities, “human contact is one of the fundamental aspects of human care” and “that replacing the human factor with robots could dehumanise caring practices” (European Parliament, 2017). h Takayama et al (2008) explored the opinions of lay people with an for occupations that require artistry, evaluation, judgment and online questionnaire and concluded that in contrast to the simplistic diplomacy. notion that robots should do dangerous, dirty, and repetitivel jobs, public opinion favours robots for jobs that require memorisation, keen perceptual skills, and service-orientation, while humans are preferred Taking control: robots and risk

4. Wider impacts 31 Robots and other digital An important concept in integrating these ideas in a technologies virtual environment is that of the “digital twin” (Tao et al, 2018), an idea first introduced by Grieves (2014) when discussing product lifecycle management in 2003.i As cyber-physical systems, robots/cobots are complex A digital twin consists of a physical product, a virtual devices. They require inter-disciplinary collaborations in product and the data that connects the physical and their design, build and execution at many levels. In digital. In the context of Industrie 4.0, a virtual addition to the core skills of sensor development, representation of the factory is created to simulate the programming and AI, coupled with mechanical and physical site. The two are kept in tandem through the electrical engineering, designers also need to consider sharing of captured data from sensors and other the wider ecosystem challenges the robotic entity would mechanisms around the physical factory. A good face, such as integration with other digital technologies. example is ABB’s RobotStudio developed based on its Many developments are currently taking place in the VirtualController, in which an exact copy of the factory Industry Internet of Things (IoT), termed Industrie 4.0 in software enables realistic simulations of the production Europe and the Smart Factory in the USA. These are robots to be tested and evaluated before they are transferred to the shop floor. discussed in further detail in the Lloyd’s Emerging Risk 2018 report Networked world: Risks and opportunities Having cobots in increasingly tightly-coupled industrial in the Internet of Things, but it suffices to say that the systems however, brings its own set of issues. future of manufacturing lies in these cyber-physical Increased data collection in such cyber-physical systems that combine human operators and machines systems, particularly through sensors on cobots, gives such as robots/cobots equipped with sensors, rise to the need for adequate storage space to hold the microprocessors and radio-frequency identification. vast amounts of data being generated. In a potential Other countries such as Japan and Korea have also future where such data is used to optimise flows inside adopted similar smart manufacturing programmes. the factory and supply chains both upstream and When placed within highly-automated plants, these downstream, manufacturers need to mitigate risks of robots/cobots can collect vast amounts of measurement (systemic) failure resulting in large losses in the event of data around processing times, queuing and set up business interruption caused by a cobot breakdown. times and error rates. In turn, data not only helps to Many firms do so by building in redundancy as well as optimise process flows, leading to better asset practicing production strategies such as contingency utilisation and improved quality; it can also inform the and forward planning and stockpiling, as well as design of the next generation of robots/cobots. increasing their use of ‘plug and play’ robots to Crucially, it enables the robot/cobot manufacturer to safeguard against breakdowns. develop alternative business models including through- Also, strong data security is paramount to safeguard life costing and outcome-based contracting. against halts in production streams from potential Recent developments have extended the scope of hacking. Many manufacturers build their own closed cyber-physical downstream to the retailer. The next and networks for their cyber-physical systems, as well as most obvious extension is into the home (Parry et al, practice segregation in restricting the data flow to and 2016), with developments such as the Hub of All Things from the robots. Interviewee #10 stated that many (HAT) offering the potential to optimise supply chains manufacturers have internal networks that are strongly from production to consumption and use. protected from integration to the wider network and therefore their access points are less easily tampered Research at the moment however, tends to focus on with. Interviewee #7 gave examples of automated specific topics and disciplines including sensors; warehouses where there is a firewall between front- communications such as 5G; networking; production office order entry and back-office picking so that only engineering; computer science; data architectures and order information is provided to the robots. ontologies. i Glaessegen and Stargel (2012) define a digital twin as an integrated product and uses the best available physical models, sensor updates, multi-physics, multi-scale, probabilistic simulation of a complex etc., to mirror the life of its corresponding twin. Taking control: robots and risk

4. Wider impacts 32 Artificial intelligence and robotics Potential cyber-risks, including hacking and data theft Artificial Intelligence (AI) is a core technology in the from systems as well as when devices are future development of robotics. Much of research in AI, communicating give rise to concerns about malicious such as deep learning for vision, speech recognition in tampering. There are concerns that processes could be interface design and the transition from supervised to changed, or a systematic defect could be introduced, unsupervised learning, are central to the future while factories and workspaces could be held at application of cobots. ransom; for instance with threats to shut down robots on a farm during peak picking seasons and hence Several leading robot and cobot experts we interviewed allowing crops to rot. There is also the potential for revealed important recent developments in AI research. deliberate unethical training of a cobot, although it Interviewee #9 identified significant developments would be the responsibility of the robot installer/systems taking place in sensing for safety, for example, where a integrator to minimise the likelihood of this occurring. knife is used on the end of a robot arm. Interviewee #2 The experts we spoke to say such incidents of using pointed out how important vision systems and predictive robots to cause damage are rare, but that the greater analysis are for ensuring worker safety where heavy risk lies in the theft of the IP and ideas by disgruntled loads are being moved around a farm or retail site, employees, particularly when they have a good while Interviewee #5 noted the challenges in developing understanding of how the systems function and how AI for interpreting images on a conveyor. different components such as the sensors work. Interviewee #8 emphasised the importance of AI for Future developments in robotics are closely linked to developing improved interface design, crucial for robotic those in AI. Indeed, all our interviewees highlighted that use in all locations but particularly in the home. the future of robotics is really constrained by the future Interviewee #8, who conducts research on of AI. The developments in AI and their potential conversational interfaces, highlighted the challenges impacts on the insurance industry are discussed in around designing an interface to minimise safety further detail in Lloyd’s report Taking control: artificial concerns where speech may be misheard or intelligence and insurance. misunderstood and could lead to potentially life- threatening situations. A second problem is making the Box 4: Developing conversational AI interface engaging and entertaining so that people will The risk with conversational AI is the potential for continue to interact with the device. Interviewee #8 systems to misunderstand the user (either through emphasised that “we are nowhere near a humanoid poor speech recognition, environmental noise, or robot” and “it is still science fiction”. Finally, although we language understanding), thus likely leading to an might be able to instruct a robot to stack a dishwasher unintended action (wrong health information or do the ironing, it would do so very slowly and only for delivered, unintended purchase, driver distraction). that specific home. It would also very unlikely be able to These factors can be somewhat mitigated by good perform any other domestic chores. conversational AI design, for example, explicitly Interviewee #10 discussed future cobots in confirming user requests at crucial moments. This unsupervised learning within a factory or home, is currently the subject of much ongoing research. highlighting the issue as to who owns any new In 2017, Amazon launched the annual Alexa Prize, Intellectual Property (IP) that is developed: the firm, the inviting teams of university students to develop a owner, the systems integrator or even the robot itself? chatbot capable of conversing with humans on a As far-fetched as it may sound, it does raise the wider variety of subjects. Building on existing Amazon issue of AI-enabled cobots becoming independent Alexa software, the teams would create a socialbot, actors. Interviewee #10 also foresees security and advancing knowledge acquisition, natural language privacy concerns for firms whose IP lies in understanding, natural language generation, manufacturing. Having a device that can understand context modelling, common sense reasoning and how their entire factory/supply chain works and dialogue planning. analysing that data remotely might limit its adoption. The tech giant believes the way humans interact with machines is at an inflection point, and that conversational AI is at the centre of the transformation (Amazon, 2018). Hence it is investing substantially in the prize which it hopes will help advance the field. Taking control: robots and risk

A study of four sectors – Industrial – Agriculture and food – Healthcare – Retail Emerging Risk Report 2019 Technology

5. A study of four sectors 34 5. A study of four sectors To develop a more compelling picture of the robotics One example is the increased adoption of cobots for landscape we have selected four sectors in which robot transportation purposes in collaborative environments. and cobot solutions have been applied at varying Just as the first industrial robots were installed away scales. We will consider each of the sectors across the from humans in factories and laboratories, the first medium term (up to three years). vehicular robots were found in warehouses where they could be used to manipulate larger payloads or Industrial dexterously configure smaller quantities of merchandise. We are now seeing more transport Robots are commonly used in industrial environments cobots operating in closer proximity to humans with due to their ability to perform operations and processes mobile robots used in a range of applications, extending quickly, repeatedly and accurately. Traditional industrial the concept of Automatic Guided Vehicles (AGVs) to robots are used for fabrication processing, foundries, Autonomous Mobile Robots (AMR) (Schneier & welding, painting, coating and sealing, flexible fixturing Bostelman, 2015). The difference now is an important and workpiece handling, as well as material handling one. Whereas the standard AGV, which has been and warehousing, assembly of mechanical and around for almost 50 years, follows fixed routes, the electronics, quality assurance, maintenance and repair robotic AMR uses sensors and on-board computers to and re-manufacturing (Nof, 1999). In addition to understand its operating environment and dynamically traditional robots operating in ‘caged’ or highly- navigate using a map. controlled environments, there is increasing deployment Cobots in collaborative environments must be designed of smaller, lighter cobots within both SMEs and large to minimise safety concerns. Important design factors manufacturers, in environments where they are close to (Kildal et al 2018) include proximity detection systems or interacting directly with humans. As the Loup that slow down or stop movement, and collision Ventures report indicates, the market for these robots detection systems that operate through force- or torque- will continue to grow at a rapid rate due to increased sensing in the robot’s joints, enabling subsequent labour costs and the need to avoid dirty, dangerous and reaction to that collision. Another safety factor difficult jobs (Murphy, 2017). (according to Interviewee #9) is hand-guiding where the Robots, cobots and mobile devices are increasingly robot is under the direct control of an operator. This being developed for industrial tasks outside the requires sensor and actuator systems that enable the traditional factory environment. Interviewee # 6 pointed robot to be physically manipulated by the user without resistance. Also important are a projection-based space to over £90m of funding by the UK’s EPSRC into four major centres for robotics research outside of the monitoring system, safety zones around the cobot’s factory. immediate area of influence and variable stiffness in actuators (Kildal et al, 2018). Taking control: robots and risk

5. A study of four sectors 35 Box 5: Cobots in industrial environments Interviewee #9 raised an important issue; when a cobot, which may otherwise be deemed unsafe, is applied to a Some examples of cobot use in industry include: dangerous process in a way that could reduce the overall risk. For example, a worker tending a hazardous - ABB’s YuMi, a collaborative dual-arm small-parts machine could trap and lose a finger. The process is assembly robot that works alongside humans on made safer by deploying a cobot to tend the machine, ABB’s socket lid assembly line handling springs, thereby saving the worker’s finger. However, the robot child locks and child lock covers. ABB has also could collide with the worker and cause a lesser injury. announced that they will build an advanced In this instance the overall risk is reduced, but the robotics factory in Shanghai, using YuMi and company may delay or add further safeguarding and other automated machines in safe proximity to reduce productivity to mitigate the new risk the cobot humans to produce other robots (Moon, 2018). brings. According to Interviewee #9, health and safety - Adidas has opened two “Speedfactories” in staff have suggested that in such cases the cobot Bavaria and in Atlanta, where both traditional should be deployed as overall safety is improved. robots and cobots are used. The time to However, their industrial partners have been cautious, manufacture a shoe is reduced from 60 days to citing concerns over litigation for deploying an unsafe six days (Wiener, 2017). robot. - In one of the largest international implementations, Foxconn in 2016 reportedly By becoming a key part of a factory’s cyber-physical installed 40,000 robots replacing 60,000 workers environment, cobots will lead to less waste throughout (Fingas, 2016) and in February 2018 it was the supply chain and increased production capabilities. planning to invest US$4 billion into robotics and As manufacturing at scale grows, unit costs will fall, and developing automation (Francis, 2018b). alternative leasing models will be deployed. Further examples of cobots working alongside humans exist in welding (Universal Robots, 2017) and coating (Universal Robots, 2019). In fact, almost all of the application areas for traditional robots outlined by Nof 20 years ago (Nof, 1999). Taking control: robots and risk

5. A study of four sectors 36 Focus on the future Insight: Implications for Cobots in this sector are currently at the early insurance adopter stage of the life cycle, with a few major 1. Increasing adoption of cobots in manufacturers creating scale economies. As the unit environments that work closely with costs of cobots will continue to fall, the economic humans provides new markets for case for adoption will become much easier. insurance products. These markets are Furthermore, as RaaS (see Box 2) becomes more likely to be international, but as the widely available the cost of ownership will drop pressure for onshoring and responsive substantially. Demand is being led by industrial manufacturing grows, opportunities in markets such as China, with the current drive for western economies are also expected to onshoring helping to increase demand in the USA. grow rapidly. The development of RaaS business models will also expand The most pressing constraint is health and safety opportunities with SMEs, which may considerations. Once processes are developed to previously have been priced out of the address those concerns, cobot adoption in industrial robot market. applications will grow more rapidly. Interviewee #9 2. Faulty cobots have the potential to cause reported that from recent industry workshops, safety damage to property (e.g. a moving robot always comes out as the number one barrier to might drive into a supplier’s vehicle) and to adoption, followed by issues of trust and acceptance other workers. (which are closely related to safety). This requires research into safety technologies, but also 3. The risk profile of employer’s liability and engagement and upskilling of stakeholders. Market public liability could change as liability could research (The Manufacturer, 2017) highlighted lack be pushed back onto the robot product of knowledge as the main barrier to uptake of manufacturer/designer. automation, followed by the bespoke nature of 4. There is potential for large-scale losses products and concerns about the length of time to get resulting from business interruption in a return on investment. Cobots promise to provide supply chains using cobots. Removing solutions to the latter, whilst the former requires cobots from a production line might incur training and education, and closer collaboration with extensive costs and be very time- the R&D community. This also relates to the issue consuming particularly if it involves product about who has responsibility – it is not just safety re-design. technology that is required, but knowledge and training of users. 5. Increasing adoption of robots in dangerous In the next 3 to 5 years, as unsupervised learning in environments (e.g., nuclear AI develops and interface design improves, we will decommissioning and space) would reduce see more industrial processes augmented by cobots. risks as process are automated reducing Cobots will enable the benefits of the human (e.g. the number of employee injury claims. cognitive and perceptual abilities) to be combined 6. Data from cobots and Industrie 4.0 enable with those of the robot (e.g. speed, precision, a much greater understanding of risk and repeatability, lifting capacity). offer opportunities for improved models. There are opportunities to collaborate with Finally, the combination of cobots with Additive Layer clients to share data. Manufacturing (ALM) and Industrie 4.0 will enable far 7. There is an opportunity for the insurance more efficient factory and supply chain optimisation industry to work directly with manufacturers with the potential for more localised, smaller and on identifying risks around robot responsive production facilities that reduce the need deployment. This may help to address for large-scale transportation. some of the concerns of Health and Safety officers and drive up adoption. Taking control: robots and risk

5. A study of four sectors 37 Agriculture Cobots are also being used in tasks related to preparing the ground for crops. Thorvald is a commodity autonomous mobile robot used by a University of The agricultural sector has proved a challenging, but Lincoln research team as a mobile platform on which to potentially rewarding opportunity for robot developers. develop a prototype soil compaction mapping system. In the first era of robot development, the factory By producing more and better data around the soil, provided a highly-structured and relatively safe such a system can offer more focused precision farming environment where processes are relatively methods (Fentanes et al, 2018). constrained, and humans can be excluded from the Robotics in agriculture could also potentially reduce area of operations. Farms represent a far less environmental impact. Robotic devices executing structured and homogenous environment, and there is precision tasks and operating either alone or in clusters clearly more human interaction albeit with the potential can be less damaging than combine harvesters with for control over access. The potential benefits of robot their significant weight and load-bearing footprint. An use in this sector are significant; it enables longer excellent summary of the application of robotics in working hours, a stronger appetite for repetitive tasks agriculture is provided by Duckett et al (2018). and greater adaptability to unpleasant working conditions. Against these opportunities are various Beyond pure agricultural applications, robots are also challenges including the diversity of farming playing a far-reaching role along the food chain. Having environments, a lack of agile equipment, high set-up robots that can assist with bagging, packing, processing costs, and the need to learn and adapt to new and shifting makes shorter supply chains possible. The processes. Demand for agricultural robots is also being potential for integrating robots within a blockchain- driven by shortage of farm workers particularly in the enabled supply chain is increasingly being discussed as USA, Japan and a post-Brexit UK. a panacea for food traceability (Pearson et al, 2019). Cobots have already been put to a range of uses in Whilst such solutions do not prevent errors in data agricultural settings, but these tend to be highly- entry, they act as a deterrent to more traditional fraud controlled environments. For example, in 2017 the IFR mechanisms. reported that farmers around the world purchased 5,700 The adoption of cobots in the agriculture and food milking robots, which are rapidly becoming the de facto sector is significantly behind that of manufacturing technique for milking cows. This is the most successful (albeit automated milking has been around for quite robot system deployed across global farming and points some time). The sector is fraught with challenges such to the opportunity for high adoption rates once the as terrain and complexities of identifying produce technology works. Interviewee #2 stated that the next ripeness. In short, much of cobot application in most significant trend in the sector will be crop- agriculture is still at the early phases of technology harvesting robots which will need to be autonomous, readiness levels and yet to be trialled at scale. thus adding complexity and worker risk. In the US there are already fully autonomous tractors. Their application More widely, the development and deployment of in the UK might however be slower: as Interviewee #2 robotics in agriculture has some overlaps with the said, a “robot going wrong in the middle of a prairie is construction sector (e.g. working in difficult terrains, one thing, a robot going wrong in densely-populated UK complex manoeuvring and the dexterity in movement is another.” required). Interviewee #7 noted significant potential in Labour-intensive tasks performed by cobots, like the the construction sector, particularly with bricklaying picking of fruit, are still under development. Interviewee robots. But although our interviewees commented that #1 pointed out that the task of identifying and picking cobot use in construction is still at very early stages, oranges from a tree is already very challenging, and monitoring and learning of opportunities and risks in each tree is very different; a task with which a human agri-robots is likely to lead to similar applications in the picker is much more adept. Interviewee #2 noted that sector. cobots are already being used in simple tasks such as moving freshly-harvested crops, which are far more realistic short-term goals. Interviewee #2 believed that the more sophisticated use of cobots in this sector is at least five years away. Taking control: robots and risk

5. A study of four sectors 38 Focus on the future Insight: Implications for In farming and agriculture, robots are at a very early insurance stage of adoption. Many current developments are 1. With the advent of precision farming, sensor- still in the laboratory or in early stage testing with an driven data from the fields coupled with increasing number of university spin-outs. Similar to external climate and weather data will enable manufacturing, current adoption is being restricted farmers to develop algorithms that exploit by health and safety concerns and limited to highly- their land, recognising for example, local constrained environments. However, Interviewee #2 differences in the soil. This in turn might identified increasing pressure to speed up enable more bespoke and accurate crop developments and applications because of labour insurance. shortages across the sector. As this shortage can 2. Interviewee #2 pointed to a future for potentially become more severe, the pressure for agriculture where farms and fields will have firms to adopt automation and robotics will increase. robots with drones together with tractors, but As robotic devices develop they could possibly with high levels of intelligence. This scenario facilitate precision farming. As they become part of introduces new risks such as potential losses an integrated system, they may be configured to from hacking or design faults and increased handle a whole range of more specialist analytic and losses from theft of highly valuable items. technical tasks, from soil analysis to precision seed 3. Light co-bots weighing below 100kg are planting. Further benefits from robot labour include inherently more vulnerable to natural the ability to increase productivity, for example by catastrophic events. There is the potential for harvesting throughout the night when the terrain and aggregation risks with a field manned by 20 produce are much cooler and therefore less prone to robots which are more vulnerable to decay than in the daytime heat. Agri-robots are also windstorm damage compared to 20-tonne being used in hazardous situations such as the tractors. The ensuing property damage and application of UV-C to crops in place of pesticides, a business interruption losses could be large. task too dangerous for humans. Their adoption can 4. There is potential for large-scale losses only grow as demand for food without pesticide use resulting from business interruption in supply accelerates. chains using cobots. Removing cobots from In the future, farmers might start using low-cost a production line might incur extensive costs standardised robots to convert their produce into and be very time-consuming particularly if it supermarket-ready packed and sorted goods. involves product re-design. Contingency planning might become more difficult; while the loss of one or several workers in a production line can be replaced quite easily, it is harder to do so on a cobot line, as having additional but underutilised labour capacity is prohibitively expensive. From a workforce perspective however, agricultural cobots will still require a degree of specialist human support, at least in the medium term. Devices will need to be supported and maintained. New tasks will need to be programmed and taught, or at least heavily supervised. Taking control: robots and risk

5. A study of four sectors 39 Healthcare The US is believed to be leading the way in the adoption of robots in surgical and medical environments while Japan is at the forefront with developing robots for The healthcare and wellbeing sector provides a huge the home. Such cobot developments are significant, as range of potential opportunities for robotics. As robots home care is a sector predicted to face enormous become more adaptable, reliable and swifter in their labour shortages in many western economies. response times, they are better able to operate near The use of robots in hospitals is not restricted to humans. The sector offers a variety of scenarios for surgery. In the field of hospital hygiene, Westchester robot-delivered services with humans in varying Medical Centre in Valhalla, NY, trialled a Xenex UV degrees of mobility; early solutions have ranged from disinfecting robot to clean its intensive care unit, surgical procedures to rehabilitation tasks. resulting in a 70% reduction in hospital-acquired C diff Surgical robots infections (Nagaraja et al, 2015). Robot logistics In 2017, the global surgical robotics market was valued systems such as Aethon’s TUG can carry equipment or at US$56,300 million; it is expected to reach US$99,000 pharmaceuticals up to 400kg around hospitals, freeing million by 2024 at a CAGR of 8.5% during the forecast up porters and relieving nurses of carrying heavy loads. period (Allied Market Research, 2017). The use of Box 6: The breathing stone robotics in surgery has been found to shorten lengths of hospital stay, decrease complication rates and allow A somewhat extreme example of a cobot is the surgeons to perform finer tasks. The costs are longer Breathing Stone developed by start-up Biobeats. intraoperative times, equipment costs and the training The stone is a manufactured device that identifies costs associated with using the equipment (Hussain et your stress levels from variations in your heart rate. al, 2014). Current evidence points to a strong cost- It uses this information to prompt breathing benefit case in the fields of urology and gynaecology exercises to music and therefore lower your stress. (Hussain et al, 2014), while extensive, large-scale randomised clinical trials currently underway should The Breathing Stone has been adopted by Chelsea identify those procedures most appropriate for robotic and Westminster Hospital, which gives around 200 surgery (Lai, 2017). patients a month access to the device prior to At the UK’s University College Hospital, an immersive surgery. Biobeats claims that physical/mental stress 3-D monitoring system is used to provide the surgeon is reduced by around 23% after the device is used with a close-up view of the operation, while a surgical in pre-surgery waiting rooms. robot with four arms can be remotely directed with This device meets the definition of a cobot in that it considerable dexterity, resulting in reduced risk of senses (heart rate), thinks (uses AI to analyse complications as well as training benefits for others patterns of heart rate) and acts (guides breathing). observing the recorded procedure (Adams, 2018). More impressively, in 2017 it was reported that a robot dentist Although it doesn’t meet the standard image of a fixed industrial cobot, it provides an example of how in China was able to carry out the world’s first technology is adapting to meet patient needs and successful autonomous implant surgery by fitting two providing innovative applications of AI. new teeth into a woman’s mouth without any human intervention (Yan, 2017). As surgical systems continue to evolve with new technologies, uniform standards for surgical team training, advanced human machine interfaces, improved accident investigation and reporting mechanisms, and safety-based design techniques should be developed to reduce incident rates in the future. While robotic surgical systems have been successfully adopted in many different specialties, a study by Alemzadeh et al (2013) has found that while the overall numbers of injury and death events have stayed relatively constant over the years as the number of procedures has increased, device and instrument malfunctions have affected thousands of patients and surgical teams by causing complications and prolonged procedure times. Taking control: robots and risk

5. A study of four sectors 40 Robots in physical therapy Robots in the home Healthcare robotics is also used in physical With the home care sector predicted to face enormous rehabilitation therapy and support, where specialised labour shortages in many economies, robots are systems can repair limbs and other motor functions currently being developed and tested to interact with the through targeted physical support. Robotic therapy has elderly and dependent to help them with their day-to- been as helpful in physiotherapy (OTPotential, 2018), day needs (Priyandoko et al., 2017). while little difference in cost has been found between Japan is at the forefront with developing robots for the robot-assisted therapy, intensive comparison therapy home, with offerings such as SoftBank Robotics’ and the usual care costs (Wagner et al, 2011).j Pepper, designed specifically to exhibit empathy in a Much research has been conducted on specific joints whole range of human modes both in terms of for robotic solutions, but they do not always take human understanding and acting. Japanese government- models as their precedent. For instance, octopus funded research institute RIKEN has developed the tentacles can provide powerful and adaptable, in the Robear, a robot that helps people in their homes by sense of form-fitting, mechanisms (Wei et al, 2018). lifting them from their beds and into a wheelchair. In China, Shanghai Fourier Intelligence Co. has Allied Market Research (2017)considers robotic produced therapeutic robots for upper and lower limb services to be the fastest-growing market segment as rehabilitation. Each robot can treat about up to 20 the number of people with chronic conditions in the people per day. “Since there are 30,000-plus rehab global population rises. facilities in the US, over 15,000 in Europe and more Within the general area of assistance, different types of than 2,000 in Australia, you can imagine the size of the robots can offer a variety of ways to help (Hosseini & market in China,” Shanghai Fourier CEO and partner Goher, 2017). Relatively simple, short and restricted Alex Gu told Shanghai Daily (Shanghai Fourier, 2018). tasks can be carried out by professionally-programmed Social care robots such as Roomba or Navi Bot cleaner robots or feeding-aid devices like Bestic or My Spoon that can Conversational AI has immediate application where serve users in eating and nursing. They can also record robots are already fulfilling informational rather than crucial behaviours that keep a person healthy and safe physical needs. For example, in healthcare there is and immediately send an alert to health services if an research and development of text and speech anomaly is recorded. conversational bots for mental health therapy, and for Robots can also be designed to offer companionship health information provision. There is also research on and intervene when appropriate. This type of cobot is physical robots in hospitals and care homes which can generally programmed to be socially cooperative and guide patients to locations or encourage them to do come in many different forms, usually robotic rehabilitation exercises. domesticated animals such as Sony’s AIBO robot dog, One of the additional benefits that increased use of the Pleo dinosaur, and the Paro baby seal. Already robotics provides comes from the application of AI; the used in healthcare around the world, Paro is being huge amounts of data collected in robotic procedures considered as a form of emotional support for may be used to train algorithms to begin the journey astronauts in space. The robotic seal is classified as a towards fully independent action and advice. Class II therapeutic medical device by the US FDA and is utilised by the UK’s NHS as a form of non- pharmacological therapy for dementia (Chaturvedi, 2018). j In Wagner et al (2011)’s study, 127 participants were randomised to $19,746 for intensive comparison therapy, and $19,098 for usual usual care plus robot therapy, usual care plus intensive comparison care. therapy, or usual care alone. At 36 weeks postrandomisation, the total costs were comparable for the 3 groups; $17,831 for robot therapy, Taking control: robots and risk

5. A study of four sectors 41 Focus on the future Insight: Implications for Although fully-autonomous robot surgeons for insurance complex procedures may be 10 years away, the 1. New business opportunities in medical and shorter term will see an increase in remote-control healthcare robotics will continue to grow surgical procedures using multiple redundant internet relatively quickly on the back of significant channels to minimise the risk of control dropout research and VC funding. As consumers during the operation. The benefits for battlefield become used to healthcare provision through surgery and more economical procedures in cobots, there is a positive network effect developing countries and remote regions are leading to scale economies and increasing considerable. One of the major drivers in healthcare use. This opens up opportunities for insurers is the level of investment from public and private to expand offers for both medical malpractice research funders and VCs. Consequently, we can and product liability coverage. expect to see a significant growth in medical devices 2. Similarly, opportunities for insurers are in the next five years. offered by emerging use of robotics in healthcare markets in countries experiencing In healthcare robotics, most cobots are fairly basic, labour shortages and ageing populations requiring significant development in sensing, learning such as Japan, but also increasingly across and the development of improved methods of western economies. Development of interaction. Many countries tend to delay the entry of assistive robots can be used across multiple technological developments to market in the customers to provide specialist services such healthcare sector to ensure high quality, safety and as physiotherapy. cost effectiveness (Petkova, 2010). However, in the 3. Manufacturers and clinicians both owe a duty next three to five years we can expect to see of care to end consumers and patients respectively. Robotics in the healthcare extensive developments in the sector as knowledge sector can complicate the assignment of gained elsewhere is applied in healthcare. For liability. Clinicians using robots without the instance, developments in other areas such as necessary training, or incorrectly operating material sciences “will allow lighter, more may amount to medical malpractice. A robot customizable structures with more tightly integrated may be defective and covered under product actuation and sensing” (Gassert & Dietz, 2018) as liability policies. The difficulty lies when lighter materials and improved sensors are where robots are not fully autonomous and incorporated. This means we should see more there is not a consensus whether the specific and personalised systems better able to clinician is negligent, the robot is defective or both. meet patient’s needs, offer more stability and robustness and at lower prices. Taking control: robots and risk

5. A study of four sectors 42 Retail However, some customers will still prefer personalised attention, and abilities such as accurate problem diagnosis, emotion identification and picking up social The overall retail market for cobots is estimated to be cues are notoriously very difficult to automate. worth more than US$11 billion across a range of Interviewee #4 told us that a lot of work is currently applications such as shopping malls, receptionists and taking place in identifying emotion. At Carnegie Mellon guides in hotels, airports, museums and amusement University’s Robotics Institute, its director Martial Hebert parks. Cobots also have significant potential to provide has said that the challenges are not so much in the guidance in banks and to carry out simple repetitive robotics but in “Understanding people, predicting cashier functions. people, and understanding their intentions. Everything Cobots mainly used in warehouses are Autonomous from understanding pedestrians for self-driving cars, to Mobile Robots (AMRs). For example, in 2017 Amazon understanding co-workers in collaborative robot had over 100,000 robots in use worldwide, with plans manufacturing, any application that involves interaction for many more (Heater, 2017) whilst Ocado has 1,100 with people at any level.” (Anandan, 2018). in one single 18-acre facility in South East UK (Kleinman, 2018). Robots are also increasingly being used not just to move products across a space but to deliver directly to the customer, such as Best Buy’s Chloe which picks up the CDs and DVDs ordered through touchscreens and delivers to the customer. Best Buy’s ‘Tally’ robot travels through warehouse aisles and tally up stock and can work out if items had been wrongly priced or put in the wrong place. The Starship delivery robot can travel outside the warehouse on pavements at around 4 miles an hour and carry a load of around 9kg. However, Interviewee#1 pointed out that these might need to be limited to very flat landscapes within low-crime zones. In South Korea, E-Mart is using LG’s shopping carts that can follow the customer around and navigate shopping aisles (Synek, 2018). One of the future benefits of the electronic cart is the ability to scan items as they are put into the cart and therefore reducing the need for check-out assistants. Giant Food Stores are using a cobot (Bowles, 2019) called Marty that alerts customers around it when it sees something it considers a hazard. It then sends a message to humans (located, in this case, in the Philippines) who are monitoring the stores on TV screens to determine if the potential hazard is something about which they need to alert the store manager. Investment in such a cobot is economically viable because US slip-and-fall accidents can be expensive if the retailer is found to be at fault. Cobots also have the potential to transform the customer experience. A recent survey (Ismail, 2017) found that nearly half of British consumers have experienced bias because of their individual characteristics, beliefs and/or appearance. Only 8% of respondents felt that chatbots will be biased; this is despite concerns that human bias could be transferred onto modern chatbots. Taking control: robots and risk

5. A study of four sectors 43 Focus on the future Insight: Implications for Cobots are very likely to have increasing and swift insurance adoption across the retail sector as the economics 1. As a consequence of increased adoption of job replacement, particularly in back-office and high-value machines, the property risk operations, is fairly straightforward. In warehouses, profile will increase. AGVs are being replaced by more sophisticated 2. Interactions with cobots create potential devices that can adapt to hazards, other devices hazards for both employees and customers and minimise risks to human operators. In the short (e.g. being struck by an object carried by a term, the major constraint is likely to be developing cobot or by the cobot itself could cause policies and procedures to ensure health and safety bodily injury) and could result in expensive concerns are met. litigations. 3. The risk profile of employer’s liability and The opportunities are also clear in more face-to- public liability could change as liability could face environments. Cobots offer the potential for be pushed back onto the robot product 24-hour opening both in retail and also in manufacturer/designer. warehouses where they can run ‘lights out’ 4. There will be opportunities for the provision operations. There are however, significant technical of ancillary services around warehouse and challenges in robot-human interaction, as shop floors layout and design based on risk Interviewee #8 pointed out that robots misidentify reduction. things and can introduce hazards. The challenge is to make the interaction accurate, but also engaging and informative so that the customer will enjoy a safe experience. Taking control: robots and risk

Conclusions Emerging Risk Report 2019 Technology

6. Conclusions 45 6. Conclusions The market for cobots is a fast-growing sector in the The report shows how the adoption of cobots is global economy and presents potential for specialty currently constrained by safety, security, liability and insurers. The large traditional robot – challenging to physical risks. By helping insureds identifyi the risks and design, make and maintain – is being replaced with by setting out ways to mitigate them, insurance could cobots that have voice recognition, are linked to help increase and speed up cobot adoption. It suggests industrial IoT, can be set up in half a day, use open- that robotics designers and manufacturers, systems sourced code and can optimise around libraries of integrators and users should work with the insurance algorithms with learning capabilities and in many cases, industry to mitigate and transfer the risks associated are more mobile and agile. The adoption of cobots is with robotics more fully. As cobots are used in more and opening up a new world of commercial possibility for more sectors, insurers should take a proactive role in developers, suppliers, users and insurers, at the same talking to insureds and potential clients to review and time as creating new risks, some of which may be assess all risks. By leading in this space, the insurance unknown today. Other risks will reduce and change. sector will acquire the necessary knowledge to provide This will necessitate innovation in both existing and new insureds with guidance on cobots best practices, lines of business. thereby shaping the ecosystem in which they operate Safety remains the main concern and number one and the product offering. barrier to adoption, followed by issues of trust and To conclude, predicting how rapidly robot and AI acceptance (which are closely related to safety). These technology will be adopted and implemented will occur will require research into safety technologies, but also is difficult, but it is highly likely cobots will play a engagement and upskilling of stakeholders as lack of significant role in transforming many industries, sectors knowledge is the main barrier to uptake of automation, and regions across the world in the next 5 years and followed by the bespoke nature of products and beyond. Insurers can facilitate this growth by working concerns about the length of time to get a return on with sectors to develop the products and services they investment. This also relates to the issue about who need. has responsibility – it is not just safety technology that is required, but knowledge and training of users. Taking control: robots and risk

References 46 References Abbott, R. & Bogenschneider, B. (2017). Should Robots Pay Taxes? Tax Policy in the Age of Automation. Harvard Law & Policy Review, 12, 2018. http://dx.doi.org/10.2139/ssrn.2932483 Abbott, R. & Bogenschneider, B. (2018). Why We Should Start Taxing the Robots That Are Taking Human Jobs. The Conversation. Retrieved 12 Feb 2019 from http://theconversation.com/why-we-should-start-taxing-the-robots-that- are-taking-human-jobs-91295 Abd Majid, M., & Fudzin, A. (2017). Study on Robots Failures in Automotive Painting Line. ARPN Journal of Engineering and Applied Sciences. 12(1), January, 62-67. http://www.arpnjournals.org/jeas/research_papers/rp_2017/jeas_0117_5581.pdf Acemoglu, D., & Restrepo, P. (2017). Robots and jobs: Evidence from US labor markets. NBER Working Paper No. 23285. March. https://www.nber.org/papers/w23285. Adams T., (2018) The Robot Will See You Now: Could Computers Take Over Medicine Entirely? The Observer. Retrieved 19 Feb 2019 from https://www.theguardian.com/technology/2018/jul/29/the-robot-will-see-you-now- could-computers-take-over-medicine-entirely Adler, P. S. (1993). The Learning Bureaucracy: New United Motors Manufacturing, Inc. In Staw, B., Cummings, L. (Eds.), Research in Organizational Behavior, vol.15 JAI Press (pp. 111-194) Retrieved 12 Feb 2019 from: http://www-bcf.usc.edu/~padler/research/NUMMI%20ROB.pdf Ahmed, S. (2017). Cryptocurrency & Robots : How To Tax and Pay Tax on Them, 1–68. Alemzadeh, H., Raman, J., Leveson, N., Kalbarczyk, Z., & Iyer, R. K. (2016). Adverse Events in Robotic Surgery: A Retrospective Study of 14 Years of FDA Data. PloS one, 11(4), e0151470. doi: 10.1371/journal.pone.0151470 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4838256/ Allied Market Research (2017). Surgical Robotics Market by Component (Systems, Accessories and Services), by Surgery Type (Gynecology Surgery, Urology Surgery, Neurosurgery, Orthopedic Surgery, General Surgery and Other Surgeries): Global Opportunity Analysis and Industry Forecast, 2017 – 2024. Retrieved 12 Feb 2019 from https://www.alliedmarketresearch.com/surgical-robotics-market Amazon (2018). The Alexa Prize. Retrieved 14 Feb 2019 from https://developer.amazon.com/alexaprize Anandan, T. (2018). The Dream Labs of Future Robotics. Robotic Industries Association, 2019, Retrieved 12 Feb 2019 from https://www.robotics.org/content-detail.cfm/Industrial-Robotics-Industry-Insights/The-Dream-Labs-of-Future- Robotics/content_id/7533 Bakhshi, H., Downing, J. M., Osborne, M. A., & Schneider, P. (2017). The Future of Skills: Employment in 2030. London: Pearson and Nesta. Retrieved Feb 16, 2019 from https://futureskills.pearson.com/research/assets/pdfs/technical- report.pdf Bekey, G.A. (2011) Current Trends in Robotics: Technology and Ethics. In Lin, P., Abney, K., & Bekey, G.A. (eds.), Robot Ethics: The Ethical and Social Implications of Robotics. Cambridge: MIT Press, pp.17-34 BIS Research (2018). Global Collaborative Robot (Cobot) Market: Focus on Industry and Application - Analysis and Forecast, 2018-2023. Retrieved 18 Feb 2019 from https://bisresearch.com/industry-report/global-collaborative- robot-market.html Bloomberg (2018). The Researcher Building a Robot for Your Grandparents. Bloomberg Technology. Retrieved 14 March 2019 from https://www.bloomberg.com/news/videos/2018-10-18/the-researcher-building-a-robot-for-your- grandparents-video Bonnefon, J., Shariff, A. & Rahwan, I. ( 2016). The Social Dilemma of Autonomous Vehicles. Science, 352(6293), 1573- 1576. DOI: 10.1126/science.aaf2654. Retrieved 12 Feb 2019 from http://science.sciencemag.org/content/352/6293/1573 Bowles, J. (2019). Retrieved 19 Feb 2019 from https://diginomica.com/say-hello-to-marty-the-robot-hes-here-for-your- retail-job/ Bots.co.uk (2018). Collaborative Robot | Hire a Collaborative Robot from £65/day. Taking control: robots and risk

References 47 Retrieved 19 Feb 2019 from https://bots.co.uk/collaborative- robots/?gclid=CjwKCAjw9sreBRBAEiwARroYmzUraUtQnWZ6DVwzKZrqn5- FKyLOftF4Od9WYCAHCLQwUes5AIb1wxoC_PwQAvD_BwE Brooks, R. (2017). Unexpected Consequences of Self Driving Cars. Rodneybrooks.com. Retrieved 12 Feb 2019 from http://rodneybrooks.com/unexpected-consequences-of-self-driving-cars/ BSI (2014). Robots at your service: The future of personal care? A guide to the British Standard for personal care robots (BS EN ISO 13482:2014). Retrieved 12 Feb 2019 from https://www.bsigroup.com/LocalFiles/en-GB/consumer- guides/resources/BSI-Consumer-Brochure-Personal-Care-Robots-UK-EN.pdf BSI (2016a). BS 8611:2016 Robots and robotic devices. Guide to the ethical design and application of robots and robotic systems. Available at: https://shop.bsigroup.com/en/ProductDetail/?pid=000000000030320089&_ga=2.240980096.227296407.15506092 74-1258345916.1550274333 BSI (2016b). Standard highlighting the ethical hazards of robots is published. Retrieved 12 Feb 2019 from https://www.bsigroup.com/en-GB/about-bsi/media-centre/press-releases/2016/april/-Standard--highlighting-the- ethical-hazards-of-robots-is-published/ Budds, D. (2018). Can a $25,000 robot make better coffee than a barista? Curbed.com. Retrieved 19 Feb 2019 from https://www.curbed.com/2018/2/23/17041842/cafe-x-automated-coffee-robot-ammunition-design Bureau of Labor Statistics (2015). Revisions to the 2013 Census of Fatal Occupational Injuries (CFOI) counts. Retrieved 19 Feb 2019 from https://www.bls.gov/iif/oshwc/cfoi/cfoi_revised13.pdf Chaturvedi, A. (2018). Robotic companion to help astronauts tackle stress and loneliness in space. Geospatialworld.net. Retrieved 12 Feb 2019 from https://www.geospatialworld.net/blogs/robotic-seal-astronauts-tackle-stress/ Carlson J., Murphy, R. R., & Nelson, A. (2004). Follow-up analysis of mobile robot failures. IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, New Orleans, LA, USA, Vol.5. pp. 4987-4994 https://ieeexplore.ieee.org/document/1302508 Cave S., Nyrup R., Void, K., Weller A. (2019). Motivations and risks of machine ethics, Retrieved 25 Feb 2019 from https://www.researchgate.net/publication/327490194_Motivations_and_Risks_of_Machine_Ethics CB Insights (2018). Smaller Collaborative Robots Are Disrupting the Robotics Industry. Research Briefs. July 18. Retrieved 19 Feb 2019 from https://www.cbinsights.com/research/collaborative-robots-reinvent-industries/ CLAWAR (2014). New ISO 13482 robot safety standard published. Clawar.org. Retrieved 12 Feb 2019 from https://clawar.org/wp-content/uploads/2015/04/ISOStandardization.pdf Clements, S. (2018). The Cobot Revolution. LionTrust. July 9. Retrieved from https://www.liontrust.co.uk/what-we- think/blogs/The-Cobot-revolution Corbin, T. (2016). Fanuc Unleashes 35kg Payload Capacity Cobot. Packaging News. Retrieved 21 Feb, 2019 from: https://www.packagingnews.co.uk/equipment/fanuc-unleashes-35kg-payload-capacity-cobot-27-06-2016 Deloitte (2018). AI, Robotics, and Automation: Put Humans in the loop. 2018 Deloitte Global Human Capital Trends. Retrieved 14 March 2019 from: https://www2.deloitte.com/content/dam/Deloitte/at/Documents/human-capital/at-deloitte- human-capital-trends-2018-ai-robotics-automation.pdf Demaitre, E. (2018). Rising Cobot Demand Brings Universal Robots to New Heights. Robotics Business Review. Retrieved 19 Feb 2019 from https://www.roboticsbusinessreview.com/manufacturing/rising-cobot-demand-brings- universal-robots-to-new-heights/ DGUV (2015). Industrieroboter. DGUV Information 209-074. Retrieved from http://publikationen.dguv.de/dguv/pdf/10002/209-074.pdf Duckett, T., Pearson, S., Blackmore, S., Grieve, B., Wilson, P., Gill, H., Hunter, A.J., & Georgilas, I. (2018). Agricultural Robotics: The Future of Robotic Agriculture. UK-RAS White Papers, UK-RAS Network, London. arXiv preprint arXiv:1806.06762. Elliq.com (2019). Bringing The World to You and You to the World. Retrieved 14 March 2019 from https://elliq.com/pages/features Emont, J. (2017). Japan Prefers Robot Bears to Foreign Nurses. Foreign Policy. Retrieved 12 Feb 2019 from https://foreignpolicy.com/2017/03/01/japan-prefers-robot-bears-to-foreign-nurses/ EPSRC (2010) Principles of robotics. Engineering and Physical Sciences Research Council. Retrieved 12 Feb 2019 from https://epsrc.ukri.org/research/ourportfolio/themes/engineering/activities/principlesofrobotics/ EU Guidelines on Regulating Robots, 2014, Retrieved 22 Feb 2019 from http://www.robolaw.eu/RoboLaw_files/documents/robolaw_d6.2_guidelinesregulatingrobotics_20140922.pdf Eurobarometer (2012) Public Attitudes Towards Robots. Special Eurobarometer 382: European Commission. Retrieved 12 Feb 2019 from http://ec.europa.eu/public_opinion/archives/ebs/ebs_382_en.pdf. European Parliament (2017). Motion for a European Parliament Resolution with Recommendations to the Commission on Civil Law Rules on Robotics. Retrieved 12 Feb 2019 from http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//TEXT+REPORT+A8-2017- Taking control: robots and risk

References 48 0005+0+DOC+XML+V0//EN Fentanes, J. P., Gould, I., Duckett, T., Pearson, S., & Cielniak, G. (2018). 3D Soil Compaction Mapping through Kriging- based Exploration with a Mobile Robot. Retrieved from http://arxiv.org/abs/1803.08069 Financial Times (2015). Worker at Volkswagen plant killed in robot accident. [online] Ft.com. Available at: https://www.ft.com/content/0c8034a6-200f-11e5-aa5a-398b2169cf79 Financial Times (2017). Bank of England’s Andy Haldane warns 15m UK jobs at risk from robots. [online] Ft.com. Retrieved 12 Feb, 2019 from https://www.ft.com/content/3cea8516-8963-11e5-90de-f44762bf9896. Fingas, R. (2016). Apple Supplier Foxconn So Far Up to 40,000 'Foxbots' in China. Appleinsider.com. Retrieved 12 Feb 2019 from https://appleinsider.com/articles/16/10/05/apple-supplier-foxconn-so-far-up-to-40000-foxbots-in-china Fourier Intelligence (2018). Deadly Earthquake Inspires Idea of Robots as Physical Therapists. Retrieved 12 Feb 2019 from http://www.fftai.com/zixun_en/zixun_bk.php?id=264 Francis, S. (2018a). 30 Industrial Robot Manufacturers to Watch. Robotics & Automation News. Retrieved 13 Feb, 2019 from http://roboticsandautomationnews.com/2018/01/03/30-industrial-robot-manufacturers-to-watch-in-2018/15542/ Francis, S. (2018b). Foxconn to invest $4 billion in new robotics and automation technology. Robotics and Information News. Retrieved 1 Feb 2019 from https://roboticsandautomationnews.com/2018/02/24/foxconn-to-invest-4-billion-in-new-robotics-and-automation- technology/16182/ Frean, A. (2018). Apps Give Companies Help in Dealing with Mental Health Issues. Thetimes.co.uk. Retrieved 12 Feb 2019 from https://www.thetimes.co.uk/article/apps-give-companies-help-in-dealing-with-mental-health-issues- kwvfkk67g Frey, C.B. & Osborne, M.A. (2017). The future of employment: how susceptible are jobs to computerisation? Technological Forecasting and Social Change.114 (2017): 254-280. Gassert, R. and Dietz, V. (2018), Rehabilitation robots for the treatment of sensorimotor deficits: a neurophysiological perspective. Journal of NeuroEngineering and Rehabilitation, 15(46). Retrieved 19 Feb, 2019 from https://doi.org/10.1186/s12984-018-0383-x Glaessgen, E. & Stargel, D. (2012) The Digital Twin Paradigm for Future NASA and US Air Force Vehicles. Proceedings of the 53rd Structures, Structural Dynamics, and Materials Conference, Honolulu, Hawaii. Globaldata (2017). Global Thematic Research: Technology, Robotics (Vol III). Retrieved 19 Feb 2019 from https://www.globaldata.com/store/report/gdtmt-tr-s148--robotics-vol-iii-global-thematic-research/ Grieves, M. (2014) Digital Twin: Manufacturing Excellence Through Virtual Factory Replication. Retrieved 18 Feb 2019 from http://www.apriso.com/library/Whitepaper_Dr_Grieves_DigitalTwin_ManufacturingExcellence.php Gurman, M. (2018). Big Tech is Throwing Money and Talent at Home Robots. Bloomberg. Retrieved 14 Feb 2019 from https://www.bloomberg.com/news/articles/2018-07-24/big-tech-is-throwing-money-and-talent-at-home-robots Heater, B. (2019). Amazon Packages On Time. Techcrunch.com. Retrieved 18 March 2019 from https://techcrunch.com/2019/03/17/these-are-the-robots-that-help-you-get-your-amazon-packages-on-time/ Heriot-Watt (2018). Heriot-Watt represents UK in global Amazon competition. Retrieved 14 Feb 2019 from https://www.hw.ac.uk/about/news/2018/heriot-watt-represents-uk-in-global-amazon.htm Hern, A. (2016). Self-Driving Cars Don't Care About Your Moral Dilemmas. The Guardian. Retrieved 12 Feb 2019 from https://www.theguardian.com/technology/2016/aug/22/self-driving-cars-moral-dilemmas HSE (2011). New Machinery. hse.gov.uk. Retrieved 15 Mar 2019 from http://www.hse.gov.uk/work-equipment- machinery/new-machinery.htm#responsibilities Hosseini, S., & Goher, K. (2017). Personal Care Robots for Older Adults: An Overview. Asian Social Science. 13(1): 1376-82. doi: 10.1111/ijcp. Retrieved 12 Feb 2019 from http://www.ccsenet.org/journal/index.php/ass/article/view/64518 Hussain, A., Malik, A., & Halim, M.U. (2014). The Use of Robotics in Surgery: A Review. International Journal of Clinical Practice, 68(11). Retrieved 12 Feb 2019 from https://www.ncbi.nlm.nih.gov/pubmed/25283250 Inkwood Research (2018) Global Collaborative Robots Market Forecast 2018-2026. Retrieved 15 March 2019 from https://www.inkwoodresearch.com/reports/collaborative-robots-market/ International Federation of Robotics (IFR) (2018a). World Robotics 2018 Service Robots Executive Summary. Retrieved from https://ifr.org/downloads/press2018/Executive_Summary_WR_Service_Robots_2018.pdf International Federation of Robotics (2018b). World Robotics 2018 Industrial Robots Executive Summary. Retrieved from https://ifr.org/downloads/press2018/Executive_Summary_WR_2018_Industrial_Robots.pdf International Federation of Robotics (2018c). Automation without Fear of Contact. Retrieved from https://ifr.org/ifr-press- releases/news/world-robotics-report-2018 i-SCOOP (2017). Robots and Cobots in Logistics – The Next Stage of Growth – Industry 4.0. Retrieved from https://www.i-scoop.eu/robots-cobots-logistics-4 Taking control: robots and risk

References 49 Ismail, N. (2017). Artificial Intelligence and Robots Key in Enhancing Customer Service. Information Age. Retrieved 12 Feb 2019 from https://www.information-age.com/artificial-intelligence-robots-key-enhancing-customer-service- 123468933/ Jacobs, T., & Virk, G. S. (2014). ISO 13482 - The New Safety Standard for Personal Care Robots. ISR/Robotik 2014; 41st International Symposium on Robotics, Munich, Germany: 1-6. Retrieved 19 Feb 2019 from http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6840202&isnumber=6840100 Johansson, R. & Nilsson, J. (2016). Disarming the Trolley Problem – Why Self-driving Cars do not Need to Choose Whom to Kill. In Matthieu Roy. Workshop CARS 2016 - Critical Automotive applications : Robustness & Safety, Sep 2016, Göteborg, Sweden. 2016, CARS 2016 - Critical Automotive applications : Robustness & Safety. Retrieved 12 Feb 2019 from http://homepages.laas.fr/mroy/CARS2016-papers/CARS2016_paper_16.pdf Karwowski, W. & Rahimi, M. (eds). (2003). Human-Robot Interaction, New York: CRC Press. Kleinman, Z. (2018). The Ocado Robot Swarms That Pack Your Shopping. Bbc.co.uk. Retrieved March 15 from https://www.bbc.co.uk/news/technology-43968495 Kildal, J., Tellaeche, A., Fernández, I., & Maurtua, I. (2018). Potential Users' Key Concerns and Expectations for the Adoption of Cobots. Procedia CIRP, 72: 21-26. Knight, H. (2014). How Humans Respond to Robots: Building Public Policy through Good Design. Brookings Institute. Retrieved 15 Feb 2019 from https://www.brookings.edu/wp-content/uploads/2014/07/HumanRobot- PartnershipsR2.pdf Lai, P.C. (2017). The Literature Review of Technology Adoption Models and Theories for the Novelty Technology. Journal of Information Systems and Technology Management, 14(1), Jan/Apr: 21-38. DOI: 10.4301/S1807- 17752017000100002 LeVine, S. (2018). In China, a Picture of How Warehouse Jobs Can Vanish. Axios. June 13. Retrieved Feb 19, 2019 from https://www.axios.com/china-jd-warehouse-jobs-4-employees-shanghai-d19f5cf1-f35b-4024-8783- 2ba79a573405.html MarketsAndMarkets (2018). Collaborative Robot Market by Payload Capacity (Up to 5 Kg, Between 5 and 10 Kg, Above 10 Kg), Industry (Automotive, Electronics, Metals & Machining, Plastics & Polymers, Food & Beverages, Healthcare), Application, and Geography - Global Forecast to 2025. Marketsandmarkets.com. Retrieved 18 Feb 2019 from https://www.marketsandmarkets.com/Market-Reports/collaborative-robot-market-194541294.html Market Research Engine (2017). Top Robotics Market By Vertical Analysis (Automotive, Electrical and Electronics, Chemical, Rubber, and Plastic, Metal and Machinery, Food and Beverages, Precision and Optics); By Application Analysis (Logistics Robots, Domestic Robots, Medical Robots, Field Robots, Defense, Rescue, and Security Robots, Entertainment, Education, and Personal Robots); By Type Analysis (Articulated Robots, SCARA Robots, Parallel Robots, Cylindrical Robots, Cartesian Robots) and By Regional Analysis – Global Forecast by 2018 – 2024. May. Retrieved from https://www.marketresearchengine.com/top-robotics-market Matthews-King, A. (2018). Brexit Will Leave UK Short of 380,000 Care Workers By 2026, Analysis Suggests. The Independent. Retrieved 17 Feb 2019 from https://www.independent.co.uk/news/health/brexit-freedom-of-movement-plan-care-workers-uk-eu-shortage- a8501751.html McKinsey Global Institute (2017). Jobs lost, jobs gained@ worksforce transitions in a time of automation. McKinsey&Company. Moon M. (2018), Robots will build robots in $150 million Chinese factory. Engadget.com. Retrieved 19 Feb 2019 from https://www.engadget.com/2018/10/27/abb-robotics-factory-china/ Moor, J.H. (2009). Four Kinds of Ethical Robots. Philosophy Now. Retrieved 12 Feb 2019 from https://philosophynow.org/issues/72/Four_Kinds_of_Ethical_Robots Motion Controls Robotics (2019). #2 of 5 Most Commonly Asked Questions from New Customers- What happens if the cell fails after you (the supplier) have left our site? Retrieved Feb 13, 2019 from https://motioncontrolsrobotics.com/2-of-5-most-commonly-asked-questions-from-new-customers-what-happens-if- the-cell-fails-after-you-the-supplier-have-left-our-site/ Murphy, A. (2017). Industrial: Robotics Outlook 2025. Loup Ventures. Retrieved Feb 15, 2019 from https://loupventures.com/industrial-robotics-outlook-2025/ Murashov, V., Hearl, F., & Howard, J. (2016). Working Safely With Robot Workers: Recommendations for the New Workplace. Journal of Occupational and Environmental Hygiene, 13(3), D61-71. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4779796/#R12 Nagaraja, A, et al. (2015). Clostridium difficile Infections Before and During Use of fviolet Disinfection. American Journal of Infection Control. Sep 1;43(9):940-5. doi: 10.1016/j.ajic.2015.05.003. Nichols, G. (2017). Robots are coming to work. Are they safe? Retrieved 18 Feb 2019 from https://www.zdnet.com/article/robots-are-coming-to-work-are-they-safe/ Taking control: robots and risk

References 50 Nof, S.Y. (1999). Handbook of Industrial Robotics. New York: John Wiley & Sons. OTPotential (2018). Robotic Therapy Is on the Rise. Here’s Why. Retrieved 12 Feb 2019 from https://otpotential.com/blog/active-assistive-robotic-therapy Out-Law.com (2016). Replacing Humans with Robots in Your Factories? Hold on Just a Sec. The Register 2016, Retrieved 12 Feb 2019 from https://www.theregister.co.uk/2016/08/30/robots_factories_legal_considerations_out_law/ Parry, G. C., Brax, S. A., Maull, R. S., & Ng, I. C. (2016). Operationalising IoT for Severse supply: The development of use-visibility measures. Supply Chain Management: An International Journal, 21(2), 228-244. Pearson, S., May, D., Leontidis, G., Swainson, M., Brewer, S., Bidaut, L., Frey, J.G., Parr, G. Maull, R., Zisman, A. (2019). Are Distributed Ledger Technologies the Panacea for Food Traceability? Global Food Security. 20: 145- 149. ISSN 2211-9124. https://doi.org/10.1016/j.gfs.2019.02.002. Petkova, H. (2010). Barriers to Innovation in the Field of Medical Devices. World Health Organisation Background Paper, August. Retrieved 15 Mar 2019 from https://apps.who.int/iris/bitstream/handle/10665/70457/WHO_HSS_EHT_DIM_10.6_eng.pdf;jsessionid=BA997413 04345FFFD6163354184ECD47?sequence=1 Pilz (2018). Revision of the ISO Safety Standard for Industrial Robots. Retrieved 1 Feb 2019 from https://www.pilz.com/en-GB/company/news/articles/196225 Platbrood, F. & Görnemann, O. (2018). Safe Robotics – Safety in Collaborative Robot Systems. Sick AG. Retrieved 12 Feb 2019 from https://cdn.sick.com/media/docs/6/96/996/Whitepaper_Safe_Robotics_en_IM0072996.PDF Prakash, A. (2017). Why Robot Law Around Industrial Automation Varies Worldwide. Robotics Business Review. Jan 1. Retrieved 17 Feb 2019 from https://www.roboticsbusinessreview.com/manufacturing/why_robot_law_around_industrial_automation_varies_worl dwide/ Priyandoko, G., Wei, C. K., Sobirin, M., Achmad, H., Widyagama, U., & Yogyakarta, U. T. (2017). HUMAN FOLLOWING ON ROS FRAMEWORK, 2(2), 77–82. PwC (2018). UK Economic Outlook. Prospects for the Housing Market and the Impact of AI on Jobs. July. Pwc.co.uk. Retrieved 19 Feb 2019 from https://www.pwc.co.uk/services/economics-policy/insights/uk-economic-outlook/july-18.html Robotics Business Review (2017). The Chief Robotics Officer - 2017 Update. Retrieved 16 Feb 2019 from https://www.roboticsbusinessreview.com/wpcontent/uploads/2017/06/CRO_2017_Report_vF.pdf Robotics Business Review (2018). 6 Service Robot Questions to Be Answered at CES 2019. Retrieved Feb 14, 2019 from https://www.roboticsbusinessreview.com/consumer/6-service-robot-questions-ces-2019/ Robotiq (2018). ISO/TS 15066 Explained. Robotiq.com. Retrieved 12 Feb 2019 from https://blog.robotiq.com/hubfs/eBooks/ebook-ISOTS15066-Explained.pdf?hsLang=en-ca&t=1541094236194 Rodrigues, J., Cardoso, P., Monteiro, J. & Figueiredo, M. (2016).Handbook of Research on Human-Computer Interfaces, Developments, and Applications, IGI Global, 29 Jun Röhrbein, F., Veiga, G., &Natale, C. (2013). Gearing Up and Accelerating Cross‐fertilization between Academic and Industrial Robotics Research in Europe: Technology Transfer Experiments from the ECHORD Project, Springer, 23 Nov 2013 - Technology & Engineering Rogers, E.M. (2010). Diffusion of Innovations. New York: Simon and Schuster. Savela, N., Turja, T. & Oksanen, A. (2017). Social Acceptance of Robots in Different Occupational Fields: A Systematic Literature Review. Retrieved 12 Feb 2019 from https://www.researchgate.net/publication/321576378_Social_Acceptance_of_Robots_in_Different_Occupational_F ields_A_Systematic_Literature_Review Schmidt, A. (2018). Two years’ Experience with Cobots in a Low Volume Manufacturing Environment. Retrieved 19 Feb 2019 from https://www.rethink-smart-manufacturing.com/wp-content/uploads/2018/03/04_01-Dr.-Axel-Schmidt- Sennheiser.pdf Schneier, M. & Bostelman, R. (2015). NISTIR 8022 Literature Review of Mobile Robots for Manufacturing. National Institute of Standards and Technology, US Department of Commerce. Retrieved 12 Feb 2019 from http://dx.doi.org/10.6028/NIST.IR.8022 Sharkey, A, (2017), Can we program or train robots to be good?. Ethics and Information Technology, 1-13. Smith, N. (2018). Who’s Winning the Robot Race? Control Engineering Europe. Retrieved 12 Feb 2019 from http://www.controlengeurope.com/article/159599/Who-s-winning-the-robot-race-.aspx Song, V. (2018). Ecovacs Deebot R95: A Solid, Versatile Robot Vacuum. PC MAgazine (Digital Edition). Statista. (2018). Worldwide Sales of Industrial Robots from 2004 to 2017 (in 1,000 units). Retrieved from https://www.statista.com/statistics/264084/worldwide-sales-of-industrial-robots/ Statutes of the Republic of Korea (2019). Intelligent Robots Development and Distribution Promotion Act. Retrieved 18 Taking control: robots and risk

References 51 Feb 2019 from http://elaw.klri.re.kr/eng_mobile/viewer.do?hseq=39153&type=lawname&key=robot ST Robotics (2018). R12 5/6 Axis Robot Arm. Retrieved 12 Feb 2019 from http://strobotics.com/small-articulated- robot.htm Synek, G. (2018). LG is putting robot shopping carts into retail stores. Retrieved 19 Feb 2019 from https://www.techspot.com/news/77273-lg-putting-robot-shopping-carts-retail-stores.html Takayama, L., Ju, W. & Nass, C. (2008). Beyond Dirty, Dangerous and Dull: What Everyday People Think Robots Should Do. HRI 2008 - Proceedings of the 3rd ACM/IEEE International Conference on Human-Robot Interaction: Living with Robots. 25-32. Retrieved 19 Feb 2019 from doi.org/10.1145/1349822.1349827 Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., & Sui, F. (2018). Digital Twin-driven Product Design, Manufacturing and Service with Big Data. The International Journal of Advanced Manufacturing Technology, 94(9-12): 3563- 3576. Tilley, J. (2017). Automation, Robotics, and the Factory of the Future. McKinsey & Company. Retrieved 19 Feb 2019 from https://www.mckinsey.com/business-functions/operations/our-insights/automation-robotics-and-the-factory-of- the-future Trade Map (2018). International Trade Centre. https://www.trademap.org/Index.aspx . Retrieved on June 6, 2018. UL (2017). Addressing Regulatory Considerations for Medical Robotic Devices. Ul LCC . Retrieved March 18, 2019 from https://library.ul.com/wp-content/uploads/sites/40/2017/08/BNG-UL17-Medical-Robots-White-Paper-080117-1.pdf United States Department of Labor (2019). Accident Search Results. Osha.gov. Retrieved 18 Feb 2019 from https://www.osha.gov/pls/imis/accidentsearch.search?sic=&sicgroup=&naics=&acc_description=&acc_abstract=&a cc_keyword=robot&inspnr=&fatal=&officetype=All&office=All&startmonth=08&startday=13&startyear=2016&endmo nth=01&endday=01&endyear=1984&keyword_list=&p_start=&p_finish=0&p_sort=&p_desc=DESC&p_direction=Ne xt&p_show=20 Universal Robots (2017). Lightweight Cobots Take on Heavy-Duty Jobs—Even Welding. Retrieved 12 Feb 2019 from https://blog.universal-robots.com/lightweight-cobots-take-on-heavy-duty-jobs-even-welding Universal Robots (2019). Pick & Place Application and Spraying for Coating Loudspeaker Membranes. Retrieved 12 Feb 2019 from https://www.universal-robots.com/case-stories/beyerdyn mic-gmbh-co-kg/ Wagner, T.H., Lo, A.C., Peduzzi, P., Bravata, D.M., Huang, G.D., Krebs, H.I., Ringer, R.J., Federman, D.G., Richards, L.G., Haselkorn, J.K., Wittenberg, G.F., Volpe, B.T., Bever, C.T., Duncan, P.W., Siroka, A. & Guarino, P.D. (2011). An Economic Analysis of Robot-Assisted Therapy For Long-Term Upper-Limb Impairment After Stroke. Stroke, Sep;42(9):2630-2. doi: 10.1161/STROKEAHA.110.606442. Epub 2011 Jul 14. Retrieved 12 Feb 2019 from https://www.ncbi.nlm.nih.gov/pubmed/21757677 Warburton, C. (2017). Rise of the Machines. Health and Safety at Work. Retrieved 12 Feb 2019 from https://www.healthandsafetyatwork.com/safety-technology/robots-cobots-rise-machines Wei, Y., Ma, Y., & Zhang, W. (2018). A multi-jointed underactuated robot hand with fluid-driven stretchable tubes Background. Robotics and Biomimetics, 5(2). https://doi.org/10.1186/s40638-018-0086-6 Wiener, A. (2017). Inside Adidas’ Robot-Powered, On-Demand Sneaker Factory. Wired.com. Retrieved 19 feb 2019 from https://www.wired.com/story/inside-speedfactory-adidas-robot-powered-sneaker-factory/ Winfield, A. (2018). Why Ethical Robots Might Not Be Such a Good Idea After All. IEEE Spectrum. Retrieved 12 Feb 2019 from https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/why-ethical-robots-might-not-be-such- a-good-idea-after-all Workman, D. (2018). Top Industrial Robots Exporters. Worldstopexports.com. Dec 26. Retrieved from http://www.worldstopexports.com/top-industrial-robots-exporters/ Yan, A. (2017). Chinese robot dentist is first to fit implants in patient’s mouth without any human involvement. South China Morning Post. Retrieved 12 Feb 2019 from http://www.scmp.com/news/china/article/2112197/chinese-robot-dentist-first-fit-implants-patients-mouth-without- any-human Taking control: robots and risk

Appendix A: Glossary of terms – Robots 52 Appendix A: Glossary of terms - Robots Terms for robots that are used widely in this report are: − Cobots or collaborative/cooperative robots: Devices that operate near humans, usually working with them in a shared space. − Industrial robots: Robots used in manufacturing, in applications that include assembly, welding, pick and place. These are usually traditional ‘caged’ robots that are fixed, but they could also be cobots. − Service robots: All types of robots except industrial robots used in manufacturing. − Professional service robots: A sub-group of service robots used for commercial tasks, for example, medical or surgery robots in a hospital or fire-fighting robot. Usually under the control or guidance of a trained operator. − Mobile robots: Devices that are capable of moving around in their environment instead of being fixed in one physical location. Examples include Automatic Guided Vehicles (AGV) and Autonomous Mobile Robots (AMR). − Personal care/personal service robots: A sub-group of service robots that operate closely with humans, usually in direct contact, to contribute directly to their wellbeing. Can include consumer/domestic robots used in the home, such as vacuum cleaner and gardening robots. Usually controlled by a layperson. − Self-learning robot: A robot that acquires skills or adapt to its environment through learning algorithms. Note that industrial/service classification is done according to application area. None of the categories refer exclusively to cobots (however, they are very likely to be in close proximity to humans). Taking control: robots and risk

Appendix B:Safety standards 53 Appendix B: Safety standards As robots fall under the broad definition of machinery, they are subject to the EU Machinery Directive 2006/42. Where they are intended for the consumer markets, they come under the purview of the General Product Safety Directive 2001/95. The design, manufacture and operation of robots and cobots are within the scope of several layers of International Organization for Standardisation (ISO) Standards and Technical Specifications (TS): ISO 12100:2010 specifies principles of risk assessment and reduction and underlies the standards governing the design of robots. ISO 10218-1:2011 specifies requirements and guidelines for the inherent safe design, protective measures and information for use of industrial robots. It is aimed at the robot manufacturer. Part 1, 5.10 covers collaborative operation requirements, such as a visual indication when the robot is in collaborative operation. It provides for four modes of safe working (Platbrood & Görnemann, 2018): 1. Monitored safe stop - where the robot is stopped while the operator enters the safe space; 2. Manual control - where the robot is manually guided at a safe speed by the operator; 3. Force and power limitation - where contact between the robot is detected and the power and force of those contacts are limited; 4. Distance and speed monitoring - where the robot detects the presence of a person and moves away to avoid contact. ISO 10218-2:2011 specifies safety requirements for the integration of industrial robots and industrial robot systems as defined in ISO 10218-1, and industrial robot cell(s). The integration includes the design, manufacturing, installation, operation, maintenance and decommissioning of the industrial robot system or cell, necessary information for the design, manufacturing, installation, operation, maintenance and decommissioning of the industrial robot system or cell and component devices of the industrial robot system or cell. It is aimed at the robot integrator. The ISO 10218 standards are currently being revised under the regular five-year ISO review cycle, and a new version is expected in May 2021. Recognising the growth of collaborative robot use, many topics and requirements are being discussed, including listing all the relevant safety functions, developing more specific safety requirements for brakes and mobile robots, and cybersecurity (Pilz, 2018). ISO/TS 15066:2016 is a technical specification for collaborative robots. It only applies to cobots in industrial environments, although its principles are relevant to other sectors. This is a specification and not a standard, but will in time be incorporated into ISO 10218. Its main focus is to provide a comprehensive risk assessment guide of all the motions, interactions and operations a robot should perform. Every automated application where humans are present requires this risk assessment, and collaborative applications need a range of safety mechanisms to keep human workers safe. Passive safety features can include fire resistance, manual movement capability, elimination of sharp edges and protrusions, padding, speed restrictions, low inertias of moving parts to limit the effects of collisions and maximum static forces, as well as switch strips mats and vests (Karwowski & Rahimi, 2003). Taking control: robots and risk

Appendix B:Safety standards 54 ISO 13482:2014 applies to personal care robots (BSI, 2014). It provides guidance for assurance of safety in the design, construction, installation and use of the robot in three categories: 1. Mobile service robots (e.g. open curtains, doors or windows, clean or vacuum, fetch and carry items such as drinks, or plates of food, pick up objects from the floor, switch equipment on or off); 2. Physical assistant robots (getting up from a chair or out of bed, getting into and out of a bath or shower, help with getting dressed, help with basic personal care such as combing hair); 3. Person carrier robots (within their home, around public buildings, or other public spaces, between predefined locations). Manufacturers (and suppliers) that comply with ISO 13482 should identify potential risks, issue clear labelling and instructions, ensure safe movement and reduce chances of ‘bad’ decisions. For example, a person carrier robot should ensure that a passenger is correctly seated before starting to move, or that it stops in a location where it is safe for the passenger to get off. Taking control: robots and risk