Video is becoming ubiquitous

Video plays an increasingly important role in our lives. As consumers, we collectively spend hundreds of millions of hours every day watching and sharing videos on services like YouTube, Facebook or Snapchat. When we are not busy gobbling up video on social media, we produce more of it with our smartphones, GoPro cameras and (soon) AR goggles. As a growing fraction of the planet’s population is documenting their lives in video format, we are transitioning from starring in our own magazine (the still image era) to starring in our own reality TV show.

All that is arguably just the beginning. The next few years will see a proliferation of connected devices, ranging from smart, always-on home cameras to autonomous vehicles. Many of these devices will rely on a camera as the primary sensory input for understanding and navigating the world. As the technological evolution marches on, video intelligence will be crucial. It is quite clear that there are not enough eyeballs in the world to process all video data and human-powered visual understanding unfortunately does not scale. What we need is a software layer that can analyze and extract meaning from video. That requires learning algorithms that understand the physical world and the myriad of actions that are carried out by the actors within it.

Video is the next frontier in computer vision

Deep Learning has made historic progress in recent years by producing systems that rival — and in some cases exceed — human performance in tasks such as recognizing objects in still images. Despite this progress, enabling computers to understand both the spatial and temporal aspects of video remains an unsolved problem. The reason is sheer complexity. While a photo is just one static image, a video shows narrative in motion. Video is time-consuming to annotate manually, and it is computationally expensive to store and process.

The main obstacle that prevents neural networks from reasoning more fundamentally about complex scenes and situations is their lack of common sense knowledge about the physical world. Video data contains a wealth of fine-grained information about the world as it shows how objects behave by virtue of their properties. For example, videos implicitly encode physical information like three-dimensional geometry, material properties, object permanence, affordance or gravity. While we humans intuitively grasp these concepts, a detailed understanding of the physical world is still largely missing from current applications in artificial intelligence (AI) and robotics.

At TwentyBN, we believe that intelligent software for handling video is a prerequisite for enabling the most promising applications of AI in the real world. One field of applications that we are excited about is health care, and in particular elderly care. When it comes to eldercare, changes in the activities of basic living (ADLs) often precede physiological changes and can predict poor clinical outcomes. Imagine how much we could improve the care for our aged loved ones if it were possible to install a handful of smart camera devices in fixed locations to monitor changes in activities of seniors, aid their memory, and ultimately improve their health?

To enable applications like these, we need a technology step change. We need systems that can understand the context and actions occurring in visual scenes. State-of-the-art image recognition just won’t cut it. This is because life is more than a sequence of snapshots, and perceiving the world is more than recognizing cats and dogs in images. It’s about what is actually happening in the physical world as time unfolds. It’s about verbs, not just nouns.

A novel approach to video understanding

One of the most important rate limiting factors for advancing video understanding is the lack of large and diverse real-world video datasets. Many video datasets that have been published to date suffer from a number of shortcomings: they are often weakly labeled, lack variety, or underwent a high degree of editing and post-processing. A few notable exceptions, like DeepMind’s recently released Kinetics dataset, try to alleviate this by focusing on shorter clips, but since they show high-level human activities taken from YouTube videos, they fall short of representing the simplest physical object interactions that will be needed for modeling visual common sense.

At TwentyBN, we have spent the past year building a foundational data layer for the understanding of physical actions. Our approach is based on a single, rather straightforward idea: Why not leverage the amazingly precise and cultivated motor skills that human beings possess to generate fine-grained, complex and varied data at scale? After all, the vast majority of motion patterns that we observe day-to-day are actually caused by other humans.

To generate the complex, labelled videos that neural networks need to learn, we use what we call “crowd acting”. We instruct crowd workers to record short video clips based on carefully predefined and highly specific descriptions. For example, “Pushing something until it falls of the table”, “Moving object A closer to object B”, or “Sliding two fingers of your left hand up”. While we collect data for many different kinds of human actions, we naturally stress dexterous manipulation of objects using one or both hands. This is because our hands are best at generating the highly controlled, complex motion patterns needed for training networks. Instead of painstakingly labeling existing video data, crowd acting allows us to generate large amounts of densely labeled, meaningful video segments at low cost.

Today, we are excited to announce the release of two substantial snapshots from our data collection campaign: A database of human-object interactions (Something-something) and the world’s largest video dataset for classifying dynamic hand gestures (Jester). The two datasets are “snapshots” because data collection is an ongoing process. In total, we are releasing 256,591 labeled video clips for supervised training of deep learning models. Both datasets are made available under a Creative Commons Attribution 4.0 International license (CC BY-NC-ND 4.0) and are freely available for academic use. If can want to license our datasets for commercial use, please contact us.

1. “Something-something” dataset

This snapshot contains 108,499 annotated video clips, each between 2 and 6 seconds in duration. The videos show objects and the actions performed on them across 175 classes. The captions are textual descriptions based on templates, such as “Dropping something into something”. The templates contain slots of “something” that serve as placeholders for objects. This provides added structure between the text-to-video encoding for the network to improve learning.

The goal of this dataset is not only to detect or track objects in videos but to decipher the behavior of human actors as well as the direct and indirect manipulations of the objects they interact with. Predicting the textual labels from the videos therefore requires strong visual features that are capable of representing a wealth of physical properties of the objects and the world. This includes information about properties like spatial relations and material properties.

2. “Jester” dataset

This snapshot contains 148,092 annotated video clips, each about 3 seconds long. The videos cover 25 classes of human hand gestures as well as two “no gesture” classes to help the network distinguish between specific gestures and unknown hand movements. The videos show human actors performing generic hand gestures in front of a webcam, such as “Swiping Left/Right,” “Sliding Two Fingers Up/Down,” or “Rolling Hand Forward/Backward.” Predicting these textual labels from the videos requires a network that is capable of grasping such concepts as the degrees of freedom in three-dimensional space (surging, swaying, heaving, etc).

Traditional gesture recognition systems require special hardware like stereo cameras or depth sensors such as time-of-flight cameras. Using our Jester dataset, we were able to train a neural network that can detect and classify all 25 gestures from raw RGB input with a test accuracy of 82%. The system runs in real-time on a variety of embedded platforms using video input from a webcam.

Key characteristics of both datasets

• Supervised learning: In contrast to other methods that seek to acquire common sense through the use of predictive unsupervised learning, we phrase the task as a supervised learning problem. This makes the representation learning task more tractable and defined.
• Dense captioning: The labels describe video content that is restricted to a short time interval. This ensures there is a tight synchronization between the video content and the corresponding caption.
• Crowd-acted videos: In contrast to other academic datasets that source and annotate video clips from YouTube, we created our datasets using crowd acting. Our proprietary crowd acting platform allows us to ask crowd workers to provide videos given caption templates instead of the other way around. This facilitates the generation of labeled recordings rather than just the labeling of existing videos.
• Human focused: With the exception of motion “textures” like ocean waves or leaves in the wind, most complex motion patterns that we ever see are caused by humans. Our datasets are human-centered to have the complex spatio-temporal patterns that encode features of articulation, degrees of freedom, etc.
• Natural video scenes: Our videos were captured with many different devices and varying zoom factors. The datasets feature scenes with natural lighting, partial occlusions, motion blur and background noise. This assures that the datasets can transfer to real-world use cases with minimal domain shift.

The videos clips are challenging because they capture the messiness of the real world. To give you a flavor, take a look at this video clip from the Jester dataset, showing a person performing a hand gesture:

While the hand gesture is visible to the human eye, it is difficult to recognize for a computer because the video footage contains sub-optimal lighting conditions and background noise (cat walking through the scene). Training on Jester forces the neural network to learn the relevant visual cues, or “hierarchical features”, to separate the signal (hand motion) from the noise (background motion). Basic motion detection will not be sufficient.

The practical use of visual common sense

How do we go from an understanding of physical concepts to offering practical, real-world solutions? We believe the answer is to be found in a technical concept called transfer learning.

We humans are good at thinking by analogy. Taking an idea from one domain and applying it to another is “the fuel and fire of our thinking,” as Douglas Hofstadter puts it. In AI, a step towards reasoning by analogy is transfer learning. With transfer learning, we can take a neural network trained on Something-something and Jester, and transfer its capabilities to contribute to specific business applications. Specifically, networks that have internalized representations of how objects interact in the real world can transfer this intrinsic knowledge to solve problems of higher order complexity that are predicated on these fundamental concepts.

Transfer learning has already achieved astounding results on a wide array of image-based vision tasks (see here, here and here). We believe that similar breakthroughs are imminent for deep video understanding. The prerequisite for leveraging transfer learning in the video domain is, however, the availability of high-quality, labeled video data that allows neural networks to model visual common sense. This is the mission we signed up for at TwentyBN. Data collection at our company spans a spectrum from hard to recognize, but solvable (with existing proof points, like hand gesture recognition), to very hard and still not solvable. The ultimate endpoint of this spectrum is general AI.

How to get the data and where to benchmark your results

The two datasets are available for download on our website. You can find more information about the datasets and the technical specifics of our research in this technical report. If you want to benchmark the accuracy that your very own model achieves on the datasets, you will be able to upload your results to our website to be ranked in a leaderboard. If can want to license our datasets for commercial use, please reach out to us.

Our goal in releasing these two datasets is to catalyze the development of machines that can perceive the world like humans. Our work builds on the foundations of researchers past and present. We are committed to give back for the benefit of this vibrant community.

Learning about the world through video was originally published in twentybn on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

The German automotive sector is a prime candidate to be rethought through the lens of artificial intelligence (AI). In this article, I will outline why autonomous cars will drive a paradigm shift in the auto industry (part I), argue that the problem for German car manufacturers is their old-school engineering culture, not a lack of innovation (part II), and examine counter models to Germany’s current approach to self-driving cars (part III). I will conclude with a number of suggestions for improvement that would, in my view, put the German car industry back on track.

I. Why Germany’s auto industry is at risk

The automotive industry is one of the most important industrial sectors in Germany. It employs nearly 800,000 people, produces roughly 15 million passenger cars per year and generates more than €400 billion in annual revenues. More than 70% of all premium brand vehicles produced worldwide are manufactured by German OEMs — in particular by the three industry leaders BMW, Mercedes and Audi. The auto sector is also a major driver of research and development (R&D), accounting for about 40% of the economy’s R&D budget. Germany is a car-crazy nation and these numbers are testament to the fact that the industry remains absolutely central to the country’s export-oriented economy.

It is therefore all the more disturbing that the German automotive sector has come under immense pressure to develop the two key technologies of the future: electric mobility (i.e. cars that are propelled by electric motors instead of internal combustion engines) and autonomous driving (i.e. cars that are capable of sensing their environment and navigating without human input). Germany’s proud carmakers missed out on the first trend and are currently overrun by Tesla in the race to build driverless cars. While the failure to lead in electric mobility is by itself excruciating, being left behind by the self-driving revolution could put the future of Germany’s most important economic sector at risk.

Autonomous driving is a paradigm shift that will likely change everything for traditional carmakers. The gradual transition towards battery technology could soon destabilize the traditional automotive supply chain, which has its emphasis on components like the combustion engine or gearbox. Even though the dismantling of this value chain poses a threat to carmakers, electrification does not drastically change the meaning of the word “car” in the eyes of consumers. In contrast, the trend towards greater autonomy will fundamentally change what cars are. As Benedict Evans has observed, the word “car” will mean something completely different in the age of autonomy — much like the word “phone” was transformed after the launch of the iPhone. Autonomous vehicles will throw existing business models into question and challenge the very notion of car ownership. If the car is in the process of being reimagined, so too must their creators.

As if this wasn’t enough, high-tech newcomers like Tesla, Uber, Google, Nvidia and tons of startups are hacking away at the market. The march towards electrically-powered autonomous vehicles has significantly lowered the barriers to entering the auto business for two major reasons. First, batteries will be a commodity once production is up and running, as will be the hardware sensors — cameras, radar, lidar, ultrasonic sensors — that are prerequisites for building self-driving cars. Second, electric vehicles will have significantly fewer moving parts than cars powered by an internal combustion engine, which could change the whole basis of competition. Note that this does not imply that car making is becoming trivial, as demonstrated by Apple’s attempts to launch its own car project. As the key hardware components of autonomous cars become cheaper and increasingly commoditized, most of the car’s value will shift to its software components.

The shift of value away from predominantly hardware (the car’s “body”) to software (the car’s “brain”) creates an existential threat for traditional carmakers. Software is not a core competence of auto OEMs, even though it has long played a role in carmaking. According to Manfred Broy, a Professor of Computer Science at the Technical University Munich, a modern premium-class car already “contains close to 100 million lines of software code” — much more than a modern airplane or fighter jet. However, writing firmware to control the car’s air conditioning is very different from teaching a car to drive itself. For example, autonomous vehicles use incredibly sophisticated machine learning algorithms to interpret the car’s surroundings and instruct the various hardware components to perform the necessary actions. Self-driving cars are also massive data-generation machines, generating one terabyte of data per hour by some estimates. Not to mention the emerging content opportunities that arise when software takes over the car. Excelling at these areas of software development requires an entirely new skill- and mindset from the vantage point of auto companies.

It is not only car manufacturers who are in need of software skills. The paradigm shift in the auto industry is also impacting car parts suppliers who have undergone a wave of consolidation in the past two years. According to data compiled by Bloomberg, the deal volume among automotive suppliers totaled $74.4 billion in 2015 and the first half of 2016 — far exceeding the $17.7 billion spent on average in the previous ten years. In particular, the three large German tier-one suppliers have been gearing up on software: In 2015, ZF Friedrichshafen acquired U.S. software provider TRW Automotive for $12.4 billion. The same year, Continental took over the software division of Elektrobit Automotive for $660 million. Bosch meanwhile acquired ZF Lenksysteme, a supplier of advanced steering systems, and more recently ITK Engineering, a software company employing more than 800 people. Both the volume and diversity of these deals are striking but support the view that incumbent parts suppliers must also retool their software teams.

The self-driving revolution raises important questions for the entire manufacturing sector. Why are Germany’s automobile companies perceived to be lagging behind? How is it possible that German consumers suddenly put more trust in Tesla than in domestic carmakers when it comes to putting a market-ready autonomous vehicle on the road? And why is the German auto juggernaut moving so incredibly slowly? At first glance, one has the impression that the likes of Mercedes or BMW could easily crush Tesla or Uber in autonomous driving. Both camps have similar technology when it comes to semi-autonomous driving systems (see here and here). Mercedes and BMW have many more cars on the road than Tesla and Uber though, which they could leverage to build up a formidable dataset for self-driving cars. Why then aren’t they charging ahead with their powerful brands, century-old experience and massive development budgets?

II. Germany’s old-school engineering culture at fault

To answer this question, it is useful to rewind the clock. In a marketplace where Tesla’s Autopilot currently dominates the narrative, it is easily overlooked that Germany was once at the vanguard of autonomous driving. Thirty years ago, Mercedes-Benz began cooperating with Ernst Dickmanns — a little known professor at the Bundeswehr University Munich — to build a self-driving car prototype that could react autonomously in real time to road and traffic conditions. In 1986, their proof-of-concept car VaMoRs made its first autonomous drive on a closed strip of the German Autobahn. The 5-tonne Mercedes van — chosen because its huge volume could house the necessary computer equipment — was outfitted with cameras, other sensors and a range of 8-bit Intel microprocessors. A year later, VaMoRs was already able to drive autonomously in highway traffic at speeds of up to 96 km/h (60 mph).

Encouraged by these early successes, the intergovernmental research center Eureka launched the Prometheus Project in 1986, together with the Bundeswehr University, Mercedes-Benz and several other car manufacturers. The goal of the well-funded R&D project was to advance autonomous driving technologies by developing more sophisticated computer vision software. To deal with the incredibly slow computing power of the time, Dickmanns came up with a technique that could estimate spatial positioning and velocity without the need to store previously captured images (“4D-approach to dynamic machine vision”). He also experimented with saccadic gaze control for visual attention focusing and used probabilistic approaches like Kalman filters. Around the same time, researchers at Carnegie Mellon University pioneered the use of neural networks to steer autonomous vehicles.

The Prometheus project achieved its first landmark result in 1994 when two re-engineered Mercedes W140 S-Class vehicles (VaMP and VITA-2) drove autonomously for more than 1,000 kilometers (621 miles) on France’s busiest highway, the Autoroute A1 that connects Paris with Lille. The second landmark was a 1,750 kilometer round trip (1,087 miles) from Munich (Germany) to Odense (Denmark) in 1995. This time around, the cars were equipped with 18 cameras and 70 parallelized microprocessors that performed 850 million operations per second. In comparison, a 2016 Nvidia Titan X GPU gives you a compute of 11 trillion floating-point operations per second. In the light of the available hardware, the results of both trips were impressive. The Mercedes prototypes were able to drive in heavy traffic for long distances without human intervention by using computer vision to recognize, avoid and overtake other cars. The test vehicles achieved speeds of up to 185 km/h (115 mph) and performed autonomous lane changes at over 140 km/h (87 mph). At one point, VaMP drove autonomously for 158 kilometers (98 miles) without a single human intervention.

The fruitful cooperation between Ernst Dickmanns and Mercedes-Benz found its peak over 20 years ago, long before Tesla introduced Autopilot and Uber began their much-hailed test run of self-driving cars in Pittsburgh. On the one hand, it demonstrates that truly disruptive technologies usually take a long time to become an “overnight success.” Apple did not invent the smartphone but eventually reaped most of the profits with the iPhone. On the other hand, this tale is a clear indication that the problem for German premium automakers is not a lack of innovation. Germany accounts for 58% of worldwide registered patents for autonomous driving and spends a ton of money on research (Volkswagen, Daimler and BMW together spend some €22 billion on R&D per year). Rather, it is the overly risk-averse mentality of German car executives that turns out to be the problem. Instead of increasing their technological lead in self-driving cars, the VaMP research car disappeared into the Deutsche Museum in Munich. In the years that followed, German carmakers went on to focus their research on more modest topics like driver assistance (e.g. Automatic Emergency Braking or basic Lane Keeping Assistance).

Today, just as twenty years ago, the strategic dilemma for German carmakers is that they feel that fully autonomous cars might never fit with the self-image of their customer base. This is particularly problematic for the premium automakers. Their brands are inextricably linked to the relationship between the car and the driver — a fact best reflected in BMW’s longtime marketing slogan Freude am Fahren, which translates to “the joy of driving.” The core value proposition of BMW, Mercedes and Audi is Fahrvergnügen (driving pleasure), not mobility. On top of this, carmakers are rightfully cautious when it comes to their reputation. When cars drive themselves, the slightest software glitch can lead to catastrophic results. As VW learned during its emissions-cheating scandal, trust is extremely fragile. It takes years to build a reputation, but only seconds to destroy it beyond repair.

At the technical level, carmakers have also been held back by the old-school thinking of senior management, especially at middle management level. Many auto engineers in Germany were long convinced, for example, that camera-based tasks like scene understanding could only be solved with traditional computer vision approaches (e.g. thresholding and image segmentation combined with decision trees or support vector machines). Traditional computer vision relies heavily on manual feature engineering and selection, whereas the defining characteristic of deep learning is representation learning, i.e. letting algorithms learn the relevant features themselves. Until recently, the prevalent opinion in the car industry was that deep learning was not suited for autonomous cars due to a lack of training data, and the fact that neural networks can be slow to train and difficult to debug. Forward-looking engineers who wanted to switch to deep learning often had to quarrel with senior colleagues who had been working on hand-crafted approaches or unsuccessfully tried neural networks at one point in their careers. This generation conflict might explain why so many startups have popped up in the autonomous driving space recently. Fast-moving startups can start with a blank piece of paper and are not burdened by any legacy approaches imposed by old-school leadership.

With the dawn of the machine learning era, traditional carmakers face enormous difficulties to adjust to this new software model. In particular, they are struggling to attract enough of the right talent. Competition is extremely fierce for specialists in self-driving car development, as demonstrated by GM’s acquisition of Cruise Automation (40 employees) for more than a $1 billion and Uber’s purchase of Otto (91 employees) for $680 million. As a result, the going rate for self-driving talent can be as high as $10 million per person according to Sebastian Thrun. This is an inconceivably high number in the eyes of German car executives. What stands out here is that tech companies in the U.S. tend to offer much higher salaries than their German counterparts. To make matters worse, German universities are not producing enough software engineers and machine learning specialists to satisfy the talent needs of the car industry. Klaus Fröhlich, the Head of Development at BMW, was quoted last year that his company needs to get “manpower equivalent to another 15,000 to 20,000 people from partnerships with suppliers and elsewhere.”

There is also a more systemic reason why Germany has lost its lead in autonomous driving: its outdated regulatory framework. Under the Straßenverkehrsordnung, or German Road Traffic Regulation, (semi-)autonomous driving systems that allow the driver to perform tasks other than driving are currently not admissible. As a result, German manufacturers have been disincentivized to ship advanced driverless features to customers. They were also compelled in recent years to do much of their testing in the United States, which hashed out the legalities of self-driving cars much quicker. Audi, BMW, Mercedes, VW and Bosch concentrated some of their best people in the U.S. to test their cars in road traffic, exacerbating the brain drain of German computer scientists (tech companies regularly poach auto specialists from traditional carmakers). Only recently did the government allow car manufacturers to test their prototypes under real-world conditions on the A9 motorway in Bavaria. Germany is now, finally, rushing to amend its traffic laws to permit self-driving cars. However, the proposed traffic code reform is not expected to be enacted before the federal election in autumn 2017.

On a final note, Germany’s incredibly slow start into the race for self-driving cars might also be attributed to the fact that the country is perceived to be good at pure research but bad at translating technical innovations into marketable businesses. An often-cited example is the invention of the MP3 music format. German researchers at the Fraunhofer-Gesellschaft developed digital compression in the 1980s but it was Asian companies that brought MP3 players onto the market and Apple who eventually created the iPod. The type of innovation where Germany traditionally excels at involves infusing old products and processes with new ideas and capabilities. The German manufacturing industry has long thrived on this type of innovation. It approaches fundamental disruptions with caution. This mentality has served German companies well in the past as many have operated in obscure industrial niches that don’t face existential threats from innovation at the edges. Thriving in a hypercompetitive mass market like autonomous vehicles will, however, require a change of mindset to say the least.

So if autonomous driving requires a cultural shift, what is the counter model to the German car industry? In my opinion it is Tesla, a tech upstart that has claimed the lion-share of media frenzy surrounding self-driving cars. This is despite the company having an insignificant market share globally in the premium car segment. To understand the cultural shift that carmakers have yet to make, it helps to take an intensive look at Tesla’s rollout strategy and examine how the traditional auto industry has reacted to the speed and ambitious risk-taking of Tesla’s founder Elon Musk.

III. How Tesla came to dominate the narrative around self-driving cars

The race to get driverless vehicles on the roads really only intensified in October 2015 when Tesla launched its heavily promoted Autopilot feature. Autopilot is an advanced driver-assistance system (ADAS) that automatically handles steering, braking and acceleration of the Tesla Model S when it is engaged. The system is what the NHTSA and the German Transportation Ministry classify as a Level 2 semi-autonomous driving system, a combination of Lane Keeping Assist and Adaptive Cruise Control. The Tesla Model S is by no means a fully self-driving car (Level 4/5). The car requires constant monitoring by the driver who must be able to assume full control of the vehicle at any time. Although the feature was labeled “public beta” upon launch, Tesla felt that the system was safe for public use, assuming that drivers proceed with caution and due attention when it is engaged.

Unsurprisingly, the traditional car companies see things quite differently. The general consensus among German automakers is that Tesla rolled out its “experimental technology” too soon. Talk to any German car executives these days and they will likely point to Tesla’s self-driving road death in May 2016, arguing that Germany’s go-slow approach may turn out to save their customers’ lives. It is worth noting, however, that installing Autopilot shows a 40% crash rate reduction and is potentially saving more lives than traditional cars do. The critical view of car execs is mirrored by the increasingly technophobic media coverage of self-driving cars in Germany. Der Spiegel, for example, has claimed that Tesla is using “human guinea pigs” to test their cars, while Süddeutsche Zeitung dismissed Autopilot as “the great, dangerous Tesla show.”

These interpretations gravely miss the point. They take the word “beta” literally, translating it to “premature” or “dangerous”. In reality, the word “beta” is best translated as “not perfect”, cautioning users to not get too comfortable with the system. Offering Autopilot as a “beta” technology emphasized the fact that the software was expected to dramatically improve over time by leveraging the power of large-scale, distributed machine learning (“Fleet Learning”) and regular over-the-air software updates.

The results of Tesla’s radical approach are so far astounding: Over a period of just 12 months, the company has collected more than 1.3 billion miles (2.1 billion kilometers) of camera/radar data from roughly 70,000 vehicles that are equipped with Autopilot. To be clear, those are miles driven in Model S cars with Autopilot hardware, not only miles when Autopilot was engaged. In comparison, it took Google’s Waymo seven years to cover 2 million real-world miles (3.2 million kilometers) with its fleet of 58 robo-cars, which are powered mainly by a lidar array on top of the vehicle. Tesla’s approach of crowdsourcing real-world data and continuously improving its machine learning models in real time stands in stark contrast to the approach of Google and German OEMs: localized testing with a limited fleet of test vehicles. An example is BMW’s latest announcement at the Consumer Electronics Show (CES) in Las Vegas. Together with Intel and Mobileye, they plan deploy a fleet of 40 self-driving test cars in the second half of 2017.

These different data collection strategies are creating a growing chasm between Tesla and all other traditional carmakers. The state of the art in machine learning is to expose deep neural networks to large quantities of labeled training data and then incrementally improve the results. Tesla, for example, is using neural networks for its vision, sonar and radar processing software. Building autonomous cars with deep learning requires a large training dataset, ideally gathered from diverse road and weather conditions and from a large variety of real-world locations and driving behaviors. In today’s hypercompetitive market environment, speed is the key, so whoever can put more cars on the road to collect and label the necessary data will have an edge. As with other machine learning applications, data will eventually become one of the primary sources of competitive advantage for car makers. A bigger fleet of self-driving cars will provide access to more data, which in turn makes software performance improvements possible. This could one day become extremely attractive for ride-sharing companies if they can jumpstart a data network effect (where more users → more data → better self-driving capabilities → higher safety → more users).

The speed at which Tesla has updated the functionality of Autopilot over the past months is truly impressive. According to Alex Roy, editor-at-large for The Drive, Tesla’s Fleet Learning “has yielded more software improvements in the 11 months since Autopilot’s initial release than most manufacturers achieve in a traditional 3–5 year model cycle.” Tesla operates at quite a different speed to the car industry and incumbents clearly have trouble adjusting to the new pace. In the words of a Mercedes official: “You can’t compare a 130-year-old company shaped by German engineering ingenuity with a startup from Silicon Valley. It’s a different approach.”

Counterintuitively, what is holding back German auto companies is their emphasis on perfectionism. It is no coincidence that the guiding principle of Mercedes-Benz is Das Beste oder nichts, or “the best or nothing”. However, as Christoph Keese has observed in his latest book Silicon Germany, the opposite of perfectionism is not carelessness. Instead, it means bringing products to market before they are perfect (as long as they are save) and then rapidly iterating in the wild to improve them. This approach to development is even more relevant in the context of machine learning, as described by Mike Schuster, a German research scientist at Google Brain: “The machine-learning mechanism is never perfect. You need to train, and at some point you have to stop. That’s the very painful nature of this whole system. It’s hard for some people. It’s a little bit an art — where you put your brush to make it nice. It comes from just doing it. Some people are better, some worse.” To quote Voltaire, “le mieux est l’ennemi du bien.”

One might argue that Germany is a nation of machine builders, not of software engineers. Guided by the mindset of mechanical engineers, German automakers approach the design of software components the same way they would design an engine component. As a result, Mercedes’ own version of Autopilot, branded as Drive Pilot, “feels like the very best hardware solution to a problem that requires artful software thinking,” as Alex Roy so aptly put it. Although the auto critic called the Mercedes E-class “probably the world’s best ADAS”, he ended up publishing a devastating critique of Drive Pilot, arguing that “it drove like a drunk ten year old, fighting for the wheel with a drunk fourteen year old.” The brittleness of Mercedes’ software can probably be explained, in large part, to the lack of real-world training data. All evidence suggests that Mercedes does not even come close to Tesla’s dataset. It can also be because traditional car manufacturers have independent product teams writing software to control their own parts, thus creating software silos. By contrast, Tesla really excels at bringing all software systems together under the roof within a central operating system that controls every function of the car. This is a major advantage to becoming a full-stack car manufacturer vs. a third-party vendor of driverless car software.

The contrast between Tesla and German OEMs might be best explained by drawing an analogy to the different methodologies of software development: While Tesla’s Autopilot is based on an iterative approach (agile software development), the autonomous driving projects of German companies seem to follow a strictly linear, sequential process (waterfall development). As a result of this mindset, Mercedes does not plan to update Drive Pilot during service (hence no “Fleet learning”). They do not plan to leverage their existing fleet of vehicles to operate Drive Pilot in something like “shadow mode” so the software can be improved offline after downloading data from live driving environments. Neither do BMW or Audi with their planned semi-autonomous driving suites. German car executives believe they can afford to watch Tesla “experiment” with Autopilot before entering the self-driving race once their own technology is ready. I disagree.

Germany’s carmakers claim they are totally committed to winning the driverless car race, but in reality they are trying to accelerate with the handbrake still firmly engaged. While Tesla’s Level 2 system goes on to improve in the hands of its users (and is by now nearly a Level 3 car), Germany’s premium carmakers risk falling further behind. Elon Musk expects the first fully autonomous Tesla to be ready by 2018 (regulatory approval may take 2–3 more years thereafter). German OEMs anticipate fully self-driving cars much later. BMW, for example, reckons that a fully autonomous version of the iNext will be market-ready in 2025. Tesla is thus in the pole position to go autonomous first. That would send ripples through the auto industry because AI systems are, by their very nature, capable of delivering accelerating returns. Can the Germans compete against a rival like Tesla? I am convinced they can if they change their mindsets.

Conclusion

Not all is lost for the German automotive industry. In fact, there are reasons to be optimistic. Germany’s premium carmakers still have control over the industry and direct relationships with consumers. They also enjoy some advantages over tech rivals thanks to their size and operational efficiency. Indeed, it remains an incredibly difficult task to reliably manufacture high-quality cars at mass scale. This is something the Germans are really good at. Tesla, on the other hand, has yet to “cross the chasm”, i.e. transform from serving a base of predominantly early adopters to becoming a profitable full-scale producer as outlined in their 10-year master plan. Nonetheless, there is absolutely no reason for the Germans to be overconfident or complacent. Blackberry and Nokia were in similar positions on the eve of the smartphone age. To avoid their fate, the following things have to change:

• Strategy: I believe that German OEMs must adopt Tesla’s iterative approach to the development of autonomous driving systems, of course without compromising their reputation for quality and safety. They should equip all their cars with networked sensors and, more importantly, gather as much real-world driving data as possible. There is no question that every single one of them could still catch up to Tesla. After all, they can put many more cars on the road than the American automaker. They must, however, begin to take data gathering seriously. Fully autonomous cars will most likely be built on the shoulders of crowdsourced data generated by networked semi-autonomous driving suites, not via localized testing and software updates over traditional 3–5 year model cycles. This is a strategy adjustment that Daimler, BMW or Volkswagen have yet to take.
• Culture: Germany’s carmakers suffer from an old-school engineering culture that is overly hierarchical, discourages risk-taking and is rooted in a mechanical engineering mindset. Competing in the digitized automotive future requires a cultural change throughout the car industry. Fortunately, some car companies are sending encouraging signals, like Daimler whose forward-looking CEO, Dieter Zetsche, is working towards implementing a swarm organization to break down the company’s hierarchy. Others are lagging behind with their digital transformation, like VW which is still grappling with the aftermath of Dieselgate.
• Capital allocation: Electric mobility and autonomous driving are shifting the car’s value away from the powertrain (hardware) to the operating system (software). As a result, carmakers should focus an increasing amount of Capex and R&D on software development. This would, admittedly, mark a break with the past: OEMs have historically spent vast amounts of capital and R&D on proprietary hardware components because this is where their core competence lies. A shift of focus towards software could be overdue though. In recent years, many automotive hardware innovations have been either duplicative across the industry or not really discernible to customers (see slide 10 of this presentation by Sergio Marchionne). If Germany’s carmakers plan to survive the autonomous future, they must develop autonomous driving capabilities in-house (ideally in Germany) and consequently equip their R&D staff predominantly with computer programmers. These structural changes need not be impossibly hard to enforce but they are necessary if carmakers want to avoid becoming the Foxconn of carmakers.
• Branding: The automotive market is split between mobility customers (those who use ride-sharing services) and driving customers (those who still purchase cars). In the next few years, autonomous driving will most likely accelerate the transition from car ownership to shared mobility. This is a problem for Germany’s luxury carmakers as their brands cater predominantly to driving customers. In my eyes, there is really only one way to solve this conflict: car manufacturers have to spin off their mobility products under sub-brands and position these brands for the autonomous future. Mercedes and BMW have taken first steps into that direction with subsidiaries like car2go or DriveNow. However, they are hardly consistent in their branding strategies, as shown by the BMW i sub-brand (the i3 is a mobility play, the i8 is for driving pleasure). Without clear brand positioning, carmakers risk brand dilution.
• Regulation: Autonomous driving is not a field where the government can play catch-up. Unless the pace of regulatory change picks up, Germany risks falling behind in setting global standards for an industry that is strategically important for the economy. Fortunately, first steps have been made in recent months towards a reform of the German traffic code. First, the Ministry of Transport launched an expert roundtable on automatic driving and recently published a white paper called “Strategy for Automated and Connected Driving”. Second, the Ministry of Economic Affairs launched a research project devoted to the standardization of testing and admission for highly automated driver assistance systems. However, these initiatives will hardly be enough to make Germany a front-runner in regulatory matters. No traffic code reform is to be expected before the 2017 federal election and when the reform arrives, it will most likely be half-baked.

Taken together, I believe that the disruption of the car industry holds important lessons for other manufacturing sectors. I am convinced that the self-driving revolution is a harbinger of things to come for corporate Germany. My contention is that many other industries could soon experience their own “Tesla moment” — i.e. an aggressive data-driven tech company like Tesla intruding a hardware market that was traditionally controlled by just a handful of established companies. For Germany, whose manufacturing sector comprises 22.5% of GDP and employs 15 million people, this will pose a challenge. Benedict Evans, among others, is convinced that “it is easier for software to enter other industries than for other industries to hire software people.” As software evolves from a supplementary function in machines to its key component, this shift could pose an existential threat to some Mittelstand companies and large-sized enterprises. I am nonetheless hopeful and optimistic that Germany will overcome the challenges posed by AI.

Thanks to Nathan Benaich, Thibault Févry, Hendrik Brackmann, Karl Kurzer, Philipp Graf and Christian David Martón for proof reading this article.

Germany asleep at the wheel? was originally published in twentybn on Medium, where people are continuing the conversation by highlighting and responding to this story.

Access to high-quality training data is critical for startups that use machine learning as the core technology of their business. While many algorithms and software tools are open sourced and shared across the research community, good datasets are usually proprietary and hard to build. Owning a large, domain-specific dataset can therefore become a significant source of competitive advantage, especially if startups can jumpstart data network effects (a situation where more users → more data → smarter algorithms → better product → more users).

Consequently, one of the key strategic decisions that machine learning startups have to make is how to build high-quality datasets to train their learning algorithms. Unfortunately, startups often have limited or no labeled data in the beginning, a situation that precludes founders from making significant progress on building a data-driven product. It is therefore worth exploring data acquisition strategies from the outset, before hiring the data science team or building up a costly core infrastructure.

Startups can overcome the cold start problem of data acquisition in numerous ways. The choice of data strategy/source usually goes hand-in-hand with the choice of business model, a startup’s focus (consumer or enterprise, horizontal or vertical, etc.) and the funding situation. The following list of strategies, while neither exhaustive nor mutually exclusive, gives a sense for the broad range of approaches available.

Strategy #1: Manual work

Building a good proprietary dataset from scratch almost always means putting a lot of up-front, human effort into data acquisition and performing manual tasks that don’t scale. Examples of startups that have used brute force in the beginning are plentiful. For instance, many chatbot startups employ human “AI trainers” who manually create or verify the predictions their virtual agents make (with varying degrees of success and a high employee turnover rate). Even the tech giants resort to this strategy: all responses by Facebook M are reviewed and edited by a team of contractors.

Using brute force to manually label data points can be a successful strategy as long as data network effects kick in at some point so that humans no longer scale at an equal pace with the customer base. As soon as the AI system is improving fast enough, unspecified outliers become less frequent and the number of humans who perform manual labeling can be decreased or held constant.

Interesting for: More or less every machine learning startup

Examples:
* Many chatbot startups (including Magic, GoButler, x.ai and Clara)
* MetaMind (manually collected and labeled dataset for food classification)
* Building Radar (employees/interns manually label pictures of buildings)

Strategy #2: Narrow the domain

Most startups will try to collect data directly from users. The challenge is to convince early adopters to use the product before the benefits of machine learning fully kick in (because data is needed in the first place to train and fine-tune the algorithms). One way around this catch-22 is to drastically narrow the problem domain (and expand the scope later if needed). As Chris Dixon says: “The amount of data you need is relative to the breadth of the problem you are trying to solve.”

Good examples of the benefits of a narrow domain are again chatbots. Startups in this segment can choose between two go-to-market strategies: They can build horizontal assistants — bots that can help with a very large number of questions and immediate requests (examples are Viv, Magic, Awesome, Maluuba and Jam). Or they can create vertical assistants — bots that try to perform one specific, well-defined job extremely well (examples are x.ai, Clara, DigitalGenius, Kasisto, Meekan — and more recently GoButler/Angel.ai). While both approaches are valid, data collection is dramatically easier for startups that tackle closed-domain problems.

Interesting for: Vertically integrated businesses

Examples:
* Highly-specialized vertical chatbots (such as x.ai, Clara or GoButler)
* Deep Genomics (uses deep learning to classify/interpret genetic variants)
* Quantified Skin (uses customer selfies to analyze a person’s skin)

Strategy #3: Crowdsourcing / Outsourcing

Instead of using qualified employees (or interns) to manually collect or label data, startups can also crowdsource the process. Platforms like Amazon Mechanical Turk or CrowdFlower offer a way to clean up messy and incomplete data using an online workforce of millions of people. For example, VocalIQ (acquired by Apple in 2015) used Amazon’s Mechanical Turk to feed its digital assistant thousands of user queries. Workers can also be outsourced by employing other independent contractors (as done by Clara or Facebook M). The necessary condition for using this approach is that the task can be clearly explained and is not too long/boring.

Another tactic is to incentivize the public to voluntarily contribute data. An example is Snips, a Paris-based AI startup that uses this approach to get its hands on a certain type of data (confirmation emails for restaurants, hotels and airlines). Like other startups, Snips uses a gamified system where users are ranked on a leaderboard.

Interesting for: Use cases where quality control can be easily enforced

Examples:
* DeepMind, Maluuba, AlchemyAPI and many others (see here)
* VocalIQ (used Mechanical Turk to teach its program how people talk)
* Snips (asks people to freely contribute data for research)

Strategy #4: User-in-the-loop

A crowdsourcing strategy that deserves its own category is user-in-the-loop. This approach involves designing products that provide the right incentives for users to give data back to the system. Two classic examples of companies that have used this approach for many of their products are Google (autocomplete in search, Google Translate, spam filters, etc.) and Facebook (users tagging friends in photos). Users are often unaware that they provide these companies with labeled data for free.

Many startups in the machine learning space have drawn inspiration from Google and Facebook by creating products with a fault-tolerant UX that explicitly encourage users to correct machine errors. Particularly notable are reCAPTCHA and Duolingo (both founded by Luis von Ahn). Other examples include Unbabel, Wit.ai and Mapillary.

Interesting for: Consumer-centric startups with constant user interaction

Examples:
* Unbabel (community translators correct machine-generated translations)
* Wit.ai (provided dashboard/API for users to correct translation errors)
* Mapillary (users can correct machine-generated traffic sign detection)

Strategy #5: Side business

A strategy that seems to be particularly popular among computer vision startups is to offer a free, domain-specific mobile app that targets consumers. Clarifai, HyperVerge and Madbits (acquired by Twitter in 2014) have all pursued this strategy by offering photo apps that gather additional image data for their core business.

This strategy is not completely without risk (after all, it costs time and money to successfully develop and promote an app). Startups must also ensure that they create a strong enough use case that compels users to give up their data, even if the service lacks the benefits of data network effects in the beginning.

Interesting for: Enterprise startups/horizontal platforms

Examples:
* Clarifai (Forevery, photo discovery app)
* HyperVerge (Silver, photo organization app)
* Madbits (Momentsia, photo collage app)

Strategy #6: Data trap

Another approach to gather useful data exhaust is to build what Matt Turck has called a “data trap” (Leo Polovets has given this strategy the rather uncharming title “Trojan Horse to collect data”). The goal is to create something that is valuable even without machine learning, and then sell it at a cost to gather data (even if the margin is tiny). In contrast to the previous strategy, building a data trap is a core part of a startup’s business model (not merely a side business).

A case in point is Recombine, a clinical genetic testing company that sells fertility tests to gather DNA data that can then be analyzed with machine learning. Another example is BillGuard (acquired by Prosper in 2015), a startup that offers a mobile app to help credit card users catch and dispute “grey charges.” The app helped BillGuard crowdsource a large amount of fraud data that could then be used for other purposes. One could even argue that Tesla is pursuing this strategy. With over 100,000 (sensor-equipped) vehicles on the road, Tesla is currently building the largest training datasets for self-driving cars (gathering more autopilot miles every day than Google).

Interesting for: Vertically integrated businesses

Examples:
* Recombine (sells fertility tests to gather DNA data)
* BillGuard (offers mobile app to gather fraud data)
* Tesla (collects data when drivers are using the autopilot feature)

Strategy #7: Publicly available datasets

A strategy that many startups have tried — with varying degrees of success — is mining data from publicly available sources. Web archives like The Common Crawl contain petabytes of free raw data collected over years of web crawling. In addition, companies like Yahoo or Criteo have released huge datasets to the research community (Yahoo released a whopping 13.5 TB of uncompressed data!). And with the recent proliferation of publicly available government datasets (spearheaded by the Obama administration), more and more data sources are becoming freely available.

Several machine learning startups have taken advantage of public data. When Oren Etzioni started Farecast (acquired by Microsoft in 2008), he used a sample of 12,000 price observations that he obtained by scraping information from a travel website. Similarly, SwiftKey (acquired by Microsoft in 2016) collected and analyzed terabytes of web-crawled data in the early days to create its language models.

Interesting for: Startups that can identify a relevant public dataset

Examples:
* Farecast (first version scraped information from a travel website)
* SwiftKey (crawled web text to create language models)
* The Echo Nest (crawled millions of music-related websites every day)
* Jetpac (used public Instagram data for its mobile application)

Strategy #8: License third-party data

Another way to access third-party data is to license it, either through an API that is provided by an external data provider, or by implementing SDKs in third-party mobile apps to capture data (ideally with end user consent). In both cases, startups pay an external party for access to data that was generated for one purpose, and then apply machine learning to extract new value from that data.

Farecast and Decide.com (both founded by Oren Etzioni) have successfully pursued this strategy. Open data platforms like Clearbit or Factual are examples of external data providers. Among the companies that use third-party data to mine predictive insights are also several hedge funds and algo-trading firms (which are using non-traditional datasets like satellite data from startups like Orbital Insight or Rezatec).

Interesting for: Startups that rely on third-party data (e.g. industry data)

Examples:
* Farecast (licensed airline data to generate airfare predictions)
* Decide.com (licensed e-commerce data to generate price predictions)
* Building Radar (uses ESA satellite images to monitor construction projects)

Strategy #9: Collaboration with large corporation

For enterprise startups, the data provider can be a large customer that provides access to relevant data. In this setup, the startup sells the customer a solution to a problem (such as reducing fraud) and trains its learning algorithms using the customer’s data. Ideally, the data learnings from one customer or instance can then be transferred to all other customers. Examples in the field of fraud detection are Sift Science and SentinelOne.

The challenge with this approach is to negotiate upfront that the data learnings will be owned by the startup while the data itself remains property of the customer. This can be problematic given that large corporations often have stringent compliance rules and are very sensitive of sharing proprietary data.

Interesting for: Enterprise startups

Examples:
* Sift Science (uses company-specific data to find unique fraud signals)
* SentinelOne (cybersecurity startup that sells endpoint protection software)
* Skytree (develops machine learning software for enterprise use)

Strategy #10: Small acquisitions

Matt Turck lists small acquisitions of companies as a way to get access to particularly relevant datasets (akin to acquisitions of valuable patent portfolios). IBM Watson, for example, has made four data-related acquisitions in 2015, transforming its health unit into one of the world’s largest and most diverse repositories of health-related data.

Since this approach requires the means to fund the acquisition, it might only be viable for startups with substantial funding.

Interesting for: (Later-stage) startups with enough funding

Examples: Hard to pinpoint (data is rarely the only reason for acquisitions)

There are quite possibly other data acquisition strategies which are not mentioned here (if so, please drop me a line). There are also several algorithmic tricks that startups can employ to work around the data problem (such as transfer learning, a technique used by MetaMind for instance).

Whatever strategy you pursue, the key message is: Accessing and owning a large, domain-specific dataset to build high accuracy models can be the single hardest problem that founders need to solve in the beginning. In some cases, it involves finding a quick-fix solution that does not scale, like employing humans pretending to be machines pretending to be humans (as done by many chatbot startups). In other cases, it requires to significantly delay the move from free, limited beta to public release until the benefits of machine learning kick in and customers are willing to pay for the product.

The strategies and examples were drawn from conversations with entrepreneurs as well as several blog posts, including from Nathan Benaich (here), Chris Dixon (here), Florian Douetteau (here), Leo Polovets (here), Matt Turck (here) and Boris Wertz (here).