Launching Artificial Intelligence in Everyday Life : Conversion of Expert Systems to Data Training and Inference
It is common knowledge that our society generates massive amounts of data on a daily basis. Thousands of data are created everyday through diverse sources like social media posts, search queries, messages exchanged, financial transactions, health tracks through smartwatches, weather station updates, traffic camera feeds, etc. What happens to all this data?
From the beginning of civilization, information has been the backbone of knowledge. People recorded information through carving on cave walls, writing scriptures, and authoring books. In this era of technology, all the generated data happens to be the source of immense information providing insight into a society’s finance, economy, healthcare, behavioral patterns and help improve societal aspects.
Such important information cannot stay underappreciated.
Big minds in our society knew the values from these data needed to be utilized in a beneficial way, although they didn’t know how.
Before the mid 2000’s most of the data would be discarded because the traditional databases would crash when they tried to process too much information and the hard drives that stored them were too expensive.
When tech innovators noticed the unused amount of data being wasted, they focused on data storage and engineering.
Companies like Google and Yahoo invented ‘Data swamps’ and ‘Data lakes’ which introduced a novel way to store files. Hence, Google File System (Hadoop — open source) was created which allowed them to unload raw, chaotic internet data across thousands of computers instead of the traditional process of sorting data into neat and expensive tables.
AI winter
Before talking about how this data was used to train an AI model and discussing AI inference let’s understand the AI winter. AI has been subject to funding scarcity, public skepticism, and stagnant commercial interest for roughly 12 to 19 years across two major periods.
In 1950, when Alan Turing first laid the foundational vision for the field of Artificial Intelligence through his paper, ‘Computing Machinery and Intelligence ’ Turing argued that when a human would have a text based conversation with a computer and wouldn’t be able to judge it apart from a human conversation, the machines can be officially considered as intelligent. While this concept of Turing Test laid the groundwork for AI, the early road to achieving it was more difficult than it was anticipated.
Get Tasfiaafrinmim’s stories in your inbox
Join Medium for free to get updates from this writer.
The first AI winter came in 1974 and lasted till the 1980s when early AI pioneers misjudged AI’s potential and took government funds to build complex tasks like language translation. However, the computers of the 1970s lacked the memory and processing speed to handle real world rules. When the technology failed to catch up to the claimed expectations, the funds were pulled back by the government following critical assessment like the UK’s Lighthill report
The situation slightly improved in 1980’s when corporations adopted the Expert Systems. Computer scientists and software engineers wrote thousand lines of manual codes laying out strict rules for specific situations like a giant digital guidebook. However even those approach could not be proven to be efficient. The systems were too brittle. If a customer had a situation that didn’t match the typed in logic, the multi million dollar computer would completely freeze or give an inept conclusion. Additionally, every time a policy was changed, the engineers had to manually find and rewrite the lines of rule code. This method turned out to be slow and really expensive. Meanwhile around 1987, microchip technology exploded and companies like Intel started making standard computer processors that were incredibly faster and cheaper than the existing specialized hardware. These PCs became just as fast at processing the tasks and were cost efficient. Therefore the market of ‘AI machines’ collapsed. High profile AI hardware giants went bankrupt and the industry lost most of its research and development budget. AI was deemed as an overhyped term which was not worth investing anymore. Thus started the official second AI winter.
Rise of Internet and Massive Data Generation
As the research and development of AI remained frozen in universities, the 1990s brought in the consumer internet and suddenly billions of people were found online, transferring and sharing data with each other.
This brings us to our first part of the article, the massive data generation and the need to utilize it.
For decades, during and after the AI winters, the computer scientists remained divided into two groups. The first group believed that computers just needed better logic and faster hardware to process human written rules to be optimized. The second group ( minority ) believed that computers needed to replicate the human brain using neural networks that learned from large datasets. But since neural networks needed huge computational power and big datasets to learn anything, the concept was not taken seriously and deemed to be a fantasy itself.
However in 2012, in the ImageNet competition ( a rough software testing competition ) something interesting happened. In this competition algorithms were given around millions of real life photographs where the algorithms had to identify the objects in the photograph ( for eg. cherry or tomatoes etc). For years experts systems took part in this competition and the best of them could achieve an error rate of approximately 26%. In 2012, a team from University of Toronto entered a data trained neural network named AlexNet . This network was trained on massive, multi layered neural network paired with a powerful gaming microchip that let the computer calculate and understand the mathematical pattern hence processing and recognizing the images at a very high speed.
The results produced by AlexNet was stunning with an unheard margin of an error rate of 15.3%. As documented by How AlexNet Transformed AI and Computer Vision Forever — IEEE Spectrum this incident is known to trigger the modern era in the tech industry.
Almost overnight, the AI community abandoned the rigid rule based expert systems and pivoted towards neural networking and data training. The industry finally understood the vision of Alan Turing couldn’t be achieved through writing thousand lines of instructions but through a machine with fast processor and high computational power to learn directly from the data.
Hence, the era of Data Training started.
Data Traning and AI Inference are very closely connected. AI inference comes in after when a machine is done training on a given dataset and all set to do required calculations, predictions and trend analysis. The word ‘inference’ simply means to derive a conclusion based on evidence and reasoning. In tech, inference refers to the action where a fully trained AI model takes in a piece of new information, runs it through its learned mathematical patterns and generates a prediction, answer, or decision based on those mathematical patterns for the information. Example could include the instance when you use your face to unlock your smartphone. In here, a pretrained model scans your facial structure through live camera and matches it against its locked mathematical template and immediately responds based on if you’re the true owner.
Over the years, AI’s journey has been a living testament of shifting perspectives. For decades, the industry thought the solution lay in the rigid rule based codes and logic for the computers to be smart. But it was simply a small piece of a bigger puzzle. There were additional important pieces such as consumer internet, neural network, creation of distributed data storage that stayed as complete blindspots. When those pieces of puzzle came in vision, the idea became clearer and the innovators realized the missing ingredient wasn’t better human programming but raw, chaotic data.
