This is the last chapter of this book. After discussing all the technical details of recommender systems, we hope you are ready to return to the overall architecture and understand the general knowledge framework from a higher perspective.
Back in Chapter 1, we initially described the technical architecture diagram of recommender systems; we suggested that you may temporarily ignore the details in the diagram and only keep the framework in mind. Now, as the technical details of each module were gradually revealed in subsequent chapters, you may have already filled in the architecture diagram in your own way.
To understand a certain field, it is most important to build your own knowledge framework. Only by establishing a knowledge framework can you check and make up for gaps based on this framework; only then can you see the details and dive deep without ignoring the whole picture. We hope that this book provides not only a specific method to solve your recommender systems problems, but also a high-level technical overview of the industry.
In this chapter, we will review all the technical content of this book in three ways and establish the logical connection between them.
Section 9.1 will look into further technical details based on the architecture diagram in Chapter 1, so that you can establish the final “comprehensive knowledge architecture diagram of recommender systems.”
Section 9.2 will focus on the core recommendation model part of the diagram, and review the model development in a timeline, especially the evolution process of the deep learning model.
Section 9.3 will talk about the core qualities that a qualified recommender systems engineer should possess from a practical perspective.
9.1 The Overall Architecture Diagram of Recommender Systems
Figure 9.1 is a summary framework of the entire book, which echoes Figure 1.4, and supplements most of the technical details involved in this book.
Figure 9.1 The comprehensive knowledge architecture of recommender systems.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Used with Permission.
Apache Hadoop, Hadoop, the Apache Hadoop Logo, Apache Flink, Flink, the Apache Flink Logo, Apache Spark, Spark, the Apache Spark Logo and Apache are either registered trademarks or trademarks of the Apache Software Foundation. Redis is a registered trademark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by Cambridge is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Cambridge.
This diagram can be viewed as a technical index of the entire book. Here, each module and each term in the diagram refers to a specific technical detail. There is a maxim of war that goes, “Until you see the whole battlefield, don’t make a move.” Only by understanding the whole picture can we find the best solution in “one domain” and achieve a true global optimum.
Although we try to summarize deep learning recommender systems in the diagram as much as possible, but “technical solutions are always diverse and cannot be unique.” Figure 9.1 is the enterprise-level recommender system architecture adopted by most companies, and it is not the only “correct” answer. We have our own limitations in information and may have missed some excellent approaches as well. In practical applications, machine learning engineers should take their own objective environment as the starting point to build the most “suitable” rather than the most “correct” recommender system.
9.2 Timeline of Recommendation Model Development
Figure 9.2 summarizes the timeline of the development of recommendation models introduced in this book.
Figure 9.2 Timeline of recommendation model development.
Noticeably, the iterative evolution of deep learning recommendation models has accelerated since 2016. In the meantime, more and more excellent Internet companies have participated, bringing many best practices to the industry.
While writing this book, there must still be many new excellent technologies and models being proposed and applied. The content of this book is static, but the development of technology is dynamic, and you will need to keep track of the cutting-edge content and never stop learning.
9.3 How to Become an Excellent Recommender Systems Machine Learning Engineer
In this section, we will be discussing the basic competencies that are required of an excellent machine learning engineer. This role requires not only good knowledge of machine learning, but also outstanding abilities in all aspects related to business practice.
9.3.1 The Four Competencies for Machine Learning Engineers
Regardless of the specific job requirements, let’s look at this from a higher perspective. The technical competencies of a machine learning engineer can basically be broken down into the following four aspects: knowledge, tools, analytics, and business.
The radar charts in Figure 9.3 roughly show the skills and capabilities required for several positions related to recommender systems and machine learning. It reveals the nuances of the ability requirements of these positions.
Figure 9.3 Radar charts of job skills related to machine learning.
Simply put, any machine learning engineer of recommender systems should meet the minimum requirements of four skills, because before becoming an “excellent” engineer, one should first be a qualified engineer. Not only should you have domain-related knowledge, but you should also have the ability to transform the knowledge into a practical system. One of the authors of this book once interviewed a machine learning engineer candidate who had published some papers and patents related to machine learning. From the perspective of “knowledge” in the field, he was a good candidate, but when asked about his engineering ability, he clearly expressed his unwillingness to write code. Maybe there was something hidden about his reluctance to write code at the time, but for the interviewer, the candidate’s ability to use “tools” was not sufficient, and his ability might be seriously “biased.” This person was, after all, not a qualified machine learning engineer. Generally speaking, the minimum qualifications required for a machine learning engineer should be the following:
Knowledge: Basic knowledge and understanding of recommender systems.
Tools: Ability to write programs and utilize related engineering tools.
Analytics: Basic knowledge of algorithms; ability for analytical thinking and reasoning.
Business: Ability to understand the business applications of recommender systems.
Based on these minimum qualifications, different positions have different requirements for abilities. With the skills radar charts shown in Figure 9.3, the characteristics of the capabilities of different positions are as follows:
Machine learning engineer: The ability requirements of a machine learning engineer are relatively comprehensive. As implementers and applicators of algorithm models, machine learning engineers are required to have a solid machine learning foundation, the ability to improve and implement algorithms, the ability to use tools, and insight into business.
Big data engineer: Big data engineers need to pay more attention to improving big data tools and platforms, and need to maintain the entire data flow related to the recommender system, so the ability to use tools is the most emphasized.
Algorithm researcher: Algorithm researchers are responsible for proposing new algorithms, new model structures, and other research tasks, so they have the highest requirements for knowledge and analytical abilities.
Engineers with “biased” ability: Some engineers don’t pay attention to the accumulation of knowledge in tool use and business understanding. When they are looking for a job, they rush to supplement their knowledge with coding problems. It may be effective in some interview situations, but to become an excellent machine learning engineer, one will also need to make up for their own shortcomings.
Of course, it is somewhat too general to describe the qualifications required for machine learning engineers with only four words: “knowledge,” “tools,” “analytics,” and “business.” So let’s explain these four skills:
Knowledge: This mainly refers to the knowledge and theories related to recommender systems, such as recommendation models, methods of embedding, and so on.
Tools: This is the ability to use tools to apply knowledge to actual business. The tools related to recommender systems mainly include model training tools such as TensorFlow and PyTorch, big data processing tools such as Spark and Flink, and some tools related to model servicing.
Analytics: This is the ability for analytical inference, problem-solving, divergent thinking, and the mastery of general algorithms.
Business: This is the ability to understand the recommender system’s application scenarios and business models, to discover user motivations, formulate corresponding optimization goals, and improve the algorithms.
Please note that skills learning should be targeted to the area that is most needed based on the specific positions and specific projects.
9.3.2 Depth and Breadth of the Capabilities
With a specific job, the ability of an excellent machine learning engineer should be comprehensive – be able to provide solutions with “depth” and “breadth.” For example, the company hopes to improve the current recommendation model, so you are proposing a model improvement plan with DIN as the main structure. This requires you to have a comprehensive understanding of the principles and implementation of DIN in both depth and breadth.
In terms of depth, it is necessary to understand a series of problems, from model motivation to implementation details. An example of a learning path from generalization to specificity is as follows:
What is the motivation behind the DIN model? Is it suitable for the current scenario and data characteristics of your company? (Business understanding)
What is the model structure of the DIN model? What are the engineering difficulties in its implementation? (Knowledge, tool applications)
What is the attention mechanism emphasized by the DIN model? How does the attention mechanism improve the recommendation outcome? (Knowledge, business understanding)
The DIN model generates embeddings for users and products. How is the embedding implemented? (Knowledge, analytical thinking)
Should you implement the DIN model by improving the existing model, or use a new offline training method to train the DIN model? (Tool application, analytical thinking)
What are the potential problems and solutions for online deployment and service of DIN models? (Tool application)
From this example, you can see that forming a complete set of model improvement plans requires the machine learning engineer to have an in-depth understanding of the details of the new model. Without thorough research, the improvement plan could encounter directional errors during implementation, which can be costly to correct.
In addition to an in-depth understanding of the technical details of the solutions, the engineer will also need to understand the pros and cons of various possible alternatives, so as to obtain the optimal solution under the current application scenario through comprehensive considerations. Following this example of model improvement, the engineer also needs to consider the following aspects in breadth:
What are the models similar to DIN, and are they suitable for the current application scenario?
What are the embedding methods used in the DIN model, and what are the advantages and disadvantages of these methods?
What are the methods for training and deployment of DIN models? How will they integrate with your company’s technology stack?
With an in-depth understanding of a technical solution, the understanding of other alternatives can be generalized. But the key points and characteristics of each solution must also be clear, and if necessary, through A/B testing, peer communication and consultation, prototype testing, and other methods to rule out candidates and determine the most suitable option.
In addition, the ability to balance between engineering and research mentioned in Section 6.6 is also one of the indispensable skills for a machine learning engineer. It is an essential ability for one to make a reasonable compromise between reality and ideal in order to accomplish a functional comprehensive technical solution.
9.3.3 Summary of the Competencies Required for Machine Learning Engineers
To become an excellent machine learning engineer in recommender systems, or even an excellent machine learning engineer in general, you should comprehensively improve your abilities in the four aspects of “knowledge,” “tools,” “analytics,” and “business.” Solving a problem will require the engineer to have knowledge and technical capabilities in depth and breadth, make trade-offs and compromises under practical constraints, and finally come up with a feasible and optimal technical solution.
