The Pitfalls of Incorrect Data in Taxonomies, LMIs, Labor Market Forecasts, and HR Analyses

 

In this new episode of our Uncovers Series, we delve into the critical issue of incorrect data in taxonomies, labor market information (LMI), labor market forecasts and predictions, and HR analyses. We explore the challenges posed by the use of unsupervised, unverified data from various sources, which is then applied without critical examination to processes such as matching, gap analysis, labor market forecasts, and demand/supply predictions.

The Quality of Skills and Job Data

In this post, we aim to shine a spotlight on the overall quality of skills and job data that permeate official and commercial taxonomies, language models, and various applications. Often sourced from the internet, this data is integrated into collections and models without rigorous scrutiny, leading to alarming deficiencies in its reliability and accuracy. Our investigation, presented in a revealing video, exposes the unsettling standards adopted by numerous companies and governments in their HR processes and labor market analyses. Despite the gravity of the situation, the video offers a blend of humor and disquiet, providing both insight and entertainment.

The Imperfections of Official and Commercial Data Sources

It is vital to recognize that even esteemed taxonomies and data sources, such as ESCO, CEDEFOP/Eurostat, O*Net, as well as popular collections from Lightcast, Textkernel, LinkedIN, and others, are susceptible to errors. Regrettably, these sources are widely utilized across various processes without comprehensive verification of their accuracy. While these taxonomies were developed with the noble aim of facilitating precise categorization and enhancing labor market research, their implementation has often fallen short of the mark.

Impact on Labor Market Forecasts and HR Analytics

The repercussions of working with inaccurate data, including that sourced from official taxonomies, can be profound. Such data can lead to flawed decision-making, misallocation of resources, and ineffective workforce planning. Furthermore, biased or misleading information can distort insights, resulting in misguided recruiting strategies, inadequate employee development initiatives, and suboptimal organizational performance.

Overcoming the Challenges

To address these challenges, organizations must prioritize data quality and integrity in their collection and analysis processes. Implementing robust data validation and cleansing mechanisms, utilizing multiple data sources for cross-verification, and leveraging advanced data analytics techniques, ideally by labor market experts, can significantly enhance the accuracy and reliability of labor market forecasts and HR analytics. In summary, the poor quality of data and potential errors in official taxonomies present significant hurdles for labor market forecasting and HR analytics. By addressing these challenges and placing a premium on data quality, government labor market organizations and companies can unleash the true potential of data-driven insights for informed decision-making and strategic HR management. For comprehensive insights and solutions, organizations can turn to JANZZon!, the world’s most comprehensive, complete, and hand-curated labor market data ontology, available in over 60 languages and tailored to hundreds of labor markets globally.

A Graph is not a Graph is not a Graph…

JANZZ

The superior power of manually curated knowledge graphs

In various fields, such as data science, biology, social networks, and labor markets, graphs play a crucial role in visually representing data and analyzing complex relationships and patterns. While automated graphs have their advantages, manually curated graphs stand out as more reliable and intelligent due to the human touch in their creation and maintenance. With regulations like the # EU AI Regulatory Act on the horizon, the explainability and interpretability of manually curated graphs are becoming indispensable for compliant use in areas such as labor market data, public employment services, recruiting, and human capital management.

Automated Graphs: The Pros and Cons

Automated knowledge graphs, generated using algorithms and software, offer efficiency and speed in graph creation. They can handle large volumes of data and quickly produce visualizations, making them suitable for tasks that require rapid insights. Furthermore, automated graphs can be helpful for initial exploratory data analysis, providing a quick overview of the data distribution and trends.

However, automated graphs have inherent limitations. They cannot discern contextual nuances and may present misleading visualizations if not carefully monitored. The absence of human intervention in the curation process makes automated graphs prone to errors, especially in interpreting complex data relationships. Moreover, automated graphs may oversimplify or overlook crucial details, leading to inaccurate conclusions and decisions.

Manually Curated Graphs: The Essence of Reliability

In contrast, manually curated graphs are crafted with human expertise, attention to detail, and DOMAIN KNOWLEDGE. The process involves thoughtful consideration of the data, its’ context, and the specific insights sought. As a result, manually curated graphs are more reliable in representing the true nature of the data, capturing subtle patterns, and avoiding misinterpretations.

The human touch in graph curation allows for the incorporation of domain-specific knowledge and expert judgment, ensuring that the visualizations accurately portray the underlying data relationships. Furthermore, manual curation enables the identification and correction of anomalies, outliers, and inaccuracies that automated processes usually overlook. This attention to detail enhances the reliability of manually curated graphs, making them indispensable in critical decision-making processes.

Intelligence Embodied in Manually Curated Graphs

Beyond reliability, manually curated, multilingual graphs exhibit a level of intelligence that automated graphs cannot match. The curation process involves critical thinking, problem-solving, and the application of human intuition, leading to extracting meaningful insights from the data. Human curators can identify patterns that algorithms might miss, recognize outliers that require special attention, and contextualize the data within the broader domain knowledge.

Moreover, the iterative nature of manual curation allows for the refinement and improvement of graphs over time. As new data becomes available or insights are gained, human curators can update and enhance the visualizations, ensuring that the graphs remain relevant and insightful. This adaptability and continuous improvement reflect the intelligence embedded in manually curated graphs, making them valuable assets in dynamic and evolving domains.

The Role of Human Expertise in Graph Curation

The superiority of manually curated graphs stems from the irreplaceable role of human expertise in the curation process. Domain knowledge, experience, and intuition are indispensable in understanding the intricacies of the data and translating them into meaningful graph representations. Human curators can ask critical questions, explore alternative visualizations, and communicate insights effectively, enriching the understanding of the data for diverse stakeholders.

Furthermore, the interpretability of manually curated graphs is a significant advantage, especially in complex or interdisciplinary domains. Human curators can provide context, explanations, and narratives accompanying the visualizations, making the insights more accessible and actionable for decision-makers. This human-centered approach to graph curation fosters transparency, trust, and collaboration, enhancing the overall impact of the visualizations.

Applications and Implications

The reliability and intelligence of manually curated graphs have wide-ranging implications across various fields. In scientific research, manually curated graphs are crucial in presenting findings, supporting hypotheses, and conveying the richness of complex data relationships. In business and analytics, manually curated graphs empower decision-makers with trustworthy insights, guiding strategic planning and resource allocation. In healthcare and medicine, manually curated graphs aid in understanding patient data, treatment outcomes, and epidemiological trends, contributing to improved care and public health interventions.

Furthermore, the emphasis on manual curation highlights the value of human expertise in the era of data-driven decision-making. While automation and algorithms have their place, the irreplaceable role of human judgment, creativity, and intuition in graph curation cannot be overlooked. This realization underscores the need for investment in human-centric approaches to data visualization and analysis, ensuring that the full potential of data is harnessed for the betterment of society.

Conclusion

In conclusion, the differences between automated and manually curated graphs are profound, with the latter emerging as the epitome of reliability and intelligence. As the demand for precise, meaningful, and actionable insights from data and AI applications continues to grow, the importance of manually curated graphs is also increasing, especially in areas where explainability and interpretability are indispensable prerequisites. If you are looking for the largest, multilingual and unique hand-curated knowledge graph in the field of labor market data, let our experts show you what #JANZZon! can offer and how it can address potential challenges with new AI regulations. Keep an eye out for our next post, which will provide insightful comparisons of frequently used graphs in the market.

A rose ≠ is a rose ≠ is a rose – or why matching with skills without a precise level is as good as useless.

JANZZtechnology_a-rose-is-a-rose

Having touched on the subject of skills a couple of days ago in the post Knowledge ≠ Skills ≠ Experience – or why a consistent distinction between these terms is more important than ever, I would like to take a closer look at the topic today.

Job matching is a process that has gained popularity in recent years as a tool to, for example, match individuals with job vacancies based on their skills. While the concept of matching skills seems reasonable, it does not make sense to match skills without knowing the exact level of all explicit and even more implicit skills, especially those that come from previous work experience.

But this is exactly how most systems available on the market work, the vast majority of job boards and aggregators, ATSs, and other career tools on the web and the recruiter side. The reason for this is that none of today’s popular taxonomies and knowledge graphs, such as ESCO, O*Net, Lightcast Open Skills, etc., provide such levels and thus do not provide meaningful differentiation. Together with other shortcomings, such as the still widespread use of keyword matching, and matching without context, which I have highlighted in previous posts, there are almost always incomplete or even incorrect matching results and many other negative implications that result from this inadequate process using outdated technology.

To show that skill matching without validated levels makes no sense at all, here are some examples:

Playing tennis could be found as a skill on my CV. However, we all know that my tennis skills cannot be compared or matched in any way to those of, say, Roger Federer. Even when we talk about the same thing, we mean something completely different. Let’s take another example to illustrate this: I can cook. It’s not all that bad, and it’s good enough for home use. And yet I am miles away from the professional cooking skills needed in a successful restaurant kitchen. I would definitely be thrown out of any kitchen after no more than two days because I would have disrupted the entire well-rehearsed kitchen operation with my incompetence. Not all tennis is the same and not all cooking is the same. And so not all Python programming is the same, and not all plumbing is the same, and not all writing is the same, and so on.

So one of the main reasons why matching skills without knowing the exact level of proficiency is problematic is that it can lead to a mismatch between the job requirements and the candidate’s skills. For example, if a candidate has a skill listed on their resumé but only has a basic knowledge of that skill, or the skill has not been used in the workplace for many years and is therefore no longer up to date, they may not be able to perform the job effectively. Conversely, if a candidate has a skill that is not explicitly listed on their CV, but has advanced knowledge of it, for example through continuous, practical use in work activities, they may be overlooked for a job for which they are well suited.

Another problem with matching skills without knowing the exact level of proficiency is that it can lead to candidates being pigeonholed into certain roles. For example, if a candidate has a particular skill that matches a job vacancy, they may be hired for that job, even if they have other skills that might be better suited to a different role. This can limit the candidate’s career growth and development, as they may not have the opportunity to explore other areas of interest or develop new skills that could be valuable to the organization.

It is also important to note that skills are not the only factor that should be considered when matching candidates with vacancies. Other factors such as personality, work ethic, and cultural fit are equally important and cannot be determined solely based on a candidate’s skills. A person’s work experience can shed light on their personality, work ethic, and cultural fit, which can be crucial in finding the right match.

In conclusion, job matching based on skills without knowing the exact level of all explicit and implicit skills is not an effective strategy. Skills alone do not determine a person’s suitability for a role. It is essential to consider a candidate’s work experience, personality, work ethic, and cultural fit to ensure the right match. Failure to do so will obviously lead to mismatches and missed opportunities for both the candidate and the organization.

So please stop the nonsense and the sadly widespread fixation on skills matching as being so modern and meaningful. Skills are and will remain just one of many relevant dimensions that should be included in job matching and/or any recruitment process. And if it has to be skills matching, then please only with relevant skill levels and even better with as much context as possible. You owe it not only to Roger Federer and all the talented and hard-working professional chefs in the world. Learn more about our products JANZZon! and JANZZsme! and how we can overcome the limitations of today’s inadequate job and skills matching.

Knowledge ≠ Skills ≠ Experience – or why a consistent distinction between these terms is more important than ever.

Knowledge, skills and experience are three crucial components that make up an individual’s competence in any field. Unfortunately, these terms are used interchangeably these days, but they have very different meanings.

Knowledge refers to an intellectual understanding of facts, concepts and theories related to a particular field. It is acquired through education, reading books, attending lectures and participating in training programmes. Knowledge is essential because it provides the basis for developing skills. It enables individuals to understand the why behind a particular practice or procedure. Skills, on the other hand, refer to the ability to perform a task with consistent accuracy and quality. It is the application of knowledge in a practical setting. Skills are developed through practice, repetition and feedback from experienced mentors or supervisors. The more an individual practices a skill, the better they usually become at it. Experience refers to an individual’s exposure to a particular field or area of work. It comes from work experience, internships, volunteering and other practical applications of knowledge and skills. Experience is valuable because it gives individuals a real-world understanding of the challenges they may face. It helps them identify potential solutions to problems and provides opportunities for personal growth and development.

Applied skills, on the other hand, refer to the practical use of skills in a specific job or field. They are skills that an individual has developed through practice and experience and that can be readily applied in real-life situations. Applied skills are essential because they are the only ones that enable individuals to perform their jobs efficiently and effectively. While both skills and experience are essential, experience is always better than skills alone. This is because only experience allows individuals to apply their knowledge and skills in a practical setting. It enables individuals to develop problem-solving, communication and other critical skills that are difficult to learn through reading or training alone. Experience also provides individuals with a deeper understanding of the complexities of a particular field. It makes them more adaptable to change and more likely to succeed in challenging situations. In addition, experience provides individuals with the opportunity to learn from their mistakes and develop resilience.

In summary, knowledge, skills and experience are all essential components that shape an individual’s competence in any field. While skills and knowledge are valuable, experience is always better than skills alone. The practical application of skills and knowledge gained through experience provides individuals with a deeper understanding of their field, problem-solving skills and the ability to adapt to new challenges.

This is precisely why it is so important to distinguish between knowledge, skills and experience when it comes to matching, recruiting and hiring, as each dimension brings unique value to the process. When recruiting candidates, an organisation must consider the specific requirements of the job or position being filled. For example, if an organisation is recruiting for a technical role, proven knowledge/applied skills in a specific coding language and programming may be more important than knowledge of theoretical concepts related to the field.

While knowledge and skills are essential, experience provides the most valuable and relevant insight into the job and the field. For example, a candidate with superb but more theoretical programming skills may not be the best fit for a role if they lack relevant work experience. It is also important to balance the different dimensions in an evidence-based matching and hiring process. Some organisations may place more emphasis on technical skills, while others may focus on soft skills such as communication and teamwork. Therefore, organisations need to have a clear understanding of the specific requirements of the role and the desired qualifications and determine the weighting of knowledge, skills and experience accordingly. At the same time, especially organisations with a strong focus on just hard or job-related skills should not overlook factors such as attitude, cultural fit and potential for growth. These factors can play a significant role in predicting long-term success and retention.

Therefore, a balanced matching and hiring process that considers all these dimensions holistically can help organisations to identify the best candidates for their open positions. Neither an exclusive focus on background, education and thus knowledge, nor on the skills or soft skills that are so widely propagated today, will enable accurate and sustainable artificial or human intelligence-based matching results and thus successful hiring. Let’s start dealing with these terms and dimensions in a more differentiated way, it would only benefit us all.

From guessing to knowing with JANZZilms!: Academic overqualification is one of the main drivers of the intensifying global worker shortage.

JANZZ.technology

Many countries worldwide, especially in emerging labor markets in Southeast Asia, Africa, and Latin America, are facing growing labor market challenges. More and more overqualified workers with academic backgrounds struggle to find work in their field. On the other hand, there is a shortage of skilled workers with technical or vocational backgrounds, leaving many jobs unfilled. Both are costly symptoms of an ever-increasing skills mismatch worldwide.

This trend is a result of several factors. One of the main drivers is the cultural emphasis placed on higher education. For many years, there has been a widespread belief that a university degree is the key to success and financial stability. As a result, many people pursue higher education, often at great expense, hoping to improve their employability and job prospects. However, the labor markets are constantly evolving. Many employers are now filling roles that once required a university degree with workers from technical or vocational backgrounds. Advances in technology and automation have further fueled this shift by redefining many traditional jobs and creating new positions that require highly specialized skills. The shortage of workers with technical or vocational backgrounds is a serious problem for many industries, particularly in construction, craft trades, manufacturing, transportation, and the health and care sectors. These industries require an ever-increasing number of workers with specialized skills that can often only be acquired through experience or training rather than by higher education alone.

Many countries have begun investing in technical and vocational education and training to address this challenge. These investments include funding programs to train students in technical skills, from carpentry and plumbing to computer programming and robotics. But in far too many cases, governments, policymakers, and industries are still too hesitant, programming is piecemeal, and funding is scant. This must change as quickly as possible if we want to alleviate the looming worldwide system-critical shortage in these professions before it is too late.

Overall, the challenge of overqualified workers and a shortage of workers with technical or vocational backgrounds is a complex problem that requires a multifaceted solution. By investing in future-proof, high-quality education and training and reevaluating the cultural emphasis placed on higher education, we all can help ensure that workers improve their employability, thus gaining access to fulfilling careers and contributing to our economy’s continued growth and success.

JANZZilms!, our intelligent integrated real-time labor market management system, pinpoints and quantifies precisely these types of facts at every possible level of granularity. Providing the insights and factual basis needed to initiate, monitor, and continuously improve appropriate action for truly intelligent labor market management.

JANZZilms! – from guessing to knowing.

 

AI, automation, and the future of work – beyond the usual bubbles

In recent years there have been many posts, articles, and reports on how AI and automation will shape the future of work. Depending on the author’s perspective or agenda, these pieces go one of two ways: either the new technology will destroy jobs and have devastating effects on the labor market, or it will create a better, brighter future for everyone by destroying only the boring jobs and generating better, much more interesting ones. ChatGPT, for instance, has spurred unprecedented hype, swinging like a pendulum between extremes of opinions. On the one hand, there have been overwhelmingly euphoric responses, with people fabulating about how we could leverage this allegedly superintelligent technology to enhance our work – from journalism to coding and data analysis to project management or school assignments. On the other hand, many are voicing concerns about potential misuse, including ChatGPT writing malware code and phishing emails, spreading misinformation, disclosing private information, and replacing humans in the workplace. In this post, we want to take a more nuanced view by discussing the most common arguments and claims and comparing them with the facts. But before we get into this, let us first clarify what AI-driven digital transformation is. In a nutshell, it is all about automation, using AI technology to complete tasks that we do not want humans to perform or that humans cannot accomplish – just as we did in the past, in the first, second, and third industrial revolutions.

From stocking looms to AI art

With each of these revolutions came the fear that human workers would become obsolete. So why do we want to automate? Even though in some cases, inventors are simply interested in the feat of the invention itself, more often than not, an invention or development is driven by business interests. As is widespread adoption. And no matter which era, businesses rarely have other goals than staying competitive and raising profits. 16th-century stocking looms were invented to increase productivity and lower costs by substituting human labor. Steam-powered machines in 19th-century mills and factories and farm machinery were used for the same reason. Robots in vehicle manufacture in the second half of the 20th century – ditto. Whether the technology is tractors, assembly lines, or spreadsheets, the first-order goal was to substitute human musculature with mechanical power, human handiwork with machine consistency, and slow and error-prone “humanware” with digital calculation. But so far, even though many jobs were lost to automation, others have been created. Massively increased production called for jobs related to increased distribution. With passenger cars displacing horse-powered travel and equestrian occupations and increasing private mobility, jobs were instead created in the expanding industry of roadside food and accommodation. Increasing computational power used to replace human tasks in offices also led to entirely new products and the gaming industry. And the rising wealth and population growth accompanying such developments led to increased recreational and consumption demands, boosting these sectors and creating jobs – albeit not as many as one may think, as we will see below. However, we cannot simply assume that the current revolution will follow the same pattern and create more jobs and wealth than it will destroy just because this is what happened in the past. Unlike mechanical technology and basic computing, AI technologies not only have the potential to replace cheap laborers, say, with cleaning or agricultural robots. They have also begun outperforming expensive workers such as pathologists in diagnosing cancer and other medical professionals diagnosing and treating patients. These technologies are now also taking on creative tasks such as creative writing, choosing scenes for movie trailers, or producing digital art. We should not necessarily assume the extreme of a dystopian future with fewer jobs and sinking wealth. But we must keep in mind that, in many cases, it is currently more cost-effective to replace expensive workers with AI solutions than cheap laborers such as textile workers in Bangladesh.

So, working towards a more differentiated view, let us look at the currently most common claims and how they stand up to closer scrutiny.

Claim 1. AI will create more/fewer jobs than it destroys

This is the main argument put forth in utopian/dystopian scenarios, including reports by WEF (97m new jobs vs. 85m displaced jobs across 26 countries by 2025), PwC (“any job losses from automation are likely to be broadly offset in the long run by new jobs created”), Forrester (job losses of 29% by 2030 with only 13% job creation to compensate) and many others. Either way, any net change can pose significant challenges. As BCG states in a recent report on the topic, “the net number of jobs lost or gained is an artificially simple metric” to estimate the impact of digitalization. A net change of zero or even an increase in jobs could cause major asymmetries in the labor market with dramatic talent shortages in some industries or occupations and massive worker surplus and unemployment in others. On the other hand, instead of causing unemployment – or at least underemployment, fewer jobs could also lead to more job sharing and shorter work weeks. Then again, although this may sound good in theory, it raises additional questions: How will pay and benefits be affected? And who reaps the bulk of monetary rewards? Companies? Workers? The government? It is admittedly too soon to see the effects of widespread AI adoption on overall employment or wages. But past outcomes, i.e., of previous industrial revolutions, do not guarantee similar outcomes in the future. And even the past ones show that job and wealth growth were not necessarily as glorious as often portrayed. The ratio of employment to working-age population has risen in OECD countries since 1970, from 64% to 69% in 2022.[1] However, much of this increase can be attributed to higher labor participation rates, especially among women. And the increased wealth is certainly not evenly distributed, e.g., in the US.

 

JANZZ.technology

JANZZ.technology

Sources: Illustration 1: Economic Policy Institute, https://www.epi.org/publication/inequality-2021-ssa-data/; Illustration 2: Author’s calculations based on data from Economic Policy Institute, State of Working America Data Library, “Wages by percentile and wage ratios,” 2022.  Updated March 2023
*Low
wage is 10th percentile, middle wage is 50th percentile, very high wage is 95th percentile. No data available for 2020-2022 in the 95th percentile because of issues with significant shares of the population facing BLS’ top-code value for weekly wages.

 

There are simply no grounds to assume that AI and automation will automatically make us wealthier as a society or that the increased wealth will be distributed evenly. We should thus be equally prepared for more negative scenarios and discuss how to mitigate the consequences. For instance, would it be acceptable to treat AI processes like human labor? If so, we could consider taxing them to support the redistribution of wealth or to finance training or benefits and pensions for displaced workers.

In addition, we should question any job displacement estimates on a basic level. Who can confidently say that this job will decline? How can we know what kind of jobs there will be in the future? None of these projections are truly reliable or objective – they are primarily based on some group of people’s opinions. For instance, the WEF’s Future of Jobs Report, one of the most influential reports on this topic, is based on employer surveys. But it is naive to think anyone, let alone a cadre of arbitrary business leaders, can have a confident understanding of which jobs and skills will be required in the future. One should not expect more from this than from fortunetelling at a fair. Just look at the predictions about cars in the early 19th century, remote shopping in the 1960s, cell phones in the 1980s, or computers since the 1940s. So many tech predictions have been so utterly wrong – why should this change now? And yet, these types of crystal ball predictions are a key element in the estimates for the “future of work”.

The fact is, scientifically sound research on this topic is extremely scarce. One of the few papers in this area studied the impact of AI on labor markets in the US from 2007 to 2018. The authors (from MIT, Princeton, and Boston University) found that greater AI exposure within businesses is associated with lower hiring rates, i.e., up until now, AI adoption has been concentrated on substitution rather than augmentation of jobs. The same paper also finds no evidence that the large productivity effects of AI will increase hiring. Some may say that this supports the dystopian view. However, we must also note that this study is based on online vacancy data, and thus the results should be treated with caution, as we explained in detail in one of our other posts. In addition, due to the dynamics of technological innovation and adoption, it is almost impossible to extrapolate and project such findings to make robust predictions for future developments.

And on a more philosophical side note, what would it mean for human existence if we worked substantially less? Work is ingrained in our very nature; it is a defining trait.

Claim 2: Computers are good at what we find hard and bad at what we find easy

Hard and easy for who? Luckily, we do not all have the same strengths and weaknesses, so we do not all find the same tasks “easy” and “hard”. This is just yet another extremely generalizing statement based on completely subjective judgment. And if it were true, then most people would probably consider repetitive tasks as typically easy, or at least easier. This directly contradicts the next claim:

Claim 3: AI will (only) destroy repetitive jobs and will generate more interesting, higher-value ones.

The WEF states that AI will automate repetitive tasks like data entry and assembly line manufacturing, “allowing workers to focus on higher-value and higher-touch tasks” with “benefits for both businesses and individuals who will have more time to be creative, strategic, and entrepreneurial.” BCG talks of the “shift from jobs with repetitive tasks in production lines to those in the programming and maintenance of production technology” and how “the removal of mundane, repetitive tasks in legal, accounting, administrative, and similar professions opens the possibility for employees to take on more strategic roles”. The question is, who exactly benefits from this? Not every worker who can perform repetitive tasks has the potential to take on strategic, creative, and entrepreneurial roles, or program and maintain production technology. It is simply a fact that not everyone can be trained for every role. More satisfying, interesting tasks for intellectuals (such as the advocates of a brighter future of work thanks to AI) may be too challenging for a less intellectual worker whose job – which may well have been perfectly satisfying to them – has just been automated. And not every white-collar worker can or wants to be an entrepreneur or strategist. Also, what exactly does “higher value” mean? Who benefits from this? The new jobs created so far, like Amazon warehouse workers, or Uber and Postmates drivers, are not exactly paying decent, secured living wages. And since the early 1970s, businesses have demonstrated a clear disinterest in sharing the added value from productivity gains with workers:

 

JANZZ.technology

Source: Economic Policy Institute, https://www.epi.org/productivity-pay-gap/

 

On the other hand, a vast number of the AI applications that are already available perform higher- to highly-skilled tasks based on data mining, pattern recognition, and data analysis: diagnosis and treatment of medical conditions, customer service chatbots, crop optimization and farming strategies, financial or insurance advising, fraud detection, scheduling and routing in logistics and public transport, market research and behavioral analysis, workforce planning, product design, and much more. Now add the manifold applications of ChatGPT into the mix. The full effect of these applications on the job market is not yet clear, but they are certainly not only removing mundane, repetitive tasks from job profiles.

Claim 4: We (just) need to up-/reskill workers.

While we certainly do not disagree with this statement in general, it is often brought up as a more or less simple remedy to prepare for the future AI-driven shifts in the labor market and “embrace the positive societal benefits of AI” (WEF). The fact is, this comes with several caveats that make it a far from simple solution.

First, we cannot repeat enough that it is impossible to predict the “future of work” reliably, especially which jobs will be in demand and which will not. Also, based on the effects of the previous industrial revolutions and current research, it is highly likely that the widespread adoption of AI will introduce new jobs with profiles that we cannot anticipate yet. This means we must equip current and future professionals with the skills necessary for jobs we currently know nothing about. One commonly suggested workaround to this issue is to encourage lifelong learning and promote more adaptable and short-term forms of training and education. That is certainly a valid and increasingly popular option. However, there are several aspects to keep in mind. For instance, 15–20% of the US and EU adult population[2] have low literacy (PIAAC level 1 or below), which means they have trouble with tasks such as filling out forms or understanding texts about unfamiliar topics. How can these people be trained to succeed at “more complex and rewarding projects” if they cannot read a textbook, navigate a manual, or write a simple report? In addition, around 10% of full-time workers in the US and EU are working poor.[3] These people typically have neither the time, resources, nor support from employers for lifelong learning and thus no well-informed access to efficient, targeted, and affordable (re)training.

By the time such issues have been addressed, many of these workers may have already missed the boat. In 2018, US employers estimated that more than a quarter of their workforce would need at least three months of training to keep pace with the necessary skill requirements of their current roles by 2022.[4] Two years later, that share has more than doubled to over 60%, and the numbers are similar worldwide.[5] In addition, even before the post-Great Recession period, only roughly 6 in 10 displaced US workers were re-employed within 12 months in the 2000 to 2006 period.[6] In 2019, this rate was the same in the EU.[7] With increasingly rapid changes in skills demands, combined with a lack of time and/or resources for vulnerable groups such as the working poor and workers with low literacy, not to mention lacking safety nets and targeted measures in underfunded workforce development systems, the prospects for these workers are unlikely to improve.

Moreover, the pandemic massively accelerated the adoption of automation and AI in the workplace in many sectors. Robots, machines, and AI systems have been deployed to clean floors, take temperatures or food orders, replace employees in dining halls, toll booths, or call centers, patrol empty real estate, increase industrial production of hospital supplies, and much more within an extremely short time. In the past, new technology was deployed gradually, giving employees time to transition into new roles. This time, employers scrambled to replace workers with machines or software due to sudden lockdown or social distancing orders. This is a crucial difference from the preceding industrial revolutions. Many workers have been cut loose with not enough time to retrain. Similarly disruptive events may well occur in the future – be it another pandemic or a technological breakthrough – and as a society, we need to be prepared for these events and provide affected workers with swift, efficient, and, above all, realistic support.

Claim 5: Employers should view up- and reskilling as an investment, not as an expense.

If a company replaces all its cashiers with robots, why would it want to reskill the newly redundant workers? Even governments have a hard time taking this stance on training and education. Many countries focus primarily on college or other training for young workers rather than retraining job seekers or employees. For instance, the US government spends 0.1% of GDP to help workers navigate job transitions, less than half what it spent 30 years ago – despite skills demands changing much faster than it did three decades ago. And the vast majority of businesses are primarily interested in maximizing profits – that is just how our economy works. Remember: we live in a world where even sandwich makers and dogwalkers may be forced by their employers to sign noncompete agreements to prevent them from getting a raise by threatening to move to a competitor for higher pay.

Well-performing conversational software could enable a company to take a 1,000-person call center and run it with 100 people plus chatbots. A bot can respond to 10,000 queries in an hour, far higher than any realistic volume even the most efficient call center rep could handle. In addition, a chatbot does not fall ill, need time off work, or ask for perks and benefits. They make consistent, evidence-based decisions and do not steal from or defraud their employers. So, if the quality of this software is sufficient and the price is right, there would probably be an uproar amongst shareholders if a company did not go for this offer. After all, a solution that increases efficiency and productivity while lowering expenses is a company’s dream come true. So, if this company doesn’t opt for it, its competition will. And despite the “tech for social good” propaganda we constantly hear from Silicon Valley, most companies are not interested in the future of soon-to-be-ex workers.

Beyond the bubble

The bottom line is that we cannot afford to overdramatize or simply reassure ourselves that there will be enough jobs, or we will constantly be playing catch up. Most of the commonly cited problems or solutions tend to be discussed within the academic or high-income bubble of researchers, tech entrepreneurs, and policymakers, mixed with a substantial amount of idealism. But to get ahead of these developments that – for good or for bad – have vast potential to completely transform our labor markets and society, we need to look beyond our bubble and design realistic strategies for the future based on facts and objective data.

 

[1] https://stats.oecd.org/Index.aspx?DatasetCode=LFS_SEXAGE_I_R#

[2] US: https://www.libraryjournal.com/?detailStory=How-Serious-Is-Americas-Literacy-Problem
EU: http://www.eli-net.eu/fileadmin/ELINET/Redaktion/Factsheet-Literacy_in_Europe-A4.pdf

[3] US: https://nationalequityatlas.org/indicators/Working_poor?breakdown=by-race-ethnicity&workst01=1
EU: https://ec.europa.eu/eurostat/databrowser/view/sdg_01_41/default/table?lang=en

[4] The Future of Jobs Report 2018, World Economic Forum, 2018.

[5] The Future of Jobs Report 2020, World Economic Forum, 2020.

[6] Back to Work: United States: Improving the Re-employment Prospects of Displaced Workers, OECD, 2016.

[7] https://skillspanorama.cedefop.europa.eu/en/dashboard/long-term-unemployment-rate?year=2019&country=EU#1 

The One-Eyed leading the Blind – Part 3: Farewell, Mythical Machine.

JANZZ

This is the third in a series of posts on machine learning in HR tech. If you haven’t already, we recommend you read the other two posts first: part 1 and part 2.

In the last two posts, we discussed the need for domain experts in building a knowledge graph for a job matching engine as well as the problem we want to solve on a conceptual level. In this post, we’re going to delve into the challenges of building a job matching system on a more technical level. (Don’t worry, it’s not going to get too technical – or at least not for long…) We will again focus on a job matching system as an example, but the basic ideas are relevant to many different applications in HR tech.

Based on the discussion in the last post, the goal of our system is to input raw, unstructured data like resumes and job descriptions, process the data fairly and accurately to output the best matches, and explain the results truthfully. For sake of argument, let’s say we’re matching candidates to a job. We won’t discuss potential graphical elements in resumes, say skill levels or section headers, because that’s a huge challenge in and of itself (which, by the way, we’ve recently solved here at JANZZ). Instead, we’ll focus on input data in the form of text.

In the broadest sense, there are two approaches we can take:

  1. Perform matching right on the text data, or
  2. First transform the raw text data into normalized tabular data and then perform matching on the tabular data.

Tabular data is generally not considered avant-garde, but AI-based text processing is hugely popular (pretty much anyone not living under a rock has heard of NLP, GPT-3 or conversational AI). This may be at least part of why the first approach is by far the most common in HR tech: You get to throw out all those fancy words when marketing your products – NLP, deep learning, cutting-edge, yadda yadda. So let’s try and build a matching engine like this.

If you want to perform matching on text data, your system has to deal with actual words. Since digital systems are designed to deal with digits, this means the words have to be translated into digits in some way that is meaningful to the machine. The document text must be turned into an array, or vector, of numbers. In the simplest version, you have a dictionary of all possible words in your resume/job description universe, and a formula that assigns a number to each word in the dictionary based on the document at hand. This could be, for instance, a count of how often the word appears in the document, multiplied with some weight based on relevance or other criteria. So for each document, you get an array (vector) with as many slots (components) as there are words in your dictionary, filled with numbers according to that formula. If, in addition, you want to somehow encode context to better capture the meaning of these words, you might extend your dictionary to include certain sequences of words as well.

Whether you include context or not, a significant challenge of this technique is the sheer number of potential words or phrases that candidates and recruiters can put in their resumes and job descriptions. For instance, there are tens of thousands of standardized skills in collections like ESCO or LinkedIn. And real-life people don’t just use standardized terms. So just for skills, you’ll end up with millions of different expressions, many of which are related to each other to varying degrees. And because the number of expressions corresponds to the number of components in the vectors representing the documents, you end up with huge vectors for each document, causing significant computational challenges down the road. So somehow or other, this complexity needs to be reduced, i.e., we want to condense the information contained in the documents into smaller vectors – but without losing the underlying semantics. This inevitably leads to embeddings; where complex models based on deep neural networks typically perform significantly better than simple models. In this approach, the model decides, based on its training data, how to transform candidate and job profiles into much smaller vectors that live a vector space (think points with coordinates labeling their positions in a three-dimensional space) in such a way that vectors representing similar profiles are close to each other. If you then feed it new profiles, it can embed them in the same way and just look for the ones that are closest together (nearest neighbors). Sounds fairly straightforward, right? Well, it’s not.

For our purposes, one of the key issues with embeddings (and, by the way, with neural networks in general) is their lack of interpretability: the components of the vectors no longer individually correspond to semantic concepts or other directly interpretable distinctions. There have been multiple attempts in the scientific community to address this issue. So far, however, the proposed approaches have proven computationally impractical, resulted in poor performance, or shown very little improvement in interpretability. This means that there is no way of knowing for certain which criteria the system used to determine similarity between two profiles. Instead, we have to make do with post-hoc explanations using additional methods. But all these methods do is perform yet more statistics to determine the most likely explanations for the system’s behavior. And, as studies have determined, different explanation methods often disagree in terms of the resulting explanations, showing that they rarely produce truthful insight into the decisions made by the system. In fact, to quote one study: “The higher the model complexity, the more difficult it may be for different explanation methods to generate the true explanation and the more likely it may be for different explanation methods to generate differently false explanations, leading to stronger disagreement among explanation methods.” This could become a serious liability concern in the not-too-distant future. Or this phenomenon could be exploited to avoid liability. As it turns out, explanation techniques can easily be abused for fair washing, ethics washing, white box washing, or whatever you want to call it. For instance, this study demonstrates that decisions taken by an unfair black-box model can be systematically rationalized through an explanation tool that provides seemingly fair explanations with high fidelity to the black-box model.

On top of that, no matter what modeling method you use, and what explanation method: If you ask why a particular output corresponds to a particular input, the answer depends not only on the mechanics of the particular method, but also very significantly on the distribution of the training set. Which leads us to the next point.

We are asking this system to understand similarity of expressions (e.g. synonyms, near-synonyms), of the underlying concepts (e.g. similar skills, job titles, certifications, and so on), and of the complete profiles. On the level of expressions and concepts, we can certainly use our knowledge graph, which – after reading the first post in this series – we built and curate with domain experts. But for the actual matching, our ML model needs vast amounts of high-quality data to learn the similarities between profiles. Of course, when you hear the vast numbers of resumes or job postings certain providers claim to process, you immediately think there’s ample job-related big data out there to feed into our system. But that’s like saying there are countless images on the internet so you can easily train a system to recognize images of, well, everything. In image recognition, it is well understood that a model that recognizes images of poodles cannot easily be retrained to recognize Venezuelan Poodle moths – even though they do share some similarities…

 

 

The same is true for job or skill similarity. Just like the universe of images, the labor market domain is very heterogeneous. What makes two jobs similar in one case does not necessarily transfer to another. Instead, you need a large amount of data covering similarities in each one of many different areas including niche careers such as ocularists and hippotherapists (if you don’t know what these people do, that’s exactly the point). Because this niche data simply doesn’t exist in the quantity this system needs, we have to find a way to work around this issue. And there are, of course, techniques to deal with small datasets, but they are not easy to implement, and maintaining high quality of data to achieve good results is challenging with any of the current techniques. This is a key reason most job matching systems on the market perform more or less ok when matching software engineers to roles, but fail miserably for occupations with less online coverage, or where the coverage is asymmetric between job postings and online candidate profiles, like blue collar workers in waste management.

In addition, labor markets continuously evolve, even dramatic shifts can happen very quickly on multiple levels, new occupations, new employers, new educations emerge all the time, certain skills become irrelevant, other become more important. These dynamics can cause models to go stale quickly, requiring frequent retraining to maintain performance. This requires not only an endless supply of fresh training data, with all the challenges that entails, and countless hours of highly paid work, but, due to the GPU-hungry processes involved, also comes with a significant carbon footprint.

We set out to build a system that produces fair and accurate results with truthful explanations. But so far, we can’t be sure our system can provide truthful explanations, or that we can continuously feed our hungry system with enough high-quality training data to produce fair and accurate results in all professional fields—at least not without burning through the budget and the planet at a painfully high rate. And then Molnar, author of the highly influential book Interpretable Machine Learning, tells you that in fact, when dealing with small datasets, interpretable models with good assumptions often perform better than black-box models.

Maybe we should try a different approach after all.

Tabular data may not sound very exciting, but it comes with an interesting feature: For tabular data, deep learning models generally do not perform better than simple models.[1] In other words, there are simple models that are at least as accurate as your highly complex, state-of-the-art model. And, unlike a complex model, any one of these simple models can be designed to be interpretable. Of course, we still have the challenge of obtaining and preparing high-quality training data and securing the right people for quality assurance, localization, and so on. The challenges are very similar to those for knowledge graphs discussed in the first post of this series. But at least we can eliminate the issue of unreliable explanations and easily identify and correct any bias. We could, by the way, also just build similarity into our human-curated knowledge graph and skip ML in the matching step altogether. Either way, this leaves us with the – by no means simple – problem of parsing and normalizing raw text data.

Without getting too deep into the weeds, parsing and normalizing text data is a language-based problem that requires a large amount of very careful training and a knowledge graph. With the right combination of natural language processing/deep learning models we can certainly build a powerful parser – provided we can feed it with carefully curated gold-standard training sets.

As for normalization, we need it, and we need high performance: Without good normalization, our matching system will produce less accurate results. For instance, by weeding out good candidates for using terms that even only slightly differ from those in the job description, or by giving higher weight to skills that are written in several different ways within both of the documents. Here’s a real-life example of failed normalization:

 

 

Because the terms “Audit”, “AUDIT” and “AUDITS” are not recognized as the same concept, the weight of this single skill is tripled. In fact, the eleven skills the system claims to have recognized in both the job description and the resume actually only comprise three to four distinct concepts. And even if we made our parser case-insensitive (which, clearly, any decent parser should be), it would not be able to equate the expressions “Certified Public Accountant” and “CPA”. If we now had another candidate who had ten distinct skills matching the job description, that person may rank lower than this candidate. (As a side note, “financial statements” is not a skill. Is this someone who can prepare financial statements? Or audit them? Maybe just read them?)

Of course, if we want to perform normalization, we need something to normalize the terms against. Which is where our knowledge graph comes in. And the better and more extensive the knowledge graph is, the better the normalization. By the way, the vendor from this example claims to provide the most accurate parser in the industry. So much for marketing claims. They also claim to perform knowledge graph-based normalization. Then again, their knowledge graph is built with ML…

So, what have we learnt? When it comes to job matching, complex approaches using the latest ML techniques are not necessarily a good choice. They may be more exciting in terms of marketing, but given the explainability issues, data challenges and – most importantly – the poor results, they are by no means worth the time, money or effort required. A much more promising approach may be to keep the matching step simple and instead focus on accurately processing the input data. For instance, by building a world-class parser using deep tech and gold-standard training sets – i.e., annotated by people who understand the content and context of the data – combined with a knowledge graph built and curated by people who – you guessed it – understand the content and context of the knowledge modeled by the graph. So you still get to play with cutting-edge machine learning and data science tech. But you also get to broaden your horizon and work with people who know about something other than data science and machine learning. And it will lead to better results. Not just “statistically significant single-digit percentage improvements” because, let’s be honest, single digit improvements on appalling is still far from great.

As 2022 comes to a close, maybe it’s time to reflect on how much money and effort has been spent and how little has been achieved since we first started out. And ask yourself if the time has finally come to bid farewell to the mythical beast you have been pursuing for so long.

May 2023 bring you new beginnings.

 

[1] See, for example: Rudin, Cynthia, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. “Interpretable machine learning: Fundamental principles and 10 grand challenges.” Statist. Surv. 16 (2022): 1-85 and Shwartz-Ziv, Ravid, and Amitai Armon. “Tabular data: Deep learning is not all you need.” Information Fusion 81 (2022): 84-90.

 

The One-Eyed Leading the Blind – Part 2: You can’t solve a problem you don’t understand.

“If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and five minutes thinking about solutions.” – Albert Einstein

janzz.technology

 

This is the second in a series of posts on machine learning in HR tech. If you haven’t already, we recommend you read the first post here.

In our last post, we explained why it takes more than data science and machine learning (ML) to build a knowledge graph for a job matching system. In a nutshell, to build a knowledge representation of sufficient quality, you need people who understand the knowledge you want to represent. More generally, if you want to solve a problem as opposed to putting out fires, you need people who understand the problem (and not firefighters). This is key not only to determining a good approach, but to the entire system lifecycle. So many attempts at innovative problem-solving fall flat because a) they target the wrong problem or b) the people involved did not understand the problem. So, before we get into the nitty-gritty of ML-based HR tech in the next post, let’s discuss the problem we want to solve, starting with the all-important results.

The jobs

The discussion that follows is based around a job matching system as an example of HR tech. However, this discussion is highly relevant to any kind of HR analytics.

In a job matching system, at the most superficial level, you want to feed candidate and job profiles into a machine, let the machine do its magic and spit out a list of… well, depending on the problem you want to solve, it could be:

  • a list of candidates that best match a job description
  • a list of jobs that best match a candidate
  • a list of candidates similar to a given candidate
  • a list of jobs similar to a given job

A significant part of understanding the problem is defining similarity. What makes one job similar to another? A first idea might be that two jobs are similar if they have similar or even the same job titles. So, when are job titles similar? Take a look at the following examples:

 

  • Project Manager
  • Customer Manager
  • Facility Manager
  • Software Engineer
  • Software Developer
  • Software Programmer

 

The job titles on the right all share the keyword Manager, but they are clearly not similar in any respect. On the other hand, most of the job titles on the right are similar in some sense. However, whether they are close enough depends on the problem you want to solve, and where the expressions came from. For instance, if you want to use the results of your system for, say, national statistics, these job titles may be close enough. If you want to recommend jobs to a software engineer, that person may not be interested in any of the other jobs, as they only cover a subset of the tasks and skills of a software engineer – in theory. In reality, you will find that the terms software engineer, software developer and software programmer are often confused or used interchangeably in job postings. In addition, even identical job titles often refer to very different jobs: an accountant for a global conglomerate is not the same as an accountant for a small local business; a project manager in the manufacturing department may not perform well if transferred to the marketing department; a carpenter with 18 months of training is probably not as skilled as a carpenter with 4 years of training. Business size, industry, training and many other factors all impact the skill set required or acquired in a given position. So let’s consider jobs in terms of skills.

The skills

We discussed skills in quite some detail in this post. The gist of it is that first of all, there is a lot of talk about skills, but no consensus on the most basic and pertinent question, namely what exactly constitutes a skill, i.e. the definition. Then there is the question of granularity, i.e. how much detail should be captured. The choice is highly dependent on the problem you want to solve. However, your system will typically need to understand skills at the finest level of granularity in any case so that it can correctly normalize any term it encounters to whatever level of detail you settle on. Which leads to the final point: When it comes to skills, we have a highly unfavorable circumstance of projection bias in a tower of babel. Given a term or expression describing a skill, most of us expect that everyone means the same thing when they use that term. In reality, there is significant confusion because everyone has their own interpretation based on their unique combination of experience, knowledge and other factors. We also discussed this in an episode of our podcast: Analyzing skills data. Long story short: There is much work to do in terms of skills and skills modeling.

Now, in an ideal world everyone would speak the same skills language. Realistically, this is simply not going to happen. What we can do is attempt to integrate this translation work within our system. Which is one of the key strengths of a knowledge representation. And we discussed in the last post of this series at length why a good knowledge graph cannot be generated by a system purely based on ML. So let’s suppose for a moment we have a solid definition of skills, the appropriate level of granularity for our problem and we are all talking about the same thing.

Again, we want to find a workable definition of similarity. Working with the premise that a job is determined by its skill set, if you have two jobs with the same skills, then the jobs are the same. This implies that the more skills two jobs have in common, the more similar the jobs. At the other end of the spectrum, two jobs that have no skills in common are not similar. Sound’s logical, right? So, if we feed lots of job postings into our system to analyze them all in terms of skills, our system can identify similar jobs. Easy. Let’s look at an example. The following are all required skills listed in three real-life job postings:

 

  • Job A
  • client centered
  • effective communication
  • planning, organizational skills
  • ability to work under pressure
  • prioritization, timely delivery of care
  • flexibility
  • teamwork
  • critical thinking
  • effective decision making
  • Job B
  • guest focus
  • communication proficiency
  • organizational skills
  • stress management/composure
  • time management
  • flexibility
  • teamwork orientation
  • critical thinking
  • effective decision making
  • Job C
  • sales experience
  • ordained in Nevada a plus
  • planning, organizational skills
  • ability to work under pressure
  • prioritization, timely delivery of care
  • flexibility
  • teamwork
  • critical thinking
  • effective decision making

 

According to our theory – and with an understanding that certain terms have similar meanings, like client centered and guest focus – jobs A and B must be similar and job C quite different. Think again.

 

  • Job A: Registered Nurse ER
  • Job B: Wedding Coordinator
  • Job C: Wedding Planner

 

Ok, but surely our system will discern more accurate and comparable skill sets if we feed it more data. Well, it might – if your system processes that data correctly. So far, these systems typically deliver results like the ones discussed in this post, where 1 in 15 online job postings for refuse workers in Europe apparently require knowledge in fine arts.

The more pertinent question, however, is how useful that would be to the problem you want to solve. As we have explained before, there is no such thing as a standard skill set for a given profession in a global sense. In addition, standardizing skills sets according to job titles is more or less equivalent to simply comparing job titles. So, if you have a matching or recommendation system, this approach will not be helpful. If you want to perform skills analysis, say, to develop a skills strategy for your business, it will not be helpful either. Instead, we need to come up with a viable definition of job similarity that not only consists of a set of features like job titles, skills, industries, company size, experience, education, and so on. The definition must also include aspects such as the importance of each feature depending on the context. For instance, specific training and certification is an indispensable feature of jobs for registered nurses and irrelevant for stocking and unloading jobs at a supermarket. Work experience is not helpful in matching graduates to entry-level jobs, but likely necessary when looking or hiring for a management position. Of course, if you’re designing a purely ML-based system, you probably want to leave the task of determining importance or weighting of the features to the system. However, somewhere along the line, you will (hopefully) want someone to check the results. And it shouldn’t be a data scientist.

The system

Suppose you have found a workable definition of job similarity and you have a good knowledge graph to help your system understand all the different terms used in your job-related data. Now you want to build a matching engine based on ML. Again, you need to first think about the problem at hand. Apart from the semantic aspects discussed above, there are ethical considerations. One thing you will want to avoid for obvious reasons is bias – in any use case. Also, if you want the system to be used in any part of an HR tech stack, it must be both explainable and interpretable. For one thing, there is increasing evidence that legislation will eventually dictate explainability and interpretability in ML-based HR tech (AI Bill of Rights, EU AI Act, Canada’s AIDA, and so on). Arguably more important is the fact that HR tech is used in decisions that strongly affect real people’s lives and as such, must be transparent, fair and in all respects ethical. And even if your system is only going to produce job recommendations for users, explainable results could be an excellent value proposition by building trust and providing users with actionable insights. And with any HR analytics tool, you should want to know how the tool arrived at its decision or recommendation before acting on it. But first, we need to discuss what exactly we mean by interpretability and explainability in an ML-based system, and how the two differ.

These terms are often used interchangeably, and the difference between the two is not always clear. In short, an ML system is interpretable if a human can understand exactly how the model is generating results. For instance, a matching engine is interpretable if the designers know how the various features (job titles, skills, experience, etc.) and their weights determine the match results. An explainable ML system can tell humans why it generated a specific result. The difference is subtle, but key. Think of it like this: Just because the system told you why it did something (and the explanation sounds reasonable), that doesn’t mean that you know why it did it.

In our matching engine example, an explainable system will provide a written explanation giving some insight into what went into the decision-making process. You can see, say, a candidate’s strengths and weaknesses, which is certainly helpful for some use cases. But you don’t know exactly how this information played into the matching score and thus you cannot reproduce the result yourself. In fact, depending on how the system is designed, you can’t even be sure that the explanation given really matches the actual reasoning of the system. It is also unclear whether the explanation accounts for all factors that went into the decision. There could be hidden bias or other problematic features. Apart from the fact that these issues could cause difficulties in a legal dispute, it looks a lot like transparency washing—particularly because most people are unaware of the potential inaccuracy of these explanations.

In an interpretable system, you know precisely how the results are generated. They are reproducible and consistent, as well as accurately explainable. In addition, it is easier to identify and correct bias and build trust. Simply put, an interpretable ML system can easily be made ethical by design.

However, even in an interpretable system you still face the challenge of correctly processing the data. It doesn’t really matter what color the box is, black, white or any other color under the sun. It’s still opaque if your system can’t properly deal with the input data. Because mistakes in the processing inevitably lead to spurious reasoning. To perform any kind of reliable analysis, your system must – at the very least – be able recognize and normalize the relevant information in a dataset correctly. Otherwise, we just end up back where we started:

 

JANZZ.technology

 

Of course, no system is perfect, so the question is how much trash are you willing to accept? In other words, you need benchmarks and metrics. You need experts that are actually capable of evaluating the system and the results. And maybe let’s not set those benchmarks too low. After all, being better than appalling isn’t saying much…

As a fan of ML and data science (which, by the way, we are too – as long as it’s put to good use), you may still want to build a purely ML-based machine that can correctly process the data, feed it into an interpretable system and produce fair and accurate results with truthful explanations that can be understood by humans. So, in the next post, we’re going to dive into the nuts and bolts of this ML-based machine, looking at everything from training data over natural language processing and vector embeddings to bias mitigation, interpretability and explainability. Stay tuned.

The One-Eyed Leading the Blind – Why you need more than data science and machine learning to create knowledge from data

Many job matching and recommendation engines currently on the market are based on machine learning (ML) and promoted as revolutionizing HR tech. However, despite all the work put into improving models, approaches and data over the past decade, the results are still far from what users, developers and data scientists hope for. Yet, the consensus seems to be that if we just get more even more data and even better models, and throw even more time, money and data scientists at the problem, we will solve it with machine learning. This may be true, but there is also ample reason to at least think about trying a different strategy. A lot of very clever people having been working very hard for many years to get this approach to work, and the results are – let’s face it – still really bad. In this series of posts, we are going to shed some light on some of the pitfalls of this approach that may explain the lack of significant improvement in recent years.

One of the key aspects that many experts have come to realize is that because job and skills data is so complex, ML-based systems need to be fed with some form of knowledge representation, typically a knowledge graph. The idea is that this will help the ML models better understand all the different terms used for job titles, skills, trainings and other job-related concepts, as well as the intricate relationships between them. With a better understanding, the model can then provide more accurate job or candidate recommendations. So far, so good. So the data scientists and developers are tasked with working out how to generate this knowledge graph. If you put a group of ML experts in a room and ask them to come up with a way to create a highly complex, interconnected system of machine-readable data, what do you think their approach will be? That’s right. Solve it with ML. Get as much data as you can and a model to work it all out. However, there are multiple critical issues with this approach. In this post, we’ll focus on two of these concerns that come into play right from the start: you need the right data, and you need the right experts.

The data

ML uses algorithms to find patterns in datasets. To discern patterns, you need to have enough data. And to find patterns in the data that more or less reflect reality, you need data that more or less reflects reality.  In other words, ML can only solve problems if there is enough data of good quality and an appropriate level of granularity; and the harder the problem, the more data your system will need. Generating a knowledge graph for jobs and associated skills with ML techniques is a hard problem, which means you need a lot of data. Most of this data is only available in unstructured and highly heterogeneous documents and datasets like free text job descriptions, worker profiles, resumes, training curricula, and so on. These documents are full of unstandardized or specialist terms, synonyms and acronyms, descriptive language, or even graphical representations. There are ambiguous or vague terms, different notions of skills, jobs, and educations. And there is a vast amount of highly relevant implicit information like the skills and experience derived from 3 years in a certain position at a certain company. All this is supposed to be fed into an ML system which can accurately parse, structure and standardize all relevant information as well as identify all the relevant relationships to create the knowledge graph.

Parsing and standardizing this data is already an incredibly challenging task, which we’ll discuss in another post. For now, let’s suppose you know how to build this system. No matter how you design it, because it’s based on ML to solve a complex task, it’s going to need a lot of data on each concept you want to cover in your knowledge graph. Data on the concept itself and on its larger context. For instance, it will need a large amount of data to learn that data scientists, data ninjas and data engineers are closely related, but UX designers, fashion designers and architectural designers are not. Or that CPA is an abbreviation of Certified Public Accountant, which has nothing in common with a CPD tech in a hospital.

This may be feasible for many common white-collar jobs like data scientists and social media experts, because their respective job markets are large and digitalized. But how much data do you think is out there for cleaner/spotters, hippotherapists or flavorists? You can only solve a problem with ML that you have enough data for.

The experts

Let’s suppose (against all odds) that you solved the data problem. You can now choose one of three possible approaches:

  1. Eyes wide shut: Build the system, let it generate the knowledge graph autonomously and feed the result into your job recommendation engine without ever looking at it.
  2. Trust but verify: Build the system, let it generate a knowledge graph autonomously and fine tune the result with humans.
  3. Guide and grow: Build the system and let it generate a knowledge graph using human input along the way.

Based on what’s currently on the market, one wonders if most knowledge graphs in HR tech are built on the eyes wide shut approach. We have covered the results of several such systems in previous posts (e.g., here, here and here). You may also have come across recommendations like the ones below yourself.

 

JANZZ.technology          JANZZ.technology         JANZZ.technology

 

If not, it may be that the humans involved in the process are all data scientists and machine learning experts, i.e., people who know all about building the system itself, instead of domain experts who know all about what the system is supposed to produce. Whether you fine tune the results at the end or give the system input or feedback along the way, at some point, you will have to deal with domain questions like the difference between a cleaner/spotter and a (yard) spotter, what exactly a community wizard is, or whether a forging operator can work as an upset operator. And then of course, all the associated skills. If you want a job matching engine to produce useful results, this is the kind of information your knowledge graph needs to encode. Otherwise you just perpetuate what’s been going on so far, namely:

 

JANZZ.technology

 

You simply cannot expect data scientists to assess the quality of this kind of knowledge representation. Like IKEA said at the KGC in New York: domain knowledge takes years of experience to accumulate—it’s much easier to learn how to curate a knowledge graph. And IKEA is “just” talking about their company-specific knowledge, not domain knowledge in a vast number of different industries, different specialties, different company vocabularies, and so on. You need an entire team of domain experts from all kinds of different fields and specialties to assess and correct a knowledge graph of this magnitude and depth.

Finally, what if you want a knowledge graph that can be used for job matching in several languages? Again, an ML expert may think there’s a model or a database for that. Let’s look at databases first. Probably one of the most extensive multilingual databases for job and skills terminology is the ESCO taxonomy. However, apart from not being complete, it is riddled with mistakes. For instance, one of the skills listed for a tanning consultant is (knowledge of) tanning processes. Sound good, right? However, if you look at the definitions or the classification numbers for these two concepts, you see that a tanning consultant typically works in a tanning salon while tanning processes have to do with manufacturing leather products. There are many, many more examples like this in ESCO. Do you really want to feed this into your system? Maybe not. What about machine translation? One of the best ML-based translators around is DeepL. According to DeepL, the German translation of yard spotter is Hofbeobachter. If you have very basic knowledge of German, this may seem correct because Hof = yard and Beobachter = spotter. But Hofbeobachter is actually the German term for a royal correspondent, a journalist who specializes in reporting on royalty. A yard spotter, or a spotter, is someone who typically moves or directs, checks and maintains materials or equipment in a yard, dock or warehouse. The correct German term would be Einweiser, which translates to instructor or usher in DeepL. There is a simple explanation for these faulty translations: there is just not enough data connecting the German and English terms in this specific context. So, you need an entire team of multilingual domain experts from different fields and specialties to assess and correct these translations – or just do the translations by hand. And simply translating isn’t enough. You need to localize the content. Someone has to make sure that your knowledge graph contains the information that a carpenter in a DACH country has very different qualifications to a carpenter in the US. Or that Colombia and Peru use the same expressions for very different levels of education. Again, this is not a task for a data scientist. Hire domain experts and teach them how to curate a knowledge graph.

Of course, you can carry on pursuing a pure ML/data science strategy for your HR Tech solutions and applications if you insist. But – at the risk of sounding like a broken record – a lot of very smart people having been working on this for many years with generous budgets and no matter how hard the marketing department sings its praise, the results are still appalling. Anyone with a good sense of business should realize that it’s time to leave the party. If you’re still not convinced, keep an eye out for our next few posts in this series. We’re going to take down this mythical ML system piece by piece and model by model and show you: if you want good results anytime soon, you’ll need more than just data science and machine learning.

“Dear passengers, please do care…” On consumer responsibility

janzztechnology_consumer-responsibility

Read the last article in our series on current events in the labor market, written from the perspective of an airport employee. We conclude it by turning to price pressure and the associated issue of consumer responsibility, and illustrate this with the example of the struggling airline industry. In doing so, we also show that this sector and its troubles are ubiquitous and should be of interest to all of us – not just because of its impact on the environment.

 

Dear readers,

Over the past few weeks, I have followed with great interest (and often also a dash of horror) the reporting by various media channels on the ongoing crisis situation in the aviation industry that is caused by a shortage of staff. Even though I did not always agree with everything that was said – especially in the comments sections – I feel that it is very important that this matter is being addressed publicly. (Disclaimer: I myself have been working as a baggage handler in this very industry for many years, have a lively exchange with my work colleagues at the airport and am therefore speaking from actual experience). In said news contributions, I have read, seen or heard almost everything, from management salary analyses to insurance tips for travelers. However, one, in fact obvious, aspect of the whole mess seemed to be missing in the conversation. You may have already guessed it, I’m referring to the role of consumers, who coincidentally (?) often also belong to the target group of cited media. In the following, I would therefore like to say a few words on this subject. Before you now dismiss it, because you do not feel addressed as a non- or non-frequent flyer, I would like to add this preliminary remark: My experience in the airline industry is just one instance of a problem that extends far beyond into other areas. No matter whether we talk about aviation or the import of cheap goods. The luxury of some people always comes at the expense of others. And be aware that we look at this very fact through the lens of the rich developed nation of Switzerland – what it means for other countries, most people do not even want to imagine.

One more preface: I would really like to thank those who publicly speak out for better working conditions and wages in my industry. This verbal support is an important start to improving circumstances in sectors and professions like mine. But it is also just that; verbal and a start. Is it laudable to use your voice to stand up for workers who due to the holiday season have other things to do than argue about long waiting times in the comments? Or to say thank you to them? Absolutely. But unfortunately, at the end of the day, I still get very little out of it if my salary  is based on the (still applicable) crisis collective wage agreement and allows me to live just above the poverty line. In addition, the authors of such statements lose a great deal of sincerity and credibility in my eyes when in real life you still end up encountering them hunting for the cheapest bargain or loudly proclaiming their annoyance over a lost suitcase. It’s like “I really do think that those poor bastards who carry my luggage deserve more. But…”. As soon as they get to feel the pain themselves, solidarity comes to an abrupt end for many people. Even better, of course, are those who don’t even think about people like me and just think that “it’s so cool” to party every weekend in a different European city…

In the current discussion about aviation, you can hear a lot of moaning about the mismanaging, greedy executives of the airlines and airports. Believe me, I probably agree with you on many points there. Of course, a general price increase will not guarantee an automatic positive impact on employees’ wages. But this does not change the fact that most people are not willing to pay higher prices for their flights in the first place – not always just because they could not afford it. After all, I’ve already stumbled over the word “human right” several times in relation to flying nowadays. “Fun” fact from the TV program Kassensturz: In order to cover all the costs actually incurred, air ticket prices would actually have to be at least twice as high. These actual costs refer to the circumstance that the airline industry still benefits from a number of outdated special rights, resulting in reduced costs: Unlike for cars, for example, there is no mineral oil tax for airlines. And unlike train tickets, no value-added taxes are levied on plane tickets.

If the calls for fairer wages from those outside the industry are meant to be sincere, the real cost of air travel in the broader sense should also include the almost universally mediocre pay of many employees in my industry. Since the beginning of the 1980s and the worldwide deregulation of commercial aviation, flying has become steadily cheaper, so it is no wonder that today’s consumers have become very sensitive to cost increases – they are not used to anything other than low-cost carriers. But they should be (or become) aware that the price pressure they induce is often passed on unfiltered to the staff in the air and on the ground. As I said, there can’t only be winners in such equations. Just as a small example: A cabin crew member in Switzerland earns between CHF 3,145 and 4,500 gross, depending on seniority. After social deductions and health insurance, this is hardly enough to live on, especially if you are not allowed to live more than an hour away from the airport because of your job. Let alone to provide for old age. It’s therefore not surprising that it’s becoming increasingly difficult to find people who will work under such conditions. Clearly, it is also up to those “price-sensitive” customers to recognize their responsibility for the consequences of their demands and to redefine their point of view. Not to mention the hypocrisy of certain passengers who claim to environmentally compensate their flight to Bali with a Meatless Monday and some fair fashion…

And one more thing: As is already becoming apparent, it is not only the airline industry’s environmental impact that ultimately affects us all. What the situation at many international airports is showing us right now is that stingy behavior that is neither ecologically nor economically sustainable will ultimately backfire. Sooner or later not only the ‘direct losers’ in the equation will have to bear the resulting costs, but the general public as well. Be it in the form of delays, lost luggage or when taxpayers’ money has to be used to pay the health insurance premium subsidies of underpaid stewardesses and the national pension scheme has ever larger funding holes due to the inadequate salaries. We are also already seeing what the consumers’ reaction to the cancelled flights can bring: many people switch to driving by car (not to trains that would have been more climate-friendly) and traffic jams are currently forming the likes of which we have not seen for a long time. Now whether that’s better, you tell me.

That is why, ladies and gentlemen, I plead for more consumer (co-)responsibility, also in the case of aviation. Of course, it is easier and more pleasant to deny any involvement in problematic developments and to point fingers at others or to describe everything as a structural problem (and therefore unsolvable by the individual). But this does not resolve the issue and is of limited help to the employees directly affected by the crisis. The scenario is similar to that in other sectors of the economy; just think of the food service sector or the fast-fashion paradox. In my opinion, being unaware (or is it calculated ignorance?) does not fully shield you from complicity, and as long as I continue to overhear passengers getting upset about the smallest increases in fees and prices, I cannot take their appreciation and compassion 100 percent seriously. Therefore, dear consumers, please do care, because you will feel the effects of the current mess not only during the next strike.

Mario V., Reader [1]

 

At JANZZ, it is not only important to us that the best job candidates, regardless of location and industry, get the best match with an advertised position, but also that they are compensated appropriately for their work. This is one of the many reasons why we are a trusted partner to an ever-growing number of Public Employment Services (PES) in various countries around the world. We develop evidence-based solutions and have been successfully deploying them since 2010. Our job and skills-matching solutions are fair and non-discriminatory and provide completely unbiased results in accordance with the OECD principles on AI.

Would you like to take a step towards fairer labor markets? Then contact us at info@janzz.technology, via our contact form or visit our product page for PES.

 

[1] Mario V. is a fictional character from our series on current events in aviation and their impact on the labor market. See here for Part 1 and Part 2. Any similarities with real persons, living or dead, are purely coincidental.