AI and Data Research Blog

When Alignment Kills Creativity

Bhoomika Lohana — Mon, 01 Jun 2026 15:33:23 GMT

We were deep into training a creative model. SFT done, reward models built across multiple dimensions of creativity, originality, emotional resonance, narrative structure, and GRPO training underway. The focus had been clear from the start: teach the model what makes creative work actually creative, not just fluent. Score it, reward it, reinforce it.

Then a teammate dropped a paper in our group chat, "Artificial Hivemind: The Open-Ended Homogeneity of Language Models," with a line that made me pause: "The bottleneck is not a lack of capability, but a collapse of creativity. RLHF-aligned general LLMs are architecturally biased against the creative tails of the distribution."

I read the paper. It showed that 70+ current models, across every major family, cluster into two or three output groups on open-ended prompts. Same metaphors, same structures, same safe creative choices.

And that raised a question I hadn't asked before: we'd been focused on making the model more creative, but what if training itself is systematically killing the diversity that creativity requires? What makes us confident ours won't collapse into the same clusters?

I didn't have an answer. So I kept reading.

The paper tested over 26,000 open-ended prompts across 70+ models. When asked to write a metaphor about time, nearly every model, across every family, produced some variation of "time is a river" or "time is a weaver." Not similar responses. Structurally identical ones. GPT, Llama, Qwen, DeepSeek, all arriving at the same place independently.

That was interesting, but what I kept coming back to was a different finding. The paper showed that reward models are miscalibrated on open-ended tasks. At first I wasn't sure what that meant practically. But the more I sat with it, the clearer it became. When two responses are both high quality but creatively different, reward models don't see a tie. They pick a winner. They assign diverging scores to responses that human annotators rate as equally good but simply different. Not randomly either. Systematically. One style gets favoured over another, even when humans collectively say both are fine.

And that got me thinking about our own pipeline. We were using reward models to train through GRPO. If the reward model is the teacher, and the teacher consistently scores one creative direction higher than another even when both are valid, then the model learns to always go that direction. Across millions of examples, that doesn't make the model less capable. It makes it less diverse. The creativity might still be in there somewhere, buried under layers of training that taught it to play safe.

I wasn't sure yet if this was the full picture, but the question had shifted for me. I'd started by asking how to make a model more creative. Now I was asking something different: how do you stop post-training from collapsing the diversity that creativity depends on?

With that question sitting in my head, I started looking into what already existed. Who else had noticed this, and had anyone actually tried to solve it?

That's when I came across three papers, all recent, all connected to each other. Each one referenced the others or built on top of the previous work. And each one tried a different approach to the same problem: how do you train a language model to produce diverse outputs without losing quality?

The first approach I came across was from Meta. DivPO's approach was about data selection. Instead of always picking the highest quality response as the training winner, pick the most diverse high quality response. Pair it against the least diverse low quality one as the loser. The model then learns to prefer outputs that are both good and different.

The results were real. 45% more diverse persona attributes and 74% more diverse stories, without losing quality.

The core insight: the data you train on shapes the output distribution as much as the algorithm does. If every training winner is the safest high-scoring response, the model learns to be safe. If the winners are varied, the model learns to vary.

DivPO also offers an LLM-as-a-judge option for measuring diversity between responses. But this happens during data curation, not during training. They run the judge once, select preference pairs, freeze them, and train with standard DPO. That's an important architectural decision. It makes the LLM judge affordable because it's a one-time cost, but it also means diversity is baked into the data and never updated. As the model evolves during training, what counts as "diverse" shifts, but the frozen pairs can't reflect that.

Follow-up research also pointed out a length bias. The filtering tends to select shorter responses as "diverse" because entropy correlates negatively with length. So the model partly learns to be diverse by being shorter, which isn't the kind of diversity you actually want.

The practical constraint worth noting: DivPO needs to generate around 64 responses per prompt to have a large enough pool for filtering. If your dataset has limited completions per prompt, this approach can't get off the ground.

The second paper, from Midjourney's research team, went a level deeper. Instead of filtering which data the model sees, they modified the loss function itself.

They introduced a deviation score for each training response, measuring how different it is from all other responses to the same prompt using sentence embeddings. That score then multiplies the standard DPO loss. Rare, high quality responses get a louder training signal. Common ones get faded out.

One multiplier on the existing loss. Minimal code change. But their 8B model achieved human-level output diversity while maintaining quality on par with GPT-4o.

The concept of deviation was the real takeaway for me. The idea that you can explicitly tell the training process "learn more from the unusual good outputs, learn less from the typical ones" felt like the right principle regardless of which training algorithm you apply it to.

An important detail: they tried PPO and it failed. The reason matters. PPO requires a reliable reward model that can score outputs in real time during training. For creative tasks, quality is subjective, and the reward model becomes noisy and unreliable. DPO sidesteps this because it learns from pre-labelled preference pairs rather than chasing a live reward signal. If you're working in creative domains, this is a real architectural consideration for choosing between offline and online methods.

They also showed clearly that inference-time fixes don't solve this problem. You can crank temperature up and outputs get incoherent before they get meaningfully diverse. The fix has to happen at the training level.

The limitation: deviation scores are pre-computed and frozen before training starts. The diversity you get is bounded by the diversity that already exists in your training data. The model can't explore beyond what's already there. And computing deviation requires multiple responses per prompt to compare against, which not every dataset has.

The third paper DARLING, Diversity-Aware Reinforcement Learning in Generative Models, brought these ideas into the place where I think they actually belong: online reinforcement learning.

DARLING built directly on top of GRPO. Instead of pre-computing diversity from a static dataset, it computes diversity live during training. At each step, the model generates a group of responses per prompt, and a learned semantic classifier partitions them into equivalence clusters. Not surface-level word differences, but whether two responses are genuinely exploring different directions.

The choice to use a learned classifier instead of embedding distance is deliberate. The Diversified DPO paper measured deviation through sentence embeddings, but embeddings can miss meaningful differences. Two stories with different vocabulary but the same plot would score as "diverse" by embedding distance even though they're creatively identical.

DARLING's classifier is trained to detect semantic equivalence, which is a deeper measure. Though it has its own limitation: the classifier is trained once before RL begins and doesn't update as the model changes, so its notion of "equivalent" stays fixed even as the model's creative range evolves. If a response is unlike most others in the group, it gets a diversity bonus.

The design choice that mattered most: they multiply quality reward by diversity score, not add them.

additive: reward = quality + α × diversity

multiplicative: reward = quality × diversity

This distinction is important. With addition, a very high quality but repetitive response can still get a strong training signal just from its quality score alone. With multiplication, if diversity is near zero, even a perfect quality score gets crushed. The model has to be both good and different. There's no way to game one dimension at the expense of the other.

The result that changed how I was thinking about our own training: diversity didn't trade off against quality. On every benchmark they tested, creative writing and competition math, explicitly optimizing for diversity led to higher quality outputs than quality-only training.

The reason, once I thought about it, made sense. Standard GRPO finds one good response pattern and exploits it. Diversity-aware GRPO is forced to explore different response modes, and in doing so it discovers better solutions that quality-only optimization would never reach because it stopped exploring too early.

What I'm Taking Forward

Each paper left me with something concrete.

From DivPO: data selection shapes the output distribution as much as the algorithm does. Diversity won't emerge on its own, the training signal has to reward it explicitly. Their LLM-as-a-judge approach for measuring diversity during data curation is practical when used as a one-time process.
From Diversified DPO: the deviation concept. Amplify the rare high quality outputs, dampen the common ones. Their method pre-computes deviation from a static dataset, but the principle transfers regardless of where deviation is computed.
From DARLING: compute diversity online inside the GRPO loop, multiply it with quality reward, so the model has to be both good and different. And the finding that diversity actually improves quality through better exploration.

The open question I keep coming back to is how to measure diversity during online training in a way that's accurate, fast, and captures actual creative difference.

Embedding distance is too shallow. Two responses can be far apart in embedding space but creatively identical.
A learned classifier is better but trained once and doesn't evolve with the model.

One direction I want to experiment with: if you already have reward models scoring responses across multiple creativity dimensions like originality, emotional resonance, and narrative structure, the distance between those reward profiles within a generated group might itself work as a diversity signal. Two responses with similar profiles are probably doing the same thing. Two with different profiles are creative in different ways.

Whether that captures the right kind of diversity, and whether it holds up inside a live GRPO loop, is what I'm looking to find out next.

My teammate's message started with a claim: the bottleneck is not capability, it's collapse. Four papers later, that framing feels right. And I think the next model to break out of the hivemind won't be the one with the most parameters or the best benchmark scores. It'll be the one trained to be interesting, not agreeable.

References

Artificial Hivemind: The Open-Ended Homogeneity of Language Models. Jiang et al., NeurIPS 2025 Best Paper.
DivPO: Diverse Preference Optimization. Lanchantin et al., Meta, January 2025.
Diversified DPO/ORPO. Chung et al., Midjourney and NYU, March 2025.
DARLING: Diversity-Aware Reinforcement Learning in Generative Models. Li et al., CMU and Meta, September 2025.

The Impact of Big Data Analytics: Key Findings on Improving Business Efficiency and Decision-Making

Bhoomika Lohana — Sat, 28 Dec 2024 10:49:26 GMT

Introduction:

In today’s competitive landscape, businesses that fail to use big data risk falling behind. This case study explores how companies are staying ahead by transforming raw data into actionable business intelligence. The study titled "Big Data Analytics and its Applications in Improving Operational Efficiency and Decision Making: A Case Study of Central Business District (CBD)" by Tibamwenda Christopher and Kamugisha Nelson investigates the impact of big data analytics on organizations within the dynamic Central Business District.

The objectives of this study are threefold: to examine how big data analytics enhances business intelligence, predictive modeling, and decision-making; to investigate the challenges and best practices for implementing big data analytics solutions; and to assess the impact of big data analytics on operational efficiency, cost optimization, and the creation of new products and services.

Background of the Study:

Advancements in data analytics have fundamentally transformed how organizations approach operational efficiency and decision-making. Traditionally, businesses relied on historical data and static reports, often leading to delayed insights and reactive strategies. For instance, banks and financial institutions used to depend on monthly or quarterly reports to make decisions about loans and investments.

Despite the potential of big data analytics to enhance operations and decision-making, many organizations face significant challenges in its implementation. These include managing vast amounts of diverse data, a shortage of skilled personnel to interpret it, and concerns about data privacy and security.

As a result, many businesses struggle to fully utilize big data, leading to missed opportunities for optimization and improvement. This shift from past-focused to timely strategies enables organizations to respond rapidly to emerging trends and market changes, significantly enhancing their operational efficiency and decision-making capabilities.

Methodology

The research process was divided into two main phases: qualitative analysis and quantitative analysis.
A survey was administered to 150 organizations across various industries that had implemented big data analytics solutions.
The survey aimed to collect data on key metrics, including: The extent of big data analytics adoption, Improvements in operational efficiency, and Changes in decision-making processes.
The collected data was analyzed using statistical techniques to identify patterns and correlations. Descriptive statistics provided an overview of the implementation landscape, while regression analysis was used to assess the relationship between big data analytics adoption and improvements in operational efficiency.

Results:

Table 1: Univariate Data Analysis on the Role of Big Data Analytics.

Description	Frequency	Percentage (%)
Enhanced Business Intelligence	75	50.0
Improved Predictive Modelling	65	43.3
Data-Driven Decision-Making	70	46.7
Implementation Challenges	80	53.3
Best Practices in Analytics Implementation	60	40.0
Improved Operational Efficiency	85	56.7
Cost Optimization	55	36.7
Development of New Products and Services	50	33.3

Findings:

Univariate analysis examines a single variable to understand its distribution, central tendency, and variation. In this study, eight distinct variables related to big data analytics were analyzed. The findings highlight the frequency of responses from organizations regarding each variable:

Enhanced Business Intelligence: Reported by 50% of respondents, indicating improvements in analyzing and interpreting data.
Improved Predictive Modeling: Experienced by 43.3%, showing advancements in forecasting capabilities.
Data-Driven Decision-Making: Adopted by 46.7%, reflecting a shift towards decisions based on data analysis.
Implementation Challenges: Noted by 53.3%, with many organizations facing difficulties during deployment.
Best Practices in Analytics Implementation: Identified by 40%, emphasizing the importance of effective strategies.
Improved Operational Efficiency: The most reported benefit at 56.7%, highlighting significant operational improvements.
Cost Optimization: Observed by 36.7%, indicating some organizations achieved lower costs.
New Product Development: Reported by 33.3%, showing a less common but notable benefit.

Example

A retail chain utilized big data analytics to enhance inventory management. By analyzing customer buying trends, they improved their ability to forecast demand, which helped prevent both overstock and shortages. This approach led to better operational efficiency and cost savings. Additionally, they discovered new product opportunities based on customer preferences. However, the company faced challenges, such as integrating data from various sources and training employees on the new system.

Table 2: Bivariate Data Analysis on Enhanced Business Intelligence and Improved Predictive Modelling

Predicted variable	Predictor variable	Frequency	Count
Enhanced Business Intelligence	Improved Predictive Modelling	Yes	55
Enhanced Business Intelligence	No	15
Not Enhanced	Improved Predictive Modelling	Yes	10
Not Enhanced	No	70

Findings:

Bivariate analysis examines the relationship between two variables to uncover patterns or associations. In this case, we’re analyzing the connection between Enhanced Business Intelligence (predicted variable) and Improved Predictive Modelling (predictor variable).

Enhanced Business Intelligence and Improved Predictive Modelling: The table reveals that among organizations reporting enhanced business intelligence, 55 also experienced improved predictive modeling, while 15 did not. This suggests a strong positive relationship: organizations with better business intelligence are more likely to achieve advanced predictive modeling capabilities. Conversely, among organizations without enhanced business intelligence, only 10 experienced improved predictive modeling, while 70 did not. This indicates that the benefits of predictive modeling are less prevalent among organizations lacking enhanced business intelligence.

Overall, the data indicates that improved business intelligence is associated with better predictive modeling, emphasizing a beneficial link between these two aspects.

Example:

Consider a retail company that adopted advanced predictive analytics to forecast customer purchasing patterns. Improved Predictive Modelling (predictor variable) enabled the company to anticipate trends more accurately, which in turn enhanced their Business Intelligence (predicted variable) by integrating these forecasts into their inventory and marketing strategies.

In contrast, another retail company used predictive analytics but didn’t adjust its inventory or marketing plans based on the insights gained. As a result, they saw limited improvement in their business intelligence. This demonstrates that while improved predictive modeling can enhance business intelligence, effective application and integration of these insights are crucial to achieving significant benefits.

Table 3: Shows the Results of Bivariate Analysis based on Table 2.

Test	Chi-Square Value	p-Value	Fisher’s Exact Value	p-Value (Fisher’s)
Chi-Square Test	29.00	< 0.001	15.30	< 0.001

Chi-Square Test: Determines if there is a significant association between two categorical variables by comparing observed and expected frequencies. The Chi-Square value is 29.00 with a p-value of < 0.001. This indicates a highly significant relationship between Enhanced Business Intelligence and Improved Predictive Modelling. The low p-value (< 0.001) suggests that the observed relationship is unlikely to have occurred by chance, confirming a strong association between the two variables.

Fisher’s Exact Test: Assesses the significance of the association between two categorical variables, especially useful for small sample sizes. The Fisher’s Exact value is 15.30 with a p-value of < 0.001. The low p-value here also indicates a significant relationship between Enhanced Business Intelligence and Improved Predictive Modelling

Table 4: Multivariate Analysis on the Predictors of Improved Predictive Modelling

Predictor Variable	Odds Ratio	t-Value	p-Value	95% Confidence Interval
Investment in Analytics Technology	2.50	3.00	0.004	[1.45, 4.30]
Employee Training on Analytics	1.80	2.20	0.03	[1.05, 3.12]
Organizational Culture Support	1.60	2.00	0.05	[1.00, 2.60]

Findings:

Multivariate analysis explores how multiple variables collectively influence an outcome, revealing complex interactions that single-variable analyses might miss. In this case, the analysis of predictors for improved predictive modeling included three factors:

Investment in Analytics Technology: This factor has the highest odds ratio of 2.50, showing it greatly increases the likelihood of improved predictive modeling. This suggests that investing in advanced analytics tools is a strong driver of enhanced forecasting capabilities.
Employee Training on Analytics: With an odds ratio of 1.80, this factor also positively impacts predictive modeling, though its effect is slightly less significant compared to technology investment. Effective training equips employees with the skills needed to leverage analytics tools more effectively.
Organizational Culture Support: This predictor has the lowest odds ratio at 1.60, indicating it contributes to better predictive modeling, but its impact is the smallest among the three factors. A supportive culture is beneficial but less influential compared to technology and training.

All predictors are statistically significant, with p-values below 0.05, underscoring their importance in enhancing predictive modeling.

Conclusion:

The study shows that big data analytics can greatly improve how organizations operate and make decisions. Companies that use big data effectively see better business insights, forecasting, and decision-making. While there are challenges in managing and integrating data, those who overcome these obstacles see significant gains. The link between better business intelligence and improved forecasting is clear, and key factors for success include investing in technology, training employees, and having a supportive culture. In summary, organizations that focus on data and invest in the right tools and skills can streamline their operations, gain a competitive advantage, and reach their goals.

Personal opinion:

Businesses that embrace enhanced business intelligence have seen significant improvements in predictive modeling, leading to better operational efficiency and smarter decision-making. However, for organizations that haven't yet adopted big data, here are key areas to focus on:

Successful big data analytics isn't just about technology—it needs an organization to be flexible. Leaders play a key role in creating a data-driven culture by motivating teams to use new technologies and decision-making methods. Without strong leadership to guide this change, resistance can reduce the benefits of data analytics.
In today’s fast-changing tech world, closing the skill gap is crucial for organizations to use big data effectively. Ongoing learning and training programs should aim to improve data skills at all levels, so employees can understand and use data to meet the organization’s goals.
Big data offers useful insights, but organizations must focus on using it ethically, especially with data privacy. By following regulations and taking steps to secure data, businesses can build trust with customers and stakeholders. This ensures they can benefit from analytics without compromising ethical standards.
Many organizations face challenges like limited budgets, outdated infrastructure, or a lack of skilled staff, making it hard to fully use big data. Recognizing these limits helps set realistic expectations, as it may take time and resources to address these issues.
Big data should be viewed as a long-term strategy, not a quick fix. By constantly updating analytics approaches and aligning them with future goals, businesses can stay competitive and get the most from their data investments.

ResearPaper: https://www.researchgate.net/publication/367590982_The_Role_of_Big_Data_Analytics_and_Decision-Making_in_Achieving_Project_Success

Research Insights: How AI and Big Data are Shaping the Future of Data Analytics

Bhoomika Lohana — Sat, 28 Dec 2024 10:43:52 GMT

From chaos to clarity—AI and Big Data are transforming complex datasets into actionable strategies. As organizations generate and accumulate ever-growing volumes of data, traditional analysis methods frequently struggle to keep pace with the complexity and scale of these datasets. Conventional approaches often fall short of uncovering detailed patterns and actionable insights, leaving businesses at a disadvantage.

For instance, a hospital collects extensive data on patient symptoms, treatments, and outcomes. Traditional methods might only identify broad health trends, missing subtle indicators of emerging health issues. Advanced techniques can analyze this complex data to uncover patterns that help doctors predict potential health problems before they arise.

In exploring these advancements, I drew insights from the article: AI and Big Data: Leveraging Machine Learning for Advanced Data Analytics’ by Xiang Chen offers valuable insights. It highlights how advanced algorithms such as deep learning and reinforcement learning are enhancing the ability to process and analyze large datasets.

Introduction

The future of business is data-driven. But with data growing at unprecedented rates, how are companies leveraging AI and machine learning to stay competitive?

As organizations collect more data than ever before, traditional methods of analysis are no longer sufficient. The integration of AI and machine learning is transforming the way businesses extract value from Big Data, enabling them to uncover insights, predict trends, and make smarter decisions in real-time.

For instance, imagine a retail company trying to understand customer buying habits. Using traditional tools like spreadsheets, they can only analyze small amounts of data, which leads to slow insights and missed opportunities. As more data comes in from online sales, store purchases, and customer reviews, these old methods can’t handle it all, leaving the company with outdated information. This makes it hard to offer timely recommendations or spot new trends in what customers want.

Challenges and Solutions of Big Data

As organizations deal with the vast complexities of Big Data, they face significant challenges due to its key characteristics: volume, variety, and velocity. Traditional methods often fall short in addressing these issues. However, the integration of Big Data with AI and the use of machine learning are providing powerful solutions to these problems, transforming how we handle and analyze large datasets.

Volume: The sheer amount of data generated by organizations—ranging from social media interactions to sensor data—creates a challenge for traditional storage and processing systems, which struggle to handle such scale.

Solution: To address this, distributed computing systems like Apache Hadoop and cloud-based platforms (e.g., AWS, Microsoft Azure) provide scalable storage and processing capabilities. These systems allow for dynamic resource allocation, enabling organizations to efficiently store and analyze vast datasets without sacrificing performance.

Variety: Big Data comes in many forms, from structured tables to unstructured text, images, and videos, which complicates integration and analysis. Traditional systems aren’t built to handle this level of diversity.

Solution: To manage variety, advanced data processing techniques such as natural language processing (NLP) for text and image recognition for visual data help convert unstructured data into analyzable formats. Data integration tools further ensure that different types of data can be harmonized, allowing for more comprehensive insights.

Velocity: The speed at which data is generated—particularly in fields like finance or IoT—requires real-time or near-real-time processing. Traditional systems often can’t keep up with the rapid influx of data, leading to missed opportunities.

Solution: Real-time data streaming platforms such as Apache Kafka, combined with real-time processing tools like Apache Spark Streaming, enable businesses to process data instantly. These technologies ensure that data is analyzed as it is generated, supporting fast and accurate decision-making.

Approximately 402.74 million terabytes of data are created each day.
181 zettabytes of data will be generated in 2025.
Video is responsible for over half (53.72%) of all global data traffic.

Deep Learning for Complex Data

One of the most transformative aspects of AI is deep learning, particularly for unstructured Big Data. Deep learning models like recurrent neural networks (RNNs) or long short-term memory (LSTM) networks excel at time-series data and pattern recognition.

For example: Imagine predicting daily stock prices. An LSTM network can analyze past stock prices to understand patterns and trends over time, allowing it to forecast future prices more accurately. By learning from historical data, the LSTM can make better predictions about future stock movements, helping investors make informed decisions.

Future Trends and Implications

Several emerging trends in AI, Big Data, and machine learning are set to transform data analytics. These trends promise to drive innovation, improve capabilities, and address current challenges:

1. Advancements in AI: AI is getting smarter, with models like transformers now leading the way in understanding and generating text

2. Edge Computing: Edge computing processes data closer to its source, reducing delays.

3. Explainable AI (XAI): This makes AI decisions more transparent. For example, in healthcare, XAI can show why a model predicts a certain diagnosis, helping doctors trust and interpret AI recommendations better

4. Automated Machine Learning (Auto ML): Simplifies model creation and makes advanced analytics accessible. For instance, tools like Google (Auto ML) allow users to build models without needing deep technical expertise, making advanced analytics accessible to more businesses.

These trends are driving innovation and improving how we handle and analyze data, paving the way for smarter, more efficient solutions in the future.

Conclusion

In summary, the integration of AI and Big Data through machine learning marks a transformative advancement in data analytics. By leveraging machine learning algorithms, organizations can efficiently process and analyze vast and complex datasets, uncovering valuable insights, predicting trends, and making more accurate data-driven decisions. This powerful combination addresses the inherent challenges of volume, variety, and velocity in Big Data while driving innovation and operational efficiency across multiple industries. As these technologies continue to evolve, they will further enhance the capabilities of data analytics, shaping the future of informed decision-making and strategic planning.

Personal Opinion

The combination of AI, Big Data, and machine learning is a game-changer for data analytics. These technologies help companies process huge amounts of data quickly, find useful insights, and make smarter decisions, which is crucial in today’s fast-paced world. AI can also handle repetitive tasks, giving people more time to focus on important and creative work.

But there are challenges too. Data privacy is a big concern, and many AI systems can be hard to understand. This is especially important in areas like healthcare, where knowing how decisions are made is vital. Without the right protections, there’s a risk of data being misused.

Overall, while the benefits are huge, companies need to use AI responsibly. Clear rules and ethical practices will be needed to make sure we get the most out of these technologies without putting privacy or trust at risk.

Paper link: https://academicpinnacle.com/index.php/acs/article/view/230

What Skills Pay More? Insights into Evolving Skill Demand and Wage Returns for Professional Workers

Bhoomika Lohana — Sat, 28 Dec 2024 10:21:20 GMT

In a world where rapid advancements in AI and Technology are reshaping industries and accelerating shifts in the labor market, understanding how these changes are redefining the skills rewarded today is crucial for both workers and employers. The study adopted an approach focusing on the research of the evolving nature of work and the changing demand for skills. It examined two key periods: 2014-2015, marking the onset of the fourth industrial revolution, and 2018 to the first quarter of 2020, summarizing its progression using job flow data of professionals in the United States.

The study aims to analyze how the value or price of specific skills has evolved and how the need (or Demand ) for these skills has shifted in the labor market For example: in the early 2000 knowing the use of Microsoft Office was a valuable skill for many office jobs, but now such skills are considered basic and expected in most professional roles, with no or little wage premium, instead advanced skills in cloud computing, big data analytics or AI are in high demand and command much higher wages.

The data on demand for skills was obtained from online job advertisements posted on Linkup, focusing on ads for professional jobs, which made up 52.5% of all the job ads they studied. The keywords for skills were developed by filtering job ads and conducting a literature review. The initial list was narrowed to the most relevant skills and then expanded by adding related skills using pointwise mutual information (PMI) to identify strong associations between skills. Synonyms were grouped into categories, resulting in 166 keywords. Principal Component Analysis (PCA) was then used to analyze patterns among these keywords, reducing them to 9 distinct skill groups.

These 9 skills were categorized into cognitive and non-cognitive groups. Of the 9, 2 were classified as non-cognitive skills: collaborative leadership and interpersonal & organized skills. The remaining 7 were cognitive skills: big data, Cloud computing, Programming, Machine learning, Research, Mathematics, and Analytical skills. Between 2014 and Q1 2020, a total of 1.3 million job postings were analyzed using Bidirectional Encoder Representations from Transformers (BERT), a machine-learning method for natural language processing. BERT classified the job descriptions into five categories: responsibilities, skills, education, legal requirements, and others.

It highlights that the non-cognitive skill 'collaborative leadership' (including attributes such as strategic thinking, leadership, influence, collaboration, creativity, negotiation, and coaching) increased from 50.14% in the first period to 61.07% in the second period. Similarly, the 'interpersonal and organizational' skill rose from 25% in the first period to 30% in the second period.

The study also analyzed the interaction between each skill with the other 8 skill groups to analyze the nonlinear returns to skills groups and their interaction refers to studying how the relationship between skills and earning is more complex than a simple direct increase or decrease.

The table provides a snapshot of the interactions for 'collaborative leadership' and 'research' skills. Such as the interaction between 'collaborative leadership' and 'research' grew by 3.7 percentage points, from 9.5% to 13.2%. This highlights how, with increasing automation, the complementarity between social skills (such as collaborative leadership) and cognitive skills (like research) is growing. For instance, financial analysts now need to combine technical skills with strong collaboration skills including teamwork and communication skills. While algorithms handle the heavy data lifting, analysts must interpret the results and explain them to clients.

To align the skills group data with wage data, the job data obtained from Linkup was matched with wage data from the U.S. Bureau of Labor Statistics (BLS), organized by Standard Occupation Classification (SOC) codes. This enabled the study to link job postings with corresponding wage data for the same occupation and location.

The table shows the top 5 and bottom 5 occupations occurring in the field of research such as the share of skills of Computer and Information Research scientists rose to 69% from 1^st period to 76% in 2^nd period, showing the rise in hourly wages from $59.67 to $62.98.

In this study, the researchers first analyzed how the share of each skill group correlates with hourly wages, using a regression model that controlled for factors like education, experience, industry, and state demographics, and to explore how interactions between the nine skill groups influenced wages, focusing on how these skills might complement one another. To do this, they used a Lasso regression model, which selects the most relevant variables by minimizing prediction errors. This deeper analysis revealed how skill combinations, especially in the context of automation and technological change can impact wage growth. By examining these interactions across two key time periods, the study sheds light on how the evolving demand for skills is shaping the future of work.

In the results, we observe a critical connection between skills and wages. Non-cognitive skills, such as collaborative leadership and interpersonal & organizational skills, show different impacts on wages. Collaborative leadership skills have gained importance, leading to a 0.3% wage increase for a 10% rise in these skills by 2020. In contrast, the interpersonal & organizational skill group was associated with a 0.73% drop in wages. For example, time management, which is part of this skill group, has seen declining demand as automation reduces the need for such skills.

When it comes to cognitive skills, the demand for advanced tech skills like machine learning saw the most significant wage increase, with a 10% rise in this skill boosting wages by 5.83% between 2018 and Q1 2020. We also found that the return on data science skills is constantly evolving, but the legacy skills associated with it, such as ‘big data,’ initially contributed to a 1.85% wage increase in 2014-15. However, by 2018-20, these same skills led to a 1.21% decrease in wages, highlighting how quickly technology evolves and the importance of staying current in the fast-paced digital economy.

CONCLUSION

The study showed a noticeable shift in the value of skills over time specifically collaborative leadership has become increasingly important, correlating with a positive wage premium. The emphasis on this skill has increased in modern workplaces because mastering this skill not only drives individual and organizational performance but also fosters an inclusive environment where innovation thrives.

We also found that the return on data science skills is constantly evolving, but the legacy skills associated with it, such as ‘big data,’ initially contributed to a 1.85% wage increase in 2014-15. However, by 2018-20, these same skills led to a 1.21% decrease in wages, highlighting how quickly technology evolves and the importance of staying current in the fast-paced digital economy.

Personal Opinion

Reflecting on the findings of this study, understanding the shift in demand for skills is crucial because, in a world where skills requirements are rapidly changing, businesses can use these insights into hiring practices with the evolving market, focusing on skills like collaborative leadership and current data science competencies. The study also highlighted that labor shortages are often caused by a lack of job-relevant skills rather than a lack of workers. So, by identifying and investing in skills that drive wage premiums and also align with market needs, both employers and individuals can better navigate the complexities of the fourth industrial revolution.

Research Paper Link: What Skills Pay More? Insights from 'The Changing Demand and Return to Skills for Professional Workers'