ChatGPT is not ready to take on employee salary benchmarking

ChatGPT, the AI technology that's set off an artificial intelligence gold rush, is not yet ready to determine that companies are extending the right salaries to attract and retain talent. 

We know this because we tested it. When asked "What salary should I offer a software engineer in the U.S. with five years experience?" ChatGPT came back with $100,000 to $120,000 a year, and maybe more than $150,000 a year in high-cost cities such as San Francisco, New York, and Seattle. 

That's not bad. But our anonymized data from Ceridian clients shows the U.S. salary for people with "software engineer" or "developer" in their title is closer to $87,000 at the base. That's also with five years of experience. 

We ran a second test on cashier salaries — and ChatGPT fared better.  

ChatGPT found that the average base rate for a cashier with five years of experience in the U.S. is $10 to $13 an hour, or $20,800 to $27,040 a year, based on a 40-hour workweek. 

Our salary benchmark data shows the average base salary for a U.S. cashier with five years of experience to be $29,854, or $14.35 an hour.  

Read more:  GPT AI checks in at the doctor's office

When setting salaries, paying even a little under the market rate can lead to failure to fill jobs, taking longer to do so, or loss of employee talent -– leading to holes in a workforce and increased hiring costs. Overpaying can also have downsides for your organization, such as increasing your operating costs. Also, if you start using a new tool that varies from what you were using, you need to make sure that you are checking the new salaries against those of existing employees to ensure there isn't a discrepancy. That's why it is important to have good data upon which to base salaries and good insight into exactly what that data entails. 

Looking at the Right Factors 
ChatGPT is coming along rapidly, and it did consider many of the important factors in its analysis.  

For instance, when coming up with the estimate for software engineers, ChatGPT didn't only consider high-cost cities. It also noted that salaries can vary depending on the size and type of company. Large tech companies or engineers with machine learning or AI expertise may command more, it noted. It also found that the individual's expertise and accomplishments mattered, including certifications, and that there is usually room for negotiation. Talented software engineers with a strong track record and in-demand skills could potentially command a higher salary, it reported. It also noted that many companies offer various forms of compensation such as stock options, bonuses, and benefits packages that can boost overall compensation. 

The Question Matters 
How one talks to ChatGPT also makes a difference. Its response was more accurate when I phrased my question more succinctly: "Hi ChatGPT, I want you to act as a member of our human resources compensation team. How much should I pay a software engineer or software developer in the U.S. with five years of experience? What would be the average base rate and average salary?" 

Its answer: "As a member of the compensation team, I can provide you with some information on the average base rate and salary for a software engineer or software developer in the United States with five years of experience. The average base rate… is around $40.00 to $60.00 per hour. This means that the software engineer or developer would earn an annual salary of approximately $83,200 to $124,800 based on a standard 40-hour workweek."  

 It also noted that there would be differences based on location, type of company, exact skill level and experience and education, and upped its range from $80,000 to $150,000 a year.  

 This change in answer occurred because I asked ChatGPT to act as a member of the HR team. It was then more discerning about the data that it chose from, likely choosing more HR-focused sources. An internal study conducted by Ceridian in 2019 found that self-reported salaries from sites like Paysa can have inflated salary data, with only 44% of self-reported salaries falling within the actual salary range, based on Ceridian client data.  

Read more:  No more ChatGPT? Here's what the 'pause' on generative AI means for the workplace

The Data Matters, Too 
When using ChatGPT, always keep in mind what data it was trained on and the age of that data. Currently, the data ChatGPT was trained on only goes up to 2021. So, unless the model is updated and retrained, the data could have become outdated.  

You can already start to see this in the data for cashiers. U.S. cashiers have seen significant gains in starting wages given the labor shortage. Cashier median pay increased 10% between 2019 to 2021, according to government data. If ChatGPT doesn't feed in new salary information, the data for cashiers will be too low by 2024.   

Again, though, when asked to consider itself part of the compensation team, the ChatGPT program working on cashier salaries also noted that cashiers in higher-end retail or specialized stories may earn more, as will cashiers with additional responsibilities, such as supervising others, and that education level and other benefits or incentives, matter, too. 

Read more:  ChatGPT can make websites more accessible for employees with disabilities

What's Ahead 
Based on how fast ChatGPT took off, one can expect big growth spurts ahead. When ChatGPT launched in November 2022, it gained more than one million users in five days. Looking forward, we expectsome roadblocks, especially around privacy concerns and correctness of answers.  

 As we discovered in our test, ChatGPT is, for now, another source of information, but not likely well suited to be the only source of analysis for something as important as compensation benchmarking. 

 As ChatGPT improves, HR teams should also learn how to use it and update it. To get the best results, you have to know how to talk to ChatGPT, and ChatGPT needs to be continually trained with new data so that it stays relevant.