An Objective Review Of Reboot’s AI-Content SEO Experiment
| PlacidPlace | Pixabay |
Artificial Intelligence (AI) is taking the world by storm. The release of ChatGPT, OpenAI’s Natural Language Processing model, ensured a massive shift in forward-thinking, but with it comes a fair share of concerns.
At the forefront of many people’s minds is how this progressively improving technology fits into a world run and maintained by human content. Understandably so, with marketing techniques like Search Engine Optimization (SEO) playing a major role in what online content is being seen while remaining relatively unregulated.
And marketers have already taken the dive, with some experts finding an overwhelming 82% of marketers claiming AI content to be “just as good as human content”.
But where does Google stand in managing AI’s sudden popularity and mass production? How does the search engine handle ranking human-generated content against AI-generated content?
Reboot, an online marketing company in the UK known for its industry-changing experiments set its sights on answering these questions.
Before we dive into the experiment though, let’s take a look at some of the components of the experiment.
Search Engine Optimization (SEO)
Search Engine Optimization has become a key phrase in a growing empire built on keywords and phrases. Online businesses have made it a focus to utilize the science behind optimizing content, pushing their websites and article links to the front page of search results. How do they do this?
Trending keywords, careful outlining, and metadata.
With AI becoming more and more advanced, this is a process that is becoming increasingly easier to attain. Marketers can provide prompts for Artificial Intelligence programs to:
- Optimize an article
- Implement search engine optimization practices into a pre-written article
- Write an entire SEO-focused content article
All within seconds. This can significantly decrease the time spent on content writing by a human while minimizing overhead costs.
AI-Generated Content
Natural Language Processing (NLP) models have hit the ground running with highly advanced “chatbots” like ChatGPT and Microsoft Bing AI, capable of producing hundreds of lines of text and near-instantaneous revisions.
This software works by taking in a prompt from the user and feeding it into an AI program trained on large amounts of text gathered from internet sources, books, magazines, etc.
Next-word prediction is used via an internal ranking system. This seems simple, but being trained on such a massive amount of information allows it to output several lines of complex text at incredibly high accuracy and precision.
Google’s Stance on AI-generated Content
Google states plainly that content is at the forefront of the company’s focus, whether man-made or AI-made. Content created by Artificial Intelligence has been around for a long time and can be utilized to increase quality and efficiency for readers and writers alike.
Their developers also state that AI content isn’t ranked in any special way, instead following the same paradigm: “useful, helpful, original, and satisfies aspects of E-E-A-T”. Whether human or AI-generated, the “winning” content will be that which ranks highest according to the E-E-A-T.
How truthful is this claim, however? Such is the basis behind Reboot’s experiment. Through deep research, statistical testing, and articulate reporting and discussion, they sought to answer that very question.
AI vs. Humans: The Reboot Experiment
The experiment began in the early months of 2023 behind the idea that AI- and human-generated content could be ranked differently on the Google search engine results. They wanted to create a controlled environment where all aspects of their content were the same except how it was created: human or AI.
The researchers utilized ChatGPT to create eleven meaningless words with no search history or domain existence. They chose ‘flemparooni’ as the keyword around which the content would be written.
They then developed and deployed ten basic websites using the remaining ten words. These websites were created to host the content articles.
After outlining a 7-step guideline to promote replicability and equal optimization across all content they began the writing process.
Five of their company’s in-house content writers wrote an article each based on the keyword ‘flemparooni’ and the guidelines given to them. The researchers then used GPT4, the latest version of OpenAI’s ChatGPT, to feed the keyword, the guidelines, and the prompts needed to output five articles meeting the specified criteria.
Finally, data analysis was performed from information collected through keyword position tracking, and statistical significance tests were performed to determine the significance of the results.
AI vs. Humans: The Results
Interestingly, Reboot found that of the ten articles created (five by humans and five by GPT-4), four of the top-five ranked articles were human-generated, with the only AI-generated article amongst that list ranking third.
Four of the bottom five ranked articles were AI-generated, with the only human-generated article ranking that low falling to number nine.
The researchers concluded that AI-generated content held an average ranking of 6.6, whereas human-written content held an average rank of 4.4, showing that content created by humans tended to outrank content created by AI. Data is only as good as its significance, however, and this is where the word “Science” in “Data Science” comes in.
Data Analysis comparing the rankings of websites hosting human-generated content vs. AI-generated content. (Reboot, 2023)
Reboot researchers tested the data using the Mann-Whitney U test.
Omitting the complex mathematical mumbo-jumbo, the Mann-Whitney tests the numerical data against a null hypothesis, in this case, that the probability of AI- and human-generated content are ranked equally is 100%. After some behind-the-scenes calculations, the test returns a number indicative of the difference in mediums (or middle numbers) between the two populations (AI-generated vs. human-generated content).
A higher result indicates websites hosting human-generated articles rank higher than their AI counterparts, and a lower result indicates that AI-generated articles rank higher.
After a total of 25 comparisons, the null hypothesis was rejected about 92% of the time, telling us that the results of the experiment were significant. This means that the results they found didn’t happen by chance.
Furthermore, the results showed that ranking tended to favor human-generated content over AI content, assuming total SEO equality throughout all content.
Conclusion
The results of this experiment are interesting considering the implications.
With SEO-based companies and marketers focusing on ChatGPT to maximize their content optimization while saving on the overhead cost involved in human content creators, this experiment shows that their competitors may maintain an edge if they stick with human-generated content.
A theory is only as good as its replicability, of course, and research should be continued by various sources for more in-depth confirmation; however, the results are a good outlook for us human content creators and should be a flag for all those online marketing decision-makers pushing for total content automation.
The answer lies somewhere between human and AI content generation.
One should not supersede the other, rather each should be utilized in harmony to promote engaging, human-to-human knowledge and content while maximizing efficiency through AI editorship, outlining, and keyword generation.
Experiments like this show us that we are not ready for complete automation, and companies that think we are will likely do themselves more harm than good.
Humans and AI Should Work Together For You
Your content should be geared towards you and your needs. AI can’t do this alone, and human creators can only do so much. Scripted does a fantastic job unifying the two in a way that maximizes speed and efficiency while providing high-branded human-to-human content. Providing a pool of carefully selected subject experts and a one-stop platform for all your content creation needs, marketers need look no further. AI Empowered, Human Perfected: Craft Authentic Content with Scripted.