Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.
Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.
The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.
Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.
These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.
Right and all the dogs in the race are now focused on neural networks and llms, which means for now, all the effort could be focused on a dead end. Because of the way capitalism is driving AI research, other avenues of AI research have almost effectively halted, so it will take the current AI bubble to pop before alternative research ramps up again
Like every time there’s an AI bubble. And like every time changes are that in a few years public interest will wane and current generative AI will fade into the background as a technology that everyone uses but nobody cares about, just like machine translation, speech recognition, fuzzy logic, expert systems…
Even when these technologies get better with time (and machine translation certainly got a lot better since the sixties) they fail to recapture their previous levels of excitement and funding.
We currently overcome what popped the last AI bubbles by throwing an absurd amount of resources at the problem. But at some point we’ll have to admit that doubling the USA’s energy consumption for a year to train the next generation of LLMs in hopes of actually turning a profit this time isn’t sustainable.
The issue I have with referring to the current situation as a bubble is that this isn’t just hype. The technology really is amazing, and far better than what people had been expecting. I do think that most current attempts to commercialize it are premature, but there’s such a big first-mover advantage that it makes sense to keep losing money on attempts that are too early in order to succeed as soon as it is possible to do so.
I think that’s intentional. Nation states and other powers that be have working propaganda mechanisms.
A real AGI is a change most important in the sense of power, not in the sense of economy (because we know how to make new humans and educate them, it wouldn’t be a qualitative change there).
All this AI gaslighting is intended to stall real advancements there.
The Web in some sense was produced in the context of AI research. In general semantic and hypertext systems were. And look what it has done to the world. They may just not want another such cataclysm.
EDIT: Also notice the shift from the hypertext paradigm to the application platform paradigm in the Web.