UK government trial of M365 Copilot finds no clear productivity boost A UK government department's three-month trial of Microsoft's M365 Copilot has revealed no discernible gain in productivity – speeding up some tasks yet making others slower due to lower quality outputs. The Department for Business and Trade received 1,000 licenses for use between October and December 2024, with the majority …
So the UK government paid Microsoft to learn that Copilot can spit out email drafts and meeting summaries but collapses on anything complex. And how exactly is an AI supposed to know the nuance of a call, or what was actually important, without losing it in a bland summary? Billions go into this hype, and the only thing Copilot does consistently is generate invoices - not productivity.
You've obviously not used it. It's remarkably good at summarising Teams calls and providing transcripts with annotation.
I've found it terrible for practically everything else, but in my very humble opinion, your comment is completely invalid and based on you having not used the product.
That might be true if you weren’t paying attention in the meeting. The danger is when people outsource attention itself. Summaries that subtly miss or distort key points seed slow, catastrophic drift in understanding. In a large organisation, reconciling those misunderstandings can take weeks, sometimes months, especially if work has already begun on flawed assumptions.
Well it's not there for you to avoid the meeting. It's there if you want it summarised or to give you recaps. If I was building a nuclear submarine I probably wouldn't rely on it. If I was working in the public sector and there was a procurement meeting about revenue and benefits software I think it would do a sterling job.
Not Copilot, as I avoid Teams whenever possible, but the AI summaries provided for Zoom calls are sometimes useful. I still make my own notes for anything I need, but the transcript allows me to go back through to check I didn't miss something I should have picked up (and, ironically, missing things is easy when you're concentrating on getting accurate notes). The transcript can sometimes be humorous, when it misinterprets someone (especially those with strong local accents) but it provides cues to where other things may need following up.
The UKGov exercise proves that what individuals see as productivity doesn't always coincide with what the organisation (and others/colleagues) see. Take the case of emails - AI writing them may save the author time (so AI is productive) but the quality of said emails doesn't convey the necessary information (so AI is a PITA). Does it come as a surprise? No, as passing the composition of communications to incompetent minions has always resulted in miscommunication.
The rules for public sector procurement are very simple. You to spend as much as possible (because large budgets make you look important) for things which won't be delivered until you have moved on elsewhere. On time and under budget are equally disastrous for career progression.
Spot On !!!
If you need to task an 'AI' to summarise a meeting that 'YOU' attended, then you wasted your time being there !!!
You should understand what the meeting was for and what the end result was ... because you were paying attention.
If you did not pay attention then you did not need to be there !!!
This is a solution looking for a problem to solve.
Total waste of money and peoples time.
Quite impressed that someone actually tried M365 Copilot out first rather than running full speed ahead with implementing it, which would be the approach I would expect from Govt agencies.
Perhaps the 'Hints' will be read by others that are being tempted by the 'full on' hard sell and people will save time & money by NOT trying to 'fit' their 'problems' to this 'Solution' !!!
:)
I don't disagree.
Yet I find myself in a few meetings I don't need to be in I could do with a summary of because I'm not needed and find myself working on other things and phasing out.
I also find myself not in, not available, or not invited to lots of meetings where a summary would be helpful and a transcript of I need to delve deeper after the fact.
If I attended everything that MIGHT fall into the second set I'd be working 28 hour days and produce nothing myself.
I see a potential for value in transcripts and summarisation - tasks it does well. It may never be realised though and some of it needs a change of culture beyond a small trial. The wider trial and culture change might never prove worthwhile.
I find myself in the same situation, in particular related to meetings I OUGHT have been invited to - aka "deities, must we counter this crap after the fact AGAIN?!".
However:
- Our Teams meetings are always in English
- This is because our teams using Teams teams are diverse (sorry for swearing) and functionally monolingual*
- The QUALITY of the English ... let's just say it ain't RP? If you've ever listened to a Swede from Scania** speaking English*? There you go.
- The meetings themselves are, how to phrase this, not commonly well organised (oh, the meeting room has hard walls walls walls ... oh, the laptop mic is pzzzzoor?)
The "AI" doing the transcription in Teams is, in one word, CRAP. It can't handle the truth, much less transcription of human speech. I'm glad you see the value at what it can do today. I'd agree, if I thought it'd get better, faster, but right now there's no worth in keeping the transcription we get out of it. A local TV channel, doing real-time subtitling of newscasts with a AI*** keep claiming it'll get better and keeps getting complaints 'cause it ain't.
Mileage and so on.
* We have Danes, Swedes, Norwegians, Indians, Iranians, Latvians ... y'know. Real life. We even got, hang on to your hat, a *Brit*. Kid you not.
** Skåne.
*** Sucks old wellies. To be polite.
Maybe transcription and translation with the use of native language is part of that culture shift that might never be worth it but could actually be the standout benefit of the technology. Hell just having the excuse of getting out of group video calls from echoey offices definitely is one in my books.
I mean it definitely struggles with acronyms casually pronounced and things at the best of times, but generally I've found automated transcripts of some non native speakers helpful where I particularly struggle with the accent.
I don't mean real time transcripts when I say I actively find them useful despite proposing that in paragraph 1 though. I've never used them in realtime but I strongly suspect feedback of context from later or clearer conversation actually fixes mistakes made that just aren't possible for true realtime transcription. And I suspect that's part of the problem with news (I've definitely seen subtitles be fixed on news, but I assume that's human manual correction)
And for a government department, the grunt grade (EO) works out at an hourly rate of £22 an hour. Across a month there's easily that much saved (or an equivalent value generated) from using it a few times. We all know it isn't reliable, insightful, or intelligent, we know we need to check the output. But even so, there's a range of time occupying tasks that I'll not do when I should, or do badly, that AI can support me on. Doesn't take away from planning, thinking, innovation, doesn't free up enough resource to get rid of staff, but comparing what it can do to what's currently being charged it's a reasonable exchange.
On this topic I will point out that the Civil Service is planning 15-20% average headcount reductions (all covered in the press); It is likely that when the actual numbers are published as to how many have gone, that some plonker will claim that AI has been a successful enabler of that reduction. I can assure you it isn't, the headcount targets are arbitrary, and entirely unsupported by any new operating model, or any understanding of where government can save money. As an example, the Civil Service comprises about 1.2% of total government spending, around £16bn a year. Unfortunately, that £16bn is peanuts compared to the collective cost of government projects and decisions that go badly wrong. Instead of shaving off £2.5bn from their pay bill, they'd save more by introducing performance bonuses* for all staff, clearer leadership and targets, introducing staff accountability for reasonably forseeable bad outcomes (even years after a decision), and a culture where not hiding bad news is encouraged. Taxpayer's Alliance are big axe grinders on the topic, and reckon government waste is around £200bn a year. I can credit that. Let's say that half of that is addressable, and 60% of the addressable could be saved. That would give a saving of £60bn a year. Little of that will be saved by slashing £2.5bn reducing an already underpaid and demoralised workforce. We all know government is the biggest single spender in the entire economy; logically if trying to effectively manage annual spend of £1.2 trillion, the workforce should be well paid, it should be a respected job that attracts high quality candidates, with good turnover to and from the private sector. None of those apply at the moment, and government then pay through the nose to contractors and large suppliers to fill gaps, without having the expertise to manage those effectively.
* Only exist for the most senior grades, "because they're worth it".
> doesn't free up enough resource to get rid of staff
But is a handy excuse to do so. See all those companies replacing their support teams with AI chatbots. Which actually are very friendly, very supportive, but as useless as the obligatory "FAQ" full with marketing selling points you'll find in most support pages ("How do I renew my subscription" "How do I buy more options?" "How do I bequeath my whole fortune to the company?").
I was interacting with FinTech Revolut’s yesterday which TVH is better than most. I was asking after their (announced in April) Cell disruptor offering. The Chatbot hallucinated it had launched and they would get to me soon. An actual human said it was stuff in development and early next year
I would shudder to let it near any money in my account.
Even if we accept the questionable savings, they look like rushed box-ticking to justify a decision already made. The real issue is data sovereignty. Copilot is covered by the US Cloud Act, which means a rogue US president can order access to everything it’s trained on. With one swipe of a pen, sensitive UK government data could be shipped straight to Washington - or worse, leaked on to Moscow. We’re trading sovereignty for the illusion of efficiency, and that’s a catastrophic bargain.
> We’re trading sovereignty for the illusion of efficiency
Or for the illusion of convenience?
It's not like most things AI does can't be done by any halfway intelligent office worker. It's just that said office worker dreams about AI doing his job and he still getting paid (which obviously won't happen. If you're not essential you'll be fired).
> It is obvious that a drone generating slides of various KPI for the higher levels is essential, but the grunts doing the work are not.
Clearly. "Essential" means the boss considers you an asset for some unspecified and not necessarily rational reason which hasn't necessarily anything to do with the job. That been said, an AI bot being able to do your job just as well (if not better) is still a huge risk nevertheless, unless the boss really, really likes you.
"sensitive UK government data could be shipped straight to Washington - or worse, leaked on to Moscow"
Well, to be fair the Oxbridge elite have a far longer track record of leaking UK secrets to Moscow than US tech companies do. Also, I remain to be convinced that the UK is anything other than the 51st state when it comes to doing dodgy Uncle Sam's bidding, or handing over everything in our pockets voluntarily (although without the entirely nominal protections of the Fabled Bog Roll).
"With one swipe of a pen, sensitive UK government data could be shipped straight to Washington - or worse, leaked on to Moscow."
Currently, there is little evidence to prove IF there is any actual difference in consequences for either destination.
(Additionally, there is little supporting evidence to verify that there are TWO 'independent' destinations rather than ONE destination 'cleverly disguised' as TWO !!!)
:)
Thank you for this !!!
It clearly states what I have thought for decades.
Motivating people to do a better job is MUCH better than swinging an axe and hoping that the 'Fear factor' encourages everyone to work 'harder' !!!
The lack of accountability and the ability to engage 'Sloping Shoulders Mode' is almost 'genetically' built into the Civil Service.
This is NEVER challenged as the 'kick the can down the road' tactic always enables the issue to be avoided until another Govt is installed and it can be ignored until someone accidentally 'trips over the can' in the future.
The Civil Service runs Govt ... period.
The elected officials et al are only there to 'officially' take the blame !!!
It does not matter which 'colour' flag the current/new Govt stands under ... you always get the SAME Civil Service !!!
:)
"The lack of accountability and the ability to engage 'Sloping Shoulders Mode' is almost 'genetically' built into the Civil Service."
In about a third of the senior civil service (that's SCS grades 1-4, Permanent Secretary down to Deputy Director), the other two thirds of SCS are very competent and committed public servants. However, an interesting result of years of intentional grinding down of the civil service has been the outcome of the fact that we can't get the "on paper" talent in at entry level. In consequence we sift through many young people who would be rejected out of hand elsewhere. And you know what? I can assure you (from 30 years or blue chip private sector experience) that I've encountered and had the privilege of coaching some incredibly capable young people in my short time as a civil servant, far more so than entry level in the private sector. Youngsters whose parents certainly are nowhere near proud enough of their remarkable children.
"The Civil Service runs Govt ... period."
But not in the way you think. The role of the civil service is to administrate the will of politicians. We do that to the best of our abilities (notable fuck ups recognised), and whether we agree with the ministers will or not. I've worked in strategy/policy roles in big companies for most of my career, I now work in the civil service, and I can say with certainty that too many people have watched "Yes, Minister!" mistaken it as a documentary. If a minister says "Jump" we say "Yes sir. How high? How frequently? Using how many legs? What should we do about people who don't have two legs?" In the private sector things are a lot less literal, but the same principles apply: The boss is the boss, and their will prevails unless actively delegated.
"It does not matter which 'colour' flag the current/new Govt stands under ... you always get the SAME Civil Service !!!"
No, collectively the populace always get the same Uniparty LabCon government that they vote for. This also reflects the minimal political involvement of around 97% of society. If the biggest political engagement people can offer is maybe to use a wax crayon to put an X in a box every five years then I don't think they can complain about the shabby state of government.
"The elected officials et al are only there to 'officially' take the blame !!!"
Au contraire, the purpose of the civil service is to take the blame without answering back. We are the whipping boys and girls for the decades of shit headed politicians that you voted for. Look at HS2. Do you think any civil servants give a toss about it? No, we're there to listen to feeble intellect ministers demand that we investigate and create a business case to justify a decision they've already made, and after the event not to tell the public that this was forced through by the Right Honourable XXXXX XXXXXXX with made up evidence.
Remember this always: As a civil servant I do not work for you; I work for the minister of my department. If they want evidence manufacturing to support their choices, that's what we'll do. This is how democracy works - or would you rather we vetoed the will of minsters because their choices are stupid? Now THAT would be government by civil service.
“But not in the way you think. The role of the civil service is to administrate the will of politicians.“
Em.. this is not Trumpistan with politically appointed sycophants. You are also there to work the wheels of Government, obey the law and work to the Civil Service Code (in the same way Ministers are supposedly bound by the Ministerial Code on obeying the law …).
https://www.gov.uk/government/publications/civil-service-code/the-civil-service-code
“‘integrity’ is putting the obligations of public service above your own personal interests
‘honesty’ is being truthful and open
‘objectivity’ is basing your advice and decisions on rigorous analysis of the evidence
‘impartiality’ is acting solely according to the merits of the case and serving equally well governments of different political persuasions”
Political shenanigans/descisions are entirely owned by the partisan monkey in charge.
https://www.gov.uk/government/publications/ministerial-code
You have in your comprehensive comment highlighted the problem by the 'nice' bolded part of the last paragraph.
"I do not work for you; I work for the minister of my department."
This is what is part of my 'Sloping Shoulders mode' definition.
The Civil Service plays ALL sides, supporting the minister/dept & supporting the Civil Service via the Civil Service code.
If there is any problem the 'I was following orders' line is used, IF that does not work then you throw the minister/dept 'under the bus' by giving testimony to any enquiry ... which is usually so many years after as to be useless in terms of correcting the 'issues'.
Your comment effectively states that there is little for the Civil Service to fear in the immediate timeframe, and by time any 'theoretically possible' enquiry is performed the civil Service staff involved are usually safely retired or very close to retirement ... hence their pension(s) are safe !!!
'Yes [Prime] Minister' is still far to close to reality to be believed !!!
[Apologies for using this 'old' meme ... but it does apply so well !!]
:)
>"'Yes [Prime] Minister' is still far to close to reality to be believed !!!"
Yes, the vast majority of MPs and Ministers are clueless, short of a few brain cells and most certainly lacking in experience.
I suspect if MPs had to have some level of competency in management and leadership we might get Ministers who didn't need to be handheld by Civil Servants. and who could actually lead, effect change and still be in office to handle the consequences of their actions...
I feel I should clarify that when I refer to the Civil Service I am referring to the whole entity called 'The Civil Service'.
This is NOT an attack on the majority of the people who are doing the 'Grunt Work'.
Unfortunately, the Civil Service has views & attitudes that are part of the 'machine' itself, these are maintained and supported across decades of existence.
There is because of the historic tendency to employ and promote a certain class of people, a certain political bias to the working of the Civil Service.
THAT 'machine' is what I am attacking, the people at the lower levels in the Civil Service have little ability to change/influence the way the Civil Service works.
A nod and a 'hat tip' is made to modernising the Civil Service in line with societal change BUT it, in general, stays the same !!!
:)
>"The lack of accountability and the ability to engage 'Sloping Shoulders Mode' is almost 'genetically' built into the Civil Service."
Sloping aka Teflon shoulders also seems to be endemic in many companies, encountered it in US companies and seems to be cultural in Indian IT companies.
>"This is NEVER challenged as the 'kick the can down the road' tactic always enables the issue to be avoided until another Govt is installed"
That's the Westminster politicians, specifically the Conservatives (mainly because they have been the ones in power for the vast majority of the time since 1979), but both Labour and Reform (local councils) exhibit the same tendancy.
>"It does not matter which 'colour' flag the current/new Govt stands under ... you always get the SAME Civil Service"
It's called continuity. You only need to look at the stupidity of DOGE and the US political appointees system under Trump specifically, to see the disadvantages of politicising the civil service.
Teams Transcription is worse than auto-generated subtitles on YouTube.
Considering it’s based on Dragon Dictate/Nuance tech that Microsoft bought 3-4 years back I’m surprised there hasn’t been any public litigation of their medical notes premium product for wrong leg/body part removed/operated on.
Whilst I'm not using it for anything nearly as important as conveying which leg to amputate...
I find it generally useful, even if the words don't match the sound shapes are usually good enough that I can follow what was said better than if I didn't have them on. Of course I don't rely on it for transcribing ticket/task IDs, because it has an amazing ability to screw up numbers, but even with a variety of accents it manages well enough to be a good support.
The real challenge is working out whether I could read the transcript without the audio and understand what was going on, and the answer to that is pretty clearly not. But I'm used to subtitles and lipreading supplementing my compromised hearing - so adding them all together is generally useful.
However, since I can't read the auto subtitles and come up with anything useful - I would be extremely sceptical that a LLM could come up with a useful summary either.
> It's remarkably good at summarising Teams calls and providing transcripts with annotation
Yes and no. It's good at summarising what was discussed but completely ignores what was presented. So if there was something critical noted on a slide - but no one spoke about because it was "so obvious" then it gets excluded from the summary.
And, inevitably, someone using the summary to catch-up (after being on holiday say) won't necessarily pull up - or even have a copy of - the slides.
Artificial, as in sweetener.
Don't get me wrong, I like Diet Coke as much as the next girl. But despite decades of research the food industry has yet to come up with diet pies (or diet tiramisu) that bear the same resemblance to the non-diet article as Diet Coke does to the sugar-infused phosphoric acid fix I occasionally remove my tooth enamel with.
I am not hopeful that AI will get significantly more intelligent. I fear that won't stop us balancing a house of cards of societal expectations on that GenAI mirage.
... and the rest were either unsure or didn't.
Unfortunately, that doesn't mean they weren't there.
This sort of survey is not exactly rigorous since there's no objective assessment of accuracy or the genuine equivalence of tasks undertaken by the two groups. Yet the productivity differences still seem to amount to little more than noise - with the added ingredient of potentially undetected hallucination.
It's not exactly a convincing value proposition.
Telling that users were disappointed when the play date with their mechanistic chums was over - perhaps all they need to do is pair up the staff working on these various tasks so they get some genuine personal interaction.
Speaking as somebody who's been a guinea pig in these government AI trials, nobody I know.
We're all aware that AI does a few minor things reasonably well, and that ANYTHING it proposes where accuracy is required needs to be checked.
I've used a few AI tools now and tbh found CoPilot to be the wakest by a considerable margin.
Do I find GPT useful? Actually yes. But not for single big bang impacts. What I fiond useful for are many smalls things, thing I generally could do but don't have time to.
Research and summarise a niche area of a topic I am knowlegable on, or generate a checklist for a team on this thing, or write a guid on this subject, restructure this document for this group.....
So things I could do but would take time, but also things I can review quickly with my own knowledge.
I also find it useful with fault finding, I'm getting more into using Linux (thanks to W11) but my knowledge is not great, I find GPT very usful with fault finding Linix issues.
I'm also having to do a little coding but I'm not qualified, but again GPT is useful to fault find and correct what I'm doing.
Is that worth $20 a month, I'm not sure. I'm not doing anything that I couldn't do or that ultimately I could do without, but it is allowing me to do more or add a bit to what I do.
Is it worth the $200 a month that is suggested needs to be charged? Nope no way.
Also businesses want to have a nice single thing that shows GPT costs are justified, and I think they'll struggle with that.
i can't say which stock exchange i worked for which could be close to El Reg's $HOME, but the inclusion of Copilot led to an over 80% rejection/turn off by my ex-employers staff. it was a billion times worse than Clippy ever was and totally useless at summarising ANYTHING!
Often poor results are blamed on AI when the problem lies with vague instructions. The people who get most out of LLM are those who have paid attention, understand how LLM work in principle, and have learned the art of prompt engineering and refinement. Still needs checking, still nowhere near intelligent, but used well it's surprising what LLM can do.
"The people who get most out of LLM are those who have paid attention, understand how LLM work in principle, and have learned the art of prompt engineering and refinement."
This ^^^^^^^^^^ is exactly why 'AI' is useless !!!
By the above definition of a 'GOOD' user of 'AI', the rest of the users (the majority) are not going to get much out of using an 'AI' and the sales pitch that sold the concept is obviously a lie.
This is just like the advert showing a person using a 'FOLDABLE Phone camera' to image a whole shelf of products & then ask the 'AI' to select the best product for the user.
In the advert it does this in seconds and highlights in the image the exact product on the shelf while 'speaking the result' via the speaker.
THE SMALL PRINT AT THE BOTTOM OF THE ADVERT, WHICH DISAPPEARS IN 5 SECONDS, STATES THAT THE 'REAL-LIFE PROCESS' IS SLOWER AND THAT THE ABILITY TO SELECT THE CORRECT PRODUCT IN THE IMAGE IS NOT YET AVAILABLE BUT IS EXPECTED IN THE NEXT QTR !!!
Once again, 'AI' is being sold on a 'hoped for future capability' with the functionality 'sped up' because in real-life the usability is suspect.
(Asking a question via this whizzy method is not very good IF you have to wait 10/15/20+++ seconds for the answer to appear on the screen)
Bells & Whistles to entertain you BUT usability/functionality are somewhat dubious.
:)
The advert is also selling the concept of all personal information being available for use in making that decision, including medical & financial.
Allow me to make your decisions, I'll organize your life to be as good as it can get, imagine, fresh Soylent deliveries when you need them without needing to do anything is only the beginning!
I've found Copilot useful for fine-tuning my resumé and for generating presentation bullet points. For actually creating the documents and presentations itself, it was total crap (which is kind of weird insofar as one would think that Microsoft AI could generate PowerPoint slides), but just producing the feedback and content I got was useful. Whether it's worth the premium is hard to say; I certainly wouldn't pay it myself without having a specific use case.
In all honesty, whilst LLMs might give a summary overview, often with errors. If you compare them to an able and intelligent human, they are seriously lacking in ability. The Harvard study recognised that 90% of AI Agent projects fail.
The reality is, anyone who thinks AI is amazing probably has an IQ of 70 or less, which seems to include many CEOs.
To be fair, it is NOT to do with any level of IQ BUT rather to do with how gullible the person maybe.
Anyone can be tricked, even more so IF you want the 'Flying Pig' to be real !!!
If you really want 'AI' to be true you may unknowingly allow yourself to be convinced by what is being sold now !!!
Sometimes you need to rein back your enthusiasm and closely examine the reality before you.
Far too many people are letting the 'future possibilities' overwhelm the actual 'AI' that is here today.
'AI' right now, as delivered today, is a scam ... period !!!
'AI' in the future may be all that you could hope for BUT 'Futures' are a gamble and may actually deliver 'absolute Zero' !!!
:)
With what Gartner Research will tell you off the record if you have a contract with them.
Copilot saves about 15mins per day at best - and is not worth paying essentially for 2 office seats per user. As a $5 add on it’s reasonable. At the current RRP of $30 it’s mostly unsellable. At even higher prices for MS to actually break even on the costs - it’s laughable.
Also because it’s only 15mins most users take the time to make another coffee or have a chat with a colleague, hence no material productivity gains,
Also based on the trial I’ve run. Free Copilot Chaf has about 80% of the gains of the paid version. All you lose is the clippy functionality in the Office suite which is marginal. The ENTRA/MS graph intergration is good and where the other ~20% comes from.
"Copilot saves about 15mins per day at best - and is not worth paying essentially for 2 office seats per user. As a $5 add on it’s reasonable. At the current RRP of $30 it’s mostly unsellable. At even higher prices for MS to actually break even on the costs - it’s laughable."
Let's say your 15 minutes is a 100% overstatement and there's an average of savings of 7 minutes per day; and assume a fully loaded labour cost of a mere $30 an hour. Over a month of 21 working days that around 2.5 hours, worth $75. Not sure where you get "2 office seats per user", but all things considered I'm giving you zero out of ten for maths.
Thanks John. You’re correctly pointed out what the AC missed which is tech costs aren’t measured at the same scale as corporate pay, An E3 license is £19 so have to copilot on each one increases your office license cost by 150%.
Your example is a good one, another one. Copilot costs $360 a year, or about 1/2 the replacement cost of a corporate latptop. How many Co’s do we know that can/would spare the cash to replace their entire suite of laptops every two years? 7 mins per day is probably the kind of benefit you’d get from a decrufted new laptop.
“Le mieux est l'ennemi du bien” the corollary of which is that “good enough” is sufficient.
My son has a robo vacuum cleaner and mopper, called Jerry. Jerry is not the sharpest tool in the box but is eager and enthusiastic.
Since the last software update my son maintains that Jerry has been lobotomised and his mopping skills are not what they were. However, I digress, suffice only to point out the degree of control that he has over Jerry.
We all know that Jerry’s vacuuming and mopping skills are average and that we could do better. We also know that what Jerry does is tedious and that Jerry can do it more often than we would ever want to. So we put up with mediocrity for the sake of convenience.
And so it is with Copilot et al.
Jerryisation : Mediocrity, Homogeneity, Banality - a fitting replacement for Liberté, Égalité, Fraternité
But only if we allow it.
I've found Gemini 2.5 pro to be better than Copilot for a lot of tasks. I've access to all the major paid for LLMs.
I seem to be getting way better results than my colleagues.
I suspect that I'm better at defining what I require and at asking the right type of questions in the most effective way.
I've found that if you sketch out what you want in reasonable detail you can iterate to a usable starting point quite quickly.
That starting point will still require validation and editing.
Yep - well chosen, and crafted, prompts do seem to generate mostly usable starting points.
Though hallucination is still a serious issue, and there is a requirement for genuine intelligence to be placed between the output and the task at hand.
You do have to ask a question for which you're pretty confident the answer is somewhere in the likely training data for the LLM.
I'm sure there are plenty of questions I could ask which would only every generate hallucinations, but by being careful about what type of questions I ask I find that an LLM often comes up with better documentation than the documentation for certain projects.
Yep - well chosen, and crafted, prompts do seem to generate mostly usable starting points.
Or as I used to tell my students at university: if you take the time to ask the right question you'll usually find you know the answer. Or in this case, if you tell the LLM the answer it will repeat it back to you. Wow.
There is a learning curve to getting decent result from the tools. It's not a particularly steep curve but you certainly develop key phases that work best to herd the AI's 'thinking'.
I often describe dealing with GPT as a bit like working with a hyperintelligent autistic 5yr old. It might be able to collate and summarise a 10000 document in 5 seconds but it still doesn't know that it's best to hold a knife by the blunt end or that cats have 4 legs.
And what % percentage of basic office users can write a macro or create a power automate script or a PowerBI report? All similar tasks similar in complexity to basic prompt engineering.
You’re just proving LLMs aren’t consumer grade.
Copilot - please summarise the sentiment in the government departments
Treasury: The coffers are empty, expect tax rises. Most often used phrases: "The former conservative government", "Liz Truss' Mini Budget"
Defence: We don't have enough of everything
Health: Terminal case
Home Office: Illegal immigrants OUT, ID cards IN
In the UK, commercial prices range from £4.90 per user per month to £18.10, depending on business plan. This means that across a government department, those expenses could quickly mount up.
I suppose so considering I paid £12/week for bedsit in 1978. Meaning, it's not expensive, currently. It's not the expense in this case, it's whether it will get used or not, and if its gets used, will it improve performance in typical civil service office work. You might have different results with younger workers.
You might have different results with younger workers.
Get them Tiktok and Instagram accounts and run the government on those platforms.
If Farage gets in, he'll replace those with Twitter/X and the Trump organ for government business. You'll recognise his missives by him following Trump with messages in ALL CAPS
Sounds like it improved productivity by a few % in a couple of minor areas, like writing emails and summarising meetings. So, the question is, is it worth the hundreds of billions invested in the tech?
At best, AI seems to be a.n.other tool that yields relatively tiny, incremental gains in some parts, like autocomplete in an IDE, writing emails without mishtakes, and speech-to-text. It's not, as they say, 'all that.'
I refer the honourable members to the original question. Is it worth the hundreds of billions of dollars invested? Nah, mate. From what I understand, it's not likely to get much better, because of finite training resources and the nature of the way LLMs are built. So, will it be better in the future, and have more uses? Bring more productivity to us proles? The people selling it say 'yes', and talk about AGI like it's just around the corner. But then, they have to don't they? Mega-yachts don't just buy themselves.
Isn't it a lot more likely that by the time AI can become a paradigm shift and [insert chosen hyperbole here], we'll all be treading water, looking for food and wondering why there are so many hurricanes, droughts, forest fires, displaced people looking for somewhere cooler/drier to live and in general, what the f*ck is happening?
It's pretty obvious when adminidroids are using Copilot because suddenly short concise answers become 3 page essays with references back to an obscure email someone sent 2 years ago. I'm sure they are delighted with their improved "productivity" despite the fact that it wastes everyone else's time reading the dross.
I find it hard to believe no productivity increase. Less than expected? Yes. Not worth the money? Perhaps. But none!
It would be interesting to hear a summary of why not. Maybe it was the type of tasks and people it was assigned to because LLMs do get things wrong or misunderstand context so the human using them has to recognise this. If you give it to a lazy drongo that just copies and pastes its output without scanning and understanding, it's going to be a disaster.
Far too many responses to these types of trials/results go immediately towards questioning the capabilities of the users using the 'AI'.
These results ARE 100% 'true to life' ... the users of these 'AI' products will NOT be trained in the right way to get the right sort of answers etc etc.
IF only the 'trained' can hope to get anything useful back from the 'AI' products then the products ARE USELESS.
This is just like saying ... "IF the users could understand and code in C++/Rust/Java/BASIC/APL etc :) , they could write the own programs to answer ALL their problems !!!"
If you need to meet a whole raft of criteria to be able to use and get meaningful answers back from these 'AI' products then they are not fit for purpose as they are sold as being for everyone.
:)
Microsoft, like a lot of modern businesses, is marketing driven. This means a lot of obvious business activity is going to be focused around the needs of marketing which, to an outsider, looks like a lot of information collection, projections of future sales and the production of communication materials such as slide decks and brochures / websites. Not surprisingly these appear to be the tasks that Copilot appears to be well suited to.
The snag is that just isn't the totality of business or government activity. There are lots of activities that require decision making to not just come up with an answer but 'show the working', the reasoning involved. Even today you really can't administer laws based on hunches.
https://americandiplomacy.web.unc.edu/1997/12/technology-and-foreign-affairs-the-case-of-the-typewriter/#:~:text=Tradition%2Dbound%20critics%20opposed%20the,usage%20linger%20to%20this%20day.
"Tradition-bound critics opposed the use of typewriters on legal and even health grounds ..."
As soon the enthusiasts let it go, the AI will reveal its true utility, but you gotta let people throw it around and see where it sticks.
Computers themselves, and lot of other tech had that same psychology trigger, people wanna use it for everything, even impractical things.
Tablets were the tools of yesteryear, for example.
Having a calculator on your wristwatch made a full comeback 40 years later, on a worse solution, a watch with a battery that lasts only 48 hours at most, instead of at least 2 years, but that's the symptom of "look, I can cram a smartphone cpu and screen on my wristwatch".
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
