Even Netflix struggles to identify and understand the cost of its AWS estate Keeping track of the amount of cloudy resources an org uses, and the cost of doing so, is notoriously tricky – so tricky, indeed, that even Netflix isn't on top of it. We know this because on Wednesday US time the vid-streamer blogged about its cloud efficiency measures. The post – penned by senior analytics engineer " …
so basically, everything they produce and everything they engineer is wholly dependent on AWS keeping the lights on and playing nicely with them
and they have no alternative provider strategy in place (?) and are actually engineering even deeper into the aws stack
AWS subsiduary in all but name
Even worse: it's the reason for Netflix's constant price increases (let's not mention constant selection quality depreciation). Netflix's enshitification now has a reason, they're giving most of their infrastructure to AWS to "manage" and Amazon, being Amazon AWS, is *more* than happy to increase your bill due to a single error in configuration without saying a word to you.
So, Netflix's major "infrastructure" for their own operations is mostly virtual. Nice job. Pass the buck, pass the bill [on to the end consumers], it's someone else's problem to keep us going, thankyouverymuch.
Nah, the entshitification and costs is due to Netflix changing from an aggregator of other people's contents to trying too hard to be its own studio. A model the other streamers have followed.
Think about it: 20 yrs ago would you have expected all of ABC/CBS/NBC shows, or those of any Euro chains, to be all good? Of course not, a good deal of them are going to be duds.
Netflix started out re-cycling shows and movies, which did not cost it all that much. But, key point, they also got the best stuff from a lot of sources, since they were an additional revenue stream over DVDs for the studios.
Over time though the studios learned that a) Netflix was making a lot of money and b) Netflix was killing their DVD biz. Prices went up, studios each wanted to become a streaming service.
So now you have a lot of shit shows that need $$$ to film. Studios holding better TV series or old film titles hoard their wares like dragons of old. The availability of classic or indy movies, compared to a decently-stocked 2000's DVD rental store has collapsed. Streamers are in a bidding war to see who can fund the most expensive re-warmed blockbuster crap like House of the Dragon or Wheel of Time, out of your wallet. Meanwhile unlike the good old days where your cable company gouged you once, you now need to subscribe to half a dozen streamers, if you want to watch all the "important shows" as they come out.
I mean, sure, Amazon not running Netflix's backend out of charity isn't helping. But that's really not the core issue.
The enshitification of modern entertainment, is everybody believes in celebrity names rather than actually good stories. They are all lazy, and think they can buy greatness because unfrotunately a lot of entertainment is about celebrity.
20 years ago, many shows were quality, nowdays basically e erything is garbage in comparison. Perfect example are the indy movies from the 80s compared to the modern ones, theres simply no comparison which is great and which is garbage built only one names instead of quality.
Yes, I fully agree and why I don't bother watching / reading fiction any more (well, I made only a few exceptions: James Bond (wish I didn't, the Daniel Craig Bonds just *aren't fun*), Marvel but only up to Endgame, Inception, and a few animateds). The [modern] writers seem to be all lazy hacks who will, almost instantly, fall into a trope to solve a plot issue or character conflict rather than actually write.
I don't think modern shows and movies are necessarily much worse. TV has gotten a lot "deeper" since good ol' Starksky and Hutch and most of the Bonds were never that great to start with - that's due to the very format and concept for the franchise - although Connery did make it work at times. Likewise, it is hard to see The Witch, White Lotus or Vast of Night as somehow inferior to an idyllic past.
No, the problem is a race for high production value content to fill the libraries of too many competing streamers, with a lot of shows getting the green light that really did not need to be made. To make the numbers work, the answer is obvious: sequels, sequels, sequels. Derivative, derivative, derivative (from similar content that worked). Ditto theatres: they are not doing well, so need to make big-screen content. Then you can another new development: best way to make an epic seeming movie is: CGI, CGI and more CGI. Even cheap CGI can make a positive difference - Equilibrium - but even great CGI won't save a crap movie though it might motivate enough eyeballs for ROI.
To start with, how many superhero films do we need every year? Star Wars? Star Trek? How much breathless coverage do we need to see about a great new fantasy adaptation? How many CSIS redos? I tried to watch Squid Game (Battle Royale) or Money Heist (pompous crime genius with hot supporting actor hookups to keep the suspense). How many nitwit near-future spy agency stories? Add to that a huge selection of online websites who seem to get paid to promote the pap (looking at you Ars Technica, Vulture, etc...)
There's plenty of good stuff coming out, it's just lost in the noise and you're stuck paying for the shit in your subscriptions.
Which knowing AMZN, will be the size of an empire state building sized pile of benjamins.
Dealing with any part of the Bezos empire is like dealing with the devil jnr.
Devil Snr is reserved for Musk who now wants to buy the UK (via Farage)
Both can suck on this [see icon]
I don't think they're relying on AWS for storage of important things. All their video assets have copies on their self-built CDN nodes, and almost certainly on massive storage systems that are at a scale where renting them doesn't help. The pain in trying to leave is taking all the services that are built on AWS components and trying to run them somewhere else. That doesn't happen if they only rented IaaS with everything custom above that, but from their description, they use other services that you can't easily just move. For data egress, the stuff they would have to pay to copy is likely a bunch of logs of every customer's every action, and those are probably quite large files, but not compared to all their videos.
Customer data is probably more important to them then content which will have duplicated across hundreds of distribution points. Its those services which will be running on AWS along with billing, marketing and public facing website and most likely their internal systems like HR, software development, etc and almost certainly most of that will not be IaaS.
They potentially have a lot of issues coming as Trump doesn't believe in Net Neutrality opening the door for ISPs to fleece Netflix for carrying their traffic.
Nobody runs their own datacentres. I work for a large corporation. We had 19 data centres, all provided by and managed by different service companies at different depths of hands-on. Barely a month went by without an issue at one or other of them. Since migrating to GCP (multiple regions around the world) we've had almost nothing that was GCP related (our punishments have been self-inflicted). There's lots of pain in the cloud, but reliability isn't generally one of them, at least not if you architecture properly. Now, I've no doubt some of our lift-and-shift is suboptimal and there would be a scramble to recover it, but we've had those scrambles multiple times with our 'own' DCs.
Who runs the DC? Is it your people all the way down?
The banks I know outsource the infra to a service company that owns the building, power, racking, network connections (usually sub-let to Telcos).
And is there no-one else in your DC?
Perhaps the US is different, but when we had our 'own' DCs, they weren't ours. They were somewhere between IAAS and PAAS; there was always someone else doing the basic running of the building and utilities.
Plenty run their own DC's. In fact, I believe this esteemed organ recently published a piece on AWS users moving services back to such DC's. The corporate execs drank the koolaid, got the hangover and now realise the extent of the deception imposed by faddish Public Cloud groupthink ...but eventually money will talk.
Amazon has often been described as having become a computing infrastructure business with a gift store on the side.
Maybe Netflix should similarly consider becoming their own computing infrastructure business with a content (ugh) licensing / production business on the side?
They could then start touting spare capacity to current AWS customers, and see if they can offer a better deal… «gets popcorn»
I used to work at a large Japanese corporation (you know the one, last seen at a UK government inquiry near you) that sneered at the cloud computing efforts of "an online bookshop" and laughed as they predicted the future when the IT services companies would crush their efforts. I guess that Azure is doing pretty well. Anyone ever hear of K5?
... that most Netfilx costs arose at the edge - pushing large video files from CDNs to telly boxes - and the bit in the middle processing payments and making it impossible to find anything worth watching would be comparatively much less costly.
But, I suppose, a lot could be analytics. Everyone is so convinced that collecting ever-more data about your customers is the key to economic success that no-one is ever going to question the cost of collecting and processing it, so maybe it's better simply not to know.
Ask Netflix for a copy of your data. They track and record every interaction. Hit pause for a pee? Recorded. Hit back up to watch a funny thing again? Recorded. Stopped watching and went and did something else? Recorded. All of it.
I'm sure this might provide some sort of valuable "insights" about video consumption but, for the love of god, just stick me in a box labelled ADHD and give up.
Well I guess if you licensed boat loads of film and TV you’d want to know what your customers are engaging with. They spent time and money getting some film or series that someone thought was good, only nobody gets past the first ten minutes. Time to unlicense that.
I'm sure this might provide some sort of valuable "insights" about video consumption but, for the love of god, just stick me in a box labelled ADHD and give up.
Netflix does videos? Not sure about valuable insights given the latest exciting new videos are a 'shelf' titled "Popular mobile games for you".
This.. puzzles me as I've never watched Netflix on a mobile, mostly via PC browser and PS5. And in typical scream.. I mean streaming service, doesn't give me the ability to hide 'shelves' or re-organise content in any way that I might want. So there are 'shelves' for horror, scarey and a bazillion more 'curating' content into overlapping categories, often featuring the same movie.. That I may already have watched. Oh, and of course there's a 'shelf' for watch again.
Meanwhile, potentially useful stuff like a plot synopsis got squeezed down to a 1-liner in most cases.
I suspect this is a cunning plan to make it appear that Netflix (and Prime) have more content than they actually do, especially as relevant 'shelves' like new stuff usually gets buried. I think both are trying to meet the definition of doom-scrolling, along with the old adage of 'down, not across. But if the way they manage UIs and categorise content is anything to go by, it's no great suprise they struggle to understand their own data or AWS costs.
"I've never watched Netflix on a mobile"
I do and it's pretty much the same as you describe. The "shelves" (not sure that's a good word but I can't think of a better one) are often full of fairly recent things or popular things and some of the more niche things only turn up if searched for or by accident.
"a cunning plan to make it appear that Netflix (and Prime) have more content than they actually do"
Perhaps, but given that it is an always-changing thing it's not entirely useful as a metric. unogs.com is your friend.
Oh, and I wouldn't lump Netflix into the same category as Prime when talking about UIs. Netflix is mostly usable, Prime ought to be a textbook example of how not to design such a UI.
"plot synopsis got squeezed down to a 1-liner in most cases."
I'm less bothered by that and more bothered by when the synopsis says something like "Variety says this modern classic is not to be missed". Yeah, and? What's it about!?
"I think both are trying to meet the definition of doom-scrolling"
Doom scrolling videos? You sure you're not in the Fox News app by mistake?
For me it's more like: Oh, that looks interesting. [add to watchlist] Oh, no, not for me. Nope to that too. Oh, that looks interesting. [add to watchlist] Oh, no, not for me. Oh, that looks interesting. [add to watchlist] Oh, that looks interesting. [add to watchlist] Half an hour later: Oh, I'm too bloody tired, time for bed...
I do and it's pretty much the same as you describe. The "shelves" (not sure that's a good word but I can't think of a better one) are often full of fairly recent things or popular things and some of the more niche things only turn up if searched for or by accident.
I think 'shelves' is the official term, and is meant to describe the chaos that Netflix and other streaming services use to describe their content organisation. I have real shelves bracketing this desk, with books and DVDs on them. Difference is they're organised the way I want, not how Neflix thinks I want them.. Which is my gripe. I want the power to organise 'shelves'. So as an example, the first 'shelf' in my current view is-
"Because you watched Cold Pursuit.."
And then 9 videos that seem very vaguely related, and have metadata like 'visually striking', and 3 of the recommendations I've already watched. And then other 'shelves' are full of genres I probably don't want to watch, and an odd mix of videos that it knows I've watched, watched a bit of and abandoned, or watched but Netflix seems to have forgotten.
Perhaps, but given that it is an always-changing thing it's not entirely useful as a metric. unogs.com is your friend.
But it's useful to me. So I want to have some control over the impersonal personalisation, and organise content the way I want. Netflix and other services collect a lot of user data in an attempt to profile us, but the results generally suggest they don't know me at all. I shouldn't have to rely on 3rd party services to try and find something to watch, and I shouldn't have to rely on wiki to find out what a video is about. And apparently Netflix currently only has 40 horror movies or shows, which is probably a limit imposed by Netflix rather than the reality.
But it's also probably indicative of the chaos behind the scenes. AWS is quite content to charge Netflix for all the queries generated by users like me, desperately trying to find something to watch, or just generating shelves full of stuff I don't want. So I'm not suprised Netflix is struggling to understand AWS, and curious how much it could reduce their bills by just giving me some kind of control.
The costs at the edge are relatively constrained. They make deals with with all the major telcos to place Netflix boxes inside their networks which has the massive advantage to the telcos that their interconnects and core routes aren't being saturated with Netflix traffic.
I'm currently working on a small application that, by corporate policy, is spread over a family of AWS services. There is some logic to having it configured so it costs nothing while it's not being used, but the engineering overhead of a whole lot of services and all the connections between them means that setting up deployment has almost certainly taken more engineering resources than writing the whole application would have if it was a simple API running on a single VM.
It's fine from my point of view - some handy additions to my CV - but I don't know that it's doing the business any good.
Wait for the business' response when your colleague's code spins up an extra million or so instances due to a mistyped bounding condition. All of a sudden the management overhead that stops that sort of thing looks like money well spent. Ask me (or better still, my colleague) how I (we) know.
I've played around with AWS and the complexity of simple things such as where they put everything is purposely obfuscating in my opinion. That's just me playing about. I can imagine if you are in an organisation such as Netflix with multiple users it gets even more infinitely complex to keep track of. It's why we have cloud software and people employed solely to keep track of spend. The question is could AWS make it straight forward? Sure but why would they do that when a single mistake can make them £1000's. It is absolutely all by design.
I work for a company who is a big user of AWS - large setups in multiple regions, prod, dev, sandbox environments, you name it, we've got it. And we've been asked to put our name in any resources we create to make it easier to track who is using what.
I'm not agreeing with you,nm just saying there are easier things to follow than AWS...
The stuff between the originator of a service and the user, that makes all the cash.
Auction houses have always done this. They charge both the buyer and seller a fee and make money even if something does not sell.
Economically, it is like subscribing rather than buying. For a bit of convenience you pay through the nose and lose the ownership rights and control. Your balls are on someone else's chopping block and you have to keep handing over as much cash as they ask for.
It requires a team, and a big project, to construct an analysis framework to just be able to predict the bill.
And it will need constant maintenance, as no doubt AWS fiddle with the billing/admin interfaces all the time, so you end up playing whack-a-mole on AWS' attempts to make the data you need harder to find or be less analysable.
It doesn't take a massive service to predict the bill. It takes a massive service to figure out where all the bill came from. To some extent, that's not unique to cloud services either. The thousands of different products that AWS or someone else offers makes it a bigger task, but you can have hideous snarls with any other large system. Trying to figure out who is using what kind of server resource can be complex even if you own the servers. Either you have clustered them and individual teams can provision resources, in which case you know when you're taxing them and need to expand but you don't necessarily know how it is being used or how necessary each load is, or you require them to plan this out for specific machines, in which case you know what everyone is using but it makes each increase in resources a drawn out process whenever teams need to expand.
Oh, that happens. That's how you have companies with a hundred AWS accounts for each division. Sometimes, they also separate dev, test, and prod into different accounts, so make that three hundred. IAM becomes a headache then, but I think IAM is nearly always that way.
They clearly thought they were paying for X simultaneous connections, but really there were ACTUALLY only paying for for X connections from the very same building.
If they wanted X connections, they would need to buy a load of Y connections, from multiple locations, for double the cost,
Yes I cancelled my for that very reason....and the fact its pretty much the same shitty content, just with a different name
We have OVH, Hetzner, 1und1, Schwarz IT/StackIT and quite a few more cloud providers here in Germany and France.
Hetzner now also has a DC in Finnland, where leccy is cheaper than Germany.
OVH can use competitive french nuclear power and has DCs in several countries.
Why be locked in to the giants if smaller and easier to understand competitor exist ?
I can recommed Hetzner, fixed, predictable price and excellent reliability and customer service.
Edit: Hetzner has even more DC locations: https://www.hetzner.com/de/unternehmen/rechenzentrum/
You are still thinking IaaS, that is just moving hosting from on-prem to off prem. Modern apps which netflix and most enterprises use don't use IaaS, they won't be wanting to deploy windows or linux. No mention of containers or K8 or anything thar supports modern architecture. Also no mention of multi-zones or HA.
Depending on what you're doing and how it's architected, those services can in fact be a competitive option. Specifically, if you would ordinarily be buying normal VMs from the big cloud providers. If your AWS bill is just EC2, S3, and bandwidth, or for the sake of not having a monopoly in this sentence your Azure bill is just VMs, storage accounts, and bandwidth, then these other companies can probably be used in the same way without too much difficulty. If there are more things on that bill that you intend to keep, it is not as easy.
The large clouds aren't just renting out servers. They're also renting out lots of software running on those servers. That software is used to develop and scale things. You could build similar functionality yourself, but development time and financial costs increase and inevitable bugs become a serious problem for those developers. Larger companies often have the resources to do that themselves and small projects often don't need it, but the medium-sized products are where there are significant reasons why people choose to design with those features, and sometimes those medium-sized companies become large ones and, although they may well eventually build it all themselves and stop using the cloud providers, they have enough inertia that it takes a long time to do so. Managed databases, functions as a service, notification systems for distributed applications, load balancers, and many other things are popular with some people. Individually, you don't really need the cloud provider to charge you to run each of those, but if you're using a lot of them, it can be costly to do it yourself. If you're running the servers in your own building or renting them from an IaaS company like the ones you mentioned, you are still doing that part yourself.
Seems to me a lot of comments going on - justifiably - about the vagaries of public cloud is somewhat missing a major point.
Netflix isn't 37 Signals, which brought stuff back in house and showed how much they saved.
Netflix has to provide reasonable last-mile data transit times to hundreds of millions of people around the world. That means hosting a lot of very heavy data in a lot of very dispersed locations, with a lot of unpredictable spiking when something exciting drops and catches the zeitgeist. The decision matrix for doing this is not the same as that of a large bank pondering self-hosting, even though they do stuff like the Netflix OCAs.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
