This is an amazing discovery, article, and fix proposal. Fantastic work, very impressive and also very instructive on how things work on modern PCs and how far you can actually dig to get at stuff that is "supposed" to be hidden.
As someone who has written embedded firmware for many years (not for PCs), I can only dream of an end user being this capable to discover a bug. I want to live in the world where Asus immediately send an e-mail offering some kind of short-term contracting work to fly in and talk to their firmware people for a few days and get $FIVE_FIGURES or something, and leave with an updated laptop running their new production BIOS.
Obviously this bug has gone un-fixed for four years so that is not the world we're in. That makes me sad. :|
The technical RCA is fascinating, but im also interested in the business processes RCA.
- this sounds ubiquitous and reproducible. How did this not get fed back through tech support/RMA channels? Was there so little evidence that it wasn't correlateable, or did ASUS look and arrive at an incorrect conclusion, eg batch of bad silicon? Could it be that they had plentiful evidence and were negligent or incompetent?
- it sounds like this is plainly evident when using the machine. What is the QA process? This should not have been possible to miss?
- now that they know, what will they do?
Imo, the ceo calculus here is clear. If you're a luxury good with elastic demand, you fix the issue and fix the perception (two separate things). Multi-year, multifaceted issues like this have the potential to ruin a brand. I've bought ROG in the past, and I'm inclined to never do so again.
EDIT: on further reflection, the firmware bug itself is pretty troubling. the other bugs i get - hardware assumptions were changed, or good code was reused that didnt know or support the gpu mux, i see how those errors comes about. the method sleeping an interrupt... is awful? how did that get reviewed? what is that firmware test suite?
It doesn’t matter that the consumer/gamer laptop is a piece of shit, because all of the competitors are too. Consumer hardware is a volume business, and the actual end-user experience matters very little compared to endorsement deals and marketing strategies.
Every one of the affected ASUS laptops probably got a glowing 5/5 review from the usual suspects, and consumers have little hope of getting a fair deal
> What is the QA process? This should not have been possible to miss?
Have you used consumer goods [or virtually anything] from the last couple decades? By and large, nobody cares. Look at the timeline here; clearly nobody cares.
As a user and programmer, I can only dream of being this knowledgeable about things. There's a ton of domain knowledge embedded in the article, it's pretty amazing.
I managed to reverse engineer a lot of my laptop's features but hit a wall when it came to this ACPI stuff. I dumped the tables and decompiled the code but all I got was stub code. I wanted to be the guy who wrote the Linux drivers for his own laptop but I just didn't manage it. Massive respect for anyone who can do this.
Yeah the best way to go is to buy Linux preinstalled and supported. Though, as with Windows in this case, that still won't save you if the system integration and firmware teams don't do their job.
Yeah. ACPI's AML bytecode is sort of a mixed blessing. It allows for reverse engineering and end user analasys/fixing of bugs like this.
It's also just a terrible disaster of a programming environment, with a very large (terrifyingly so, given the limited capability) interpreter that needs to live at the highest privilege level of the kernel.
And it's generally used like a hatchet by system integrators for tricks like this, with pretty much exactly the code quality you'd expect. Almost always the path to writing a Linux driver for some oddball laptop subsystem starts with "throw away the ACPI stuff".
As far as I know there are three ACPI AML stacks, the reference intel one, linux uses this, miscrosoft has one, and those crazy hackers over at the openbsd project decided to make their own.
I think that's right. Though my understanding is that the Windows code is derived from the original Intel one too and has evolved in tandem with the Intel-maintained driver. And... yeah, acpica (drivers/acpica/acpica) is just huge; I checked again and it's at 2.5MB of source code. All for a DSDT table parser and a virtual machine with about the capability of a 6502.
Yeah, sorry, that was a bit unclear. I just meant that the article went as far as propose rather clearly what is needed to fix the issue ("don't sleep() in an interrupt service routine").
Impressive that they managed to ship crippling stuttering for 4 years in gaming laptops specifically. Makes you wonder about the end user psychology, evidently they didn't get a show stopping rate of product returns.
A quote from one of the linked reddit threads. I wonder if the warranty trip is part of their scheme.
"I did everything you suggested , but nothing changed. I send it back via garante. I am curious what they do whit it."
"what was it at the end? did they respond?"
"They have claimed that the plato works perfectly. So basically i just got use to it. I am using bluetooth earbuds all the time so i cant notice the problems."
I've owned 2 gaming laptops in my lifetime and both had similar issues that were never fixed.
One was the first gen Alienware M17 with two GTX 270M GPUs (yes two) and an onboard nvidia GPU whose specific model I can't remember. That one suffered from stutters and audio crackling, etc. It was sort of fixed by disabling SLI and the onboard GPU and sticking to a specific driver that was modded, the driver was by someone on the notebookcheck forums IIRC. Later on I think it got somewhat patched with a bios update that let you use SLI without the stutters, but I think the laptop reached EOL without it being fully fixed.
The second was an ROG ASUS laptop with a GTX 460m (I can't recall the laptop model). Pretty much the same story as the OP but I didn't have the knowhow to go deep into the ACPI code. The only change from the story is that latencymon kept attributing the latency spikes to multiple dlls, sometimes it was some wifi driver, other times it was an nvidia one. I don't remember the full fix for that one, but it involved me changing the wifi card and disabling the dGPU (not the onboard one) when I was not gaming so I could watch videos and such without it crackling. Funnily enough it didn't crackle much when actually playing games (it still happened, just very rarely).
I stopped buying gaming laptops after that. Seeing this story makes me think things haven't changed one bit.
I am frequently asked for hardware purchasing advice by family and friends. Starting around 2017 or 2018, if asked to recommend a "gaming" laptop, I have refused. I never had a good experience myself, and more often than not, what I had recommended over the previous years ended up flawed or outright broken. Across every OEM and brand. I tell them to settle for a professional/business SKU with a low-tier dedicated GPU, or give up on laptop gaming entirely. Is it worth the money to pay a "business" premium for a weaker system? No, and I'll tell them that. It's not a good deal in on-paper-dollar-for-performance terms. But at least there's a chance that all of the components function and are supported!
What use is a "good price", when what you get is a quality and support minefield?
I've owned two gaming laptops and never had issues like this.
First one was a Clevo (rebranded as Medion) with a GTX 970m that I bought in 2017. An absolute beast, I lugged it in a backpack around the world for 4 years, including to places you really shouldn't bring a laptop like beaches and rainforests. I passed it my girlfriend's nephew and it is still going strong and being used every day. I repasted once in that time.
My current laptop is an MSI GE66 with an RTX 3070m bought in 2022. It's loud, I've repasted recently because it started overheating. It had some problems with the screen connector which they fixed under warranty fairly quickly. But aside from that it's solid.
One thing about both of these laptops - they are very easy to open and it looks like I could repair/replace pretty much every removable component easily. No glue.
The only thing I consider a real problem is the MSI fan noise. Well, that and the power brick which is the size of a literal brick.
Glad to see that there are laptops that don't suffer like this. But I think the combo of having a steam deck + business laptop beats buying a gaming laptop. Assuming you already own a gaming rig at home.
20 years ago, I had a midrange laptop with a dGPU and while it played games with mediocre results, the laptop experience itself was also mediocre. Stupidly, 10 years ago, I bought a laptop with a dGPU again, but because NVidia didn't play nice with Linux back then, I don't think I've used that GPU for more than an hour or so.
Never again. A laptop with a dGPU runs counter to the things a laptop should be. Keeping gaming activities on a desktop is the best option in my experience.
A few months ago, I started working at an e-waste recycling company, and discovered that used Microsoft Surface tablets are what I've been looking for. My work "laptop" is a Surface Pro 5 with Debian (my work desktop is an Optiplex micro). I'm typing this on a Surface Go (with BlissOS) that I bought for myself. The cameras don't work on either and the work Surface never knows it's battery status, but I don't care (it lasts an entire afternoon with a barcode scanner, good enough for me).
> Well those are just so underpowered it literally hurts. All of disk size, expansion options and memory.
That’s not a complaint have heard before. My needs aren’t huge and it has a lot more of everything than I need.
> And booting something that isn't a funny variant of a locked down OS is relatively hard.
I wouldn’t want anything else in it, but with a Mac mini I really wish it would run something Linux more easily. They are a great headless server, but the OS is really limiting.
I can tell you how I got duped into buying their product - a Zephyrus G14:
ASUS at the time had an exclusive deal with AMD to ship their Ryzen 4xxxHS line.
Initially it worked fine, but two years later performance was already much worse and dominated by thermal throttling. Repasting, though necessary due to the state of the paste, only helped partially.
I still don't know the root cause of the issue, but I investigated declining battery performance and it turned out that the iGPU was going full throttle at all times. Setting the dGPU as the preferred device actually improved battery life somewhat.
When mechanical failures started accumulating I switched to a FW16 and never looked back. I don't care what gaming laptop manufacturers have on offer and for how little if I can't buy having them give a shit about their products and customers.
I bought a Clevo-based gaming laptop for programming because it had a good CPU. It was a really bad decision.
It will thermal throttle itself to uselessness within seconds of a load being placed on it. The dGPU idles at about 15 W, the entire power budget of a single board computer, and it's one of those problematic nvidia GPUs that will never be properly supported on Linux. The Windows app that controlled things like fans and keyboard LEDs was so obnoxiously bad they required over one minute to show a window on the screen, reverse engineering that thing was one of the best things I've ever done. Mercifully the firmware wasn't broken by default but I still didn't manage to reverse engineer the ACPI nonsense, I dumped the tables and decompiled the code but there was nothing useful.
Looks like Apple has a monopoly on good taste and giving half a shit about the quality of the products they sell. I wish the Apple silicon macbooks existed at the time.
I was considering such a device since surely those huge vents meant good thermals, right? Glad I didn't go for it.
With the ASUS I had a setup with a cooling pad where the metal grid cover was removed and the sides were sealed with foam to enhance flow from the pad's fan and only with that I could maybe get 30-45min of gameplay until throttling started.
Meanwhile the Framework has overall much higher power consumption, but still manages to whoosh all that hot air out. I can't take these companies seriously if a much smaller business that is not focused squarely on gaming is running circles around them.
My mother rocks an M1 Air which she got for pennies and it's a great all around home computer.
This flaw only happens in Ultimate mode, when the user explicitly tells the mux to switch to the discrete GPU. This is an extra feature only users who primarily use the laptop for gaming with an external display care about.
The laptop works fine in Optimus mode even with external displays, you just lost a bit of performance and you're missing out on some display features like G-Sync. So it is highly likely that most users always use the laptop in Optimus mode. If you primarily use the laptop as a laptop you probably wouldn't even know the mux feature existed.
The problem is Asus shipping extra features in their hardware that are not properly QA tested. It looks like they only thoroughly tested the golden path.
Asus doesn't even test basic features, nevermind the extra ones. I have the 2017 Zephyrus GX501, which came with a Nvidia GTX 1080 which introduced HDMI 2.0 support. The Asus Zephyrus marketing material is boasting about HDMI 2.0 capability, the manual talks about HDMI 2.0 etc. However, in reality the device is limited to HDMI 1.4 bandwidth.
The problem isn't limited to some units, there was plenty of discussion online of this issue at the time of release. [1]
Asus never recalled, fixed, or even responded to the issue. Indeed, even the marketing page [2] still talks about how you can use HDMI 2.0 to connect 4K TVs at 60Hz.
It was also an interesting showcase of laptop reviewer incompetence. All the reviews just regurgitated Asus marketing material on how it has HDMI 2.0, but apparently nobody actually tested it.
Imagine if we had strong consumer protection laws. Companies would fear claiming HDMI 2.0 that was really HDMI 1.2. Instead they just figure “meh! If anyone complains we’ll let them return it.”
Seems to me this can't be concluded from the information at hand. This investigation used the mode, but the linked reddit posts I read complaining about ACPI latencies reportedby latencymon don't mention it.
Gaming laptop market feels like the most exploitative segment of entire laptop market to me.
They're like four-seater off-road motorcycles. You have to NOT understand how sketchy that concept is to consider one. The engineers has to know that they're guilty to be involved in it.
What's sad is that a lot of buyers are falling for it from the presumption that laptops are the most standard and regular type of computers. But I guess there's little we could do about it.
My 2013 Macbook also had a display mux that switched the laptop display itself between the integrated and discrete GPU. In some cases it would fail to properly switch the display over causing it to remain permanently off until the next reboot. I was not the only Macbook user with the problem.
Apple is just as guilty for shipping laptops with hardware issues that you just have to work around. And unlike this Asus issue the Macbook mux was on by default. You had to turn it off in the settings if you wanted to entirely avoid the issue and then you would have no way of using the discrete GPU.
As did my 2015 Thinkpad - same problem, both an integrated and an external GPU, and I was constantly fighting with the drivers to get it to just make use of the external GPU. I'm plugged into a wall whenever the laptop isn't suspended/shut down for transport, stupid machine, stop making Autodesk and my games crash because you want to boost your battery lifetime numbers!
They had a special Lenovo driver that would occasionally become overriden by Windows updates but could be reinstalled manually, I dual-booted Debian though and getting the system to work properly under that was a nightmare. There were a couple years when I simply gave up, I got it to work with the iGPU and I wasn't running anything more graphics intensive than a browser so I simply left the discrete GPU idle while running Linux.
Incredibly frustrating.
I think all these manufacturers are desperate to get their published specs for battery life estimates to double-digit hours that can't be reached while running the discrete GPU at full speed all day. Heck, they can't be reached while running the CPU at full speed all day, you're not going to run a 35W processor and a 55W graphics card and a 20W display (10W when you arbitrarily reduce the max brightness when on battery power) all day.
You've got like 90 watt-hours available in the battery, at 100% usage on everything the real capacity is gone in under an hour...which is unacceptable. So Asus and Apple and Lenovo and everyone else have to come up with some hack to turn it off whenever it's on so that the spec sheet says you can get 8, 10, 12 hours of runtime.
We had dozens of 2013 MBP with discrete GPU mux where I worked. I bought one afterwards and used it until the M1 laptops came out, for pretty much everything you can use a laptop for. Never had or saw this problem in any of them, FWIW!
Yeah, that's not shocking. Linux uses the Intel ACPI infra, whereas Windows uses Microsoft's. It's as good as they can do, but it's not going to be possible to perfectly replicate Windows, let alone improve such clearly buggy firmware, at least generally.
It is possible on Linux to override some of the firmware (most notably the DSDT, e.g https://wiki.archlinux.org/title/DSDT because so much hardware is broken). So, if you can make or get a fixed version, you should be good. A wholesale replacement of all the ACPI assets, though, seems unlikely. I could well be wrong, though.
Anyway, in this case, I suspect the poster was advocating for Macs.
As if Apple land was free of issues, remember those wonderful keyboards, Snow Leopard, Tahoe among many other examples, or any Linux laptop for that matter.
Or the Installer.app, at least from 2015 to whenever they finally switched to image-based updates.
A percentage of users were "unlucky" to hit a bug where Installer.app would end up in infinite loop trying to unpack a pkg file when updating OSX. My personal record is minor update that ultimately took a week.
Apparently this happens only if you set the laptop to discrete GPU only mode, which most people won't do.
However, this is not the only problem with Asus bioses. My daughter has one and it randomly locks up if you add an extra SSD, sooner or later depending on the SSD. You'll blame the SSD's firmware, but the most locking one was one that I have in two desktops with no problems...
This makes sense, but what does not make sense is who tested this 'ultimate mode', I mean they went to the trouble of adding a physical hardware switch on the motherboard for this, surely when testing there was some kind of benchmark or comparison to show this feature was an advantage. Maybe they don't test, or they have 'internal firmware' that is not what the user gets, but it's a serious fail either way.
I don't know if anyone cares, is the problem. I used to have a £7000 Dell workstation laptop, specced to the moon - i9, 3080Ti, 128GB of ram.....that kind of thing.
The laptop was absolutely useless at playing games, because it would throttle itself thermally after about 30 seconds. Which was ironic given that I used to work at a games development company and the ability to play games was actually a core feature of the product. I then used to have a Razer Blade 15 which wasn't as bad but would also eventually start throttling hard - just inadequate cooling imho.
Funnily enough I have a much cheaper MSI gaming laptop now with an i7 and a 3070Ti and that never throttles, I can run games without it slowing down. But clearly the cooling system in it is massively overbuilt, which is great.
> I have a much cheaper MSI gaming laptop ... that never throttles, I can run games without it slowing down. But clearly the cooling system in it is massively overbuilt
Maybe they learned their lesson. I had an MSI gaming laptop a while back, and it ran horribly, I never realised until long after it was possible for me to return it, that it was just poorly designed, and could never run beyond ~50% of its gaming performance. Within minutes of starting a game it would be thermally throttled and that was that; it also sounded like it was about to take off, to the point you could barely drown it out with headphones.
I think they did - I used to have an MSI gaming laptop years ago with a GTX560M if I remember correctly, and that used to throttle every several seconds, so every game you played stuttered. This one - not a single slowdown, but if you open it it has like 10 heat pipes and two fans that sound like jet engines - but it clearly works.
I'm glad to hear at least one manufacturer is trying. I can't imagine myself buying a gaming laptop again, perhaps for my kids when they're old enough, but I stick to desktop now.
My main drive in choosing the MSI I did back then was the thinness and lightness, which was counter-productive to good cooling performance, mine had a GTX970M but was about 1cm thick; the bottom of the case got so hot it would burn you if you touched it after a while of gaming.
It always puzzles me how apparently well known flaws never mentioned by product reviewers, even genuinely pro-consumer and generally well respected, like rtings or notebookchecks.
You buy product after stellar review, encounter problem, search for solution, find reddit thread where everyone is "yeah, it is always like that, why do you act surprised?"
Usually their revenue is from affiliate links. YouTube channels have ad spots, merch sales etc. There are many shills of course, but some are doing their best to give accurate reviews and don't shy away from bashing manufacturers for their missteps.
Yet widely known (to enthusiasts) problems, like stutters from the OP, are often not mentioned at all.
LinusTechTips doesn't depend on ASUS money in any meaningful way, but still failed to mention stutters in their Zephyrus G16 review. Some might say LTT is not a reviewer, he is an entertainer, but he undeniably thrives to be accurate while doing so.
This happens on a 2019 MSI gaming laptop (GS65 Stealth) I have too. Latencymon shows >10ms stutters within a minute of being open. Found out disabling all ACPI devices fixes the stutter. Also found out disabling all ACPI drivers disables the dedicated GPU. It's possible this affects a large portion of gaming laptops with dGPUs.
I wonder if the "programmer" (and I use this term very loosely) who wrote that sleep-in-an-interrupt code ever tested the code personally, or if it was some other distant responsibility-diluted department of a hundred other lamers who didn't care "because the automated tests all pass". This is a situation where dogfooding, in the original Microsoft sense, would definitely be beneficial as among the developers experiencing this on their own machines, surely one would be tempted to fix it.
A long time ago I did some contract work writing firmware for a major hardware manufacturer in Taipei. I quickly learned to ignore bugs, because reporting them would get me reprimanded for doing things other than the task I was assigned. Even worse, the hardware team saw the firmware/driver/software devs as lowly servants and dismissed any feedback outright.
It's crazy how the hardware sector just can't grasp that software is actually important too. Everywhere I've worked at we were just second class citizens, last to get a new hire or any budget, and then it was our fault when the software was subpar. It seems like they think going down to our level will actually sully them.
Short version: don't buy ASUS gaming laptops until this is definiteively fixed, and if you one under warranty, file a warranty claim, being prepared to go to Small Claims Court.
Unless you specifically bought that model because you want low-latency output to an external display with G-SYNC.
I've used Asus motherboards in my gaming PCs for years, and their BIOS/UEFI firmwares there are equally awful, their Ryzen AGESA stuff has been a complete mess.
My G14 functioned just fine with G-Sync in Optimus mode via nVidia connected USB-C port (and didn't need reboot to switch between GPUs) so I never found real use for the MUX mode. It's telling that most other gaming laptops don't spend on that piece of hardware.
The switch solves a fundamental trade-off: an external port can only be physically connected to one GPU. If the dGPU you get GSYNC/etc but high power consumption if you connect an external display.
If the iGPU, then the dGPU is basically sending its frame buffer through the iGPU and is limited by it.
It sounded to me like they were multiple bugs. Some were specific the mux mode, but sleeping in an interrupt, and artificially re-arming it is problematic either way.
Sometime around 2015 I promised myself to never buy a laptop with switchable graphics again. This has worked well so far.
But it never ceases to amuse me watching brands that position themselves as 'premium' spending pennies on firmware development team somewhere down in a basement compared to millions they spend on shiny marketing.
The big difference is Apple can't hold other companies accountable since they delivered the software, the hardware and the integration and they have an incentive to fix their stuff.
The few issues I've had with Macs the last decade did get fixed within a reasonable amount of time. Lots of PC hardware bugs simply don't get fixed at all since there is no party that really cares about them. The author of the linked write-up should have been an Asus employee instead of a disgruntled user...
Hardware issues are tough to fix, but they did iterate on the butterfly keyboard and eventually gave up and switched to a different design. My experience with smaller software and compatibility issues is they get fixed eventually if enough people notice and report them.
I had a mac at work for 8 years. Overall things mostly worked ok, but I had two big issues.
a) one time charging stopped working... thankfully I had a pretty full charge when I noticed and was able to migrate my data to a spare machine and not have to deal with it... removable storage would have been super handy.
b) for a whole year, there was about a 25% chance of loud static instead of music when I started playing a stream in iTunes; pause and play again would fix it most of the time. It started when I installed a named OS revision, and it stopped when I installed the next version. Did not have issues with sounds from other apps. Of course, there was no information to be found anywhere about this, because 'macs just work'
Less big, but if Outlook was running when I put the laptop to sleep, there was a good chance it would continue to eat battery and generate heat in my bag. Outlook is a travesty, but when the corp runs Exchange, Outlook is less effort to make work compared to fighting to make auth work with anything else and then still having to use Outlook from time to time.
Reminds me of when MSI laptops were getting properly bricked after users ran `rm -rf /` because of a UEFI bug where the board could not boot after some variables got deleted https://github.com/systemd/systemd/issues/2402
Congratulations ASUS for 0.01% of your marketing budget you could have fixed the experience of your millions of users, reduced the amount of replaced computers, improved your brand positive views... This once again proves that many companies are absolutely mismanaged and think marketing will make them more efficient than good engineering...
My Lenovo X1E regularly burns 20% of its CPU cycles on some high frequency recurring interrupt. I did get pretty far with debugging it, but eventually gave up since I can't justify spending so much time on fixing a 'professional' laptop that I paid top dollar for.
It also has a multi-GPU setup that has never worked reliably under Linux, which is ironic as I opted for Lenovo due to its supposedly good Linux compatibility.
Switching between GPU modes is a hit or mis, waking up from stand-by often results in a blank screen, screen flickering, sporadic high fan speeds, etc. And then there's the coil whine, which seems to be fixed in some BIOS versions, then returns in the next. Supposedly it has something to do with power-saving measures.
Since I owned it there have been at least 20 BIOS version releases for 'improved performance and security', but none seem to actually fix anything. It's such a mess.
It says it affects all ASUS laptops since 2021, making them stutter at the most basic tasks.
Which I'm ready to believe, knowing the state of most laptops... but this entire thing is pretty clearly generated by Gemini with its over-the-top dramatic style, italics emphasis, and -isms like "It's not just X, it's Y", which was unable to handle the article of this size and started looping over. Not sure I should believe any of it, or at least be sure that it didn't mess up the specifics. Why would one do this in a technical writeup?
I don't know why everyone uses AI for important writing now. The other day management showed us a ChatGPT presentation apologising for being terrible at managing.
I have an older ASUS laptop from 2015 which also has (more minor than this!) ACPI state management bugs. I initially bought that machine because it was a pretty high-end and was somewhat disappointed about both the build quality and the firmware/software support.
Somehow laptop makers all write complete garbage firmware.
I'm sure dell does the same terrible handling of DGPU power and badly written ISRs that pointless raise system latency. I had shoddy crashes for months that would cause my dell laptop to BSOD and burn up in my backpack because the DGPU got stuck on I a loop during some ridiculous windows modern standby wakeup.
I'm a big fan of my Deck, but I think Steam-related software projects always have a distinctive low-level stink.
I use mine on a train that has Wi-Fi with a captive portal and attempting to join it makes the whole UI unresponsive. Using the overlay with a guide in a game always resets the scroll location. These are the kind of things I can live with, but also things I don't expect ever to be fixed unless Valve come out with some wholesale replacement for some overt new strategic goal.
Lenovos in my experience are pretty flawless in both Windows and Linux -- my current laptop is a Lenovo and sleep/wake has worked flawlessly in Debian and Windows 11. My work laptop is a HP and sleep/wake also works fine in Windows. I've had the occassional power issues (including failures to wake from sleep) with Intel MacBooks previously for both personal and work laptops.
I have a Steam Deck and love it but it still cannot suspend and wake without the audio crackling issue - other than that it is magical though. (The pause games decky plugin is a third party solution that fixes it!)
Macbook essentially involved Apple hardcoding 11th hour fixes selected by product code in their OS level drivers after utterly fucking up ACPI tables in ways that would make the discussed bug blush. Pretty much most of what BootCamp did on later models, and what various hackintosh "special drivers" did, was fix things like HDA tables located in completely wrong place.
Steam Deck is just... extensively tested and debugged in ways that I don't think Apple did unless they got an egg on the face in national media (remember "you're holding it wrong?")
I was pretty disappointed when my work MacBook ran into the same "Bluetooth headset connected, set as audio sink, but the OS refuses to acknowledge its existence further and routes audio through the speakers" but that I've seen on every other desktop OS. I really hoped Apple would be better, considering their hardware costs more than twice as much.
I also ran into weird Wi-Fi issues that required a reboot, and getting that thing to recognize external displays without corrupting video is some kind of dark art while my Lenovo and Steam Deck work just fine with the same USB C plug.
Apple beats some brands for sure (especially the cheap "consumer" lines with a starting price lower than Apple's headphones) but their computers are hardly flawless.
I have yet to run into issues with my Steam Deck, which is very impressive, but I'm sure I'll run into an issue at some point. No computer works flawlessly.
Surface Pro 8s have a bug where the battery just gives up randomly and won't ever charge again. 2, 3,4 and 7 do not have this - one sitting on my shelf two years charged right up from 10%.
Replacing the battery costs like $200-500 because the screen likes to explode when removing it.
Lenovo docks of a specific gen will have the USB hub/billboard just crash and stop doing display port.
older Dell docks would pollute arp tables and crash switches.
Computers have always had some wild flaws, some worse than others. They are built to a price point typically and by humans under politics so the best design or parts usually don't make it -- cost and profit.
> I used an LLM for wording. The research, traces, and AML decomp are mine. Every claim is verified and reproducible if you follow the steps in the article; logs and commands are in the repo. If you think something's wrong, cite the exact timestamp/method/line. "AI wrote it" is not an argument.
That's a pretty big if in technical writeups like this, all you do by rewriting those is obfuscate the actual inputs you had. Was is generated from scattered notes? Entirely vibe-written? How many details are actually verified to be correct by a human? Seeing how even the structure seems generated, it's clear that there was little input, and I'm not sure about any of the above.
I can deal with poor writing, and in case of ESL it's enough to tell the LLM to proofread/rephrase the piece (and check it yourself afterwards). But lazy generations just make you trust the article less.
> Was is generated from scattered notes? Entirely vibe-written?
Why does that matter? Maybe the person hates writing. You need people to suffer and put effort for the end result to be worth your time?
> How many details are actually verified to be correct by a human?
I mean.. assume the best..? The author could also have written it by hand and just lied. Or he's a paid troll from an Asus competitor - it's all just made up and it's a work of fiction. You implicitly have to assume the author tried their best and be okay there might be some errors.
If the writing is more clear thanks to an LLM then you're likely to catch errors more easily.
If you feel the thing has errors, then engage with the material and point out the errors.
This is literally the same discussion when some people insisted that nothing on the internet can be trusted and physical books are the only reliable source of information.
I'm honestly a little surprised that ACPI has a Sleep()-like function at all, I can't really imagine many situations where the firmware actually wants the user operating system to wait for any real length of time, as that would block any other ACPI events, even if it wasn't in an interrupt handler. I feel there's pretty easy ways to deadlock the system in that sort of situation.
And also surprised that windows actually allows the ACPI driver to sleep in an interrupt handler - on Linux that'll immediately BUG()... Unless windows doesn't and the above ACPI blocking is what they're measuring here.
Reminds me of a cursed HP keyboard I had at work once. You plug it into USB, then your irq rate goes way, way up. All keyboard events telling you the key pressed last millisecond is still pressed. You literally gained 5 to 10% speed when compiling without it. Apart from that, and the way its sleep-state-destroying irqs ate battery, it was a nice keyboard, so some data entry person got it in the end.
Out of curiosity, why not release BIOS mod with a fix? Atleast personal laptops (out of warranty) can benefit out of it until Asus fixes their sht.
People blame Windows being slow and etc but most of the times hardware manufactures don't even get into this level to make best out of thier hardware. This is the reason why Apple is so successful, they control hardware, software while in open world, software like Linux/Windows is written by someone while hardware is designed by someone else.
Maybe the risk is too high ?, bricking UEFI means time for soldering/reflowing, because you won't be able to software-recover it(unless it has dual bios mechanism).
Right, and Windows probably has something similar (maybe it requires loading a driver). This is BIOS-style code interpreted by the OS, so you can patch it at the OS level.
So if I understood it right, every single Asus gaming laptop have easily measurable stutters? That's insanity. And I thought about buying Asus RoG Strix laptop, because for some reason marketing made me believe they're the best gaming laptops out there.
FWIW, I'm using an X13 for some standard gaming, but mostly VR and it performs as advertised for about a year and half now. I'd have sent it back immediately if it stuttered every 30 sec with cracking sound.
I won't defend Asus, they have a crowd of skeletons in the closet and need to pay more attention to feedback, I'm just saying there will be a set of use cases were issues won't be plagging the experience.
My HP screen (HP Aero 13, not a gaming laptop, with a integrated gpu only) does flicker, turning completely off and then on, and this issue doesn't appear when connecting to external monitor.
The same happens under linux as well.
This post had me curious about the ACPI now... maybe I can follow along !
I do not have the same technical depth to dig this far as the author, but this kind of problem seems pretty common on laptops, especially those with "switchable" iGPU/dGPU setups.
I had an Acer laptop about 7-8 years ago with almost the exact same latency symptoms. In the end I just disabled the dGPU in the BIOS (since I only used it for office work), and that instantly solved the issue.
This kind of thing is very infuriating because not only is it hard to track down the root cause (which I am very grateful the author did), but it is also even harder to get the vendor to actually acknowledge or fix it.
Intel laptops with igpu and Nvidia chips and thunderbolt docks are absolutely a nightmare between the 3 trips video has to go to even leave the laptop and the drivers and firmware mismatches.
I've had weird issues with this setup since the core2duo days upgrading once a year.
when it works it's amazing, however.
AMD igpu and dgpu work super well together but I feel like over time, since I use this configuration the most things either improve or go to hell with driver updates. depends on the laptop OEM really.
This all said, where the hell are the strix points igpus where they rival desktop cards (yes the lower mid end) at laptops where stuff just works without compromise ...if there is power and cooling.
Side note - I have a rog g14 that until I loaded a beta bios for thunderbolt over usb4 would reboot when shut down and shut down randomly. (amd CPU and GPU)
This really underscores how complex a computer is these days. This complexity is why you get a ton of little glitches when you slap Linux on a Windows PC--and this is even on Windows, as designed!
I have one of these, a Zephyrus G15. That it had an AMD CPU and Nvidia GPU should have been a red flag that support would be really poor. Only a year out of warranty, it is a brick on a shelf because the thermals are so atrocious it pretty much burned itself out, and even with a thorough new application of thermal paste through a multi hour process there just isn't any way to get it to perform within spec. Supposedly, if you RMA it through ASUS they will charge you something like $700 and be unlikely to fix it. They have an insane dud rate, and even when it does work the hardware is barely hanging on. Several acquaintances have had similar problems.
It drastically reduced my perception of Asus as a brand - I wanted something I could game with, it promised the moon of portability and performance but they couldn't pull it off.
Curious, my experience could not have been more different. I still use Asus Zephyrus G15 (GA503QR from 2021) as my main development machine, and also game occasionally on it. It's AMD Zen 3 CPU with GeForce 3070.
As I write this, the fans are either completely stopped or so slow that I cannot hear them. The fans can get a little "whooshy" under load, but nothing out of line with other Windows laptops, as far as I can tell.
I'm running under Windows profile (not one of the ASUS-specific ones in Armory Crate). I also limit the battery to 60%, since I'm mostly using it plugged-in, with external monitor and keyboard. A month ago, I upgraded RAM to 40 GB and added another 4 TB SSD (and cleaned the fans in the process, so it's even quieter). I think I'll keep this machine for a few more years...
That being said, I did experience occasional cursor stuttering, but mostly when the machine is under load (typically during Visual Studio build).
I believe I have the same one! For me, almost out of the box I needed to disable CPU boost due to the thermals and instability. That worked for a while, though I considered replacing it and didn’t because there was just no others available at the time due to the pandemic. Apparently my issue is common but not the majority - a lot of folks seem to just have gotten a bad batch. I’m curious though, have you not even had any issues with sleep mode on the laptop? It seems extremely common that it just doesn’t work/crashes.
I got a good 2.5 years out of it, but I was hoping to use it for at least 6 or 7.
> have you not even had any issues with sleep mode on the laptop?
Now that I think of it, I did disable sleep and just use hibernate instead. I no longer remember why - perhaps I too had some issues. Maybe I was just annoyed by Windows restarting in the middle of a night and (c)losing my open applications whenever it wishes to update? Or did I disable Wake Timers for that? Sorry, my memory is a bit fuzzy.
As it is, hibernate works perfectly for me, is fast enough, and it never closes my applications behind my back.
OTOH, I had sleep issues with pretty much all PCs that I ever had (be it laptop or desktop), so maybe it was just inertia to always use hibernation instead. :)
Now I wonder if this is why my laptop (Dell) is so goddamn annoying too.
LatencyMon does show elevated ISR and DPC max exec times (i.e. reports that for realtime tasks my system is busted), and they're also primarily carried out by CPU0. But it's on the order or 2 ms, not 30 ms.
Still a long time though for interrupts I guess. ACPI.sys taking the lead too, with NDIS.sys being close second, and then pretty much nothing else.
I have a 2024 Zephyrus G14 and it has bursts of stuttering which seem to be directly linked to running off USB-C power. It doesn't do it on the original power brick, but on a 70W USB power brick, it slows down massively every now and then, to the point where the mouse cursor is only updating every few seconds and any playing audio starts underrunning buffers. Unplugging USB power immediately clears the issue up for a while. It's fine running off battery, and it's fine when I plug USB power back in, even straight away.
It does other stupid things with power management, too:
- There seems to be some "cooldown" logic that keeps it awake with the fan running for a while (sometimes minutes) after closing the lid. If I just unplug the laptop stick it straight in a backpack, it'll keep doing this (getting hotter and hotter, and burning half of the battery capacity) until it hits the critical high temp shutdown. It's great fun taking it out at the start of a plane flight and finding out it's on low battery and has bbq'd itself.
- Even if I do wait for the fan to turn off before stashing the laptop, when I open the lid and wake it up, it immediately goes into hibernate mode, and I have to wait for it to finish hibernating, turn it back on, and wait for it to boot up, which is really frustrating.
The solution to both of these (for me) is to reassign the power button to be 'hibernate' instead of 'sleep', and to explicitly hibernate it every time I'm packing it up. It's still stupid and annoying, and a damn shame because it's otherwise a really nice laptop. The OLED screen is beautiful and the build quality feels great. I just wish it wasn't crippled.
Dunno about 2024, but 2023 G14 explicitly documents it can't work at full power when powered by USB-C, because it's USB-C PD supports only 100W compared to the >200W draw of the system. So it throttles just like macbooks with i9 (but those throttled always because one of the problems was overheating VRMs)
I don't expect max performance gaming on USB-C, but it should at least run as fast as it does on battery alone. It should definitely be able to update a mouse cursor and play an MP3.
This isn't just throttling, it's unusable. And it instantly goes away for a while when you disconnect and reconnect the USB-C power supply, even when gaming etc.
Oh, I fully admit that it could be tuned better. At least the 2023 G14 works reasonably well under linux this way, though I'm thinking of going through the ACPI and fixing some of that...
Your story reminds me a little bit of an issue with the late-Intel macbook pros. If your USB-C charger was plugged in on the left side, and you had an external display connected as well, you'd get crazy thermal throttling. I believe this was because the GPU was on the left side, and so the heat from the charging chip + the GPU resulted in the Macbook deciding to throttle the CPU and crank the fans to max. The solution was to charge on the right side.
Unfortunately you need to disable signing for that, which will trigger anticheat in online games and makes Windows nag you about it.
Someone wrote a bootloader specifically to patch ACPI on boot in CSM mode if you're okay with reinstalling Windows on an older system that can't play certain games: https://github.com/MovAX0xDEAD/ACPI-Patcher
This is what I was going to ask, but I assume that the ACPI.SYS driver (which wraps the ASUS ACPI BIOS implementation) is signed by Microsoft, which means "not unless you allow unsigned drivers", which I don't even know if that's possible in current Windows (my deep Windows-fu is about 20 years out of date).
The ACPI.SYS driver itself is not at fault here, it’s just a generic bytecode interpreter that’s the same in every Windows install. The problem lies with the bytecode—the “ACPI tables”—that the firmware of these specific devices feeds it.
There is nothing physically forcing it to run the code that’s stored in the motherboard flash, though; it could, say, use a patched version instead. The equivalent function is well-supported on Linux, because Linux uses a different interpreter (the reference one from Intel, in fact) and in general manages the hardware differently enough to regularly expose bugs in the ACPI code of manufacturers whose QC pass condition is “boots Windows” (all of them) and who can’t be bothered to fix bugs not affecting Windows (almost all of them).
>i'm caving for a world where hw manufacturers exist who fully embrace opensource development,
Those do very much exist! My go-to is System76. There are others, e.g. arguably Framework.
> and are rewarded for that...
Oh well, one can dream anyway.
The freedom Linux gives you also gives you the freedom to slap Linux on some random bit of Windows kit and then blame Linux for failing to work around the broken firmware. Apparently this is preferable to buying hardware that works with Linux.
... and people are looking forward to signed UEFI and ACPI on ARM systems too. How do they expect an ACPI written in a chinese sweatshop will work if Asus quality is this low?
The firmware is amalgamated from a bunch of vendors with core solution coming from few common ones.
Sometimes you get hilarious errors, like Intel not having any way to verify if their driver is actually loading a dumped memory image (intel rapid start), so if you forgot to disable Rapid Start and installed anything on a drive in the bay that was specified for rapid start, on boot the intel driver would just... blit it into RAM and be happy dumb
I don't follow. My X13s is a Snapdragon, has full uefi, and works like a normal laptop, mostly. It can't stop from overheating and has no cooling but that's besides the point.
I have a Zephyrus G14 and have experienced these issues. Audio crackling, lots of stuttering when playing back video on the iGPU. Throttling has been less of an issue for me. I'm on Linux Mint, so the problems are definitely BIOS related and not OS-dependent. Stuttering continued even after I manually expanded the iGPU's available memory pool in the AMD kernel settings.
The laptop was a nice deal on fire sale, but I guess I get what I pay for.
This is an amazing discovery, article, and fix proposal. Fantastic work, very impressive and also very instructive on how things work on modern PCs and how far you can actually dig to get at stuff that is "supposed" to be hidden.
As someone who has written embedded firmware for many years (not for PCs), I can only dream of an end user being this capable to discover a bug. I want to live in the world where Asus immediately send an e-mail offering some kind of short-term contracting work to fly in and talk to their firmware people for a few days and get $FIVE_FIGURES or something, and leave with an updated laptop running their new production BIOS.
Obviously this bug has gone un-fixed for four years so that is not the world we're in. That makes me sad. :|
Edit: s/fix/fix proposal/.
The technical RCA is fascinating, but im also interested in the business processes RCA.
- this sounds ubiquitous and reproducible. How did this not get fed back through tech support/RMA channels? Was there so little evidence that it wasn't correlateable, or did ASUS look and arrive at an incorrect conclusion, eg batch of bad silicon? Could it be that they had plentiful evidence and were negligent or incompetent?
- it sounds like this is plainly evident when using the machine. What is the QA process? This should not have been possible to miss?
- now that they know, what will they do?
Imo, the ceo calculus here is clear. If you're a luxury good with elastic demand, you fix the issue and fix the perception (two separate things). Multi-year, multifaceted issues like this have the potential to ruin a brand. I've bought ROG in the past, and I'm inclined to never do so again.
EDIT: on further reflection, the firmware bug itself is pretty troubling. the other bugs i get - hardware assumptions were changed, or good code was reused that didnt know or support the gpu mux, i see how those errors comes about. the method sleeping an interrupt... is awful? how did that get reviewed? what is that firmware test suite?
It doesn’t matter that the consumer/gamer laptop is a piece of shit, because all of the competitors are too. Consumer hardware is a volume business, and the actual end-user experience matters very little compared to endorsement deals and marketing strategies.
Every one of the affected ASUS laptops probably got a glowing 5/5 review from the usual suspects, and consumers have little hope of getting a fair deal
> What is the QA process? This should not have been possible to miss?
Have you used consumer goods [or virtually anything] from the last couple decades? By and large, nobody cares. Look at the timeline here; clearly nobody cares.
As a user and programmer, I can only dream of being this knowledgeable about things. There's a ton of domain knowledge embedded in the article, it's pretty amazing.
I managed to reverse engineer a lot of my laptop's features but hit a wall when it came to this ACPI stuff. I dumped the tables and decompiled the code but all I got was stub code. I wanted to be the guy who wrote the Linux drivers for his own laptop but I just didn't manage it. Massive respect for anyone who can do this.
Yeah the best way to go is to buy Linux preinstalled and supported. Though, as with Windows in this case, that still won't save you if the system integration and firmware teams don't do their job.
Yeah. ACPI's AML bytecode is sort of a mixed blessing. It allows for reverse engineering and end user analasys/fixing of bugs like this.
It's also just a terrible disaster of a programming environment, with a very large (terrifyingly so, given the limited capability) interpreter that needs to live at the highest privilege level of the kernel.
And it's generally used like a hatchet by system integrators for tricks like this, with pretty much exactly the code quality you'd expect. Almost always the path to writing a Linux driver for some oddball laptop subsystem starts with "throw away the ACPI stuff".
As far as I know there are three ACPI AML stacks, the reference intel one, linux uses this, miscrosoft has one, and those crazy hackers over at the openbsd project decided to make their own.
I think that's right. Though my understanding is that the Windows code is derived from the original Intel one too and has evolved in tandem with the Intel-maintained driver. And... yeah, acpica (drivers/acpica/acpica) is just huge; I checked again and it's at 2.5MB of source code. All for a DSDT table parser and a virtual machine with about the capability of a 6502.
sorry, what fix? the linked github page ends in "here is everything, ASUS, please fix it", right?
He knows how to fix it, and he fixed his system. It's not universal, you need a custom patch for each model and fw version.
Yeah, sorry, that was a bit unclear. I just meant that the article went as far as propose rather clearly what is needed to fix the issue ("don't sleep() in an interrupt service routine").
Truly awesome analysis, it's great that Asus spends this effort to quality check their hot garbage.. oh wait..
Impressive that they managed to ship crippling stuttering for 4 years in gaming laptops specifically. Makes you wonder about the end user psychology, evidently they didn't get a show stopping rate of product returns.
A quote from one of the linked reddit threads. I wonder if the warranty trip is part of their scheme.
"I did everything you suggested , but nothing changed. I send it back via garante. I am curious what they do whit it."
"what was it at the end? did they respond?"
"They have claimed that the plato works perfectly. So basically i just got use to it. I am using bluetooth earbuds all the time so i cant notice the problems."
I've owned 2 gaming laptops in my lifetime and both had similar issues that were never fixed.
One was the first gen Alienware M17 with two GTX 270M GPUs (yes two) and an onboard nvidia GPU whose specific model I can't remember. That one suffered from stutters and audio crackling, etc. It was sort of fixed by disabling SLI and the onboard GPU and sticking to a specific driver that was modded, the driver was by someone on the notebookcheck forums IIRC. Later on I think it got somewhat patched with a bios update that let you use SLI without the stutters, but I think the laptop reached EOL without it being fully fixed.
The second was an ROG ASUS laptop with a GTX 460m (I can't recall the laptop model). Pretty much the same story as the OP but I didn't have the knowhow to go deep into the ACPI code. The only change from the story is that latencymon kept attributing the latency spikes to multiple dlls, sometimes it was some wifi driver, other times it was an nvidia one. I don't remember the full fix for that one, but it involved me changing the wifi card and disabling the dGPU (not the onboard one) when I was not gaming so I could watch videos and such without it crackling. Funnily enough it didn't crackle much when actually playing games (it still happened, just very rarely).
I stopped buying gaming laptops after that. Seeing this story makes me think things haven't changed one bit.
I am frequently asked for hardware purchasing advice by family and friends. Starting around 2017 or 2018, if asked to recommend a "gaming" laptop, I have refused. I never had a good experience myself, and more often than not, what I had recommended over the previous years ended up flawed or outright broken. Across every OEM and brand. I tell them to settle for a professional/business SKU with a low-tier dedicated GPU, or give up on laptop gaming entirely. Is it worth the money to pay a "business" premium for a weaker system? No, and I'll tell them that. It's not a good deal in on-paper-dollar-for-performance terms. But at least there's a chance that all of the components function and are supported!
What use is a "good price", when what you get is a quality and support minefield?
I've owned two gaming laptops and never had issues like this.
First one was a Clevo (rebranded as Medion) with a GTX 970m that I bought in 2017. An absolute beast, I lugged it in a backpack around the world for 4 years, including to places you really shouldn't bring a laptop like beaches and rainforests. I passed it my girlfriend's nephew and it is still going strong and being used every day. I repasted once in that time.
My current laptop is an MSI GE66 with an RTX 3070m bought in 2022. It's loud, I've repasted recently because it started overheating. It had some problems with the screen connector which they fixed under warranty fairly quickly. But aside from that it's solid.
One thing about both of these laptops - they are very easy to open and it looks like I could repair/replace pretty much every removable component easily. No glue.
The only thing I consider a real problem is the MSI fan noise. Well, that and the power brick which is the size of a literal brick.
Glad to see that there are laptops that don't suffer like this. But I think the combo of having a steam deck + business laptop beats buying a gaming laptop. Assuming you already own a gaming rig at home.
20 years ago, I had a midrange laptop with a dGPU and while it played games with mediocre results, the laptop experience itself was also mediocre. Stupidly, 10 years ago, I bought a laptop with a dGPU again, but because NVidia didn't play nice with Linux back then, I don't think I've used that GPU for more than an hour or so.
Never again. A laptop with a dGPU runs counter to the things a laptop should be. Keeping gaming activities on a desktop is the best option in my experience.
A few months ago, I started working at an e-waste recycling company, and discovered that used Microsoft Surface tablets are what I've been looking for. My work "laptop" is a Surface Pro 5 with Debian (my work desktop is an Optiplex micro). I'm typing this on a Surface Go (with BlissOS) that I bought for myself. The cameras don't work on either and the work Surface never knows it's battery status, but I don't care (it lasts an entire afternoon with a barcode scanner, good enough for me).
I daily drive Ubuntu on this laptop. It has issues, but the Nvidia GPU is not one of them. Times have changed.
If you don't like a dGPU in a laptop, that's fine. But people have different needs. I travel a lot and do 3d content creation work.
Almost any Ryzen laptop these days will have faster integrated graphics than a SteamDeck just due to the age of the chip set Valve still uses.
I've had 2 now from different manufacturers and the firmware seems alright due to the integrated nature of the API making them all fairly homogenous.
Exactly, you can get decent integrated graphics with any recent Ryzen and you have lots of choice in form factor.
Well, I do play games on my laptops. But that's just a nice perk, what I bought them for is 3d content creation.
Also, comparing a steamdeck to a modern gaming laptop is like comparing a $1 water pistol to a super soaker.
That is my current approach as well. A steamdeck/switch/none + FW13 while traveling.
Laptops are vile piece of shit devices, as a rule.
All the complexity of a PC, in a package the size of a book, with the engineering quality of a Happy Meal toy.
I’m very had very different experiences, with MacBooks.
Well those are just so underpowered it literally hurts. All of disk size, expansion options and memory.
And booting something that isn't a funny variant of a locked down OS is relatively hard.
> Well those are just so underpowered it literally hurts. All of disk size, expansion options and memory.
That’s not a complaint have heard before. My needs aren’t huge and it has a lot more of everything than I need.
> And booting something that isn't a funny variant of a locked down OS is relatively hard.
I wouldn’t want anything else in it, but with a Mac mini I really wish it would run something Linux more easily. They are a great headless server, but the OS is really limiting.
MacBook Pro with 8TB of storage and 128GB of unified memory here.
Skill issue.
That is what happens when the industry has spent decades educating users that is normal computers are broken.
In any other industry everyone would be returning their acquisitions day one.
About 35 years ago, I had a teacher asserting computers are like buying shoes that randomly explode when tying them.
Thankfully consumer laws are finally happening.
GOD, thank you for saying this, I was thinking I was alone and going crazy.
How come people have become so obedient ?! That's crazy.
> How come people have become so obedient ?! That's crazy.
Most people have always been that way, will always be that way. IME the vast majority of people who don't stand for this shit are autistic.
I can tell you how I got duped into buying their product - a Zephyrus G14:
ASUS at the time had an exclusive deal with AMD to ship their Ryzen 4xxxHS line.
Initially it worked fine, but two years later performance was already much worse and dominated by thermal throttling. Repasting, though necessary due to the state of the paste, only helped partially.
I still don't know the root cause of the issue, but I investigated declining battery performance and it turned out that the iGPU was going full throttle at all times. Setting the dGPU as the preferred device actually improved battery life somewhat.
When mechanical failures started accumulating I switched to a FW16 and never looked back. I don't care what gaming laptop manufacturers have on offer and for how little if I can't buy having them give a shit about their products and customers.
I bought a Clevo-based gaming laptop for programming because it had a good CPU. It was a really bad decision.
It will thermal throttle itself to uselessness within seconds of a load being placed on it. The dGPU idles at about 15 W, the entire power budget of a single board computer, and it's one of those problematic nvidia GPUs that will never be properly supported on Linux. The Windows app that controlled things like fans and keyboard LEDs was so obnoxiously bad they required over one minute to show a window on the screen, reverse engineering that thing was one of the best things I've ever done. Mercifully the firmware wasn't broken by default but I still didn't manage to reverse engineer the ACPI nonsense, I dumped the tables and decompiled the code but there was nothing useful.
Looks like Apple has a monopoly on good taste and giving half a shit about the quality of the products they sell. I wish the Apple silicon macbooks existed at the time.
I was considering such a device since surely those huge vents meant good thermals, right? Glad I didn't go for it.
With the ASUS I had a setup with a cooling pad where the metal grid cover was removed and the sides were sealed with foam to enhance flow from the pad's fan and only with that I could maybe get 30-45min of gameplay until throttling started.
Meanwhile the Framework has overall much higher power consumption, but still manages to whoosh all that hot air out. I can't take these companies seriously if a much smaller business that is not focused squarely on gaming is running circles around them.
My mother rocks an M1 Air which she got for pennies and it's a great all around home computer.
This flaw only happens in Ultimate mode, when the user explicitly tells the mux to switch to the discrete GPU. This is an extra feature only users who primarily use the laptop for gaming with an external display care about.
The laptop works fine in Optimus mode even with external displays, you just lost a bit of performance and you're missing out on some display features like G-Sync. So it is highly likely that most users always use the laptop in Optimus mode. If you primarily use the laptop as a laptop you probably wouldn't even know the mux feature existed.
The problem is Asus shipping extra features in their hardware that are not properly QA tested. It looks like they only thoroughly tested the golden path.
Asus doesn't even test basic features, nevermind the extra ones. I have the 2017 Zephyrus GX501, which came with a Nvidia GTX 1080 which introduced HDMI 2.0 support. The Asus Zephyrus marketing material is boasting about HDMI 2.0 capability, the manual talks about HDMI 2.0 etc. However, in reality the device is limited to HDMI 1.4 bandwidth.
The problem isn't limited to some units, there was plenty of discussion online of this issue at the time of release. [1]
Asus never recalled, fixed, or even responded to the issue. Indeed, even the marketing page [2] still talks about how you can use HDMI 2.0 to connect 4K TVs at 60Hz.
It was also an interesting showcase of laptop reviewer incompetence. All the reviews just regurgitated Asus marketing material on how it has HDMI 2.0, but apparently nobody actually tested it.
--
[1] https://rog-forum.asus.com/t5/rog-zephyrus-series/gx501-zeph...
[2] https://rog.asus.com/laptops/rog-zephyrus/rog-zephyrus-gx501...
Imagine if we had strong consumer protection laws. Companies would fear claiming HDMI 2.0 that was really HDMI 1.2. Instead they just figure “meh! If anyone complains we’ll let them return it.”
> The problem is Asus shipping extra features in their hardware that are not properly QA tested.
For this specific case, getting no fix since the issues have been reported in 2021 is tough to brush over.
Asus already has a spotty reputation regarding to customer repairs and business practices, so this issue piling on top of that is unfortunate.
Seems to me this can't be concluded from the information at hand. This investigation used the mode, but the linked reddit posts I read complaining about ACPI latencies reportedby latencymon don't mention it.
Gaming laptop market feels like the most exploitative segment of entire laptop market to me.
They're like four-seater off-road motorcycles. You have to NOT understand how sketchy that concept is to consider one. The engineers has to know that they're guilty to be involved in it.
What's sad is that a lot of buyers are falling for it from the presumption that laptops are the most standard and regular type of computers. But I guess there's little we could do about it.
Windows laptop users are just conditioned to the fact that they don't work properly and just deal with the issues.
My 2013 Macbook also had a display mux that switched the laptop display itself between the integrated and discrete GPU. In some cases it would fail to properly switch the display over causing it to remain permanently off until the next reboot. I was not the only Macbook user with the problem.
Apple is just as guilty for shipping laptops with hardware issues that you just have to work around. And unlike this Asus issue the Macbook mux was on by default. You had to turn it off in the settings if you wanted to entirely avoid the issue and then you would have no way of using the discrete GPU.
As did my 2015 Thinkpad - same problem, both an integrated and an external GPU, and I was constantly fighting with the drivers to get it to just make use of the external GPU. I'm plugged into a wall whenever the laptop isn't suspended/shut down for transport, stupid machine, stop making Autodesk and my games crash because you want to boost your battery lifetime numbers!
They had a special Lenovo driver that would occasionally become overriden by Windows updates but could be reinstalled manually, I dual-booted Debian though and getting the system to work properly under that was a nightmare. There were a couple years when I simply gave up, I got it to work with the iGPU and I wasn't running anything more graphics intensive than a browser so I simply left the discrete GPU idle while running Linux.
Incredibly frustrating.
I think all these manufacturers are desperate to get their published specs for battery life estimates to double-digit hours that can't be reached while running the discrete GPU at full speed all day. Heck, they can't be reached while running the CPU at full speed all day, you're not going to run a 35W processor and a 55W graphics card and a 20W display (10W when you arbitrarily reduce the max brightness when on battery power) all day.
You've got like 90 watt-hours available in the battery, at 100% usage on everything the real capacity is gone in under an hour...which is unacceptable. So Asus and Apple and Lenovo and everyone else have to come up with some hack to turn it off whenever it's on so that the spec sheet says you can get 8, 10, 12 hours of runtime.
We had dozens of 2013 MBP with discrete GPU mux where I worked. I bought one afterwards and used it until the M1 laptops came out, for pretty much everything you can use a laptop for. Never had or saw this problem in any of them, FWIW!
Was it a Fall 2013 or Spring 2013 Macbook?
The quip against Windows users is besides the point of the article:
>Even installing Linux, only to find the problem persists. [...] >The problem is far deeper, embedded in the machine's firmware, the BIOS.
Anyway it's not as if the Linux laptop user experience in general were much better.
Yeah, that's not shocking. Linux uses the Intel ACPI infra, whereas Windows uses Microsoft's. It's as good as they can do, but it's not going to be possible to perfectly replicate Windows, let alone improve such clearly buggy firmware, at least generally.
It is possible on Linux to override some of the firmware (most notably the DSDT, e.g https://wiki.archlinux.org/title/DSDT because so much hardware is broken). So, if you can make or get a fixed version, you should be good. A wholesale replacement of all the ACPI assets, though, seems unlikely. I could well be wrong, though.
Anyway, in this case, I suspect the poster was advocating for Macs.
As if Apple land was free of issues, remember those wonderful keyboards, Snow Leopard, Tahoe among many other examples, or any Linux laptop for that matter.
Or the Installer.app, at least from 2015 to whenever they finally switched to image-based updates.
A percentage of users were "unlucky" to hit a bug where Installer.app would end up in infinite loop trying to unpack a pkg file when updating OSX. My personal record is minor update that ultimately took a week.
I don't see Mac or Linux Laptops being that much different in that regard.
A HN comment slandering people because of their OS/laptop choice? Never heard that one before.
[dead]
Apparently this happens only if you set the laptop to discrete GPU only mode, which most people won't do.
However, this is not the only problem with Asus bioses. My daughter has one and it randomly locks up if you add an extra SSD, sooner or later depending on the SSD. You'll blame the SSD's firmware, but the most locking one was one that I have in two desktops with no problems...
This makes sense, but what does not make sense is who tested this 'ultimate mode', I mean they went to the trouble of adding a physical hardware switch on the motherboard for this, surely when testing there was some kind of benchmark or comparison to show this feature was an advantage. Maybe they don't test, or they have 'internal firmware' that is not what the user gets, but it's a serious fail either way.
I don't know if anyone cares, is the problem. I used to have a £7000 Dell workstation laptop, specced to the moon - i9, 3080Ti, 128GB of ram.....that kind of thing.
The laptop was absolutely useless at playing games, because it would throttle itself thermally after about 30 seconds. Which was ironic given that I used to work at a games development company and the ability to play games was actually a core feature of the product. I then used to have a Razer Blade 15 which wasn't as bad but would also eventually start throttling hard - just inadequate cooling imho.
Funnily enough I have a much cheaper MSI gaming laptop now with an i7 and a 3070Ti and that never throttles, I can run games without it slowing down. But clearly the cooling system in it is massively overbuilt, which is great.
> I have a much cheaper MSI gaming laptop ... that never throttles, I can run games without it slowing down. But clearly the cooling system in it is massively overbuilt
Maybe they learned their lesson. I had an MSI gaming laptop a while back, and it ran horribly, I never realised until long after it was possible for me to return it, that it was just poorly designed, and could never run beyond ~50% of its gaming performance. Within minutes of starting a game it would be thermally throttled and that was that; it also sounded like it was about to take off, to the point you could barely drown it out with headphones.
I think they did - I used to have an MSI gaming laptop years ago with a GTX560M if I remember correctly, and that used to throttle every several seconds, so every game you played stuttered. This one - not a single slowdown, but if you open it it has like 10 heat pipes and two fans that sound like jet engines - but it clearly works.
I'm glad to hear at least one manufacturer is trying. I can't imagine myself buying a gaming laptop again, perhaps for my kids when they're old enough, but I stick to desktop now.
My main drive in choosing the MSI I did back then was the thinness and lightness, which was counter-productive to good cooling performance, mine had a GTX970M but was about 1cm thick; the bottom of the case got so hot it would burn you if you touched it after a while of gaming.
All the evidence points to corporations, in general, not caring about making products that actually work.
It always puzzles me how apparently well known flaws never mentioned by product reviewers, even genuinely pro-consumer and generally well respected, like rtings or notebookchecks.
You buy product after stellar review, encounter problem, search for solution, find reddit thread where everyone is "yeah, it is always like that, why do you act surprised?"
Why indeed?
It's not trivial to find it, you need to set it to dGPU only mode via mux switch and then let the LatencyMon test running for more than 120seconds.
(I'm not sure if it lags in igpu pass-through mode)
notebookcheck does report latencymon numbers and even remarks when it says the laptop is not recommended for real time audio:
https://www.notebookcheck.net/The-RTX-5090-Laptop-and-mini-L...
They don't see these extreme values though.
I really never did this question to myself, but if will happens I would start, trivially, looking for who sponsor those review sites.
Usually their revenue is from affiliate links. YouTube channels have ad spots, merch sales etc. There are many shills of course, but some are doing their best to give accurate reviews and don't shy away from bashing manufacturers for their missteps.
Yet widely known (to enthusiasts) problems, like stutters from the OP, are often not mentioned at all.
LinusTechTips doesn't depend on ASUS money in any meaningful way, but still failed to mention stutters in their Zephyrus G16 review. Some might say LTT is not a reviewer, he is an entertainer, but he undeniably thrives to be accurate while doing so.
This happens on a 2019 MSI gaming laptop (GS65 Stealth) I have too. Latencymon shows >10ms stutters within a minute of being open. Found out disabling all ACPI devices fixes the stutter. Also found out disabling all ACPI drivers disables the dedicated GPU. It's possible this affects a large portion of gaming laptops with dGPUs.
Here's a link to MSI forums also discovering the ACPI hiccups with Latencymon: https://forum-en.msi.com/index.php?threads/constant-micro-st...
Just google "nvidia gaming laptop stutter latencymon acpi"
I wonder if the "programmer" (and I use this term very loosely) who wrote that sleep-in-an-interrupt code ever tested the code personally, or if it was some other distant responsibility-diluted department of a hundred other lamers who didn't care "because the automated tests all pass". This is a situation where dogfooding, in the original Microsoft sense, would definitely be beneficial as among the developers experiencing this on their own machines, surely one would be tempted to fix it.
A long time ago I did some contract work writing firmware for a major hardware manufacturer in Taipei. I quickly learned to ignore bugs, because reporting them would get me reprimanded for doing things other than the task I was assigned. Even worse, the hardware team saw the firmware/driver/software devs as lowly servants and dismissed any feedback outright.
I am not surprised by this story.
It's crazy how the hardware sector just can't grasp that software is actually important too. Everywhere I've worked at we were just second class citizens, last to get a new hire or any budget, and then it was our fault when the software was subpar. It seems like they think going down to our level will actually sully them.
Short version: don't buy ASUS gaming laptops until this is definiteively fixed, and if you one under warranty, file a warranty claim, being prepared to go to Small Claims Court.
Most other laptops don't even have the mode you need to put the ASUS in to get this bug.
I don't think I ever put any of my laptops into dGPU-only mode via MUX, it's stupidly power hungry with little upside.
> it's stupidly power hungry with little upside
Unless you specifically bought that model because you want low-latency output to an external display with G-SYNC.
I've used Asus motherboards in my gaming PCs for years, and their BIOS/UEFI firmwares there are equally awful, their Ryzen AGESA stuff has been a complete mess.
My G14 functioned just fine with G-Sync in Optimus mode via nVidia connected USB-C port (and didn't need reboot to switch between GPUs) so I never found real use for the MUX mode. It's telling that most other gaming laptops don't spend on that piece of hardware.
The switch solves a fundamental trade-off: an external port can only be physically connected to one GPU. If the dGPU you get GSYNC/etc but high power consumption if you connect an external display.
If the iGPU, then the dGPU is basically sending its frame buffer through the iGPU and is limited by it.
I don't think I ever put any of my laptops into iGPU-only mode, it's stupidly low-performance with little upside.
It sounded to me like they were multiple bugs. Some were specific the mux mode, but sleeping in an interrupt, and artificially re-arming it is problematic either way.
Sometime around 2015 I promised myself to never buy a laptop with switchable graphics again. This has worked well so far.
But it never ceases to amuse me watching brands that position themselves as 'premium' spending pennies on firmware development team somewhere down in a basement compared to millions they spend on shiny marketing.
No wonder people end up pushing macs.
It's unbelievable that something this bad has been shipping for four years. I guess I know what I'm not buying, at least...
Apple also had issues like these. For example this one where it also first denied the issue: https://9to5mac.com/2019/10/24/efi-firmware/
They have their own issues as well, for those of us outside the distortion field.
The big difference is Apple can't hold other companies accountable since they delivered the software, the hardware and the integration and they have an incentive to fix their stuff. The few issues I've had with Macs the last decade did get fixed within a reasonable amount of time. Lots of PC hardware bugs simply don't get fixed at all since there is no party that really cares about them. The author of the linked write-up should have been an Asus employee instead of a disgruntled user...
How many years did it took to replace the keyboards, or the famous antenna issue?
And let's not even start digging on Apple Radar's backlog, which always makes for some entertaiment on Apple related podcasts.
Hardware issues are tough to fix, but they did iterate on the butterfly keyboard and eventually gave up and switched to a different design. My experience with smaller software and compatibility issues is they get fixed eventually if enough people notice and report them.
To fix something, first one needs to acknowledge it, and that took several years.
Have you missed the OS X releases that were all about bug fixing, and still there are plenty of left overs?
Lets see how many Tahoe is bringing.
I had a mac at work for 8 years. Overall things mostly worked ok, but I had two big issues.
a) one time charging stopped working... thankfully I had a pretty full charge when I noticed and was able to migrate my data to a spare machine and not have to deal with it... removable storage would have been super handy.
b) for a whole year, there was about a 25% chance of loud static instead of music when I started playing a stream in iTunes; pause and play again would fix it most of the time. It started when I installed a named OS revision, and it stopped when I installed the next version. Did not have issues with sounds from other apps. Of course, there was no information to be found anywhere about this, because 'macs just work'
Less big, but if Outlook was running when I put the laptop to sleep, there was a good chance it would continue to eat battery and generate heat in my bag. Outlook is a travesty, but when the corp runs Exchange, Outlook is less effort to make work compared to fighting to make auth work with anything else and then still having to use Outlook from time to time.
Reminds me of when MSI laptops were getting properly bricked after users ran `rm -rf /` because of a UEFI bug where the board could not boot after some variables got deleted https://github.com/systemd/systemd/issues/2402
> pushing macs
Would you push gamers/VR people to macs ??
Congratulations ASUS for 0.01% of your marketing budget you could have fixed the experience of your millions of users, reduced the amount of replaced computers, improved your brand positive views... This once again proves that many companies are absolutely mismanaged and think marketing will make them more efficient than good engineering...
Excellent sleuthing. I feel like this has some great quick debugging ideas for system wonkiness that I never knew existed.
It feels a bit of a shame to wrap it all up in an AI-written summary, but I guess if that was the only way to get the info out, so be it.
One laptop model with buggy ACPI down, 5,387 to go.
Yeah, please let this guy tackle Lenovo next.
My Lenovo X1E regularly burns 20% of its CPU cycles on some high frequency recurring interrupt. I did get pretty far with debugging it, but eventually gave up since I can't justify spending so much time on fixing a 'professional' laptop that I paid top dollar for.
It also has a multi-GPU setup that has never worked reliably under Linux, which is ironic as I opted for Lenovo due to its supposedly good Linux compatibility.
Switching between GPU modes is a hit or mis, waking up from stand-by often results in a blank screen, screen flickering, sporadic high fan speeds, etc. And then there's the coil whine, which seems to be fixed in some BIOS versions, then returns in the next. Supposedly it has something to do with power-saving measures.
Since I owned it there have been at least 20 BIOS version releases for 'improved performance and security', but none seem to actually fix anything. It's such a mess.
/rant
It says it affects all ASUS laptops since 2021, making them stutter at the most basic tasks.
Which I'm ready to believe, knowing the state of most laptops... but this entire thing is pretty clearly generated by Gemini with its over-the-top dramatic style, italics emphasis, and -isms like "It's not just X, it's Y", which was unable to handle the article of this size and started looping over. Not sure I should believe any of it, or at least be sure that it didn't mess up the specifics. Why would one do this in a technical writeup?
I don't know why everyone uses AI for important writing now. The other day management showed us a ChatGPT presentation apologising for being terrible at managing.
I have an older ASUS laptop from 2015 which also has (more minor than this!) ACPI state management bugs. I initially bought that machine because it was a pretty high-end and was somewhat disappointed about both the build quality and the firmware/software support.
Somehow laptop makers all write complete garbage firmware.
I'm sure dell does the same terrible handling of DGPU power and badly written ISRs that pointless raise system latency. I had shoddy crashes for months that would cause my dell laptop to BSOD and burn up in my backpack because the DGPU got stuck on I a loop during some ridiculous windows modern standby wakeup.
Only devices I've had which seem to work flawless are macbooks and the steam deck. The ability to properly suspend and wake without issue is so rare.
I'm a big fan of my Deck, but I think Steam-related software projects always have a distinctive low-level stink.
I use mine on a train that has Wi-Fi with a captive portal and attempting to join it makes the whole UI unresponsive. Using the overlay with a guide in a game always resets the scroll location. These are the kind of things I can live with, but also things I don't expect ever to be fixed unless Valve come out with some wholesale replacement for some overt new strategic goal.
Lenovos in my experience are pretty flawless in both Windows and Linux -- my current laptop is a Lenovo and sleep/wake has worked flawlessly in Debian and Windows 11. My work laptop is a HP and sleep/wake also works fine in Windows. I've had the occassional power issues (including failures to wake from sleep) with Intel MacBooks previously for both personal and work laptops.
I have a Steam Deck and love it but it still cannot suspend and wake without the audio crackling issue - other than that it is magical though. (The pause games decky plugin is a third party solution that fixes it!)
Macbook essentially involved Apple hardcoding 11th hour fixes selected by product code in their OS level drivers after utterly fucking up ACPI tables in ways that would make the discussed bug blush. Pretty much most of what BootCamp did on later models, and what various hackintosh "special drivers" did, was fix things like HDA tables located in completely wrong place.
Steam Deck is just... extensively tested and debugged in ways that I don't think Apple did unless they got an egg on the face in national media (remember "you're holding it wrong?")
I was pretty disappointed when my work MacBook ran into the same "Bluetooth headset connected, set as audio sink, but the OS refuses to acknowledge its existence further and routes audio through the speakers" but that I've seen on every other desktop OS. I really hoped Apple would be better, considering their hardware costs more than twice as much.
I also ran into weird Wi-Fi issues that required a reboot, and getting that thing to recognize external displays without corrupting video is some kind of dark art while my Lenovo and Steam Deck work just fine with the same USB C plug.
Apple beats some brands for sure (especially the cheap "consumer" lines with a starting price lower than Apple's headphones) but their computers are hardly flawless.
I have yet to run into issues with my Steam Deck, which is very impressive, but I'm sure I'll run into an issue at some point. No computer works flawlessly.
Did you have these issues on Apple Silicon MacBooks? I ask because they seem to have very different networking and display stacks.
Surface Pro 8s have a bug where the battery just gives up randomly and won't ever charge again. 2, 3,4 and 7 do not have this - one sitting on my shelf two years charged right up from 10%.
Replacing the battery costs like $200-500 because the screen likes to explode when removing it.
Lenovo docks of a specific gen will have the USB hub/billboard just crash and stop doing display port.
older Dell docks would pollute arp tables and crash switches.
Computers have always had some wild flaws, some worse than others. They are built to a price point typically and by humans under politics so the best design or parts usually don't make it -- cost and profit.
article is great but it feels like its rewritten by an llm. " a crucial insight", "occurring like clockwork"
The author disclosed that
> I used an LLM for wording. The research, traces, and AML decomp are mine. Every claim is verified and reproducible if you follow the steps in the article; logs and commands are in the repo. If you think something's wrong, cite the exact timestamp/method/line. "AI wrote it" is not an argument.
why is that a problem..?
Maybe the author is ESL or just not very good at writing...
If it's clearer and the information is still all correct - then isn't that great? More people can engage in clear communication with each other
>and the information is still all correct
That's a pretty big if in technical writeups like this, all you do by rewriting those is obfuscate the actual inputs you had. Was is generated from scattered notes? Entirely vibe-written? How many details are actually verified to be correct by a human? Seeing how even the structure seems generated, it's clear that there was little input, and I'm not sure about any of the above.
I can deal with poor writing, and in case of ESL it's enough to tell the LLM to proofread/rephrase the piece (and check it yourself afterwards). But lazy generations just make you trust the article less.
> Was is generated from scattered notes? Entirely vibe-written?
Why does that matter? Maybe the person hates writing. You need people to suffer and put effort for the end result to be worth your time?
> How many details are actually verified to be correct by a human?
I mean.. assume the best..? The author could also have written it by hand and just lied. Or he's a paid troll from an Asus competitor - it's all just made up and it's a work of fiction. You implicitly have to assume the author tried their best and be okay there might be some errors.
If the writing is more clear thanks to an LLM then you're likely to catch errors more easily.
If you feel the thing has errors, then engage with the material and point out the errors.
You're not judging the end result on its merits
This is literally the same discussion when some people insisted that nothing on the internet can be trusted and physical books are the only reliable source of information.
The decompilation might be interesting but the prose is full of sheen and puffery.
It's like someone took a technical report from a bug tracker and ran a linguistic obfuscator on it.
I don't even bother anymore, it feels like word vomit to me. This could probably be half its length.
I'm honestly a little surprised that ACPI has a Sleep()-like function at all, I can't really imagine many situations where the firmware actually wants the user operating system to wait for any real length of time, as that would block any other ACPI events, even if it wasn't in an interrupt handler. I feel there's pretty easy ways to deadlock the system in that sort of situation.
And also surprised that windows actually allows the ACPI driver to sleep in an interrupt handler - on Linux that'll immediately BUG()... Unless windows doesn't and the above ACPI blocking is what they're measuring here.
Friends don't let friends buy laptops with nvidia GPUs. I can't believe optimus is still such a mess.
This could also have happened on a laptop with a discrete AMD GPU and SmartAccess MUX, this time it’s not really an Nvidia fault.
The problem here is not with Optimus though.
When we're buying work laptops, them not having discrete GPUs is a feature I insist on.
On the (these days) very rare occasion I game, I have a desktop.
Reminds me of a cursed HP keyboard I had at work once. You plug it into USB, then your irq rate goes way, way up. All keyboard events telling you the key pressed last millisecond is still pressed. You literally gained 5 to 10% speed when compiling without it. Apart from that, and the way its sleep-state-destroying irqs ate battery, it was a nice keyboard, so some data entry person got it in the end.
ASUS should hire someone in QA with the original github author's skills. I know that will (most likely) never happen...
Out of curiosity, why not release BIOS mod with a fix? Atleast personal laptops (out of warranty) can benefit out of it until Asus fixes their sht.
People blame Windows being slow and etc but most of the times hardware manufactures don't even get into this level to make best out of thier hardware. This is the reason why Apple is so successful, they control hardware, software while in open world, software like Linux/Windows is written by someone while hardware is designed by someone else.
Maybe the risk is too high ?, bricking UEFI means time for soldering/reflowing, because you won't be able to software-recover it(unless it has dual bios mechanism).
not that simple to apply necessarily. lots of security junk on modern bios. keeps ya safe, ya know.
Perhaps there is firmware signing.
I dont think it even needs a bios mod, i think you could get away with updating the acpi tables ( See https://docs.kernel.org/admin-guide/acpi/initrd_table_overri... )
Right, and Windows probably has something similar (maybe it requires loading a driver). This is BIOS-style code interpreted by the OS, so you can patch it at the OS level.
So if I understood it right, every single Asus gaming laptop have easily measurable stutters? That's insanity. And I thought about buying Asus RoG Strix laptop, because for some reason marketing made me believe they're the best gaming laptops out there.
FWIW, I'm using an X13 for some standard gaming, but mostly VR and it performs as advertised for about a year and half now. I'd have sent it back immediately if it stuttered every 30 sec with cracking sound.
I won't defend Asus, they have a crowd of skeletons in the closet and need to pay more attention to feedback, I'm just saying there will be a set of use cases were issues won't be plagging the experience.
My HP screen (HP Aero 13, not a gaming laptop, with a integrated gpu only) does flicker, turning completely off and then on, and this issue doesn't appear when connecting to external monitor. The same happens under linux as well. This post had me curious about the ACPI now... maybe I can follow along !
Excellent article.
I do not have the same technical depth to dig this far as the author, but this kind of problem seems pretty common on laptops, especially those with "switchable" iGPU/dGPU setups.
I had an Acer laptop about 7-8 years ago with almost the exact same latency symptoms. In the end I just disabled the dGPU in the BIOS (since I only used it for office work), and that instantly solved the issue.
This kind of thing is very infuriating because not only is it hard to track down the root cause (which I am very grateful the author did), but it is also even harder to get the vendor to actually acknowledge or fix it.
Intel laptops with igpu and Nvidia chips and thunderbolt docks are absolutely a nightmare between the 3 trips video has to go to even leave the laptop and the drivers and firmware mismatches.
I've had weird issues with this setup since the core2duo days upgrading once a year.
when it works it's amazing, however.
AMD igpu and dgpu work super well together but I feel like over time, since I use this configuration the most things either improve or go to hell with driver updates. depends on the laptop OEM really.
This all said, where the hell are the strix points igpus where they rival desktop cards (yes the lower mid end) at laptops where stuff just works without compromise ...if there is power and cooling.
Side note - I have a rog g14 that until I loaded a beta bios for thunderbolt over usb4 would reboot when shut down and shut down randomly. (amd CPU and GPU)
This really underscores how complex a computer is these days. This complexity is why you get a ton of little glitches when you slap Linux on a Windows PC--and this is even on Windows, as designed!
I have one of these, a Zephyrus G15. That it had an AMD CPU and Nvidia GPU should have been a red flag that support would be really poor. Only a year out of warranty, it is a brick on a shelf because the thermals are so atrocious it pretty much burned itself out, and even with a thorough new application of thermal paste through a multi hour process there just isn't any way to get it to perform within spec. Supposedly, if you RMA it through ASUS they will charge you something like $700 and be unlikely to fix it. They have an insane dud rate, and even when it does work the hardware is barely hanging on. Several acquaintances have had similar problems.
It drastically reduced my perception of Asus as a brand - I wanted something I could game with, it promised the moon of portability and performance but they couldn't pull it off.
Curious, my experience could not have been more different. I still use Asus Zephyrus G15 (GA503QR from 2021) as my main development machine, and also game occasionally on it. It's AMD Zen 3 CPU with GeForce 3070.
As I write this, the fans are either completely stopped or so slow that I cannot hear them. The fans can get a little "whooshy" under load, but nothing out of line with other Windows laptops, as far as I can tell.
I'm running under Windows profile (not one of the ASUS-specific ones in Armory Crate). I also limit the battery to 60%, since I'm mostly using it plugged-in, with external monitor and keyboard. A month ago, I upgraded RAM to 40 GB and added another 4 TB SSD (and cleaned the fans in the process, so it's even quieter). I think I'll keep this machine for a few more years...
That being said, I did experience occasional cursor stuttering, but mostly when the machine is under load (typically during Visual Studio build).
I believe I have the same one! For me, almost out of the box I needed to disable CPU boost due to the thermals and instability. That worked for a while, though I considered replacing it and didn’t because there was just no others available at the time due to the pandemic. Apparently my issue is common but not the majority - a lot of folks seem to just have gotten a bad batch. I’m curious though, have you not even had any issues with sleep mode on the laptop? It seems extremely common that it just doesn’t work/crashes.
I got a good 2.5 years out of it, but I was hoping to use it for at least 6 or 7.
> have you not even had any issues with sleep mode on the laptop?
Now that I think of it, I did disable sleep and just use hibernate instead. I no longer remember why - perhaps I too had some issues. Maybe I was just annoyed by Windows restarting in the middle of a night and (c)losing my open applications whenever it wishes to update? Or did I disable Wake Timers for that? Sorry, my memory is a bit fuzzy.
As it is, hibernate works perfectly for me, is fast enough, and it never closes my applications behind my back.
OTOH, I had sleep issues with pretty much all PCs that I ever had (be it laptop or desktop), so maybe it was just inertia to always use hibernation instead. :)
What are the equivalent tools on Linux to capture those traces / events / interruption ect...?
Now I wonder if this is why my laptop (Dell) is so goddamn annoying too.
LatencyMon does show elevated ISR and DPC max exec times (i.e. reports that for realtime tasks my system is busted), and they're also primarily carried out by CPU0. But it's on the order or 2 ms, not 30 ms.
Still a long time though for interrupts I guess. ACPI.sys taking the lead too, with NDIS.sys being close second, and then pretty much nothing else.
I have a 2024 Zephyrus G14 and it has bursts of stuttering which seem to be directly linked to running off USB-C power. It doesn't do it on the original power brick, but on a 70W USB power brick, it slows down massively every now and then, to the point where the mouse cursor is only updating every few seconds and any playing audio starts underrunning buffers. Unplugging USB power immediately clears the issue up for a while. It's fine running off battery, and it's fine when I plug USB power back in, even straight away.
It does other stupid things with power management, too:
- There seems to be some "cooldown" logic that keeps it awake with the fan running for a while (sometimes minutes) after closing the lid. If I just unplug the laptop stick it straight in a backpack, it'll keep doing this (getting hotter and hotter, and burning half of the battery capacity) until it hits the critical high temp shutdown. It's great fun taking it out at the start of a plane flight and finding out it's on low battery and has bbq'd itself.
- Even if I do wait for the fan to turn off before stashing the laptop, when I open the lid and wake it up, it immediately goes into hibernate mode, and I have to wait for it to finish hibernating, turn it back on, and wait for it to boot up, which is really frustrating.
The solution to both of these (for me) is to reassign the power button to be 'hibernate' instead of 'sleep', and to explicitly hibernate it every time I'm packing it up. It's still stupid and annoying, and a damn shame because it's otherwise a really nice laptop. The OLED screen is beautiful and the build quality feels great. I just wish it wasn't crippled.
Dunno about 2024, but 2023 G14 explicitly documents it can't work at full power when powered by USB-C, because it's USB-C PD supports only 100W compared to the >200W draw of the system. So it throttles just like macbooks with i9 (but those throttled always because one of the problems was overheating VRMs)
I don't expect max performance gaming on USB-C, but it should at least run as fast as it does on battery alone. It should definitely be able to update a mouse cursor and play an MP3.
This isn't just throttling, it's unusable. And it instantly goes away for a while when you disconnect and reconnect the USB-C power supply, even when gaming etc.
Oh, I fully admit that it could be tuned better. At least the 2023 G14 works reasonably well under linux this way, though I'm thinking of going through the ACPI and fixing some of that...
Your story reminds me a little bit of an issue with the late-Intel macbook pros. If your USB-C charger was plugged in on the left side, and you had an external display connected as well, you'd get crazy thermal throttling. I believe this was because the GPU was on the left side, and so the heat from the charging chip + the GPU resulted in the Macbook deciding to throttle the CPU and crank the fans to max. The solution was to charge on the right side.
https://eshop.macsales.com/blog/61253-power-your-macbook-pro...
I hacked the ACPI firmware on my system, linux is able to apply "my firmware" rather than use the operating system supplied firmware.
Does anyone know if windows can do the same ?
Microsoft documents replacing the ACPI table with their API: https://learn.microsoft.com/en-us/windows-hardware/drivers/b... (bottom of the page)
Unfortunately you need to disable signing for that, which will trigger anticheat in online games and makes Windows nag you about it.
Someone wrote a bootloader specifically to patch ACPI on boot in CSM mode if you're okay with reinstalling Windows on an older system that can't play certain games: https://github.com/MovAX0xDEAD/ACPI-Patcher
IIRC you can resign everything with your own key, but anticheat might still explode about it.
This is what I was going to ask, but I assume that the ACPI.SYS driver (which wraps the ASUS ACPI BIOS implementation) is signed by Microsoft, which means "not unless you allow unsigned drivers", which I don't even know if that's possible in current Windows (my deep Windows-fu is about 20 years out of date).
The ACPI.SYS driver itself is not at fault here, it’s just a generic bytecode interpreter that’s the same in every Windows install. The problem lies with the bytecode—the “ACPI tables”—that the firmware of these specific devices feeds it.
There is nothing physically forcing it to run the code that’s stored in the motherboard flash, though; it could, say, use a patched version instead. The equivalent function is well-supported on Linux, because Linux uses a different interpreter (the reference one from Intel, in fact) and in general manages the hardware differently enough to regularly expose bugs in the ACPI code of manufacturers whose QC pass condition is “boots Windows” (all of them) and who can’t be bothered to fix bugs not affecting Windows (almost all of them).
i'm craving for a world where hw manufacturers exist who fully embrace opensource development, and are rewarded for that...
>i'm caving for a world where hw manufacturers exist who fully embrace opensource development,
Those do very much exist! My go-to is System76. There are others, e.g. arguably Framework.
> and are rewarded for that...
Oh well, one can dream anyway.
The freedom Linux gives you also gives you the freedom to slap Linux on some random bit of Windows kit and then blame Linux for failing to work around the broken firmware. Apparently this is preferable to buying hardware that works with Linux.
... and people are looking forward to signed UEFI and ACPI on ARM systems too. How do they expect an ACPI written in a chinese sweatshop will work if Asus quality is this low?
I assume all of these gaming laptops are largely developed by outsourced sweatshops today so I don't imagine much difference.
The firmware is amalgamated from a bunch of vendors with core solution coming from few common ones.
Sometimes you get hilarious errors, like Intel not having any way to verify if their driver is actually loading a dumped memory image (intel rapid start), so if you forgot to disable Rapid Start and installed anything on a drive in the bay that was specified for rapid start, on boot the intel driver would just... blit it into RAM and be happy dumb
I don't follow. My X13s is a Snapdragon, has full uefi, and works like a normal laptop, mostly. It can't stop from overheating and has no cooling but that's besides the point.
> I don't follow.
Open source devicetree + u-boot can be maintained independently of any manufacturer's support.
Great write up. As annoying failures are, always learn most from debugging sessions
I have a Zephyrus G14 and have experienced these issues. Audio crackling, lots of stuttering when playing back video on the iGPU. Throttling has been less of an issue for me. I'm on Linux Mint, so the problems are definitely BIOS related and not OS-dependent. Stuttering continued even after I manually expanded the iGPU's available memory pool in the AMD kernel settings.
The laptop was a nice deal on fire sale, but I guess I get what I pay for.
Back when Hackintoshes were a thing I remember that I had to tinker with ACPI bytecode stuff in order to get the battery icon displayed on the screen.
Unfortunately I can't remember the details now.
[dead]
[dead]