So ... the subtext that a lot of people aren't calling out explicitly is that this round of new APIs has been done in cooperation with the big engines. The Mantle spec is effectively written by Johan Andersson at DICE, and the Khronos Vulkan spec basically pulls Aras P at Unity, Niklas S at Epic, and a couple guys at Valve into the fold.
This begs the question: what about DirectX 12 and Apple's Metal? If they didn't have similar engine developer involvement, that's clearly a point against DX12/Metal support in the long run.
Also, the end is worth quoting in its entirety to explain why the APIs are taking such a radical change from the traditional rendering model:
Personally, my take is that MS and ARB always had the wrong idea. Their idea was to produce a nice, pretty looking front end and deal with all the awful stuff quietly in the background. Yeah it's easy to code against, but it was always a bitch and a half to debug or tune. Nobody ever took that side of the equation into account. What has finally been made clear is that it's okay to have difficult to code APIs, if the end result just works. And that's been my experience so far in retooling: it's a pain in the ass, requires widespread revisions to engine code, forces you to revisit a lot of assumptions, and generally requires a lot of infrastructure before anything works. But once it's up and running, there's no surprises. It works smoothly, you're always on the fast path, anything that IS slow is in your OWN code which can be analyzed by common tools. It's worth it.
I wonder if/when web developers will reach a similar point of software layer implosion. "Easy to code against, but a bitch and a half to debug or tune" is an adequate description of pretty much everything in web development today, yet everyone keeps trying to patch it over with even more "easy" layers on top. (This applies to both server and client side technologies IMO.)
In its early days, OpenGL benefited in many ways from being a "forward-looking" API that exposed far more functionality than was implemented by graphics hardware. Software that used those features got faster throughout the 90s with no rewriting needed as that functionality became available on GPUs. (Of course, software like games that wanted to be fast now just avoided those features.) When graphics hardware started to include features that hadn't been anticipated in OpenGL, things got messy and stayed messy.
Now that the feature set of graphics hardware is very stable and uniform across different vendors' hardware, the APIs are needed to solve an entirely different problem from what they were invented to do.
About your last paragraph, I would say that that is even more important to optimize and to make low level layers for than graphical game engines (believe it or not :)).
The thing is while gaming is seen as a "performance critical" application, where it is natural to want the maximum performance to be squeezed out of your hardware, Web development is just a behemoth of inefficiency, and we think that that is the norm, an inevitable drawback of developing for the web, an inescapable trait of web applications. Web apps are dozens or even hundreds of times slower than native applications, and we think nothing of it, because it seems to be an inherent characteristic of the web about which we can't do anything.
Your comment made me reconsider that. Maybe it doesn't have to be so. What if there was a Vulkan for the web, instead of the layers upon layers of inefficiency which are all the rage these days? What if I could run a medium complexity web page or app without it bogging my computer several times harder than a comparable native app?
Let's imagine a world where a browser is just a secure place to execute code. You get vulkan, io, networking, input, camera access in a cross platform way.
So, how do search engines find any info? How do you share links to content? How do you support braille readers? Browser extensions?
I'd argue those things are pretty essential to the web as we know it (well maybe not the extensions) and I might also argue that some of the inefficiencies of the web environment are a direct result of making the decisions that the most important part of the web is a common semi scannable and linkable format, HTML.
The web these days is very clearly diverging into two separate things: a portable application runtime/platform, and the document publishing system it started as. I don't see any way to accomplish a clean split between the two, but it looks increasingly necessary.
The new medium has been born, but the old one is still being abused, so the split isn't complete and won't be until this no longer happens: http://xkcd.com/1174/
The worst is LinkedIn, which will send you an e-mail with article links it it, and if you click the links it takes you to your browser, which does the thing that xkcd comic describes if you don't have the app installed, but if you do have the app it just launches the app and leaves you at the home screen, and the context of the link you clicked on is gone.
I'm wondering about the "can't zoom" bit. I can't zoom on many websites - because Mobile Safari has sold out control to the web developers, instead of acting as my user agent.
Good luck. Efficient 2D using OpenGL is very difficult. It will never approach the speed of a well designed 2D API. Microsoft gave a lot of insight about the hard it was to get Direct2D to be competitive with GDI/GDI+.
If you do see any performance benefits it likely will be because you support less features than what a modern day browser supports with HTML/CSS. But in my opinion you could probably achieve faster speeds with Canvas in a similar manner.
I agree it will be very difficult. I'm working with WebGL as my day gig and understand how hard it is, as we're working with embedded devices there. That's actually what motivated me to start this project, so web developers can build fast, rich and interactive UI's on the same level as those you see in current generation video game consoles.
It depends on what you are doing. Rendering thousands of sprites with independent rotation and scale is easy. Rendering text is a lot of work, but once you do it you can do all sorts of things with the text. Rendering simple things like bezier curves is very difficult. There are some things in the 2D world that HTML/CSS/GDI/GDI+ are very bad at, compared to OpenGL, and vice versa.
Efficient 2D using OpenGL is very difficult. It will never approach the speed of a well designed 2D API.
All web content on OS X and iOS is rendered using OpenGL. That's not to say it was easy to make efficient, but it definitely outperforms the default CoreGraphics implementation on most content.
Edit: Seriously, how feasible is that? Creating a GUI framework on top of WebGL and creating an abstraction layer (which the original article advises against) would be a huge performance boost comparing to current webpages, wouldn't it?
WebGL should never have been invented, at least on current architectures. It's a ticking bomb, precisely for the reasons this articles explored - drivers are complex and buggy black boxes that lie in kernel space so ring 0 access is never too far from reach for many exploits. WebGL has a potential to be exploit vector much more severe than what we've seen with other auxiliary web technologies like JavaScript or Java applets. Why not confine web to (mostly) nicely behaved x86 VMs?
Personally I think browsers will become service layers that expose the underlying OS in controlled and specific ways. Not a completely sandbox, but not a free for all either.
We've started creeping that way with a lot of the newer HTML5 stuff, but we have a long way to go.
Looking at the bug bounty record from Chrome and Mozilla it seems that fuzzing-for-bounties vulnerability research has largely moved away from WebGL. Must be at least partly due to WebGL reaching a reasonable level of robustness. Bounties may be harder to claim since bugs can be hardware/os/driver ver etc dependent, but still there used to be a lot more WebGL bounties dealt out.
To be fair, I think the takeaway isn't don't make/use abstraction layers, but don't make a one-size-fits-all abstraction and then hide the lower level stuff. A targeted abstraction that correctly supports the ___domain you are in is still a good thing. That's what the engine developers end up being here, an abstraction on top of this low level API that gives many devs exactly what they want.
Edit: To be clear, I don't think writing a web rendering engine meant to do the same think as the browser on top of WebGL will be faster in any respect, as really the HTML and CSS are just inputs fed into super optimized low level rendering engines. I don't see how JS could compete in that ___domain.
Reminds me of MIT's Exokernel- make the lower level interfaces as low-level as possible (while still providing security) and let user-level libraries provide various abstractions suited to different use cases.
No file systems, no network protocols, no processes; just a very thin layer that lets userspace implement those if/how it wants to.
Parts of CSS3 are already faster to implement "by hand" in javascript rather than using the browser's "native" implementation.
HTML/CSS was designed for requirements that don't apply to many modern websites. I would not be surprised if it were possible to implement a rendering engine that performed better for relevant use cases, e.g. perhaps assuming a grid as a first-class part of the layout.
Wouldn't that kill accessibility, selectable text, being able to call system services (i.e., on OS X hitting cmd+shift+D to get a dictionary definition of a word), any extensions which modify the page hierarchy, automatic translation/localisation, and so on.
Exactly; bypass this entire service layer maintained by thousands of engineers and try to do it in Javascript?! I think that's probably not a good idea.
It's definitely feasible! A goal for my side project is to give developers a React-like declarative UI with a WebGL rendering backend that abstracts away the graphics programming.
yup. i have started to see the same as well. having a simpler web solution is hard to achieve, over the time, it get messier and messier.
i'm now doing small and simple examples in newer language such as go to see if it is any easier there to keep software simple to 1)run 2)maintain 3) code.
> I wonder if/when web developers will reach a similar point of software layer implosion.
Will doing things the "good but hard" way force me to write tons of boilerplate? Because as a (currently iOS) programmer, as soon as I start having to write generic, structural code, I find it difficult to sustain my interest in the project. I want to go from idea to something that works as quickly as humanly possible; having to remake the universe from scratch just because somebody somewhere needed the performance is not what I got into the game for.
(I'm currently working with OpenGL for the first time, and, ugh, all the state management stuff and shader boilerplate is driving me crazy — especially after years of the relative ease of dealing with UIKit! Am I to understand that Vulkan will only make things worse in this regard? I guess it's good for engine devs, but I would not want to code like this in my day-to-day.)
600 lines of code really isn't that much. At my day job a single file can be 2-6x that long, and there are a lot of files to deal with. Any project that gets beyond "Hey here's a neat little snippet of code that does something" will very quickly begin to dwarf those 600 lines. Also, from the original post talking about the new API and current graphics drivers, every developer is already implicitly using a couple million lines of code by using the OpenGL/DirectX APIs. They're just hidden from the developer in a closed-source binary.
"This begs the question: what about DirectX 12 and Apple's Metal? If they didn't have similar engine developer involvement, that's clearly a point against DX12/Metal support in the long run."
There's several parts to this. DirectX 12 is going to run on Windows 10, and its adoption by developers is probably going to be entirely driven by how much of their target audience can be persuaded to upgrade from Windows 7. If Windows ships with better DirectX 12 drivers than Vulkan drivers (and I would bet that it will), then you'll probably see more AAA games written to DirectX12. Metal, likewise, is the preferred graphics API for reaching iPhone users, which are plentiful and (compared to Android users) lucrative.
> Many years ago, I briefly worked at NVIDIA on the DirectX driver team (internship).
I've been a full-time GPU engineer at a couple of major companies for about 6 years now. My colleagues at my current company have all been GPU engineers (and video, prior to that) in the 20--30 year range.
While a lot of the technical details are correct, there are a lot of red-flags in this guy's essay. For instance:
> Although AMD and NV have the resources to do it, the smaller IHVs (Intel, PowerVR, Qualcomm, etc)
Intel owns, what? 60? 70 percent of the desktop market? More? PVR, Qualcomm, etc. own 90% of the mobile market? Intel has an enormous number of GPU engineers---far more than AMD, and easily comparable to (or more than) NV.
Now, to OP's statement about DX12 & Metal: obviously they were developed cojointly with AAA game developers. The major title developers are in constant contact with every major IHV and OS vendor (less so for open-source) at all stages with respect to driver- and API- development.
Furthermore, it's not like the major architects and engineers of these driver teams are total tools; these guys live and breathe GPU architecture (and uarch), and are intimately familiar with what would make a good driver.
The impetus for these low level APIs is differentiation and performance. When those couldn't be had from GPU HW, the next place to look is "one up the stack": driver & API. It surprised no one in the industry that Mantle/Vulkan/Metal/DX12 all came out at about the same time: we've all been pushing for this for years.
Thanks for this. It makes sense that DX12 was developed alongside engine devs. As jsheard notes, Unity and Unreal Engine have already announced support for it, and we're already seeing tech demos of Unreal Engine on DX12:
The Xbox One is being updated to support DX12 as well, so the major engines will adopt it regardless of whether Windows 10 gets any traction - in fact, Unity and UE4 already have.
We don't know yet whether Apple will adopt Vulkan or not. Vulkan is not just a direct competitor to Metal, but also an evolution of OpenGL ES, and so far Apple has adopted OpenGL ES.
I wouldn't call it a software layer implosion -- most people are going to continue to use abstractions on top of these API's for game development. Unity is to Rails, etc ...
Regarding your last paragraph, the Extensible Web Manifesto was created to encourage new APIs to expose low-level capabilities that can be extended and smoothed-over by libraries, reducing the surface area of privileged browser code and giving web developers more control.
Whether or not the Extensible Web Manifesto is successful or not remains to be seen.
I wonder if we'll see opengl-old implemented on top of vulkan. Actually one thing I would love to see is vulkan on linux accessible from userspace without any involvement of a window server, so that you can write the rest of the APIs on top of it. Maybe even have graphical console running on it.
There's already work to do this sort of thing on top of GL... (eg: glamour the X driver that use GL for all it's drawing and works on anything mesa and the kernel support)
I don't think anything will stop someone from being able to run vulkan directly on to of libdrm like you can right now with GL. At least for open source drivers like Intel's.
That last quote really resonates with my experience of software dev in general. "easy" API's generally do it by making a ton of assumptions. Modifying those assumptions becomes a huge pain point and that's if the API even allows you to modify that particular assumption.
Exactly. This is why I like Rich Hickey's Simple Made Easy [1] so much. Basically with easy constructs it becomes harder to build simple systems, even though the simple constructs are harder to learn.
I love this talk, and Rich Hickey's talks in general, but
I think this goes beyond that.
At one point you want full control of the HW, much like you did with game consoles..
On the other you want security: This model must work in a sandboxed (os, process, vm, threads, sharing etc.) environment, along with security checks (oldest one that I remember was making sure vertex index buffers given to the driver/api must not reference invalid memory, something you would make sure is not the case for a console game through tests, but something that the driver/os/etc. must enforce and stop in a non-console game world - PC/OSX/Linux/etc.)
From little I've read on this API, it seems like security is in the hands of the developer, and there doesn't seem to be much OS protection, so most likely I'm missing something... but whatever protection is to be added, definitely would've not been needed in the console world.
Just a rant, I'm not a graphics programmer so it's easy to rant on topics you just scratched the surface...
----
(Not sure why I can't reply to jeremiep below), but thanks for the insight. I was only familiar with one I posted above (and that was back in 1999, back then if my memory serves me well, drawing primitives on Windows NT was slower than 95, because NT had to check all index buffers whether they were not referencing out-of-bounds, while nothing like this was on 98).
Security is actually much easier to implement on the GPU than on the CPU. For the simple reason that GPU code has to be pure in order to get this degree of parallelism. A shader is nothing more than a transform applied to inputs (attributes, uniforms and varyings) in order to give outputs (colors, depth, stencil).
Invalid data would simply cause a GPU task to fail while the other tasks happily continue to be executed. Since they are pure and don't interact with one another there is no need for process isolation or virtualization.
Basically, its easy to sandbox a GPU when the only data it contains are values (no pointers) and pure functions (no shared memory). Even with the simplified model the driver still everything it needs to enforce security.
You are describing GPU from 1990s. Modern GPU is essentially a general purpose computer sitting on the PCIe bus and able to do anything the CPU can. It does not have to run pure functions (e.g. see how it can be used for normal graphics tasks in [1]) and can write any ___location in the memory it can see. Securing it is as easy/hard as securing a CPU: if you screw up and expose some memory to the GPU it can be owned just like the memory exposed to a CPU task[2].
GPUs these days have MMUs and have address spaces allocated per context. It's implemented internally to the driver though so you don't see it. And it's normally mapped differently, but the point of AMD's HSA stuff is to make the CPU's and GPU's MMU match up.
This is just hn adding a delay until the reply link appears related to how deeply nested the comment is. The deeper the longer the delay. It's a simple but effective way to prevent flame wars and the likes.
> "easy" API's generally do it by making a ton of assumptions.
Well, perhaps a better way of putting it is that "easy" APIs try to model a ___domain. Sometimes the API designer just nails it, and folks forget eventually take it for granted. The problem is now "solved". For example, the Unix userspace filesystem API (open, close, read, write, etc.) is pretty damn solid for what it is. I admit that I took it for granted until I worked at a place where someone who'd never seen a filesystem API (!!) was tasked with creating one from the ground up. You can guess how that went.
The downside case is where the API's model either never cut it in the first place, or as @wtallis illuminates above with OpenGL, where the model just ages poorly[1].
Gah. At our company a bunch of early Rails apps metastasized into financial/billing modules that do DB access using ActiveRecord. With all the convenient pluralization and automatic field/method naming. And are now on the critical path for our revenue. It's a daily reminder to me of how the "easy API" thing can go wrong :-)
>I wonder if/when web developers will reach a similar point of software layer implosion. "Easy to code against, but a bitch and a half to debug or tune"
for me it was 3 years ago in web. And 6 years ago in Enterprise java.
Personally, Angular broke me. We built an app that was just a table view basically, and it refreshed the data every minute or whatever. For users whose table got to be 100 or more rows, the performance was awful, and there was nothing we could do about it besides rewrite the whole thing a different way.
I still love JS, and coding for the web. But I don't use frameworks now.
This might hold true if you're only looking at the actual design/code side of the lifecycle, but it sounds like a key point of the new APIs is to make new projects more testable, debuggable and optimizable. Given most project spend inordinate amounts of time in the those three activities (and it sounds like games moreso), this should be a win for everyone.
And even if it isn't, the video game industry just got revolutionized by most of the major engines becoming free, so indie devs don't have to drop down to a lower level of abstraction in order to have a chance at competitive performance.
Can we please stop calling Unity/Unreal "free"? Unity still sells a Pro version, and Unreal takes a cut of revenue after a certain amount of $ earned. I'm not arguing against either model, and it's GREAT for hobbyists, but these engines still cost professional studios money.
The salient point is the barrier to entry. The fact that they still have some form of revenue related to the engines doesn't elevate the up-front cost any and the royalties (where they exist) are structured in a way that cannot prevent them from being a viable option for indie studios. The engines will be making the benefits of new graphics APIs properly accessible to indie studios in all the ways that matter; it won't be a AAA-exclusive thing.
(And professional studios don't have to license the ancillary tools and services and source code access of the Pro editions in order to ship a game.)
It is already integrated into Source 2, Unity started (they also have Metal support) and Epic also going that way.
It will make it to indie engines that aren't companies funded as well eventually.
This actually gives a ton of power to current engine developers in the market so getting them involved was a big win.
I am a huge fan of OpenGL ES / WebGL (also based on OES) and was happy that we finally had mobile win that battle but with new graphics rendering layers like Metal and optimization for mobile more needs to be done. The driver mess is a big problem and limits innovation as well.
So ... the subtext that a lot of people aren't calling out explicitly is that this round of new APIs has been done in cooperation with the big engines. The Mantle spec is effectively written by Johan Andersson at DICE, and the Khronos Vulkan spec basically pulls Aras P at Unity, Niklas S at Epic, and a couple guys at Valve into the fold.
This begs the question: what about DirectX 12 and Apple's Metal? If they didn't have similar engine developer involvement, that's clearly a point against DX12/Metal support in the long run.
Also, the end is worth quoting in its entirety to explain why the APIs are taking such a radical change from the traditional rendering model:
Personally, my take is that MS and ARB always had the wrong idea. Their idea was to produce a nice, pretty looking front end and deal with all the awful stuff quietly in the background. Yeah it's easy to code against, but it was always a bitch and a half to debug or tune. Nobody ever took that side of the equation into account. What has finally been made clear is that it's okay to have difficult to code APIs, if the end result just works. And that's been my experience so far in retooling: it's a pain in the ass, requires widespread revisions to engine code, forces you to revisit a lot of assumptions, and generally requires a lot of infrastructure before anything works. But once it's up and running, there's no surprises. It works smoothly, you're always on the fast path, anything that IS slow is in your OWN code which can be analyzed by common tools. It's worth it.
I wonder if/when web developers will reach a similar point of software layer implosion. "Easy to code against, but a bitch and a half to debug or tune" is an adequate description of pretty much everything in web development today, yet everyone keeps trying to patch it over with even more "easy" layers on top. (This applies to both server and client side technologies IMO.)