It’s fascinating that the conclusion / forecast is that tools will abstract engineering problems and DE will move closer to the business . While over the last 20 years the exact opposite has happened and the toolset has actually become harder (not easier to use) but orders of magnitude more powerful and DE has moved closer to engineering, to the point where a good data engineer basically is a specialized software engineer.
The absolute pinnacle of “easy to use” was probably the Informatica / Oracle stack of the late 90’s and early 00’s. It just wasn’t powerful or scalable enough to meet the needs of the Big Data shift
Of course I guess this makes sense given the author works for a company with a vested interest in reversing that trend.
I think those days were easy to use within the zeitgeist. Even advanced versions of those tools would struggle against the data needs today which have become incredibly bespoke. My skill set extends all the way from my actual industry (finance) to the boundary of software development. I also have data, big data and cluster usage skills (slurm etc). I don’t use everything every day and obviously I cannot be a specialist in most of this stuff (I concentrate on finance more than anything else) considering the incredible range, but this is just the past 2 years for me.
I cannot imagine a less specialized future looking around today where some nice tool does 80% of my work. Not because the work I do is difficult to automate. But because the work I do won’t match the work other industries may do (beyond existing generalizations of pandas, regression toolkits and other low level stuff). There’s no point building a full automation suite just for my single work profile which itself will differ from other areas of finance.
DE's at my company actually spend all their time making easy to use interfaces ontop of these tools (Github workflows, IaC/CI-CD solutions, Dev/SecOps features, etc.) so that DW/BI/DS Developers, or even analysts in some cases, can create new pipelines quite easily.
That sounds like infra ops, not data engineering. It's a common theme, though, for DEs to slip into maintaining infra and analysts taking on the role of engineering data pipelines.
The absolute pinnacle of “easy to use” was probably the Informatica / Oracle stack of the late 90’s and early 00’s. It just wasn’t powerful or scalable enough to meet the needs of the Big Data shift
Of course I guess this makes sense given the author works for a company with a vested interest in reversing that trend.