As I come to the end of a contract I tend to start the process of looking around to see who specifically is hiring and what the current trends are in data projects, industries that seem to be recruiting.
When I first started working with data and databases, the role was more clearly defined as a Data Analyst or as a (SQL) Database Developer as mostly organisations stored their data in SQL Server DBs or Oracle DBs or similar. This has obviously changed over the years as other storage formats have become more viable.
Then we had the Business Intelligence era, more closely focused on the display of data in aesthetically pleasing reports and dashboards. I see now in projects that organisations will tend to have people who can perform this end of the process as tools like Power BI have evolved and become much more user friendly.
We have moved from on-premises to hybrid cloud solutions and now I am seeing fully cloud-native data platforms, there is a requirement for us to have at least an understanding of cloud architecture.
The patterns for handling the processing of data (ETL/ELT) have remained broadly the same though the more recent broader adoption of medallion architecture and processing provides a more easily understood process which non-technical users can more readily understand, in my experience.
As the role has evolved the name has apparently changed, not sure if a Recruiter somewhere came up with the snappy title of Data Engineer but it’s certainly taken root.
Behind the job titles and changes in technology though, it still remains a role where the key aim is to make data accessible to users so that they can be provided with insights and make more informed decisions for the organisation.