Navigating your Career in Data & AI

‘Data is the new oil’, or ‘Data is water’, or ‘AI is like electricity, or fire’ – there are a plethora of analogies out there. I don’t intend to add to them, but the key takeaway is that Data is fast becoming the central tool in any domain. Our worlds are becoming increasingly digital. Digital works with data, creates more data. Compute power has increased exponentially, cost has been driven down. Cloud computing has democratized the access to compute power, manipulate data at scales never before possible. Compute and data enable algorithms, and that has fueled innovations in AI. That’s the summary! 🙂

This article is about navigating your career in this fast evolving domain of data and AI. I’m using the term AI somewhat broadly to include analytics, algorithms, intelligence. Similarly, I’m using the term Data to include all kinds of data – structured, unstructured, big data etc. I’m sharing my thoughts here in response to the frequent questions I have received over the years about navigating one’s career in this field. And I’m drawing from the different phases of my career to offer a point-of-view.

No alt text provided for this image

In my view, the field of Data & AI can be considered as a Venn diagram consisting of 3 pillars – Domain, Algorithms, and Engineering, as shown in the figure. Since the field is vast and evolving, depending on where you are entering it from and where your interests or strengths lie, there is room for you to grow, make an impact and determine how you want to navigate your career.

The first pillar of Algorithms is about, as the name suggests, data science and machine learning. If you enjoy mathematics (especially statistics, linear algebra, calculus), and computer science (numerical techniques, optimization, algorithms), you are likely to be naturally drawn to this pillar. In this pillar, you are working on exploratory data analysis, modeling, feature engineering, etc. This is where I began my career at AT&T Bells Labs in the mid 1990’s and before that during my graduate school research, working on Recurrent Neural Networks. You may find an article I wrote some time back on the topic of Data Science interesting. (The ‘Science’ in Data Science | LinkedIn)

The second pillar of Domain is about, again as the name suggests, the area you are solving problems in. The domain could be an industry vertical like banking or retail, could be a functional area like human resources or finance, or could be a domain like drug discovery, urban planning, child mortality or biodiversity. Problem-solving and enquiry in every domain is increasingly becoming data driven. In this pillar, you are defining problem-statements, asking good questions, forming hypotheses, converting domain problems into data problems. This is where I spent the second leg of my career in management consulting, notably at the Boston Consulting Group (BCG) and during my training at Wharton. BCG taught me the skills for becoming hypothesis-led, data-driven and outcome-focused. Your problem-solving skills will increasingly be about your data literacy, and your ability to translate business problems into data problems. Data analysts, citizen data scientists, business-intelligence experts are all excellent roles to enter the world of Data & AI through the pillar of Domain.

The third pillar of Engineering is about technology/IT that allows data and algorithms to be used in practice by users, who may not themselves be data and AI experts — seamlessly and at scale. Engineering involves the various technologies and frameworks, and how to combine them to create solutions to enable a company’s business processes with data pipelines. Engineering is also about preparing the foundation that the Algorithm pillar uses to be effective. For example, data engineers are responsible for the collection, moving, storing, and preparation of the data infrastructure. Data scientists are then able to use the infrastructure for modeling or optimization. Engineers are responsible for designing, developing and maintaining data platforms, which include the data infrastructure, data applications, data warehouse, and data pipelines. If you are an IT professional, this pillar is your natural entry-point into Data and AI. As CTO of Janalakshmi Financial Services, now Jana Small Finance Bank (the 3rd leg of my career 🙂 ), this was my primary responsibility – working with IBM as our strategic partner to build a data estate for the organization to accelerate our digital transformation.

No alt text provided for this image

All 3 pillars in the field of Data & AI are inextricably interlinked, as you might expect. Much richness, value, challenge and opportunity lie in these overlaps, shown as regions ‘a’, ‘b’, ‘c’ and ‘d’ in the Venn diagram. As you navigate your career, depending on your goals and interests, you may charter your path along these regions.

Finally, Data & AI is a very interdisciplinary world. Soft skills of curiosity, communication and collaboration become increasingly important to be effective, as one advances in one’s career. For leaders, culture becomes key in order to successfully drive your organization’s roadmap in Data and AI.

(This article is To Be Continued. I will elaborate further both on the regions ‘a’, ‘b’, ‘c’ and ‘d’ of the Venn diagram as well as the topic of Soft Skills — right here in this article. Or, maybe I’ll write an addendum! 🙂 )