Gable Blog | Shifting Left in Banking

Talk Description

Good Data and not Big Data is becoming more important in today's ecosystem. Machine Learning models rely on good quality data to make their model training more efficient and effective. We have traditionally applied Data Quality checks and balances in manual, centralized way, putting a lot of onus on our customers. Shifting Left Data Quality will bring the data quality checks closer to where data is being created, while preventing bad data from flowing downstream. Also auto-detecting, recommending and auto-enforcing data quality rules will make our customers job easier, while creating a more mature and robust data ecosystem.

Additional Shift Left Data Conference Talks

Shifting Left with Data DevOps ( recording link )

Shifting From Reactive to Proactive at Glassdoor ( recording link )

Data Contracts in the Real World, the Adevinta Spain Implementation ( recording link )

Panel: State of the Data And AI Market ( recording link )

Wayfair’s Multi-year Data Mesh Journey ( recording link )

Automating Data Quality via Shift Left for Real-Time Web Data Feeds at Industrial Scale ( recording link )

Panel: Shift Left Across the Data Lifecycle—Data Contracts, Transformations, Observability, and Catalogs ( recording link )

Shift Left with Apache Iceberg Data Products to Power AI ( recording link )

The Rise of the Data-Conscious Software Engineer: Bridging the Data-Software Gap ( recording link )

Building a Scalable Data Foundation in Health Tech ( recording link )

Shifting Left in Banking: Enhancing Machine Learning Models through Proactive Data Quality ( recording link )

Panel: How AI Is Shifting Data Infrastructure Left ( recording link )

Transcript

*Note: Video transcribed via AI voice to text; There may be inconsistencies.

" Abhi, what's going on man? Yeah, it has been fascinating. I've been listening to all this great speakers talking about different things. Seems, you know, very similar to some of the things we are working on. So I'm really excited to talk about what we are doing things in the banking industry. And I just talked about, uh, things they're doing in a very regulated industry like healthcare.

Mm-hmm. And banking has been very similar lines. It's very regulated. So there are certain things that you just cannot do. Uh, and you have to be very careful with your data. So, well, you know what I'm going to talk about. So really excited to be here. Um, have been working with Chad and Gable for a while now.

So, uh, thank you for so much for inviting me to this conference and I'm really excited to be here. Of course, man. We're happy to have you here. And so feel free to share your screen. Sure. Thank you. And, um, just, uh, to give a brief introduction to myself, I'm Abby. Um, I lead the data observability and external data pipeline in, uh, at Capital One. Uh, watching your wallet.

I think everyone has seen that, uh, or heard it in multiple things. So I've been in Capital One for 13 plus years now. Um, really excited to be in this space. Um, software engineer by trade, um, have solved lot of software engineering problems at Capital One. Uh, but recently, over the last three, four years have been focused all largely on data and how we can make our data ecosystem mature.

Um, so with that, uh, let me jump right into the topic, and I think it's very important. I've heard multiple speakers talk about data quality, data observability. Um, and so would love to first just define like, what is data quality? Because without that, um, you know, we, we, that's the cross cover of the topic.

Um, so at Capital One, and I think in any big, like the, I'm hearing throughout this, um, conference, you know, trusting your data is, is core. And to deliver real time automated and intelligent experiences, you need to trust your data. And to me, here are a few key things. Like data has to be complete and timely.

The moment it is published, data has to be accurate and consistent for machine learning and data should be valued and unique to support automated decision. Now, I think all these things leads to the question is the data we are using is fit for its purpose. And we heard Andrew talk about how we are shifting left with, I think iceberg data products in ai.

We heard the panel, um, of Bar Tristan, um, talking about DataOps ability shifting left in the data ecosystem. So this is in the same vein, and we just had Anna talk about, uh, something very similar in a regulated industry. So that being said, you know, how, how do you, like, we come into, and, and that leads to inter data observability, and these are some of the common pillars of data observability.

One is freshness, and like I said, I lead not only data observability, but external data pipeline. And data pipeline is about, um, looking into, um, like data coming into Capital One. So these are the five main things that we want to worry about. So data coming through, let's say an external vendor, uh, is it arriving on time?

Is the volume accurate? Like we were receiving terabytes of data, all of a sudden we start receiving gigabytes or gigabytes, a megabyte, or all of a sudden something increases. How are we keeping track of it? The schema that we have agreed upon is that, you know, valid, uh, quality of the data. And, you know, that is one of the topic of this, um, this, this talk is, you know, it, it's, it's accurate or not.

And also the lineage. um, do we understand how data quality issues are occurred and the impact of it? And so. How does data quality impact us? And here is a good representation of, you know, how, like different personas, how it impacts us. Um, you could be a product manager, you could be a data scientist, you could be a developer, um, you could be a data analyst.

And to me, everyone has a stake in this game about the maturity of the data ecosystem. So traditionally, like we, I heard some talks around enterprise versus non-enterprise, and I think what enterprise meant was something pretty big in scale, very distributed. Uh, the data, like the company wasn't born with that mindset.

Oh, data is the first class citizen and we are going to like, build everything around data. Like Capital One was definitely like, it's an information based service. Um, and uh, it was like data has always been a prime, but not the extent that we think about it today. So like, depending on the person, uh, it can impact you in different ways.

So now let's look at shift left. And a lot of, lot of speakers before me have talked about shift left. So, uh, I don't think I need to talk about what is shift left. Everyone understands that part. But, uh, let's look at, um, a traditional way of deploying enterprise solutions. And when I say a traditional way is big enterprises where, you know, capital One is definitely one of them.

Where we have, you know, petabytes of data coming through internally, externally, like how do you like really leverage data? And the traditional way has always been okay, we take the data, we put, put it into a data warehouse and then, you know, we will do something on that data warehouse. So, so to me, like, you know, some of the traditional way I'm a user, I come in, I have some user interface, I may have a config, API, I am going to like, you know, put into the user interface some information and say, okay, I want to run data quality checks on a data and here are some of my data quality rules.

And that gets stored in a database. Now I have, you know, created some configuration. I have a deployer, I have a scheduler now that runs on an execution engine. It could be anything around Spark, um, or anything or data pipeline that you use. Um, it does some pre-processing. It's executes, it does some pre-process post-processing.

Um, it goes to the data warehouse. That's where you know you are going to run the thing and it going to store, um, those things on a, in a results, uh, database. Then, you know, there could be dashboards and monitors that people look at it. Uh, there could be bi tools. uh, you are creating bunch of reports. So to me, like a traditional enterprise um, solution, you'll find that a very common architecture pattern.

People run it, it's after the fact, like data has already landed into data warehouse. You don't know it's good, bad data. You may have already consumed it, and then you're running some data quality checks. So it has worked. It is one form of really running data quality checks, but I think it's not cutting it anymore.

Um, this, so, and here are some of the challenges, and again, it's not exhaustive, but this is like, we have a centralized installation, so it only works within centralized installation. What happens if now you buy another company and you know, capital One has been in the journey of buying a few companies, uh, and uh, or, you know, taking, taking, um, and so how do you assimilate that into it?

Uh, they may not be all connected to an enterprise solution. Uh, also the user has to come to this centrally managed solution instead of we going to the user. Uh, and then rules define are very manually in nature. Um, a lot of you are coming in, you are understanding the data. So someone really needs to understand the data and then manually enter it.

And then, like I said, this is much later into the data lifecycle. The data has already gone through your pipeline. It has landed into your data warehouse. You are running it. And you know, that has always at least caused challenges in big enterprises where, hey, I've already started using the data, but the data is not of great quality.

So I think like any other enterprise like this are some of the challenges that, uh, is industry has faced already, still facing, uh, in big enterprises. So how do you change it? And there are multiple ways I'm going to talk about, uh, we have talked about a lot of concept conceptually in some of the talks, but I'm going to go into the nitty gritty details about a little bit about how we are going to make some changes to shift left.

Uh, so one is like, um, we, you know, compostable services, and this is nothing new. Compostable services has been there in the industry for a long time. These are SDKs jars, you know, whatever you want to call it. uh, like if you are a Java developer, you work with jars, uh, SDKs have been there, there are packages.

Um, and so like the basic principles of that is, uh, it's federated modular services. So, you know, it's a smaller service. Um, it's, uh, deployed agnostic of infrastructure. So you don't need like, ah, I need a whole infrastructure to do it. Um, it's indicated with other platforms. It's, it's not only just works in silo, but it's also, uh, it could be managed.

It could be distributed, but you know, like it, it integrates with other platforms.

What's Next

We, uh, you know, continue to advance the heart of with innovation experimentation. And, and Gable has been a great partner to us.

Conclusion

Thank you. That is excellent. And there's some good questions coming through in the chat, so I'm going to fire away.