Watch agents enrich all the context your AI needs

Varun Banka

0:00 We always say everybody likes documentation, but nobody wants to do documentation. And at Atlan, because this has been the hardest problem of context layer, and we've been at it for years, from starting from making it easy for you to write low-code, no-code rules-based automation to last year, we launched trigger-based AI inputs to help you generate this. But now I'm super excited to present Context Agents. Now let me make it real for all of you.

Varun Banka

0:30 What you're seeing is we've created curated set of collections of assets. And we've made this knowing the fact that these are the assets your agents will actually need, and this is based on existing information from your data layer. One of the favorite collections for our customers has been popular SQL assets, which is a reflection of all the assets that, like, the business is already using.

Varun Banka

0:54 So what I opened is a collection of popular SQL assets. Based on this, it shows that my context has hundred assets. The coverage for metadata looks like eight percent, and we have three things that we could do. We could run enrichment on descriptions, README, SQL intelligence. And what it means is, it's as easy as clicking a button. What this means is our agents will traverse through the data map that we already have, goes through schema, column names, upstream lineage, downstream lineage, look at every query history that exists, l-look at your existing business concepts, existing metadata, and generate them. So for me, right now, let's just trigger this.

1:32 So while this is happening, while it's going through traversing through this, let me open the asset which already has something. What you're seeing right now is the same table, just that the language now makes sense. It's no longer PL and ID. It talks about exactly what descriptions exist, so that when the agent tries to traverse, it does not do guesswork.

Varun Banka

1:53 And we didn't just stop here. We, in fact, went one more step. I'm sure a lot of you have so many stale Confluence documents that nobody gets updated. Now this README gets automatically updated based on existing data layer. In fact, one thing that I really like here is how teams use data. This has been amazing to just see how different teams have been using data. And this could be super helpful for the agents when to decide what, what to pick up, what not to pick up, but from a business point of view, not just from a SQL point of view.

Varun Banka

2:23 And like I said, we didn't just stop here. We went one more step. Intelligence. This is something most people miss out on. It's not that pre-LLM or pre-talk to data, teams have not been using data. We've been using data, whether it's by ad hoc data queries, your BI layers, all your BIs are sharing queries. What we've done with Context Agents is we've reversed engineered every SQL question and given a list of questions that are getting actually asked. So what this gives is, this gives another test set, eval set, and we look at some of this of how we are gonna use this to actually build an agent later. And we didn't stop with business questions.

3:04 We looked at joins. We all know AI is not good at writing joins right now, so this makes it easy for it, ground it so that the reasoning gets better. We looked at filters so that it does not make like, does not have problems doing operands. And we looked at what other columns are connected to this. So all your historical business questions that data has been used is actually the biggest IP of companies right now. So let's go next.

Varun Banka

3:33 Now, we talked about how we used Context Agents to leverage on top of the data layer. But with large language models, something else has changed. These models now are able to process your unstructured data which has never been used before. They've been just lying around. Nobody has been able to use them, tap into them. What you're seeing right now is what we call knowledge folders. Think about them. This is like the gold layer of unstructured data because you could have hundreds of data lying in your raw storage. This is the final layer, and this has primary documentation like your actual SOPs, your PDF files that will actually determine the SOPs.

4:09 I'm just gonna open one of them. I'm sure you, you folks will be able to relate to this. This is like a refund reference. I'm sure you can imagine how the AI agent right now, when we, when we go through this building the process, how it could use some of this to actually answer the questions that Prikalpa was talking about in that scenario. Now, how do we use it? So apart from this popular SQL assets, if you guys remember, we, we used this to actually generate a ton of context for the context layer.

4:36 We also have an agent that runs through all your data assets, helps you select the right knowledge folders and knowledge files, and generate not just a data graph, but your business graphs. So for sake of demo, we've already done this, so let me go through one of them. What we've moved on to now is we moved on from a data graph, now we've moved on to a business graph. What you're seeing is a business concept of subscription. This might this document— this content of this might be lying around hundreds of document, the knowledge folders that we talked about. Our Context Agents crawls through all of that to help you create this and auto manage this.

5:15 So it help you generate this. Again, the description, what this could use. So the next time the agent wants to run, it knows the exact difference of what subscription looks like. Because just like business, we don't talk in the language of PL and ACT. We talk in the language of business, and this is what it grounds any agent on.

Varun Banka

5:34 So we went through a journey of data graph traversal, so let's, let's also go through a journey of maybe business graph traversal. What you're seeing is subscriptions. This is the entity that we created, which is determined which is created by Context Agents. Let me click. It realized that it figured out that, hey, the subscription is actually built by an entity called invoice. Maybe let me expand a little bit more. It talked about that, hey, this is this is a different concept that applies to this.

6:04 And maybe, and business concepts are not just about entities. These are also about metrics. So it's able to figure out the right metrics that are calculated, which are related to invoice. Your agent does not need to do another calculated formula and, and try to reason and hallucinate on building the right formulas. It can just ground itself on some of these metrics. So we saw the data graph, we saw the business graph. But for the context layer, we brought both of them together. So not only our Context Agents creates your data graph or your business graph, with linked assets, we're able to bring all of them together.

6:39 So not only this is the business concept of subscription, but this links together with all your existing assets in your data layer that actually references a subscription business concept.

Varun Banka

6:50 At Atlan, we always build with our customers, and I'm so happy and excited to have Adrianna with us to talk here about our journey, and I'm joined with my amazing colleague, Nandini, to take over from me to have a quick conversation with Adrianna on her experience. Nandini.

Nandini Tyagi

7:04 Awesome. Thanks, Varun. I'm thrilled to welcome Adrianna from Engine, who have been our design partners and have been leveraging Context Agents Studio. Adrianna, so glad you're here. Let's dive in. Okay. So tell us about Engine and what were you trying to accomplish? Why did this matter to you?

Adrianna Clark

7:23 Yeah. So I'm a senior data governance analyst here at Engine. We are an all-in-one modern, modern travel platform. We help companies book and manage hotel, flights, and rental cars all in one place, and we've heavily been heavily investing in AI to help drive efficiency across our organization. We wanna always ensure that our customers have the best product. We are currently exploring Atlan as the nervous system for how AI operates within our data ecosystem, and we understand that AI is only as good as the context that it has access to.

7:56 So a big focus for us has been making sure that we build something that is accurate, a scalable context layer with well-governed data that already sits inside of Atlan. At the moment, our data teams are already using Atlan's MCP server. We're getting a ton of value there.

Nandini Tyagi

8:10 Got it. Makes sense. When you first started running the agents, what were you expecting? And then what actually happened?

Adrianna Clark

8:17 Yeah. So honestly, I was not sure what to expect. We were a few days out from launching Atlan. I had a ton of work to do trying to coordinate training. We were rolling out to the entire business. I expressed this to Charlotte, who is our customer success manager, and she mentioned the Context Agents Studio. I still was skeptical about the level of effort, but once I got in and started using it it was already pulling out our most used assets, and it had collections ready for me. And so it was very efficient, and I was shocked by how fast it was. So it was a very good experience.

Nandini Tyagi

8:49 Amazing. Amazing. Thanks for sharing that. And when you actually looked at the AI-generated output, I know it can bring a lot of feelings, what surprised you?

Adrianna Clark

8:58 So I was definitely nervous going in.

Nandini Tyagi

9:00 Yeah.

Adrianna Clark

9:00 I thought that there might be, like, some gaps in the definitions or SQL but once I went through it, it was very accurate. I think one thing that really stood out to me was the ability to take some of our most popular business questions that people were already asking about our data. They wanted answers to this information. And you know, this is the type of context that we need for our agents, and it definitely enabled a sense of trust for me in the Context Agents Studio.

Nandini Tyagi

9:27 Makes sense. Makes sense. Thanks for sharing. And as I'm thinking of wrapping up, stepping back, if we zoom out what did this actually unlock for you at Engine? Like, what's now possible that wasn't possible earlier?

Adrianna Clark

9:41 So I think for years we've been taught that in order to document, document human tribal knowledge, it has to be a very manual process. You need your data stewards, you need IT, you need the business. You have to have all this buy-in. It has to be hand-in-hand. And what I experienced with Context Agent is that you can now lead with technology so you don't have to lead with the business. I was able to run this entire exercise on my own. And at Engine, we really do not have the capacity across our business verticals to have a data steward to sit down and go through, you know, manual hours to really complete this exercise.

10:18 So I think that was the big shift is you know, the old mode— old model really assumes that you have the head count, you have the people, you have the capacity. But when you don't, you know, this really makes it possible to to put a solution in place.

Nandini Tyagi

10:33 Makes sense. I know in one of our one-on-ones you mentioned that it's having your own AI team, and always I laugh when I think of that. Thank you so much for sharing all of that. It was great having you here. Shout out to everyone who participated in XData and seeing lots of really good comments. I'm gonna pass it back to Varun. Thank you so much for joining us, Adrianna.

Adrianna Clark

10:50 Thank you.

Varun Banka

10:51 Thanks, Nandini, and thanks, Adrianna. And we were equally worried when we launched this with some of our early customers, and we were like, "Hey, will, will we need, like, human-in-the-loop set up for this?"

Varun Banka

11:04 But we were just surprised the fact that nobody wanted to overwrite any of these descriptions. And I know for folks who want to hear more, reach out to, reach out to Atlan. We'll be happy to showcase what, what it's able to do. Now, but how does that happen? Why does it make it special? What's, what's happening that it's able to traverse through so much context? We believe that this is what we believe in the concept of context compounding. Step one, we talked about how we connect to your underlying data layer to bring existing context, harness existing context, process that, and bring that into Atlan.

11:35 Once that happens, our agents goes deeply through across a data source. If your, if your data is coming from Salesforce, we understand why Salesforce data was moved from Salesforce to your underlying system of analytics. We understand what's happening in your BI layer, why was that as a dashboard that in use. Understand what the what business is actually asking these questions on those columns. Understand why those filters are happening. The agents understands that, and then it helps us create everything that we talked about.

12:02 Once it does that, we get, we get another unlock because once you have that, we are able to create metrics, KPIs, and actually create one level more of insights that we'll be able to create. And once you have that, we're able to unlock not just the technical graph, we're able to unlock the business graph itself. With the power of both structured data layer and unstructured data layer, we were able to bring all this together, and it compounded.