Are you ready to push the boundaries of real-time data streaming and AI-driven innovation? The wait is over — Current Bengaluru 2025 is here!

Join the Data Streaming Revolution at Confluent’s Current Bengaluru 2025

“If Confluent is down, Swiggy is down.”

How Confluent is Helping Swiggy Deliver Millions of Orders Daily

Confluent Cloud will now be accessible on Jio Cloud Services, simplifying data streaming adoption.

California-based Confluent Partners with Jio Platforms to Advance GenAI Development in India

In a report released yesterday, Matthew Hedberg from RBC Capital maintained a Buy rating on Confluent (CFLT – Research Report), with a pri...

Confluent (CFLT) Receives a Buy from RBC Capital

Confluent received a Neutral rating from the analysts at Baird as they initiated coverage on the data streaming platform. Read more here.

Confluent initiated at Neutral due to data streaming demand: Baird

Multiple leading enterprises with open query engines have already expressed support for Snowflake Polaris Catalog, including Amazon Web Services (AWS), Confluent, Dremio, Google Cloud, Microsoft Azure and Salesforce.

Snowflake unveils Polaris, a vendor-neutral open catalog implementation for Apache Iceberg

Confluent, the data streaming giant is making significant strides in India as a crucial market for its global expansion strategy.

Confluent has 20% of its Global Workforce in India

The development comes as Confluent looks to build out governance capabilities and “make itself known” to grow its presence in the APAC region.

Confluent launches plug-and-play option for realtime streaming AI

The company is seeking data scientists and engineers to further bolster its tech team.

Data Science Hiring Process at Confluent

Confluent stock has rallied 50% in February, fueled by better-than-expected fourth-quarter results along with an optimistic 2024 outlook.

Confluent Stock Is No Nvidia But This AI Player Is Targeting 325% EPS Growth In 2024

Confluent

confluent

Fractal Designs, Simple to Complex (Source) Fractal Designs, Simple to Complex Source Complex systems emerge from simple parts. Bits form bytes, letters form words, and basic arithmetic builds into entire mathematical models. In software, microservices become platforms like Netflix. This pattern, small components combining into powerful wholes, is the foundation of complexity itself. The great gift to the world that technologies like Apache Kafka gave us was decoupling: it broke monoliths into independent services and teams that could scale.Kafka didn’t just change how data moved, it reshaped how we design systems: modular, loosely coupled, and easier to evolve. Multi-agent systems (MAS) offer a similar shift for AI. MAS introduces a way of structuring intelligence, breaking down complex AI tasks into simpler, focused agents that work together to solve harder problems. It’s a design shift that brings clear benefits: better testing, more reliable behavior, and scalable systems. Generative AI often suffers from the opposite problem: too much complexity packed into a single prompt. In this post, we’ll explore how MAS -- along with two other concepts that are important here, emergence, decomposition -- offer a better path for building intelligent, reliable AI systems. Emergence: Complexity from Simplicity Emergence is the phenomenon where simple rules or parts combine to create unexpected, higher-order behavior. It’s how the physical world is built. Atoms form molecules, molecules form cells, and cells form living organisms. None of the individual components “know” the full picture, but their interactions create it. Emergence Emergence The Biological Levels of Organization From an Atom to a Multicellular Organism (Source: Biological Macromolecules) The Biological Levels of Organization From an Atom to a Multicellular Organism (Source: Biological Macromolecules) Biological Macromolecules Biological Macromolecules We see this bottom-up complexity in countless domains. When I first studied Conway’s Game of Life in college, it completely reshaped how I thought about computation. It was part of a theoretical computer science course, and the idea that a grid of binary cells, just alive or dead, could create such rich, evolving patterns was mind-bending. It’s continued to fascinate computer scientists for decades. Conway’s Game of Life Conway’s Game of Life Tetromino Patterns from Conway’s Game of Life (Source: Cornell University’s Math Explorer Club) Tetromino Patterns from Conway’s Game of Life (Source: Cornell University’s Math Explorer Club) Cornell University’s Math Explorer Club Cornell University’s Math Explorer Club With only four simple rules, you could simulate behaviors that looked like movement, reproduction, and even computation itself. That lesson stuck with me: you don’t need to engineer complexity directly. You can build small parts with clear behaviors and let them interact. Complexity emerges. Another example is ant colonies. Each ant follows basic instructions: lay pheromones, follow trails, pick up food. There’s no leader, no central coordination, yet the colony exhibits intelligent, adaptive behavior. It organizes itself. Ant Colony Behavior (Source: Insect Lore, Ant Colony Behaviors) Ant Colony Behavior (Source: Insect Lore, Ant Colony Behaviors) Insect Lore, Ant Colony Behaviors Insect Lore, Ant Colony Behaviors This is the essence of emergence. Intelligence isn’t always about the smartest part, it’s about the smartest system. This is how we need to think about AI and this is the principle that multi-agent systems bring to AI. Instead of building one all-knowing model, MAS encourages you to design multiple simple agents, each focused on a narrow task. Like ants or Conway’s gliders, they interact to form a system that can solve problems much larger than any one part could handle alone. The Trouble with Generative AI The Trouble with Generative AI Generative AI is powerful, but it can also be chaotic. The same foundation model that can summarize a document can also write code, explain a joke, or draft an email. That flexibility is impressive, and it’s in part what drives the interest but it also introduces a problem: we’ve packed too much complexity into a single prompt-shaped interface. This is the opposite of how purpose-built AI models are designed. Traditional models are scoped: fraud detection, recommendation, and churn prediction. The inputs and outputs are known, the evaluation metrics are clear, and testing is straightforward. They lack generality, but that’s exactly what makes them reliable, testable, and deployable at scale. Generative AI flips this. It's like giving an employee fifteen unrelated tasks at once, with no prioritization, no clear inputs, and no defined outputs. It’s hard to evaluate their work, not because they’re bad at it, but because we’ve defined the problem in a way that resists structure. From a systems perspective, this is the opposite of emergence. Instead of building complexity through interaction between simple components, we’re injecting complexity all at once and hoping for coherence. It’s brittle, hard to debug, and nearly impossible to test reliably. Generative AI systems suffer from: Testing difficulties – The input space is infinite, and the output space is open-ended. How do you assert correctness in a system that can say anything?
Poor predictability of outputs – Small changes to prompts can produce wildly different results. You don’t know what you're going to get. This makes designing and testing these systems more challenging than their conventional alternatives.
Scalability limits –  Complex overloaded prompts don’t scale well organizationally. Everything becomes entangled in the prompt and the model gets confused. Testing difficulties – The input space is infinite, and the output space is open-ended. How do you assert correctness in a system that can say anything? Testing difficulties Poor predictability of outputs – Small changes to prompts can produce wildly different results. You don’t know what you're going to get. This makes designing and testing these systems more challenging than their conventional alternatives. Poor predictability of outputs Scalability limits –  Complex overloaded prompts don’t scale well organizationally. Everything becomes entangled in the prompt and the model gets confused. Scalability limits These challenges aren’t just inconvenient, they make generative AI systems hard to engineer, scale, and trust in production. But there’s another way to approach the problem. A way that embraces complexity through structure, not despite it. One that draws from emergence, modularity, and clear system boundaries. through That’s where multi-agent systems come in. multi-agent systems From Models to Agents From Models to Agents So how do we bring structure back into AI systems? We start thinking in terms of agents. An agent is more than just a model. It’s a goal-directed entity with memory, tools, and autonomy. Instead of asking a model to respond to a prompt, you give an agent a task and let it decide how to achieve it. It can plan, call APIs, retrieve information, interact with other agents, and adapt based on context. agent Agent Architecture (Inspired by https://arxiv.org/pdf/2304.03442) Agent Architecture (Inspired by https://arxiv.org/pdf/2304.03442) https://arxiv.org/pdf/2304.03442 https://arxiv.org/pdf/2304.03442 One of the most powerful things agents can do is execute deterministic processes, like querying a database, running a script, or triggering a workflow. Foundation models provide a natural language interface to those actions, removing the friction that used to require specialized knowledge. The deep database expertise of a senior analyst, how to structure the right query, filter by quarter, join across systems, can now be surfaced and reused by a non-technical executive who simply wants to know what’s happening with the revenue forecast. what’s happening with the revenue forecast This is where agents become more than just clever wrappers around LLMs. They can connect probabilistic reasoning (language) with deterministic systems (code and data), a fusion that gives language models real-world leverage. Even more powerful is when you connect multiple agents together. That’s the idea behind multi-agent systems: a framework where each agent is designed with a narrow focus, but collectively they coordinate to solve complex problems. Planning, routing, executing, refining, all handled by different agents working in concert. As explored in this article, agents offer a clean separation of concerns. explored in this article explored in this article You get modular components with defined responsibilities, interfaces, and scopes. That makes them easier to test, scale, and evolve independently, just like well-designed microservices in software architecture. And just like in ant colonies or Conway’s Game of Life, the intelligence isn’t embedded in one massive entity. It emerges from the interaction of smaller, purpose-driven units. This is where MAS mirrors nature: small agents following simple rules can collectively generate sophisticated, reliable behavior. In the next section, we’ll dig into the advantages of this approach and why it might be the most practical path to building complex, trustworthy AI systems. Why Multi-Agent Systems Work One of the key conceptual shifts MAS enables is moving from open-world to closed-world problem framing. A general-purpose foundation model operates in an open world, it can be asked anything, in any way, and is expected to respond appropriately. That’s powerful, but it also makes the system unbounded and unpredictable. In contrast, an agent operates in a closed world: it has a specific goal, access to known tools, and clear success criteria. This boundedness makes agents testable, observable, and composable, exactly what’s needed for real-world reliability. 1. Testability When each agent has a narrow scope, you can test it like any traditional software component. A retrieval agent can be validated on recall and precision. A summarizer can be benchmarked against expected outputs. You no longer have to treat the system like a black box with infinitely many valid responses, you can probe and improve each piece in isolation. 2. Observability In a monolithic LLM interaction, you get input, output, and not much else. With MAS, every agent step is visible: what data was fetched, what reasoning was applied, what decision was made. That makes debugging tractable, auditing possible, and failure modes easier to understand and fix. 3. Composability Agents are building blocks. Once you’ve built a reliable planner or a summarizer, you can reuse them across different workflows. This aligns perfectly with how modern software is built, through reusable, composable services, and opens the door to faster iteration and lower operational overhead. 4. Scalability Different agents can be developed and maintained by different teams. Each can be versioned, monitored, and deployed independently. That’s crucial in a growing organization or product where responsibilities need to be distributed. You scale the system by scaling the team without increasing the complexity for any single contributor. 5. Alignment with Real-World Constraints Real-world tasks are not free-form language problems, they’re structured workflows with known inputs, outputs, constraints, and goals. MAS allows us to encode that structure into the system itself. You can insert guardrails, fallback logic, human-in-the-loop checks—all within a coherent architecture. MAS doesn’t eliminate complexity. But it organizes it. It gives you the ability to engineer AI systems with the same rigor and reliability we expect from modern software. MAS doesn’t eliminate complexity. But it organizes it. It gives you the ability to engineer AI systems with the same rigor and reliability we expect from modern software. An Example: Revenue Forecasting with MAS An Example: Revenue Forecasting with MAS Imagine a company wants to forecast revenue across regions and product lines. Doing this well requires data retrieval, trend analysis, anomaly detection, and narrative reporting. Trying to prompt a single foundation model to do all of that in one shot is brittle and opaque, hard to test, hard to trust, and hard to scale. With MAS, we decompose the problem: A retriever agent pulls sales and financial data from internal systems.
An analysis agent applies deterministic logic or invokes a forecasting model.
A QA agent checks for missing data, outliers, or violations of business rules.
A reporting agent summarizes the results in natural language.
A planner agent orchestrates the workflow and handles errors or retries. A retriever agent pulls sales and financial data from internal systems. retriever agent An analysis agent applies deterministic logic or invokes a forecasting model. analysis agent A QA agent checks for missing data, outliers, or violations of business rules. QA agent A reporting agent summarizes the results in natural language. reporting agent A planner agent orchestrates the workflow and handles errors or retries. planner agent Each agent is scoped, testable, and reusable. As a system, it produces a more reliable and explainable result. And because the interface is natural language, a non-technical user can ask a high-level question like “How are we tracking against Q2 targets?”, and get an answer grounded in structured data and deterministic logic. “How are we tracking against Q2 targets?” This approach aligns AI with how businesses operate: modular systems, clear ownership, and verifiable outcomes. Closing Thoughts Closing Thoughts Building intelligent systems isn’t just about having powerful models, it’s about how we structure and compose them. Complex behavior doesn’t require complex parts. It requires well-designed, simple parts working together, guided by clear boundaries and interactions. That’s the essence of emergence and the core design principle behind multi-agent systems. By breaking down monolithic prompts into smaller, testable agents, we regain control. We shift AI from an experimental black box into an engineered system, observable, scalable, and aligned with real business workflows. Foundation models gave us a powerful interface. MAS gives us the architectural pattern to make it useful, dependable, and extensible. The future of AI isn't just bigger models, it's better system design. Good ol’ practical computer science. Written by Sean Falconer Written by Sean Falconer Written by Sean is an AI Entrepreneur in Residence at Confluent where he works on AI strategy and thought leadership. Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from AI to quantum computing. Sean also hosts the popular engineering podcasts Software Engineering Daily and Software Huddle. You can connect with Sean on LinkedIn. LinkedIn

What Conway, Ants, and Apache Kafka Can Teach Us About AI System Design

コンウェイ、蟻、アパッチ・カフカがAIシステムデザインについて教えてくれること

दक्ष इन्जिनीयर, आर्किटेक्चर, बिल्डर र कन्ट्र्याक्टर कर्णालीमा भित्र्याउन आवश्यक छ ।

สิ่งที่ Conway, ตุ๊กตาและ Apache Kafka สามารถสอนเราเกี่ยวกับการออกแบบระบบ AI

We recently published a series of tutorial videos and on the Apache Kafka® platform as we see it. After you hear that there’s a thing called Kafka but before you put hands to keyboard and start writing code, you need to form a mental model of what the thing is. tweets give you the basics you need to know to have the broad grasp on Kafka necessary to continue learning and eventually start coding. This post summarizes them. These videos Events Pretty much all of the programs you’ve ever written respond to events of some kind: the mouse moving, input becoming available, web forms being submitted, bits of JSON being posted to your endpoint, the sensor on the pear tree detecting that a partridge has landed on it. Kafka encourages you to see the world as sequences of events, which it models as key-value pairs. The key and the value have some kind of structure, usually represented in your language’s type system, but fundamentally they can be anything. Events are immutable, as it is (sometimes tragically) impossible to change the past. Kafka topics Because the world is filled with so many events, Kafka gives us a means to organize them and keep them in order: topics. A topic is an ordered log of events. When an external system writes an event to Kafka, it is appended to the end of a topic. By default, messages aren’t deleted from topics until a configurable amount of time has elapsed, even if they’ve been read. Topics are properly , not queues; they are durable, replicated, fault-tolerant records of the events stored in them. logs Logs are a very handy data structure that are efficient to store and maintain, but it’s worth noting that reading them is not too exciting. You can really only scan a log, not query it, so we’ll have to add functionality on a future day to make this more pleasant. Partitioning Topics are stored as log files on disk, and disks are notoriously finite in size. It would be no good if our ability to store events were limited to the disks on a single server, or if our ability to publish new events to a topic or subscribe to updates on that topic were limited to the I/O capabilities of a single server. To be able to scale out and not just up, Kafka gives us the option of breaking topics into partitions. are a systematic way of breaking the one topic log file into many logs, each of which can be hosted on a separate server. Partitions This gives us the ability in principle to scale topics out forever, although practical second-order effects and the finite amount of matter and energy available in the known universe to perform computation do place some upper bounds on scalability. Kafka brokers Kafka is distributed data infrastructure, which implies that there is some kind of node that can be duplicated across a network such that the collection of all of those nodes functions together as a single Kafka cluster. That node is called a . broker A broker can run on bare metal hardware, a cloud instance, in a container managed by Kubernetes, in Docker on your laptop, or wherever JVM processes can run. Kafka brokers are intentionally kept very simple, maintaining as little state as possible. They are responsible for writing new events to partitions, serving reads on existing partitions, and replicating partitions among themselves. They don’t do any computation over messages or routing of messages between topics. Replication As a responsible data infrastructure component, Kafka provides replicated storage of topic partitions. Each topic has a configurable replication factor that determines how many of these copies will exist in the cluster in total. One of the replicas is elected to be the leader, and it is to this replica that all writes are produced and from which all reads are probably consumed. (There are some advanced features that allow some reads to be done on non-leader partitions, but let’s not worry about those here on day five.) The other replicas are called , and it is their job to stay up to date with the leader and be eligible for election as the new leader if the broker hosting the current leader goes down. followers Kafka producers Once the Kafka cluster is up and running with its minimal feature set, we need to be able to talk to it from the outside world. A is an external application that writes messages to a Kafka cluster, communicating with the cluster using Kafka’s network protocol. is a publicly documented thing, but it would be an extraordinarily bad idea to write your own interface library when so many excellent ones are available. producer That network protocol Out of the box, Apache Kafka provides a Java library, and Confluent supports libraries in Python, C/C++, .NET languages, and Go. The producer library manages all of the non-trivial network plumbing between your client program and the cluster and also makes decisions like how to assign new messages to topic partitions. The producer library is surprisingly complex in its internals, but the API surface area for the basic task of writing a message to a topic is very simple indeed. Kafka consumers The is an external application that reads messages from Kafka topics and does some work with them, like filtering, aggregating, or enriching them with other information sources. Like the producer, it relies on a client library to handle the low-level network interface in addition to some other pretty sophisticated functionality. A consumer can be just a single instance, or it can be many instances of the same program: a consumer group. consumer Consumer groups are elastically scalable by default, but the library only manages some of the challenges associated with scale-out and fault tolerance. For example, if your consumer is stateful (and it probably is), then you’ll have some extra work to do to manage that state during failover or scaling operations. The Kafka ecosystem Let’s pause on this day and set the stage for the rest of the series. With basic pub/sub, partitioning, producing, and consuming work in hand, other needs are going to arise. These things consistently emerge from organizations making serious use of Kafka. You need data integration, schema management, and options for stream processing. The Kafka community and have solved these problems in standard ways and are likely to continue solving new common problems as they arise. Confluent community Kafka Connect is a system for connecting non-Kafka systems to Kafka in a declarative way, without requiring you to write a bunch of non-differentiated integration code to connect to the same exact systems that the rest of the world is connecting to. Kafka Connect Connect runs as a scalable, fault-tolerant cluster of machines external to the Kafka cluster. Rather than write bespoke code to read data from a database or write messages to Elasticsearch, you deploy pre-built connectors from the extensive connector ecosystem, and configure them with a little bit of JSON. Connect then reads data from source systems and writes it to sink systems automatically. Schema Registry Schema change is a constant fact of life. Any time you serialize data, put it somewhere, and hope to go get it from that place later on, changes in the format of the data are a perennial challenge. We feel this problem most acutely in database schemas, but message formats in Kafka are no exception. The exists to help manage schema change over time. When you release a new producer or a new consumer application with a modified message format, the Schema Registry will help the client application determine whether the new schema is compatible with the old one, given the expectations of other clients that have yet to be versioned. It’s an indispensable tool for a complex deployment. Confluent Schema Registry Kafka Streams Producing messages to Kafka is often fairly simple: Messages come from some source, either read from some input or computed from some prior state, and they go into a topic. But reading gets complicated very quickly, and the consumer API really doesn’t offer much more abstraction than the producer. The exists to provide this layer of abstraction on top of the vanilla consumer. It’s a Java API that provides a functional view of the typical stream processing primitives that emerge in complex consumers: filtering, grouping, aggregating, joining, and more. It provides an abstraction not just for streams, but for streams turned into tables, and a mechanism for querying those tables as well. It builds on the consumer library’s native horizontal scalability and fault tolerance, while addressing the consumer’s limited support for state management. Kafka Streams API ksqlDB Writing stream processing applications in Java is a nice thing to do if you’re using Kafka Streams, and if you’re using Java, and if it makes sense to marry stream processing functionality with the application itself. But what if you didn’t want to do those things? Or what if you wanted a simpler approach in which you just used SQL to get your stream processing done? This is precisely what is: an application-oriented stream processing database for Kafka. A small cluster of ksqlDB nodes runs continuous stream processing queries written in SQL, constantly consuming input events and producing results back into Kafka. It exposes the same stream and table abstractions as Kafka Streams and makes tables queryable through a lightweight JSON API. ksqlDB Coda I hope this video series has helped you see the breadth of the Kafka ecosystem. I want you to have a basic mental model of how Kafka works and what other components have grown up around it to help you solve the kinds of problems that will inevitably present themselves as you build event-driven systems. If you’ve got some time in the next couple of weeks, and if a bit of study helps you relax as it does for me, be sure to . check them out Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the senior director of developer advocacy. He can frequently be found at speaking at conferences in the U.S. and all over the world. He is the co-presenter of various O’Reilly training videos on topics ranging from Git to distributed systems, and is the author of . He lives in Littleton, CO, U.S., with the wife of his youth, their three children having grown up. Gradle Beyond the Basics

Apache Kafka: How Does It Work?

A Data Engineer's Guide to PyIceberg

Kafka Schema Evolution: A Guide to the Confluent Schema Registry

Big Tech Loves These 45 System Design Questions—Do You Have the Answers?

Apache Kafka provides schema management capabilities through the Schema Registry, including various levels of compatibility. In this article, we will discuss three types of transitive compatibility: BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE. Additionally, we will analyze their pros, cons, and provide code examples. BACKWARD_TRANSITIVE Description Backward transitive compatibility means that a new version of the schema must be compatible with all previous versions. This is useful in systems with a long data lifecycle. Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class BackwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "BACKWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Ensures compatibility with historical data.
✅ Useful for long-term data storage. Cons ❌ May restrict adding new fields or changing the structure. FORWARD_TRANSITIVE Description Forward transitive compatibility requires that a new version of the schema be compatible with all future versions. This is useful when consumers are updated asynchronously with producers. Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FORWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class ForwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FORWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Ensures stability for old consumers with new data. ✅ Convenient for systems with asynchronous updates. Cons ❌ Does not guarantee compatibility with historical data. FULL_TRANSITIVE Description Full transitive compatibility combines the requirements of backward and forward compatibility. A new schema version must be compatible with both previous and future versions. Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class FullTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FULL_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Maximum flexibility and resilience.
✅ Guarantees compatibility with all schema versions. Cons ❌ The strictest compatibility type.
❌ Requires careful planning for schema changes. Comparison Table Compatibility Type

Compatibility with Past Versions

Compatibility with Future Versions

Ease of Use

Applicability



BACKWARD_TRANSITIVE

✅

❌

Medium

Long-term data storage



FORWARD_TRANSITIVE

❌

✅

Medium

Asynchronous updates



FULL_TRANSITIVE

✅

✅

Low

Maximum resilience These compatibility types allow you to configure the system to meet your stability and flexibility requirements. The correct choice depends on your system architecture and use cases. Apache Kafka provides schema management capabilities through the Schema Registry, including various levels of compatibility. In this article, we will discuss three types of transitive compatibility: BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, and FULL_TRANSITIVE. Additionally, we will analyze their pros, cons, and provide code examples. BACKWARD_TRANSITIVE Description Backward transitive compatibility means that a new version of the schema must be compatible with all previous versions. This is useful in systems with a long data lifecycle. all Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class BackwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "BACKWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class BackwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "BACKWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Ensures compatibility with historical data. ✅ Useful for long-term data storage. ✅ Ensures compatibility with historical data. ✅ Useful for long-term data storage. Cons ❌ May restrict adding new fields or changing the structure. ❌ May restrict adding new fields or changing the structure. FORWARD_TRANSITIVE Description Forward transitive compatibility requires that a new version of the schema be compatible with all future versions. This is useful when consumers are updated asynchronously with producers. all Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FORWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FORWARD_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class ForwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FORWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class ForwardTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FORWARD_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Ensures stability for old consumers with new data. ✅ Ensures stability for old consumers with new data. ✅ Convenient for systems with asynchronous updates. ✅ Convenient for systems with asynchronous updates. Cons ❌ Does not guarantee compatibility with historical data. ❌ Does not guarantee compatibility with historical data. FULL_TRANSITIVE Description Full transitive compatibility combines the requirements of backward and forward compatibility. A new schema version must be compatible with both previous and future versions. Code Example curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://localhost:8081/config/<subject_name> Java Code Example import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class FullTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FULL_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
import io.confluent.kafka.schemaregistry.rest.exceptions.RestClientException;

import java.io.IOException;

public class FullTransitiveExample {
    public static void main(String[] args) throws IOException, RestClientException {
        String schemaRegistryUrl = "http://localhost:8081";
        SchemaRegistryClient client = new CachedSchemaRegistryClient(schemaRegistryUrl, 10);

        String subject = "<subject_name>";
        String compatibility = "FULL_TRANSITIVE";

        // Set compatibility type
        client.updateCompatibility(subject, compatibility);

        // Check current compatibility type
        String currentCompatibility = client.getCompatibility(subject);
        System.out.println("Current compatibility level: " + currentCompatibility);
    }
} Pros ✅ Maximum flexibility and resilience. ✅ Guarantees compatibility with all schema versions. ✅ Maximum flexibility and resilience. ✅ Guarantees compatibility with all schema versions. Cons ❌ The strictest compatibility type. ❌ Requires careful planning for schema changes. ❌ The strictest compatibility type. ❌ Requires careful planning for schema changes. Comparison Table Compatibility Type

Compatibility with Past Versions

Compatibility with Future Versions

Ease of Use

Applicability



BACKWARD_TRANSITIVE

✅

❌

Medium

Long-term data storage



FORWARD_TRANSITIVE

❌

✅

Medium

Asynchronous updates



FULL_TRANSITIVE

✅

✅

Low

Maximum resilience Compatibility Type

Compatibility with Past Versions

Compatibility with Future Versions

Ease of Use

Applicability BACKWARD_TRANSITIVE

✅

❌

Medium

Long-term data storage FORWARD_TRANSITIVE

❌

✅

Medium

Asynchronous updates FULL_TRANSITIVE

✅

✅

Low

Maximum resilience Compatibility Type Compatibility with Past Versions Compatibility with Future Versions Ease of Use Applicability Compatibility Type Compatibility Type Compatibility with Past Versions Compatibility with Past Versions Compatibility with Future Versions Compatibility with Future Versions Ease of Use Ease of Use Applicability Applicability BACKWARD_TRANSITIVE ✅ ❌ Medium Long-term data storage BACKWARD_TRANSITIVE BACKWARD_TRANSITIVE ✅ ✅ ❌ ❌ Medium Medium Long-term data storage Long-term data storage FORWARD_TRANSITIVE ❌ ✅ Medium Asynchronous updates FORWARD_TRANSITIVE FORWARD_TRANSITIVE ❌ ❌ ✅ ✅ Medium Medium Asynchronous updates Asynchronous updates FULL_TRANSITIVE ✅ ✅ Low Maximum resilience FULL_TRANSITIVE FULL_TRANSITIVE ✅ ✅ ✅ ✅ Low Low Maximum resilience Maximum resilience These compatibility types allow you to configure the system to meet your stability and flexibility requirements. The correct choice depends on your system architecture and use cases.

Transitive Compatibility Types in Schema Registry for Apache Kafka

Everything Apache Iceberg-Related Announced This Year

Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit), CDK for Terraform, and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks, Confluent, and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent, Snowflake, Databricks, and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization. Infrastructure as code (IaC) has become increasingly challenging as organization and usage scale. Inconsistent configurations, security gaps, and increasing costs remain to be hard to manage without the right level of abstraction, but what's the right level of abstraction? Infrastructure as code (IaC) From DevOps to Infrastructure as Code DevOps started with engineers clicking around the AWS console and writing scripts to automate tasks like creating EC2 instances and setting up VPCs, but as usage grew, Infrastructure as Code (IaC) tools such as Terraform and CloudFormation emerged. Instead of writing shell scripts, infrastructure engineers can write declarative code to define how the cloud resources should be deployed. This is a great step up in terms of abstraction from the one-off automation scripts. Terraform CloudFormation On one hand, the declarative nature of the IaC tools makes it easy enough for most developers to write (you can think of it as configuration instead of code); on the other, it's not abstracted enough to give platform engineers the ability to write modularized code for developers to create infrastructure across the organization. Take resource tagging for example, for both cost tracking and governance purposes, it's a very common use case to want to tag cloud resources by application, team, business unit, cost center, etc. With Terraform, application engineers need to manually specify these tags for each resource. Even if platform engineering teams create the relevant terraform "tag" module to enforce common fields, application engineers still need to provide the tag values every time. The declarative nature of IaC makes it hard for platform engineering teams to automatically populate these tag values from a central registry of applications and teams. This becomes especially problematic as the number of resources and size of the organization grows. This led to the next question: how can we create a higher level of abstraction on top of IaC that addresses these enterprise-scale challenges? Creating Platform Abstraction for Scale Creating the right level of platform abstraction enables large organizations to manage infrastructure at scale more effectively and increase developer velocity. Platform engineering teams often use a mix of strategies, depending on use cases, resources, and expertise within the organization. 1. Terraform Modules or Templates Creating reusable terraform modules or templates is a common and easy way to create higher-level abstractions. While flexible, it's a lower-level abstraction, and making organization-wide changes can be time-consuming as they require review and testing by individual application teams. For organizations lacking resources and expertise, vendors like Resourcely create templated terraform code with enforced practice and security. Resourcely 2. Imperative IaC With declarative IaC, integrating external processes into infrastructure deployment often requires workarounds. Platform teams frequently combine declarative IaC with one-off scripts or API calls in their CI/CD pipelines. Tools like AWS CDK (Cloud Development Kit) , CDK for Terraform , and Pulumi allow teams to generate IaC using imperative programming languages, enabling platform engineers to create infrastructure code alongside dynamic processes. AWS CDK (Cloud Development Kit) CDK for Terraform Pulumi While this provides greater flexibility and abstraction, it poses a higher adoption barrier for application teams less familiar with these "CDK" frameworks. 3. Infrastructure and Platform as a Service At the highest abstraction level, platforms like Databricks , Confluent , and Elastic wrap entire infrastructure deployments, exposing only relevant configurations to developers. This allows the provisioning of complex, distributed systems with minimal effort. Kubernetes similarly abstracts compute resource management, enabling developers to create deployments using high-level configurations. Databricks Confluent Elastic Kubernetes While the high-level abstraction allows platform teams to standardize provisioning, governance, and security at scale for supported use cases, it comes at the cost of reduced flexibility and potential vendor lock-in. New or custom use cases can require much longer implementation cycles. Platform Abstraction: Balancing Flexibility and Control Defining the "right" level of abstraction is never easy. Too little abstraction can lead to frequent manual updates and migrations, inconsistent standards, and security and governance implementations. However, too much abstraction can force teams to adapt their use cases to fit the platform's limitations, slowing down developer velocity. The ideal abstraction varies case by case. For real-time fraud detection or experimental features, teams often need flexibility. Templated terraform might provide sufficient guardrails while enabling developers to easily create custom deployments. In contrast, common use cases like data streaming involving Kafka, connectors, and multi-platform integrations typically require more abstraction. Platform teams often create end-to-end deployments, exposing only necessary configs. However, data pipelines can be nuanced, and platform teams often expose lower-level components, allowing application teams to create custom implementations easily and bring back these features to the platform as they mature. For less complicated scenarios like application deployment, most teams only need to specify a Docker image, CPU, and memory requirements. Platform teams can wrap up the CI/CD process without exposing lower-level components. Knowing the different implementation strategies and levels of abstraction for various scenarios, how can we build platforms that scale? Future of Platform Engineering Data and AI Platform as a Service Kubernetes has simplified infrastructure and application deployment, but data platforms have become increasingly complex. The proliferation of specialized data tools, compounded by AI and ML solutions, has made standardization, integration, security, and governance much harder to manage than infrastructure or application deployment. Vendors like Confluent , Snowflake , Databricks , and dbt are improving the developer experience with more automation and integrations, but they often operate independently. This fragmentation makes standardizing multi-directional integrations across identity and access management, data governance, security, and cost control even more challenging. Developing a standardized, secure, and scalable solution for multi-platform environments is now a fast-evolving area for platform engineering teams. Confluent Snowflake Databricks dbt Higher Abstraction for Standardization, Security, and Governance While application developers can use low-level terraform to create infrastructure on their own, this approach doesn’t scale at the enterprise level. Standardization, security, governance, and cost efficiency aren't just “nice to have” but "required" for large organizations. Many companies undertake multi-year, large-scale migrations due to a lack of abstraction where it’s needed, meanwhile slowing down developer velocity in areas where abstraction isn’t necessary. Developers are looking for a combination of flexibility and simplicity depending on the features they are trying to build. Platform teams must continue to simplify the developer experience while strategically exposing lower-level components where necessary. By exposing the right components, they provide developers with greater flexibility and reduce the time needed to integrate new features back into the platform. Imperative IaC for Platform Teams To enable platform teams to build for higher abstraction and flexibility, offload the complexity from CI/CD pipelines, which are designed for orchestration rather than abstraction. Currently, platform teams typically use either configuration-driven systems or declarative IaC, but both approaches have limitations when it comes to creating sophisticated, dynamic deployments and abstraction. The gap in tooling continues to be a challenge, especially for data and AI platform engineering. While there's a proliferation of data product vendors, few focus on simplifying multi-platform deployments. Instead, there's a surge of governance, lineage, and cataloging products to compensate for the lack of integrations. These solutions aim to solve the complexity that arises from using multiple platforms without addressing the core issue of creating streamlined and flexible deployments across data and AI products and platforms. While infrastructure development tools and frameworks are a highly competitive space, there's still room for tools that can provide the advantage of imperative programming for complex logic, while preserving the audit trails, and version control of declarative IaC for governance and compliance. Summary The value of platform engineering is often less tangible than customer-facing product features. While platform improvements may not directly contribute to top-line growth, they are one of the most effective ways to manage cloud costs, which are often one of the biggest drivers of bottom-line costs. Choosing the right toolings and abstraction levels can not only improve developer experience and productivity but also strengthen security, governance, and compliance across the organization.

Platform Engineering Abstraction: How to Scale IaC for Enterprise

Step-by-step guide on how to use the .Net Aspire Kafka component in Visual Studio. .Net Aspire framework is used to develop cloud and production-ready distributed applications. It consists of components to handle cloud-native concerns such as Redis, Postgres etc. Prerequisites Install .Net 8
Install Visual Studio 2022 version 17 or higher
.Net Aspire Workload
Container runtime such as Docker Desktop 10 Day .Net Aspire Challenge Objectives Learn how to create a starter project using .Net Aspire with the Apache Kafka component. Github Sample: The solution structure is divided into the following projects: DotnetAspireChallenge.ApiService
DotnetAspireChallenge.AppHost
DotnetAspireChallenge.ServiceDefaults
DotnetAspireChallenge.Web Getting Started Step 1: Install the following NuGet package Install the following Nuget package into the subsequent project “DotnetAspireChallenge.AppHost” dotnet add package Aspire.Hosting.Kafka In the above project, register Kafka UI as shown below var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); Step 2: Add dependency of Kafka Producer Add the dependency in the Program.cs file of the project “**DotnetAspireChallenge.**ApiService” builder.AddKafkaProducer<string, string>("messaging"); and add a relevant minimal API endpoint using the following code. public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } The endpoint takes two parameters namely key and value as route values, and produces the message on the docker-hosted Kafka server. https://localhost:7313/send?key=key&value=1 Step 3: Add dependency of Kafka Consumer Now move “DotnetAspireChallenge.Web” project wherein register as a Kafka producer builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); Note: It's mandatory to provide a default group ID. Step 4: Create a Razor Page Create a basic razor page named “KafkaConsumer.razor” to show the consumed message from the Kafka server. @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } Step 5: Configure HttpCall to the ApiService public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } Kafka Produce Demo Kafka UI Demo Kafka Consume Demo Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge
Cheatsheet: Cheat Sheets — .Net Step-by-step guide on how to use the .Net Aspire Kafka component in Visual Studio. .Net Aspire framework is used to develop cloud and production-ready distributed applications. It consists of components to handle cloud-native concerns such as Redis, Postgres etc. Prerequisites Install .Net 8 Install Visual Studio 2022 version 17 or higher .Net Aspire Workload Container runtime such as Docker Desktop Install .Net 8 .Net 8 Install Visual Studio 2022 version 17 or higher Visual Studio 2022 .Net Aspire Workload Container runtime such as Docker Desktop Docker Desktop 10 Day .Net Aspire Challenge 10 Day .Net Aspire Challenge 10 Day .Net Aspire Challenge Objectives Learn how to create a starter project using .Net Aspire with the Apache Kafka component. Github Sample : The solution structure is divided into the following projects: Github Sample DotnetAspireChallenge.ApiService DotnetAspireChallenge.AppHost DotnetAspireChallenge.ServiceDefaults DotnetAspireChallenge.Web DotnetAspireChallenge.ApiService DotnetAspireChallenge.AppHost DotnetAspireChallenge.ServiceDefaults DotnetAspireChallenge.Web Getting Started Step 1: Install the following NuGet package Install the following Nuget package into the subsequent project “ DotnetAspireChallenge.AppHost ” DotnetAspireChallenge.AppHost dotnet add package Aspire.Hosting.Kafka dotnet add package Aspire.Hosting.Kafka In the above project, register Kafka UI as shown below var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); var messaging = builder.AddKafka("messaging")
```                           .WithKafkaUI();

Then finally add a reference to both the Producer and Consumer where the producer is “DotnetAspireChallenge.ApiService” and the consumer is “DotnetAspireChallenge.Web” project respectively.
```csharp
    var apiService = builder.AddProject<Projects.DotnetAspireChallenge_ApiService>("apiservice")
        .WithReference(messaging);
    
    
    
    builder.AddProject<Projects.DotnetAspireChallenge_Web>("webfrontend")
        .WithExternalHttpEndpoints()
        .WithReference(cache)
        .WithReference(apiService)
        .WithReference(messaging); Step 2: Add dependency of Kafka Producer Add the dependency in the Program.cs file of the project “**DotnetAspireChallenge.**ApiService” builder.AddKafkaProducer<string, string>("messaging"); builder.AddKafkaProducer<string, string>("messaging"); and add a relevant minimal API endpoint using the following code. public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } public static class AspireKafkaExtension
    {
        public static void MapAspireKafkaEndpoint(this WebApplication app)
        {
            app.MapGet("/send", async (IProducer<string, string> services, string key, string value) =>
            {
                try
                {
                    var message = new Message<string, string> { Key = key, Value = value };
                    DeliveryResult<string, string>? result = await services.ProduceAsync("messaging", message);
                    return result;
                }
                catch (Exception ex)
                {
    
                    throw;
                }
    
            });
        }
    } The endpoint takes two parameters namely key and value as route values, and produces the message on the docker-hosted Kafka server. https://localhost:7313/send?key=key&value=1 https://localhost:7313/send?key=key&value=1 Step 3: Add dependency of Kafka Consumer Now move “ DotnetAspireChallenge.Web ” project wherein register as a Kafka producer DotnetAspireChallenge.Web builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); builder.AddKafkaConsumer<string, string>("messaging", options =>
    {
        options.Config.GroupId = "my-consumer-group";
        options.Config.AutoOffsetReset = AutoOffsetReset.Earliest;
        options.Config.EnableAutoCommit = false;
    }); Note: It's mandatory to provide a default group ID. Note: It's mandatory to provide a default group ID. Step 4: Create a Razor Page Create a basic razor page named “KafkaConsumer.razor” to show the consumed message from the Kafka server. @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } @page "/kafka"
    @attribute [StreamRendering(true)]
    @attribute [OutputCache(Duration = 5)]
    @using Confluent.Kafka
    <h3>KafkaConsumer</h3>
    
    @inject KafkaConsumeMessageClient kafaConsumeMessageClient
    <PageTitle>Kafka Consumed Message</PageTitle>
    
    <h1>Kafka</h1>
    
    <p>This component demonstrates showing data loaded from a backend API service.</p>
    
    @if (consumedMessage == null)
    {
        <p><em>Loading...</em></p>
    }
    else
    {
        <table class="table">
            <thead>
                <tr>
                    <th>Topic</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>@consumedMessage.Topic</td>
                    <td>@consumedMessage.Value</td>
                </tr>
    
            </tbody>
        </table>
    }
    @code {
        private ConsumeResult<string, string>? consumedMessage;
    
        protected override async Task OnInitializedAsync() => consumedMessage = kafaConsumeMessageClient.GetKafkaMessage();
    } Step 5: Configure HttpCall to the ApiService public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } public class KafkaConsumeMessageClient(HttpClient httpClient, IConsumer<string, string> _consumer)
    {
    
        public ConsumeResult<string, string>? GetKafkaMessage(CancellationToken cancellationToken = default)
        {
            ConsumeResult<string, string>? deliveryResult = null;
            _consumer.Subscribe("messaging");
            deliveryResult = _consumer.Consume(TimeSpan.FromSeconds(10));
    
            return deliveryResult;
        }
    } Kafka Produce Demo Kafka UI Demo Kafka Consume Demo Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge Cheatsheet: Cheat Sheets — .Net Github Project: GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge GitHub - ssukhpinder/DotnetAspireChallenge: 10 Day .Net Aspire Challenge Cheatsheet: Cheat Sheets — .Net Cheat Sheets — .Net

Stories, News & Mentions about CONFLUENT

Trending Topics

Classic

Neon Noir

Minty

Newspaper

HN StartUps