Open Source Summit + AI_dev Japan 2024

Recent IT paradigms like DevOps, CD, and IaC have shortened release cycles but burdened developers with mastering many tools. Platform Engineering, a field of software engineering frequently discussed in the CNCF, addresses this by providing Internal Developer Platforms (IDPs) that automate non-essential tasks. Kubernetes is widely used to run the various components that make up an IDP, thanks to its consistent API experience and high extensibility. However, Kubernetes was initially created as a general foundation for running workloads, not as part of platforms. Therefore, IDP builders must add Cloud Native technologies and customizations to Kubernetes. Despite three years since the rise of platform engineering, making the right architectural and technology choices remains challenging and requires a deep understanding of the organization's context and technologies. In this session, he will explain various architectural patterns for building IDPs using Kubernetes, including access control and multitenancy. Additionally, based on years of experience providing Kubernetes-based IDPs, he will discuss the pros/cons, operational considerations, and suitable organizational structures.

The demand for running LLMs in the cloud is growing exponentially. In this keynote, we will explore developer’s and enterprises’ urging need for open source LLMs and the best practices for deploying them in cloud-native environments. Three key approaches for LLM deployment: Python-based solutions, native runtimes like llama.cpp or vLLM, and WebAssembly as an abstraction layer. Miley will discuss the benefits and challenges of each approach, focusing on real-world applications, integration ease, portability, and resource efficiency and talk about the CNCF CNAI ecosystem landscape. This keynote aims to demystify cloud-native AI. Attendees will have practical advice and a clear roadmap for deploying LLMs in the cloud, learning about the strengths and trade-offs of different approaches, to have a better idea to select and implement the most suitable strategy

Developers often face the complex challenges of designing, debugging, and testing distributed systems like microservices. Understanding failures, identifying performance issues, and ensuring system reliability from the early stages of design and development can be daunting. Observability technologies provide valuable insights not just in production but also during design and development. In this session, we will explore OpenTelemetry, a cutting-edge observability framework, and its practical applications in the design, debugging, and testing of distributed systems. Key topics include: - Assessing the impact of incorporating a cache server on system behavior during the design phase. - Evaluating how database failures affect both backend and frontend applications during fault testing. - Detecting performance bottlenecks for specific requests during load testing. Participants will gain a clear understanding of how OpenTelemetry can revolutionize their debugging and testing processes, leading to more effective experiments and increased reliability in their distributed systems.

This presentation will explain the concept of Docker and demonstrate how it shortens the development iteration cycle. Additionally, a demonstration will showcase how Kubernetes can enhance CI/CD performance in the development process.

With the advancements in the fields of Applied AI/ML, computational power, FinOps practices, and cloud services, organizations are increasingly building and deploying AI/ML workloads on Kubernetes. Cloud Computing gives the impression that the underlying resources (like Virtual Machines, CPU, RAM, Storage, etc.) are infinite, and hence many individuals don’t focus on the count of machines and resources when training their AI/ML models and proceed with training fairly large models with millions and billions of parameters. The energy-intensive nature of model training and iterations, coupled with the growing scale of AI/ML deployments, is contributing to a considerable environmental impact in the form of Carbon Emissions.

This talk introduces some key open-source cloud agnostic tools and techniques to build eBPF-powered observability solutions for practicing Sustainable Computing. By analyzing and tuning performance and energy metrics from Kubernetes resources like Pods and Nodes, AI/ML Practitioners and Platform Engineers can optimize workloads, reduce energy consumption, lower costs, and minimize environmental impact while maintaining performance.

As the importance of data utilization for AI grows, enterprises aim to securely exchange data with their customers and leverage external data.Many services and open-source software related to data sharing and governance have attracted attention and extensive research and development, such as Snowflake Marketplace and Databricks Delta Sharing, among others. However, sharing data between different companies presents numerous challenges in terms of data security and efficiency, including efficient access to geographically dispersed data and access control for data managed by multiple organizations. To address these challenges, we have developed virtual data lake system that achieves efficient and secure data sharing using federated cloud storage. In this approach, virtual data integration is performed by collecting and managing only metadata without collecting the original data. In this session, we’ll take a look at how to build a safe and efficent data lake system by using existing OSS for data governance and data federation tools, such as LinkedIn DataHub and Alluxio ..etc.

Traditional testing methods like unit and integration tests are great for functional validation in isolation, but are they enough for distributed systems? Distributed systems need to deal with real-world failures such as network issues, hardware errors, and race conditions. One of the proven ways to test these systems is to inject failures during testing and see if the system still works as expected. This is called robustness testing, where you run the system like it would be used in real life. Jespen is one of the first frameworks to test distributed systems by simulating such real-world scenarios and validating the operational history. Inspired by Jespen, etcd, the backbone of Kubernetes, built its own testing framework. This framework is written in Golang and for Golang projects, allowing even more failure types on the fly and verifying if the data stays consistent using Porcupine. As etcd contributors, we will share our challenges in writing tests to force failure via gofail and our journey of developing, leveraging, and debugging issues caught by this ever-evolving framework, so that you can apply the findings to your projects with minimal tweaks.