Open Source Summit + AI_dev Japan 2024

As the importance of data utilization for AI grows, enterprises aim to securely exchange data with their customers and leverage external data.Many services and open-source software related to data sharing and governance have attracted attention and extensive research and development, such as Snowflake Marketplace and Databricks Delta Sharing, among others. However, sharing data between different companies presents numerous challenges in terms of data security and efficiency, including efficient access to geographically dispersed data and access control for data managed by multiple organizations. To address these challenges, we have developed virtual data lake system that achieves efficient and secure data sharing using federated cloud storage. In this approach, virtual data integration is performed by collecting and managing only metadata without collecting the original data. In this session, we’ll take a look at how to build a safe and efficent data lake system by using existing OSS for data governance and data federation tools, such as LinkedIn DataHub and Alluxio ..etc.