Open Source Summit + AI_dev Japan 2024

Every conversation on AI starts with models and ends with data. Data preparation is emerging as a very important phase of the GenAI journey, as high quantity and quality text and code corpora for GenAI model training have shown to play a crucial role in producing high performing Large Language Models (LLMs). The data preparation phase in the Generative AI lifecycle aims to clean, filter, and transform the datasets of text and code that are acquired from various sources into a tokenized form that is suitable for the training of LLMs, be it pre-training, or constructing LLM apps via fine-tuning or instruct tuning. The latter poses unique challenges, as each use case may necessitate tailored data preparation approaches. Given the enduring and evolving demand for data preparation techniques in LLM applications, we are introducing Data Prep Kit as an open-source software asset. This endeavour is geared towards fostering collaborative efforts within the community, enabling collective development and utilization, and ultimately reducing time to value. DPK has been instrumental in powering the IBM open-source Granite models.