Jupyter Notebook Writing

Skill: Jupyter Notebook Writing

Write Milvus application-level Jupyter notebook examples as a DevRel workflow. Uses a Markdown-first approach — AI edits .md files, then converts to .ipynb via jupyter-switch.

Prerequisites: Python >= 3.10, uv (uvx command available)

---

When to Use

The user wants to create or edit a Jupyter notebook example, typically demonstrating Milvus usage in an application context (RAG, semantic search, hybrid search, etc.).

---

Core Workflow: Markdown-First Editing

Jupyter .ipynb files contain complex JSON with metadata, outputs, and execution counts — painful for AI to edit directly. Instead:

1. Write/Edit the `.md` file — AI works with clean Markdown 2. Convert to `.ipynb` — using jupyter-switch for runnable notebook 3. Keep both files in sync — the .md is the source of truth for editing

Format Convention

In the .md file:

• Python code blocks ( `python ... ` ) become code cells in the notebook
• Everything else becomes markdown cells
• Cell outputs are not preserved in .md (they get generated when running the notebook)

Conversion Commands

# Markdown -> Jupyter Notebook
uvx jupyter-switch example.md
# produces example.ipynb

# Jupyter Notebook -> Markdown
uvx jupyter-switch example.ipynb
# produces example.md

• The original input file is never modified or deleted
• If the output file already exists, a .bak backup is created automatically

---

Step-by-Step

Creating a New Notebook

1. Create example.md with the content (see structure below) 2. Convert: uvx jupyter-switch example.md 3. Both example.md and example.ipynb now exist

Editing an Existing Notebook

1. If only .ipynb exists, convert first: uvx jupyter-switch example.ipynb 2. Edit the .md file 3. Convert back: uvx jupyter-switch example.md

Testing / Running

1. Resolve the Jupyter execution environment

Before running any notebook, you must determine which Python environment to use. The system default jupyter execute may not have the required packages installed.

Step A — Detect available environments.

# Discover conda/mamba environments
conda env list 2>/dev/null || mamba env list 2>/dev/null

# Discover registered Jupyter kernels
jupyter kernelspec list 2>/dev/null

# Check system default Python
which python3 2>/dev/null && python3 --version 2>/dev/null

# Check for local virtual environment in the working directory
ls -d .venv/ venv/ 2>/dev/null

# Check if a uv-managed project (pyproject.toml + .venv)
test -f pyproject.toml && test -d .venv && echo "uv/pip project venv detected"

Step B — Ask the user which environment to use. Present a numbered list of choices. Include all detected environments:

1. System default — run jupyter execute as-is, no --kernel_name 2. Each detected conda/mamba environment — show name and path 3. Each registered Jupyter kernel — show kernel name 4. Local venv (if .venv/ or venv/ found in working directory) — the Python inside that venv 5. Custom — let the user type a Python path or environment name

Note on uv projects: If the working directory has pyproject.toml + .venv/ (a uv-managed project), the local venv option covers this case. The user can also run uv run jupyter execute example.ipynb directly if jupyter is a project dependency.

Example prompt:

Which Python environment should I use to run this notebook?

1. System default (jupyter execute as-is)
2. conda: myenv (/path/to/envs/myenv)
3. Jupyter kernel: some-kernel
4. Local venv (.venv/)
5. Custom — enter a path or environment name

Step C — Apply the chosen environment:

2. Prepare the notebook for execution

Before running, comment out "setup-only" cells in the .md file — cells that are meant for first-time users but should not run in an automated test environment. Specifically:

• `pip install` cells — dependencies should already be installed in the chosen Jupyter environment. If any packages are missing or need upgrading, install them externally in the target environment (with --upgrade), not inside the notebook.
• API key / credential placeholder cells — e.g. os.environ["OPENAI_API_KEY"] = "sk-***********". Instead, set environment variables externally before running (export in shell, or inject via code before jupyter execute).
• Mock / demo-only cells — any cells that exist purely for illustration and would fail or interfere in a real run.

To comment out a cell, wrap its content in a block comment so the cell still executes (producing empty output) but does nothing:

# # pip install --upgrade langchain pymilvus
# import os
# os.environ["OPENAI_API_KEY"] = "sk-***********"

This keeps the notebook structure intact (cell count, ordering) while preventing conflicts with the external Jupyter environment.

For environment variables: either export them in the shell before running jupyter execute, or prepend them to the command:

OPENAI_API_KEY="sk-real-key" jupyter execute --kernel_name=<name> example.ipynb

3. Convert and run

1. Convert .md to .ipynb if needed 2. Install any missing dependencies in the target environment externally: <env-python> -m pip install --upgrade <packages> 3. Run: jupyter execute --kernel_name=<name> example.ipynb (omit --kernel_name if using system default) 4. If errors found, fix in the .md file, uncomment setup cells if needed for debugging, and re-convert

---

Notebook Structure Template

A typical Milvus example notebook follows this structure:

# Title

Brief description of what this notebook demonstrates.

## Prerequisites

Install dependencies:

` ``python
!pip install pymilvus some-other-package
` ``

## Setup

Import and configuration:

` ``python
from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")
` ``

## Prepare Data

Load or generate example data:

` ``python
# data preparation code
` ``

## Create Collection & Insert Data

` ``python
# collection creation and data insertion
` ``

## Query / Search

` ``python
# search or query examples
` ``

## Cleanup

` ``python
client.drop_collection("example_collection")
` ``

---

Reference Documents

This skill includes two reference documents under references/. Read them when the task involves their topics.

Bootcamp Format (references/bootcamp-format.md)

Read this when the user is writing a Milvus integration tutorial for the bootcamp repository. It covers:

• Badge format (Colab + GitHub badges at the top)
• Document structure: Header -> Prerequisites -> Main Content -> Conclusion
• Dependency install format with Google Colab restart note
• API key placeholder conventions ("sk-***********")
• Each code block should have a short text introduction before it

Milvus Code Style (references/milvus-code-style.md)

Read this when the notebook involves pymilvus code. Key rules:

• Always use `MilvusClient` API — never use the legacy ORM layer (connections.connect(), Collection(), FieldSchema(), etc.)
• Always define schema explicitly (create_schema + add_field) — do not use the shortcut create_collection(dimension=...) without schema
• Include has_collection check before creating collections
• Add commented consistency_level="Strong" line in create_collection()
• No need to call load_collection() — collections auto-load on creation
• First MilvusClient connection must include the blockquote explaining uri options (Milvus Lite / Docker / Zilliz Cloud)

---

Important Notes

• Always edit the `.md` file, not the .ipynb directly. The .md is easier for AI to read and write.
• Keep both files — .md for editing, .ipynb for running/sharing.
• After editing .md, always re-run uvx jupyter-switch example.md to sync the .ipynb.

Milvus Integration Tutorial Format Specification

This document outlines the standard format for Milvus integration tutorials in the Milvus bootcamp repository.

File Naming

• Use lowercase with underscores
• Keep provider names simple and recognizable
• This guide can be used for both markdown and ipynb files.

Document Structure

0. Badges

Add two badges to the top of your notebook. Note that the suffix of the notebook path should be ipynb, even if it is a markdown file since we will finally convert it to a jupyter notebook file.

<a href="https://colab.research.google.com/github/milvus-io/bootcamp/blob/master/bootcamp/relative_path/to/your_notebook.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>   <a href="https://github.com/milvus-io/bootcamp/blob/master/bootcamp/relative_path/to/your_notebook.ipynb" target="_blank">
    <img src="https://img.shields.io/badge/View%20on%20GitHub-555555?style=flat&logo=github&logoColor=white" alt="GitHub Repository"/>

1. Header Section

• Title
• Start with "# " as the markdown heading level.
• Introduction
• One or two paragraphs explaining the purpose of the tutorial, the integration provider and the key technologies.

2. Prerequisites/Preparation

• Dependencies:
• Pip install command with all required packages, always include --upgrade flag
• After installing pip in Google Colab, always add this note:

> If you are using Google Colab, to enable dependencies just installed, you may need to **restart the runtime** (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

• API Keys:
• Instructions for obtaining API keys
• Environment variable setup code block
• Use placeholder format: "sk-***********" or "***********"

3. Main Content Sections

This is the main content of the tutorial. It should be divided into several hierarchical sections, each section should be a logical unit of the tutorial.

Before each code block, add a short introduction to the code block, which can avoid the reader to be confused about the code block.

(Optional) In the text description, if there are concepts or terms that are not familiar to the reader, you can properly add links to the text.

At the end of the article, the conclusion section should be as simple as possible, such as several sentences to summarize the tutorial.

---

Example

Below this divider, here is an example of a Milvus integration tutorial: ---

<a href="https://colab.research.google.com/github/milvus-io/bootcamp/blob/master/integration/langchain/rag_with_milvus_and_langchain.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a> <a href="https://github.com/milvus-io/bootcamp/blob/master/integration/langchain/rag_with_milvus_and_langchain.ipynb" target="_blank"> <img src="https://img.shields.io/badge/View%20on%20GitHub-555555?style=flat&logo=github&logoColor=white" alt="GitHub Repository"/> </a>

Retrieval-Augmented Generation (RAG) with Milvus and LangChain

This guide demonstrates how to build a Retrieval-Augmented Generation (RAG) system using LangChain and Milvus.

The RAG system combines a retrieval system with a generative model to generate new text based on a given prompt. The system first retrieves relevant documents from a corpus using Milvus, and then uses a generative model to generate new text based on the retrieved documents.

LangChain is a framework for developing applications powered by large language models (LLMs). Milvus is the world's most advanced open-source vector database, built to power embedding similarity search and AI applications.

Prerequisites

Before running this notebook, make sure you have the following dependencies installed:

! pip install --upgrade langchain langchain-core langchain-community langchain-text-splitters langchain-milvus langchain-openai bs4

If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the "Runtime" menu at the top of the screen, and select "Restart session" from the dropdown menu).

We will use the models from OpenAI. You should prepare the api key OPENAI_API_KEY as an environment variable.

import os

os.environ["OPENAI_API_KEY"] = "sk-***********"

Prepare the data

We use the Langchain WebBaseLoader to load documents from web sources and split them into chunks using the RecursiveCharacterTextSplitter.

import bs4
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Create a WebBaseLoader instance to load documents from web sources
loader = WebBaseLoader(
    web_paths=(
        "https://lilianweng.github.io/posts/2023-06-23-agent/",
        "https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/",
    ),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
# Load documents from web sources using the loader
documents = loader.load()
# Initialize a RecursiveCharacterTextSplitter for splitting text into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)

# Split the documents into chunks using the text_splitter
docs = text_splitter.split_documents(documents)

# Let's take a look at the first document
docs[1]

As we can see, the document is already split into chunks. And the content of the data is about the AI agent.

Build RAG chain with Milvus Vector Store

We will initialize a Milvus vector store with the documents, which load the documents into the Milvus vector store and build an index under the hood.

from langchain_milvus import Milvus
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

vectorstore = Milvus.from_documents(
    documents=docs,
    embedding=embeddings,
    connection_args={
        "uri": "./milvus_demo.db",
    },
    # drop_old=True,  # Drop the old Milvus collection if it exists
)

For the connection_args:

- Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.

- If you have large scale of data, you can set up a more performant Milvus server on docker or kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.

- If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.

Search the documents in the Milvus vector store using a test query question. Let's take a look at the top 1 document.

query = "What is self-reflection of an AI Agent?"
vectorstore.similarity_search(query, k=1)

from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI

# Initialize the OpenAI language model for response generation
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

# Define the prompt template for generating AI responses
PROMPT_TEMPLATE = """
Human: You are an AI assistant, and provides answers to questions by using fact based and statistical information when possible.
Use the following pieces of information to provide a concise answer to the question enclosed in <question> tags.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
<context>
{context}
</context>

<question>
{question}
</question>

The response should be specific and use statistics or numbers when possible.

Milvus Code Style Guidelines

Installation

Install pymilvus and milvus-lite as separate packages — do NOT use the extras syntax:

# GOOD
pip install pymilvus milvus-lite

# BAD — do not use the extras bracket syntax
pip install "pymilvus[milvus_lite]"

The pymilvus[milvus_lite] extra is outdated and may pull in the wrong version. Always install them separately.

API: Always Use MilvusClient (Not ORM)

pymilvus has two sets of APIs:

• `MilvusClient` (recommended) — the current, simplified interface. All new tutorials and examples should use this.
• ORM layer (connections.connect(), Collection(), FieldSchema(), CollectionSchema(), etc.) — the legacy interface. Do not use ORM in new code. It is verbose, harder to read, and no longer recommended.

If you see existing code using ORM patterns like connections.connect(), Collection(name, schema), or utility.has_collection(), rewrite it to use MilvusClient.

Quick comparison

# BAD — ORM style (do not use)
from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType, utility
connections.connect("default", host="localhost", port="19530")
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
    FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=768),
]
schema = CollectionSchema(fields)
collection = Collection("demo", schema)
collection.create_index("vector", {"index_type": "AUTOINDEX", "metric_type": "COSINE"})
collection.load()

# GOOD — MilvusClient style (always use this)
from pymilvus import MilvusClient
client = MilvusClient(uri="./milvus.db")
# ... see collection creation below

Create Collection

Always define the schema explicitly so readers can see the data model clearly.

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="./milvus.db")

schema = client.create_schema(auto_id=False, enable_dynamic_field=True)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=768)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=65535)

index_params = client.prepare_index_params()
index_params.add_index(
    field_name="id",
    index_type="AUTOINDEX"
)
index_params.add_index(
    field_name="vector",
    index_type="AUTOINDEX",
    metric_type="COSINE"
)

client.create_collection(collection_name="demo_collection", schema=schema, index_params=index_params)

Always use `AUTOINDEX` as the index type. It is compatible across Milvus Lite, Milvus server (Docker/K8s), and Zilliz Cloud, so notebooks work in all deployment modes without changing the index config.

In practice, you can specify different configurations for the schema and the index parameters depending on the scenario.

Other format advice

• You'd better include a has_collection check before creating a collection to avoid creating a collection that already exists.
• You'd better add a comment line setting consistency_level="Strong" in the create_collection() method.

```python

client.create_collection(

collection_name=collection_name,

dimension=5,

# Strong consistency waits for all loads to complete, adding latency with large datasets

# consistency_level="Strong", # Supported values are ("Strong", "Session", "Bounded", "Eventually").

)

```

But if you find this description too verbose, you can just simply omit these lines.

• No need to explicitly load the collection using load_collection() after creating it, because the collection is loaded automatically when it is created.

• When the first time you connect to a Milvus server using uri or token, you need to provide the blockquote block for the explanation of the connection options, like this:

As for the argument of MilvusClient:

- Setting the uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.

- If you have large scale of data, you can set up a more performant Milvus server on docker or kubernetes. In this setup, please use the server uri, e.g.http://localhost:19530, as your uri.

- If you want to use Zilliz Cloud, the fully managed cloud service for Milvus, adjust the uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.