ragnar 0.3.0

We’re happy to announce that ragnar 0.3.0 is now available on CRAN. ragnar is a tidy, transparent toolkit for building trustworthy retrieval-augmented generation (RAG) workflows: ingest documents, build a store, retrieve relevant chunks, and inspect exactly what’s being fed to a model.

If you’re new to ragnar, the quickest way to get oriented is the Getting Started vignette. If you’ve already built a store with ragnar 0.2, this release focuses on making it easier to scale ingestion, use more embedding providers, and connect your store to the tools you already use.

You can install ragnar from CRAN with:

install.packages("ragnar")

This post covers the biggest user-facing changes in ragnar 0.3.0. For a complete list of changes, see the NEWS.

library(ragnar)

A quick refresher

If you’re already familiar with ragnar, feel free to skip this section.

ragnar helps you build retrieval-augmented generation (RAG) workflows by turning your trusted documents into a local store that you can query with both vector search (embeddings) and keyword search (BM25).

At the “front door”, read_as_markdown() can ingest web pages, PDFs, Office documents, images (via OCR), archives, and even YouTube URLs (via transcripts), so you can usually start from the same sources you’d use for manual research.

At a high level, a typical ragnar workflow has three parts:

1. Build a store:
   • Collect document sources (URLs or files) and convert them to Markdown with read_as_markdown().
   • Split documents into chunks with markdown_chunk() (optionally adding context).
   • Embed and store chunks in a DuckDB-backed RagnarStore.
2. Query and inspect the store:
   • Retrieve chunks directly with ragnar_retrieve(). It returns a tibble with scores, source information, and the chunk text (including columns like origin, cosine_distance, bm25, context, and text), so you can inspect exactly what will be passed downstream.
   • Use the Store Inspector or Embedding Atlas ( ragnar_store_inspect() and ragnar_store_atlas()) to understand what’s working, then iterate and go back to step 1 as needed.
3. Connect the store to tools:
   • Register a retrieval tool with an ellmer chat so an agent can search the store on demand.
   • Serve retrieval over MCP so external tools and agents can query the store directly.
   • Write your own loop using ragnar_retrieve() or lower-level helpers like ragnar_retrieve_vss() and ragnar_retrieve_bm25().

What’s new

This release focuses on four big improvements:

• Faster ingestion for large corpora with ragnar_store_ingest().
• Better retrieval with multi-query support and better de-duplication and de-overlapping of results.
• New embedding providers: Azure OpenAI and Snowflake.
• New integrations and tooling: serve a store over MCP, plus improved inspection with the Store Inspector and embedding atlas.

In the sections below, we’ll walk through each change in more detail.

Faster ingestion with ragnar_store_ingest()

Ingestion is usually the slowest part of building a knowledge store. ragnar_store_ingest() parallelizes the document preparation step with mirai, and then writes prepared chunks to the store in the main process. It’s designed to make it easy to ingest hundreds (or thousands) of pages without hand-rolling your own parallel pipeline.

Only preparation (reading, chunking, and optionally embedding) is parallelized; store writes still happen in the main process.

store <- ragnar_store_create(
  "docs.ragnar.duckdb",
  embed = \(x) ragnar::embed_openai(x, model = "text-embedding-3-small")
)

paths <- ragnar_find_links("https://quarto.org/sitemap.xml")

ragnar_store_ingest(store, paths, n_workers = 4, prepare = \(path) {
  path |> read_as_markdown() |> markdown_chunk()
})

Better retrieval: multiple queries and fewer duplicates

Retrieval is where ragnar tries to be pragmatic: we run both semantic search (embeddings) and keyword search (BM25) because they fail in different ways. This release makes it easier to do that intentionally.

• ragnar_retrieve() now accepts a vector of queries, so you can pass one query tuned for semantic search and one tuned for keywords.
• ragnar_register_tool_retrieve() uses a new default tool name prefix: search_{store@name} (instead of rag_retrieve_from_{store@name}).
• When registered with ellmer, ragnar’s retrieval tool continues to avoid returning previously returned chunks, enabling deeper searches via repeated tool calls.
• BM25 result ordering was corrected to sort by descending score.
• Duplicate rows from ragnar_retrieve() when running multiple queries were removed.

ragnar_retrieve(
  store,
  c(
    "How do I subset a data frame with a logical vector?",
    "subset dataframe logical vector"
  ),
  top_k = 10
)

New embedding providers: Azure OpenAI and Snowflake

ragnar’s embedding helpers continue to expand so you can use the infrastructure you already have:

• embed_azure_openai() supports embeddings from Azure AI Foundry.
• embed_snowflake() supports embeddings via the Snowflake Cortex Embedding API.

These integrate the same way as the other providers: you choose an embed function when creating a store, and ragnar uses it during insertion and retrieval.

Better document reading (including YouTube transcripts)

read_as_markdown() is now more robust across common inputs, so you get higher-quality documents without having to hand-fix edge cases.

• Substantial improvements to HTML-to-Markdown conversion, including correct handling of nested code blocks, plus a range of other robustness fixes driven by real-world failure cases.
• read_as_markdown() once again fetches YouTube transcripts and now supports a youtube_transcript_formatter so you can include timestamps or links in the transcript output.
• Reading plain text with non-ASCII content was fixed.
• read_as_markdown() gained an origin argument to control what gets recorded on returned documents.

Together, these changes make ingestion more reliable, which helps improve retrieval quality downstream.

New integrations: serve a store over MCP

mcp_serve_store() lets you expose a RagnarStore as an MCP tool. This is particularly useful if you already have a local store and want an MCP-enabled client (like Codex CLI or Claude Code) to query it directly.

For example, with Codex CLI you can add something like this to ~/.codex/config.toml:

[mcp_servers.my_store]
command = "Rscript"
args = [
  "-e",
  "ragnar::mcp_serve_store('docs.ragnar.duckdb', top_k=10)"
]

This runs a long-lived R process that exposes retrieval over MCP.

New ways to inspect a store

ragnar now has more tools to help you understand what your store contains and why retrieval is (or isn’t) working:

• The Store Inspector received a number of usability improvements (keyboard shortcuts, improved preview, better metadata display, and general bug fixes).
• ragnar_store_atlas() integrates with the Embedding Atlas project to visualize your embedding space (via reticulate).

The Store Inspector makes it easy to iterate on retrieval: try a query, compare vector search and BM25, and inspect the underlying chunks and metadata that were returned. The screenshots below show a store built from the Quarto documentation.

If you’re not sure whether a store “looks right”, ragnar_store_atlas() gives you a high-level view of how your documents cluster in embedding space. It’s a useful way to spot outliers, see which areas of the space match a query, and explore how clusters relate back to your sources.

Get started

Install ragnar with install.packages("ragnar"), then work through the Getting Started vignette. For details on individual functions, see the function reference. For the full changelog, see NEWS.

ragnar is designed to help you build trustworthy RAG workflows by making it easy to inspect what gets retrieved and what ultimately gets sent to your model. If you try ragnar 0.3.0, we’d love to hear what you’re using it for in GitHub Discussions.

Acknowledgements

Thanks to everyone who contributed to ragnar 0.3.0 through code, issues, testing, and feedback: @agricolamz, @AlekFisher, @bianchenhao, @brooklynbagel, @bshashikadze, @christophscheuch, @cstubben, @dfalbel, @eschillerstrom-usfws, @grantmcdermott, @howardbaik, @jeroenjanssens, @jhbrut, @JosiahParry, @jpmarindiaz, @luisDVA, @mattwarkentin, @Rednose22, @shikokuchuo, @smach, @SokolovAnatoliy, @t-kalinowski, @thisisnic, and @vrognas.