Skip to main content

Getting Started

2 min read

Introduction to Needle​

Needle is the easiest way to build a RAG pipeline. In fact there's no pipeline you need to build since Needle take care of it for you! You only need to provide the data, either by uploading or using connectors, and in several minutes you can start searching your data semantically.

What You Can Achieve with Needle​

Here are some examples of what you can do with Needle:

  • Prompt your technical documentation and get instant answers.
  • Find answers to customers' requests instantly from your help/FAQ page.
  • Search your company's internal documents (Legal, HR, Financial Reports, Designs, etc.) in seconds.
  • Talk to all research papers in your area.
  • Build chatbot assistants specialized in one topic.

Good thing about Needle is that you get back answers always with references without any hallucination, so that you can trust the results.

Core concepts​

As mentioned, the most common actions you need to take are 1) upload data and 2) search. See the information below for the terminology and concepts you need to know.

Collections​

A collection is a group of files that you put together and can perform a search on. A lot of action is happening around the collections in Needle. You can upload files to a collection, list the files' status in a collection, perform search, etc.

Files​

A file is in-fact self-explanatory. Needle currently expects files that contain unstructured text data. Currently supported formats are PDF, DOCX, HTML, TXT. We are constantly working on adding more formats.

Indexing​

When a file is added to a collection, it automatically goes through the indexing process. Once the indexing is done, the file is ready to be searched. This may take several minutes depending on the size of the file. You can see and track the status of the files in the collection, either in the web UI or by querying from the REST API.

Chunks​

A chunk is a part of the file. When a file is indexed, it is split into chunks. When you perform a search, Needle finds relevant chunks for your prompt and returns them as search results.

Connectors​

We know how painful it is to upload files manually and keep them up-to-date if the data you work with is changing. Because of this, Needle provides builtin connectors to automate this process. Currently supported connectors are Google Drive and Web Connector. We are working on adding more connectors to make your life easier, you can soon expect to see connectors for Dropbox, OneDrive and more.

Connectors read and synchronize data from the source to Needle on a regular basis. You can create connectors in the web UI - and in most cases, you will need to configure a connector only once.

Usage Limits​

In order to ensure fair usage of our services, we implemented some limits. All functionality is rate limited to prevent abuse. Furthermore we have a limits that apply to free tier of Needle. You can find the details in the pricing page.

Next Steps​

Now that we have covered the core concepts, you are ready to start using Needle. In the next section, we will guide you through how to enhance SQL documentation using Needle, so that you, as a data analyst, can find answers to your SQL questions instantly.