Ingesting with Metadata

This guide covers how to upload multiple documents to Activeloop and interact with your data through the Chat Completions API.

Directory Structure

Organize your documents clearly by category and author:

philosophy/
├── hegel/
│ ├── 1.pdf
│ ├── 2.pdf
│ └── 3.pdf
└── kant/
    ├── 1.pdf
    ├── 2.pdf
    └── 3.pdf

mathematics/
├── euclid/
│ ├── 1.pdf
│ ├── 2.pdf
│ └── 3.pdf
└── pythagoras/
    ├── 1.pdf
    ├── 2.pdf
    └── 3.pdf

Uploading Data

This section handles batch uploading PDF files from philosophy and mathematics categories providing extra metadata.

import os
import json
import requests

url = 'https://api.activeloop.ai/files'
token = os.getenv("ACTIVELOOP_TOKEN")
headers = {'Authorization': f'Bearer {token}'}

for category in ['philosophy', 'mathematics']:
    for author in os.listdir(category):
        files = [('file', (pdf, open(f'{category}/{author}/{pdf}', 'rb'))) for pdf in os.listdir(f'{category}/{author}')]
        
        # specify any extra metadata 
        metadata = { 
            'category': category,
            'author': author
        }
        data = {'metadata': json.dumps(metadata)}
        
        # upload files 
        response = requests.post(url, headers=headers, files=files, data=data)
        print(response.status_code)

You can learn more on supported document type by going over Modalities.

Talking with the Data

Query your data directly using Activeloop's Chat API:

payload = {
    "model": "activeloop-l0",
    "messages": [
        {
            "role": "user",
            "content": "What is logic?", 
        }   
    ]
}
        
response = requests.post(url, headers=headers, json=payload)

# Print the response
print(response.json())

Next, you'll learn how to filter files before querying your data.