Predibase

Use Predibase’s open-source and fine-tuned LLMs through Portkey.

Quick Start

Get started with Predibase in under 2 minutes:

Python

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @predibase provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@predibase/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai'

// 1. Install: npm install portkey-ai
// 2. Add @predibase provider in model catalog
// 3. Use it:

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY"
})

const response = await portkey.chat.completions.create({
    model: "@predibase/llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL

# 1. Install: pip install openai portkey-ai
# 2. Add @predibase provider in model catalog
# 3. Use it:

client = OpenAI(
    api_key="PORTKEY_API_KEY",  # Portkey API key
    base_url=PORTKEY_GATEWAY_URL
)

response = client.chat.completions.create(
    model="@predibase/llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

import OpenAI from "openai"
import { PORTKEY_GATEWAY_URL } from "portkey-ai"

// 1. Install: npm install openai portkey-ai
// 2. Add @predibase provider in model catalog
// 3. Use it:

const client = new OpenAI({
    apiKey: "PORTKEY_API_KEY",  // Portkey API key
    baseURL: PORTKEY_GATEWAY_URL
})

const response = await client.chat.completions.create({
    model: "@predibase/llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }]
})

console.log(response.choices[0].message.content)

# 1. Add @predibase provider in model catalog
# 2. Use it:

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -d '{
    "model": "@predibase/llama-3-8b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Add Provider in Model Catalog

Before making requests, add Predibase to your Model Catalog:

1. Go to Model Catalog → Add Provider
2. Select Predibase
3. Enter your Predibase API key
4. Name your provider (e.g., predibase)

Complete Setup Guide

See all setup options and detailed configuration instructions

---

Predibase Capabilities

Serverless Endpoints

Predibase offers LLMs like Llama 3, Mistral, Gemma, etc. on its serverless infrastructure that you can query instantly.

Sending Predibase Tenant ID

Predibase expects your account tenant ID along with the API key in each request. With Portkey, you can send your Tenant ID with the user param while making your request.

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    user="PREDIBASE_TENANT_ID"  # Required: Your Predibase tenant ID
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello!" }],
    user: "PREDIBASE_TENANT_ID"  // Required: Your Predibase tenant ID
});

console.log(response.choices[0].message.content);

Using Fine-Tuned Models

Predibase allows you to deploy and use fine-tuned models with adapters. Use the special format with your model identifier from your Predibase dashboard:

Fine-Tuned Model Format:
model = base_model:adapter-repo-name/adapter-version-number

For example: llama-3-8b:sentiment-analysis/1

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b:sentiment-analysis/1",  # Base model + adapter
    messages=[{"role": "user", "content": "This product is amazing!"}],
    user="PREDIBASE_TENANT_ID"
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b:sentiment-analysis/1",  // Base model + adapter
    messages: [{ role: "user", content: "This product is amazing!" }],
    user: "PREDIBASE_TENANT_ID"
});

console.log(response.choices[0].message.content);

Dedicated Deployments

Route requests to your dedicated deployed models by passing the deployment name in the model parameter:

Python

from portkey_ai import Portkey

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="my-dedicated-mistral-deployment",  # Your deployment name
    messages=[{"role": "user", "content": "Hello!"}],
    user="PREDIBASE_TENANT_ID"
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "my-dedicated-mistral-deployment",  // Your deployment name
    messages: [{ role: "user", content: "Hello!" }],
    user: "PREDIBASE_TENANT_ID"
});

console.log(response.choices[0].message.content);

JSON Schema Mode

Enforce JSON schema for all Predibase models by setting response_format to json_object with your schema:

Python

from portkey_ai import Portkey
from pydantic import BaseModel, constr

# Define JSON Schema with Pydantic
class Character(BaseModel):
    name: constr(max_length=10)
    age: int
    strength: int

portkey = Portkey(api_key="PORTKEY_API_KEY", provider="@predibase")

response = portkey.chat.completions.create(
    model="llama-3-8b",
    messages=[{"role": "user", "content": "Create a character profile"}],
    user="PREDIBASE_TENANT_ID",
    response_format={
        "type": "json_object",
        "schema": Character.schema()
    }
)

print(response.choices[0].message.content)

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: 'PORTKEY_API_KEY',
    provider: '@predibase'
});

const response = await portkey.chat.completions.create({
    model: "llama-3-8b",
    messages: [{ role: "user", content: "Create a character profile" }],
    user: "PREDIBASE_TENANT_ID",
    response_format: {
        type: "json_object",
        schema: {
            properties: {
                name: { maxLength: 10, title: "Name", type: "string" },
                age: { title: "Age", type: "integer" },
                strength: { title: "Strength", type: "integer" }
            },
            required: ["name", "age", "strength"],
            title: "Character",
            type: "object"
        }
    }
});

console.log(response.choices[0].message.content);

---

Supported Models

Predibase provides access to various open-source and fine-tuned models:

• Llama 3 (various sizes)
• Mistral
• Zephyr
• Your custom fine-tuned models

Check Predibase’s documentation for the complete model list and fine-tuning options.

---

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Predibase requests

Prompt Library

Manage and version your prompts

Fine-Tuning Logs

Track performance of fine-tuned models

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation