LLM OS

The LLM OS proposes that LLMs are the CPU/Kernal of an emerging operating system and can solve problems by coordination multiple resources. Andrej Karpathy talks about it in this tweet, this tweet and this video. Here’s a video of me building the LLM OS. Using this template, we can run the llm-os locally using docker and in production on AWS.

Setup

Create a virtual environment

python3 -m venv ~/.venvs/aienv
source ~/.venvs/aienv/bin/activate

python3 -m venv aienv
aienv/scripts/activate

Install phidata

pip install -U "phidata[aws]"

pip install -U "phidata[aws]"

Install docker

Install docker desktop to run your app locally

Export your OpenAI key

export OPENAI_API_KEY=sk-***

setx OPENAI_API_KEY sk-***

You can get an API key from here.

Create your codebase

Create your codebase using the llm-os template

phi ws create -t llm-os -n llm-os

phi ws create -t llm-os -n llm-os

This will create a folder llm-os with the following structure:

llm-os                      # root directory of your llm-os
├── ai
    ├── agents.py       # AI Agents
├── app                     # Streamlit app
├── api                     # FastApi routes
├── db                      # database settings
├── Dockerfile              # Dockerfile for the application
├── pyproject.toml          # python project definition
├── requirements.txt        # python dependencies generated using pyproject.toml
├── scripts                 # helper scripts
├── utils                   # shared utilities
└── workspace               # phidata workspace directory
    ├── dev_resources.py    # dev resources running locally
    ├── prd_resources.py    # production resources running on AWS
    ├── secrets             # storing secrets
    └── settings.py         # phidata workspace settings

Set Credentials

We use gpt-4o as the LLM, so export your OPENAI_API_KEY. You can get one from OpenAI if needed.

export OPENAI_API_KEY=sk-***

setx OPENAI_API_KEY sk-***

If you’d like to use the research agent, export your EXA_API_KEY. You can get one from here if needed.

export EXA_API_KEY=***

setx EXA_API_KEY ***

Run LLM OS

We’ll build a simple front-end for the LLM OS using streamlit. Start the app group using:

phi ws up --group app

phi ws up dev:docker:app

Press Enter to confirm and give a few minutes for the image to download (only the first time). Verify container status and view logs on the docker dashboard.

Open localhost:8501 to view your LLM OS.
Enter a username.
Add blog post to knowledge: https://blog.samaltman.com/what-i-wish-someone-had-told-me and ask: what did sam altman wish he knew?
Test Web search: Whats happening in france?
Test Calculator: What is 10!
Test Finance: What is the price of AAPL?
Test Finance: Write a comparison between nvidia and amd, use all finance tools available and summarize the key points
Test Research: Write a report on Hashicorp IBM acquisition

Optional: Serve your LLM OS as an API

Streamlit is great for building micro front-ends but any production application will be built using a front-end framework like next.js backed by a RestApi built using FastApi. Your LLM OS comes with ready-to-use FastApi endpoints.

Enable FastApi

Update the workspace/settings.py file and set dev_api_enabled=True

workspace/settings.py

...
ws_settings = WorkspaceSettings(
    ...
    # Uncomment the following line
    dev_api_enabled=True,
...

Start FastApi

phi ws up --group api

phi ws up dev:docker:api

Press Enter to confirm

View API Endpoints

Open localhost:8000/docs to view the API Endpoints.
Test the v1/assitants/chat endpoint with

{
  "message": "Whats 10!",
  "agent": "LLM_OS"
}

Build an AI Product using the LLM OS

Your llm-os comes with pre-configured API endpoints that can be used to build your AI product. The general workflow is:

Call the /assitants/create endpoint to create a new run for a user.

{
  "user_id": "my-app-user-1",
  "agent": "LLM_OS"
}

The response contains a run_id that can be used to build a chat interface by calling the /assitants/chat endpoint.

{
  "message": "whats 10!",
  "stream": true,
  "run_id": "372224c0-cd5e-4e87-a29d-65d33d9353a5",
  "user_id": "my-app-user-1",
  "agent": "LLM_OS"
}

These routes are defined the api/routes folder and can be customized to your use case.

Message us on discord if you need help.

Delete local resources

Play around and stop the workspace using:

phi ws down

phi ws down --env dev --infra docker

phi ws down dev:docker

or stop individual Apps using:

phi ws down --group app

phi ws down --group api

phi ws down --group db

Congratulations on running your LLM OS locally. Next Steps:

Run your LLM OS on AWS
Read how to update workspace settings
Read how to create a git repository for your workspace
Read how to manage the development application
Read how to format and validate your code
Read how to add python libraries
Chat with us on discord

Templates

How to

Workspace

Apps

Resources

LLM OS

Setup

Create your codebase

Set Credentials

Run LLM OS

LLM OS

Optional: Serve your LLM OS as an API

Build an AI Product using the LLM OS

Delete local resources

Next

​Setup

​Create your codebase

​Set Credentials

​Run LLM OS

​LLM OS

​Optional: Serve your LLM OS as an API

​Build an AI Product using the LLM OS

​Delete local resources

​Next

Setup

Create your codebase

Set Credentials

Run LLM OS

LLM OS

Optional: Serve your LLM OS as an API

Build an AI Product using the LLM OS

Delete local resources

Next