Let’s run the LLM OS in production on AWS.

​
AWS Setup

1

Update Credentials

To run on AWS, you need one of the following:

  1. The ~/.aws/credentials file with your AWS credentials
  2. or AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY environment variables

To create the credentials file, install the aws cli and run aws configure

2

Update region and subnets

Add 2 subnets to the workspace/settings.py file (required for ECS services)

workspace/settings.py
ws_settings = WorkspaceSettings(
    ...
    # -*- AWS settings
    # Add your Subnet IDs here
    subnet_ids=["subnet-xyz", "subnet-xyz"],
    ...
)

Please check that the subnets belong to the selected aws_region

​
Update Secrets

1

Streamlit App Password

Update the streamlit app password in workspace/secrets/prd_app_secrets.yml

workspace/secrets/prd_app_secrets.yml
APP_PASSWORD: "admin"
# OPENAI_API_KEY: "sk-***"
2

RDS database password

Update the RDS database password in workspace/secrets/prd_db_secrets.yml

workspace/secrets/prd_db_secrets.yml
# Secrets used by prd RDS database
MASTER_USERNAME: ai
MASTER_USER_PASSWORD: "ai9999!!"

​
Create AWS resources

Create AWS resources using:

This will create:

  1. ECS Cluster for the application.
  2. ECS Task Definitions and Services that run the application on the ECS cluster.
  3. LoadBalancer to route traffic to the application.
  4. Security Groups that control incoming and outgoing traffic.
  5. Secrets for managing application and database secrets.
  6. RDS Database for Knowledge Base and Storage.

Press Enter to confirm and grab a cup of coffee while the resources spin up.

  • The RDS database takes about 5 minutes to activate.
  • These resources are defined in the workspace/prd_resources.py file.
  • Use the ECS console to view services and logs.
  • Use the RDS console to view the database instance.

​
LLM OS App

Open the LoadBalancer DNS provided when creating the Streamlit App

  • Enter the APP_PASSWORD from the prd_app_secrets.yml file (default: admin)
  • Enter a username.
  • Add blog post to knowledge: https://blog.samaltman.com/what-i-wish-someone-had-told-me and ask: what did sam altman wish he knew?
  • Test Web search: Whats happening in france?
  • Test Calculator: What is 10!
  • Test Finance: What is the price of AAPL?
  • Test Finance: Write a comparison between nvidia and amd, use all finance tools available and summarize the key points
  • Test Research: Write a report on Hashicorp IBM acquisition

​
LLM OS API

1

Enable FastApi

Update the workspace/settings.py file and set prd_api_enabled=True

workspace/settings.py
...
ws_settings = WorkspaceSettings(
    ...
    # Uncomment the following line
    prd_api_enabled=True,
...
2

Create FastApi resources

Press Enter to confirm

3

View API Endpoints

  • Open the LoadBalancer DNS + the /docs endpoint to view the API Endpoints.
  • Test the v1/assitants/chat endpoint with
{
  "message": "Whats 10!",
  "agent": "LLM_OS"
}
  • Integrate with your front-end or product.

​
Update Production

Follow this guide to update your production application. You'll need to:

  1. Create a new image
  2. Update the ECS Task Definition and Services.

​
Delete AWS resources

Play around and then delete AWS resources using:

or delete individual resource groups using:

​
Next

Congratulations on running your LLM OS on AWS. Next Steps:

Was this page helpful?