Build a Junior Data Engineer

• Write python scripts for processing data, creating charts
• Analyze CSV, Parquet or JSON data using DuckDb.
• Write complex queries including joins and filtering.
• Run data transformations and export the results.
• Explain analysis step-by-step.

​

Setup

python3 -m venv ~/.venvs/aienv
source ~/.venvs/aienv/bin/activate

python3 -m venv aienv
aienv/scripts/activate

pip install -U "phidata[aws]"

pip install -U "phidata[aws]"

export OPENAI_API_KEY=sk-***

setx OPENAI_API_KEY sk-***

​

Create your codebase

phi ws create -t junior-de -n junior-de

phi ws create -t junior-de -n junior-de

junior-de                     # root directory for your junior-de
├── ai                      # directory for AI components
    ├── duckgpt             # directory for the DuckDb DE
    ├── pygpt               # directory for the Python DE
    ├── agents          # AI agents
    ├── knowledge_base.py   # agent knowledge base
    └── storage.py          # agent storage
├── app                     # directory for Streamlit apps
├── db                      # directory for database components
├── notebooks               # directory for Jupyter notebooks
├── Dockerfile              # Dockerfile for the application
├── pyproject.toml          # python project definition
├── requirements.txt        # python dependencies managed by pyproject.toml
├── scripts                 # directory for helper scripts
├── utils                   # directory for utilities
└── workspace               # phidata workspace directory
    ├── dev_resources.py    # dev resources running locally
    ├── prd_resources.py    # production resources running on AWS
    ├── jupyter             # jupyter notebook resources
    ├── secrets             # directory for storing secrets
    └── settings.py         # phidata workspace settings

​

Set OpenAI Key

export OPENAI_API_KEY=sk-***

setx OPENAI_API_KEY sk-***

​

Run Junior DE locally

phi ws up

phi ws up dev:docker

​

DuckGPT: Automate Data Analysis using DuckDb

• Open localhost:8501 to access your Junior DE.
• Your first Junior DE is DuckGPT that can write and run SQL queries using DuckDb.
• Click on DuckGPT and enter a username.
• Message “Show me revenue over time”
• See your Junior DE work through the problem.
• Message “Save it” to save the query to the ai/duckgpt/scratch folder.

​

Add your data

• You can add csv, json or parquet files stored locally or on s3.
• You can also add txt files to provide more information to the Agent.
• Click the Update Knowledge Base to load the knowledge base.

​

How DuckGPT works

​

PyGPT: Automate Data Analysis using Python

• Your next Junior DE is PyGPT that can write python scripts for processing data, create charts and more. Click on PyGPT.
• Message “Show me a chart of revenue per year”
• Each script that PyGPT creates and runs is saved to the ai/pygpt/scratch folder for reference
• See your Junior DE work through the problem.

​

Add your data

• You can add csv, json or parquet files stored locally or on s3.
• You can also add txt files to provide more information to the Agent.
• Click the Update Knowledge Base to load the knowledge base.

​

How PyGPT works

​

Optional: Run Jupyterlab

1

Enable Jupyter

Update the workspace/settings.py file and set dev_jupyter_enabled=True

workspace/settings.py

...
ws_settings = WorkspaceSettings(
    ...
    # Uncomment the following line
    dev_jupyter_enabled=True,
...

2

Start Jupyter

terminal

shorthand

phi ws up --group jupyter

Press Enter to confirm and give a few minutes for the image to download (only the first time). Verify container status and view logs on the docker dashboard.

3

View JupyterLab UI

• Open localhost:8888 to view the Jupyterlab UI. Password: admin
• Open notebooks/duckgpt to play with DuckGPT.

​

Delete local resources

phi ws down

phi ws down --env dev --infra docker

phi ws down dev:docker

phi ws down --group app

phi ws down --group api

phi ws down --group db

phi ws down --group jupyter

​

Upcoming Upgrades

• Add Snowflake Agent.
• Ask questions via slack
• Add Airflow Agent.
• Add ability to write and test data pipelines.

​

Next

• Run your Junior DE on AWS
• Read how to update workspace settings
• Read how to create a git repository for your workspace
• Read how to manage the development application
• Read how to format and validate your code
• Read how to add python libraries
• Chat with us on discord