Dataset creation at scale

Using LLMs in bulk to enrich datasets is still underrated. With structured outputs, you can add useful columns at scale to describe data better, analyze faster, and speed up feature engineering.

Pattern: one row in, one structured object out. Each row is an input to an LLM call, and the output matches a schema you define (JSON). My go‑to library for this is instructor.

Quality depends on three things:

• Input + prompt: Vague prompts yield inconsistent outputs; over‑specified prompts overfit and break on high‑variance data.
• Schema design: Attribute names matter. Make them clear and informative to guide extraction.
• Model reasoning: Stateless runs across tens of thousands of rows introduce drift; expect some inconsistency.

Practical trick: run the job twice with two different models and keep rows where both agree. It’s a simple agreement‑based filter that lifts reliability without heavy post‑processing.