Software Architect

MyCloset is a fashion startup building an AI-driven outfit recommender. They needed an architecture that could collect product data from across the web, make it searchable by similarity, and serve personalized outfit recommendations to users — but had no existing infrastructure to build on. I came in as the architect: designed the system end-to-end, specified how each stage of the pipeline should work, and built the AI pipelines myself. The goal was a layered architecture where scraping, embedding, and serving were decoupled enough to evolve independently.

System Architecture

MyCloset had no existing infrastructure when I joined — just an idea and an ambition to recommend outfits using AI. The first question was structural: where does the data come from, where does it live, and how does it get to the user? Without a clear answer, every later decision gets tangled with the ones before it.

I designed the system as three independent layers: a scraper that collects product data from the web, an embedding pipeline that turns products and user preferences into searchable vectors, and a serving layer that takes a user's request and returns ranked outfit recommendations. Each layer has its own data store and its own deployment lifecycle. The scraper can be reworked — new sites added, old ones retired — without touching the recommender. The embedding model can be swapped without re-scraping. The serving layer can change its recommendation strategy without invalidating the data underneath.

This separation wasn't theoretical. It let the team iterate on the scraping strategy during my engagement without me re-engineering the embedding or serving stages. New product attributes could be added to the data model without breaking the recommender. The architecture is what makes that possible.

Each layer exposes a clean contract: a schema of what it accepts and what it produces. Adding a new layer later — say, a feedback loop where user picks retrain the embeddings — becomes an integration problem, not a redesign problem.

Scraping Pipeline

Web scraping for fashion is not a solved problem. Sites are visually rich, often JavaScript-rendered, frequently change their markup, and actively try to detect and block automated access. A scraper that works against one site will break against another, and a uniform approach fails on both.

I built the scraper in Python as a configurable set of per-site adapters rather than a single general-purpose crawler. Each adapter knows the structure of one site: how to enumerate product listings, how to extract attributes (title, image, price, category, sizes), and how to handle that site's specific anti-scraping measures. Common patterns — rate limiting, retries with exponential backoff, request rotation — are shared infrastructure that every adapter inherits.

The output is normalized into a single product schema regardless of source site. A dress from Site A and a dress from Site B end up with the same shape in the database: same fields, same types, same downstream treatment. This means the embedding pipeline never has to know which site a product came from.

The pipeline is idempotent and re-runnable. Running the scraper a second time updates rather than duplicates. New sites are added by writing a new adapter and registering it; the rest of the system is untouched. The same pattern extends to data sources beyond fashion sites — the abstraction isn't tied to any one category.

Embedding & Recommendation

The recommendation problem at MyCloset is two-sided: the user wants outfits that match their preferences, and the team wants those recommendations grounded in real, available products rather than invented suggestions. Both halves need to work together.

I built the embedding pipeline to ingest scraped products and turn each one into a vector representation using OpenAI embeddings. The vectors capture semantic properties — style, occasion, color family, formality — that a simple text search would miss. Vectors are stored in pgvector, Postgres's vector extension, so the system uses one database for both relational product data and similarity search. There's no second system to operate.

The recommender takes a user profile and finds candidate products by similarity, then ranks them using an OpenAI-powered reasoning pass over the retrieved set. The reasoning layer is what turns 'products similar to what the user liked' into 'outfits that make sense together'. It grounds the LLM in real catalog data so suggestions stay plausible.

The pipeline is end-to-end observable. Every stage logs what it did and why — which products were embedded, which embeddings were retrieved, which candidates the reasoning pass saw. When the recommendation quality drifts, the team can trace it back through the pipeline. The architecture I delivered made this debuggability a first-class concern, not an afterthought.