Software Development With Devin: Security, Deployment, Maintenance (Part 4)

Welcome back! We’re almost there: Slack is chirping, Jira has tickets, tests are green, and our CI blocks every sloppy PR. Part four is the final push where we’ll be turning the fp-ts playground into something we could, in theory, point the public at.

You can access all the tutorials in the Devin series here:

1. Setup and First Pull Request (Part 1)
2. Shipping a Vertical Slice with Devin (Part 2)
3. Integrations, Testing, and CI/CD (Part 3) 
4. Security, Deployment, Maintenance (Part 4)

Here’s the plan for this fourth tutorial:

• Auth: No more anonymous UUIDs; we’ll drop in NextAuth credentials, JWT cookies, and a GqlAuthGuard so progress is tied to real users.
• Eyes on production: One line of Sentry code in the web app, one in the Nest API, and every exception lands in the dashboard.
• Push-button deploy: The front-end goes live on Vercel with preview URLs on every PR. (The API will stay local for now, we’ll fake prod with Docker.)

We will also reflect on this pair programming with Devin and see where it shines and where a human still has to tidy up loose ends before the world sees the repo. Ready for the last stretch?

From Anonymous UUID to Real Users

What I asked Devin to do

In short, I asked Devin to rip out the anonymous UUID flow and give me a real auth stack:

• NextAuth v6 credentials provider in apps/web
• NestJS AuthModule with Argon2 hashing and JWT cookies
• Prisma migration that adds a User table and changes Progress.sessionId → userId (non-null)
• Copy any existing UUID progress rows to a placeholder user so learners keep their badges
• Do not switch Postgres for SQLite

What came back (mostly good)

Here’s a summary of the results:

Another SQLite migration

Guess what I saw pop again in Devin’s little thinking box.

Successfully migrated from PostgreSQL to SQLite for local development 🥳

Exactly what I’d told it not to do. The Prisma schema was rewritten for SQLite, and connection strings and types were changed (from TIMESTAMP to DATETIME). I caught it mid-session:

1. Paused the session.
2. git checkout main apps/api/prisma/schema.prisma (restore Postgres schema).
3. Deleted the new dev.db file.
4. npm run prisma:migrate --workspace=apps/api to re-apply Postgres migrations.
5. Resumed Devin; it detected the revert and continued without complaint.

Damage: 0.8 ACU and about ten minutes.

Breaking-change checklist for teammates

I really appreciate the instructions Devin puts in PRs. Here it told us to:

# 1  Install new deps
npm install --workspaces

# 2  Run the Postgres migration
npm run prisma:migrate --workspace=apps/api

# 3  Restart everything
npm run dev:api   &   npm run dev:web

Architecture snapshot

In the wiki, I found this helpful diagram of our new auth system:

With real users in place and the database firmly back on Postgres, we’re finally ready to track genuine learner progress, ship preview builds to Vercel, and catch runtime errors with Sentry.

Front-End on Vercel, Preview Links on Every PR

With secure log-ins in place, the next obvious milestone was to show the world something clickable. We decided to deploy only the Next.js front-end for now. API and Postgres will remain on my local machine until we pick a host that fits the budget.

Manual account hookup (0 ACUs)

Connecting Devin to my personal Vercel account via native integrations isn’t possible yet. So I did it manually, via the Vercel dashboard:

1. Logged into Vercel and clicked “New Project → Import Git Repository”.
2. Selected the fp-ts-exercises repo, left the build command as npm run build --workspace apps/web, and hit Deploy.
3. Copied the resulting VERCEL_PROJECT_ID and VERCEL_TOKEN into GitHub Secrets.

Let Devin update the CI workflow (0.4 ACU)

Following my prompt, Devin added a single job to the bottom of the existing workflow and a tiny shell step that curls the generated URL to Slack:

- name: Deploy to Vercel
  if: ${{ success() && github.event_name == 'pull_request' }}
  run: npx vercel deploy --prod --token ${{ secrets.VERCEL_TOKEN }}
  env:
    VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
    VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}

Result

Push a branch → CI green → Slack ping:

✅ Preview ready: https://fp-ts-exercises-git-my-branch.vercel.app

Opening the link showed our playground, full auth flow, and learner dashboard. No environment tweaking required on Vercel’s end yet, because the front-end talks to my local API over http://localhost:4000.

A Closer Look at Devin’s Secrets System

Before integration with Sentry, I had a look into Devin’s secrets system to figure out if I should use it. Devin offers a built-in “Secrets” vault so you can give the agent API keys, DB URLs, or OAuth tokens without checking them into Git and without exposing them in chat. Think of it as a lightweight equivalent to GitHub Actions secrets or Vercel’s Environment Variables, but scoped to Devin’s own cloud workspaces.

Adding a secret

Here are the steps you need to follow to add a secret:

1. Open Team → Secrets in the Devin dashboard.
2. Click “Add Secret” and enter a name (SENTRY_DSN_WEB) and its value.
3. Devin confirms storage; the key is now available as $.SENTRY_DSN_WEB.

Using a secret in a task

Task: update API Sentry init; Set DSN to $SECRET.SENTRY_DSN_API

Devin substitutes the value when it writes main.ts. In the shell tab, you’ll see the literal DSN, but the chat log masks it.

Pros and cons

Pros:

• No risk of leaking keys into PRs or Timelapse logs.
• Organization-scoped secrets work across sessions: one upload, many uses.
• Zero credit cost to store or fetch.

Cons:

• Keys aren’t available locally unless you copy them into .env. Test runs outside Devin need manual setup.
• No per-environment scoping (e.g., “only in production”). All sessions get all secrets.

When to use it

I find it easier to use Devin’s vault when Devin itself needs the key (e.g., pushing to Vercel, hitting a private registry, or wiring Sentry during build). I would keep using .env / GitHub Actions secrets for everything that also runs in your local dev shell or CI runners.

Devin Integration With Sentry

Here’s the prompt I used: Add Sentry to both apps so every exception shows up in my dashboard. Use environment variables in .env. No other behaviour changes.”

Devin’s work (0.6 ACU, one session)

Here’s a summary of the result:

My manual steps

Here are the steps I took:

1. Copied DSNs from my Sentry project into .env.
2. Restarted both dev servers.
3. Hit http://localhost:4000/graphql with a deliberately bad query to check it worked. An issue appeared in Sentry within seconds.

Why I skipped Devin’s secrets feature

This is why I skipped Devin’s secrets feature:

• Simpler mental model: same .env file everywhere.
• Easier to run preview builds on Vercel (those env vars are already set there).
• No ACU burn for secret-store operations.

Reflections: Where Devin Shines and Where It Stumbles

Four tutorial parts ago, this repo was a dusty pile of fp-ts exercises—now, it has:

• Modernised codebase: Latest dependencies, strict TS config, lint & Prettier rules (Part 1).
• Browser-based playground: Next.js 14 + Sandpack editor with live Vitest feedback (Part 2).
• NestJS + Postgres backend: GraphQL API, Prisma migrations, anonymous → JWT progress tracking (Parts 2 & 4).
• Green-lit pipeline: CI workflow that lints, type-checks, runs Jest & Playwright, and blocks every failing PR (Part 3).
• Team integrations: Slack status pings, Jira-label analysis, and preview URLs on every branch (Part 3).
• Production observability: Sentry wired into both web and API, capturing every exception (Part 4).
• Preview deploys: Vercel spins up a shareable URL for each pull request (Part 4).

We spent about ≈ 70 ACUs total (~$157 on the Core plan) and about two working days of hands-on review.

What Devin does frighteningly well

Here’s where I think Devin is very good:

Where a human still beats the agent

This is where I think Devin still needs improvement:

My opinion

Devin’s own docs never promised a product builder. They outline a much narrower sweet spot, and in hindsight, I should have stayed inside it. The official “What are Devin’s strengths?” page lists four headline uses:

The docs call feature development that spans UI, API, and data flows “experimental”. They are described as large, open-ended tasks where the agent may “require significant human review and iteration.” This is exactly what we saw when Sandpack hard-coded tests or when the dashboard looked good but didn’t actually compute data.

So, as a rule of thumb:

Use Devin for anything a senior developer could automate with a script, and keep the steering wheel for tasks that need product sense.

Conclusion

We reached the end of our four-part tutorial:

1. Setup and First Pull Request (Part 1)
2. Shipping a Vertical Slice with Devin (Part 2) 
3. Integrations, Testing, and CI/CD (Part 3) 
4. Security, Deployment, Maintenance (Part 4)

Here are a few next steps we can take from here:

1. Host the API so preview builds (and future users) hit a real backend.
2. Make sure core functionality works and fix the 30 bugs I know are in there
3. Polish the UI
4. Add more content and exercises
5. Release for people to use!

If you follow the same path, remember: you can let the agent lay the concrete, but you still have to design the house. Happy coding!