mrvitej.in

Archive · DevOps

On-Prem Platform, Part 2: Angular, Node.js, Presigned URLs, CDN, SMTP and Notifications

Deploy the application layer on k3s: an Angular UI with runtime config, a Node.js API, direct-to-S3 uploads with presigned URLs, CDN delivery, email over SMTP and Kafka-driven notifications.

11 min read

  • Angular
  • Node.js 24 LTS
  • AWS SDK for JavaScript v3
  • LocalStack
  • Amazon S3
  • CloudFront
  • Mailpit
  • Apache Kafka 4
  • k3s

Part 1 built the platform: WSL2 Ubuntu, Docker, k3s and the data stores. Part 2 deploys the application on it and covers the features that are hardest to test without a cloud account: file uploads to S3, CDN delivery, email and notifications.

The same code runs locally and in AWS. Only configuration changes: the S3 endpoint, the SMTP host and the URLs.

Architecture of the application layer

flowchart LR
  U[Browser] --> T[Traefik ingress<br/>app.local.test]
  T --> UI[Angular UI<br/>nginx]
  T --> API[Node.js API]
  API --> MY[MySQL]
  API --> MG[MongoDB]
  API --> RD[Redis]
  API -- signs URL --> S3[S3 or LocalStack]
  U -- PUT file with presigned URL --> S3
  U -- GET via CDN --> CDN[CloudFront or nginx cache]
  CDN --> S3
  API -- publish event --> K[Kafka topic notifications]
  K --> W[Notification worker]
  W --> SMTP[Mailpit locally<br/>SES or relay in cloud]
  W --> IN[In-app notifications<br/>MongoDB]

1. LocalStack and Mailpit as cloud stand-ins

# 20-platform.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: localstack, namespace: platform }
spec:
  replicas: 1
  selector: { matchLabels: { app: localstack } }
  template:
    metadata: { labels: { app: localstack } }
    spec:
      containers:
        - name: localstack
          image: localstack/localstack:4
          env:
            - { name: SERVICES, value: "s3" }
            - { name: PERSISTENCE, value: "1" }
          ports: [{ containerPort: 4566 }]
          volumeMounts: [{ name: data, mountPath: /var/lib/localstack }]
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: localstack-data }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: localstack-data, namespace: platform }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 5Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: localstack, namespace: platform }
spec:
  selector: { app: localstack }
  ports: [{ port: 4566 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: mailpit, namespace: platform }
spec:
  replicas: 1
  selector: { matchLabels: { app: mailpit } }
  template:
    metadata: { labels: { app: mailpit } }
    spec:
      containers:
        - name: mailpit
          image: axllent/mailpit:latest
          ports:
            - { name: smtp, containerPort: 1025 }
            - { name: web, containerPort: 8025 }
---
apiVersion: v1
kind: Service
metadata: { name: mailpit, namespace: platform }
spec:
  selector: { app: mailpit }
  ports:
    - { name: smtp, port: 1025 }
    - { name: web, port: 8025 }

Mailpit accepts every email and shows it in a web inbox, so nothing reaches real people during testing. Pin its tag once you have a working version.

Create the bucket with CORS, which browsers need for direct uploads:

kubectl -n platform port-forward svc/localstack 4566:4566 &
export AWS_ENDPOINT_URL=http://localhost:4566 AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=ap-south-1
aws s3 mb s3://shop-uploads
aws s3api put-bucket-cors --bucket shop-uploads --cors-configuration '{
  "CORSRules": [{
    "AllowedOrigins": ["https://app.local.test"],
    "AllowedMethods": ["PUT", "GET"],
    "AllowedHeaders": ["Content-Type"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }]
}'

2. Uploads with presigned URLs

Sending large files through the API wastes memory and bandwidth and ties up workers. Instead, the API authorises the upload and the browser sends the file straight to S3.

sequenceDiagram
  participant B as Browser
  participant A as Node.js API
  participant S as S3
  participant K as Kafka
  B->>A: POST upload-url with name, type, size
  A->>A: check user, type and size limits
  A->>B: presigned PUT URL valid 5 minutes plus object key
  B->>S: PUT file using the URL
  S->>B: 200 OK
  B->>A: POST uploads complete with key
  A->>S: HEAD object to confirm size and type
  A->>K: publish file uploaded event

The API

// api/src/storage.js
import { S3Client, PutObjectCommand, GetObjectCommand, HeadObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'node:crypto';

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  // set only locally, e.g. http://localstack.platform.svc:4566
  ...(process.env.S3_ENDPOINT && { endpoint: process.env.S3_ENDPOINT, forcePathStyle: true }),
});
const BUCKET = process.env.UPLOAD_BUCKET;
const ALLOWED = new Set(['image/png', 'image/jpeg', 'application/pdf']);
const MAX_BYTES = 10 * 1024 * 1024;

export async function createUploadUrl(userId, { contentType, size }) {
  if (!ALLOWED.has(contentType)) throw Object.assign(new Error('type not allowed'), { status: 400 });
  if (size > MAX_BYTES) throw Object.assign(new Error('file too large'), { status: 400 });

  const key = `users/${userId}/${randomUUID()}`;          // never trust the client's file name
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({ Bucket: BUCKET, Key: key, ContentType: contentType }),
    { expiresIn: 300 },
  );
  return { key, url };
}

export async function confirmUpload(userId, key) {
  if (!key.startsWith(`users/${userId}/`)) throw Object.assign(new Error('forbidden'), { status: 403 });
  const head = await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));
  if (head.ContentLength > MAX_BYTES) throw Object.assign(new Error('file too large'), { status: 400 });
  return { size: head.ContentLength, type: head.ContentType };
}

export function createDownloadUrl(key) {
  return getSignedUrl(s3, new GetObjectCommand({ Bucket: BUCKET, Key: key }), { expiresIn: 600 });
}
// api/src/routes/uploads.js (Express)
router.post('/uploads/url', auth, async (req, res, next) => {
  try { res.json(await createUploadUrl(req.user.id, req.body)); } catch (e) { next(e); }
});

router.post('/uploads/complete', auth, async (req, res, next) => {
  try {
    const meta = await confirmUpload(req.user.id, req.body.key);
    await db.files.insertOne({ userId: req.user.id, key: req.body.key, ...meta, createdAt: new Date() });
    await publish('notifications', { type: 'file.uploaded', userId: req.user.id, key: req.body.key });
    res.status(201).json(meta);
  } catch (e) { next(e); }
});

Rules that keep presigned URLs safe:

  • Short expiry (minutes, not days).
  • The server generates the key; the client’s file name is stored as metadata only.
  • Content type is signed into the URL, so the browser cannot upload a different type.
  • Confirm with HeadObject after upload; a URL that was issued does not mean a file arrived.
  • Keep the bucket private. Downloads use either a presigned GET or the CDN.

The Angular side

// ui/src/app/upload.service.ts
import { Injectable, inject } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';
import { APP_CONFIG } from './app.config';

@Injectable({ providedIn: 'root' })
export class UploadService {
  private http = inject(HttpClient);
  private cfg = inject(APP_CONFIG);

  async upload(file: File): Promise<void> {
    const { key, url } = await firstValueFrom(
      this.http.post<{ key: string; url: string }>(`${this.cfg.apiUrl}/uploads/url`,
        { contentType: file.type, size: file.size }));

    // PUT directly to S3; the Content-Type must match what was signed
    await firstValueFrom(this.http.put(url, file, { headers: { 'Content-Type': file.type } }));

    await firstValueFrom(this.http.post(`${this.cfg.apiUrl}/uploads/complete`, { key }));
  }
}

If your HTTP interceptor adds an Authorization header to every request, skip it for the S3 URL. S3 rejects a request carrying both a presigned signature and a bearer token.

The UI loads config.json at start-up (see the multi-region Angular post), so the same build works locally (https://app.local.test/api) and in the cloud.

3. CDN delivery

In AWS, CloudFront sits in front of the private bucket using origin access control, so only CloudFront can read it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCloudFrontOAC",
    "Effect": "Allow",
    "Principal": { "Service": "cloudfront.amazonaws.com" },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::shop-uploads/public/*",
    "Condition": {
      "StringEquals": { "AWS:SourceArn": "arn:aws:cloudfront::111122223333:distribution/E123EXAMPLE" }
    }
  }]
}
  • Public assets (product images) go under public/ and are served by the CDN with long cache lifetimes. Use content-hashed or versioned keys, so a new file gets a new URL instead of needing an invalidation.
  • Private files (invoices, documents) are served with short-lived presigned GET URLs, or CloudFront signed URLs if they must also be cached.

Locally, an nginx cache in front of LocalStack imitates the CDN well enough to test cache headers:

proxy_cache_path /var/cache/nginx keys_zone=cdn:10m max_size=1g inactive=7d;
server {
  listen 80;
  location /public/ {
    proxy_cache cdn;
    proxy_cache_valid 200 7d;
    add_header X-Cache $upstream_cache_status;
    proxy_pass http://localstack.platform.svc:4566/shop-uploads/public/;
  }
}

4. Notifications through Kafka

The API should not send emails itself. If the SMTP server is slow, the user’s request would wait. Instead the API publishes an event and a worker does the slow work, with retries.

// worker/src/index.js
import { Kafka } from 'kafkajs';
import nodemailer from 'nodemailer';
import { MongoClient } from 'mongodb';

const kafka = new Kafka({ clientId: 'notify', brokers: process.env.KAFKA_BROKERS.split(',') });
const consumer = kafka.consumer({ groupId: 'notification-worker' });
const producer = kafka.producer();
const mail = nodemailer.createTransport({
  host: process.env.SMTP_HOST,                 // mailpit.platform.svc locally
  port: Number(process.env.SMTP_PORT),         // 1025 locally, 587 for a real relay
  secure: false,
  auth: process.env.SMTP_USER ? { user: process.env.SMTP_USER, pass: process.env.SMTP_PASS } : undefined,
});
const mongo = await new MongoClient(process.env.MONGO_URL).connect();
const inbox = mongo.db('shop').collection('notifications');

await Promise.all([consumer.connect(), producer.connect()]);
await consumer.subscribe({ topic: 'notifications' });

await consumer.run({
  eachMessage: async ({ message }) => {
    const event = JSON.parse(message.value.toString());
    const attempts = Number(message.headers?.attempts ?? 0);
    try {
      // idempotency: skip if this event was already handled
      const id = `${event.type}:${event.key ?? event.orderId}`;
      const done = await inbox.findOne({ _id: id });
      if (done) return;

      const user = await mongo.db('shop').collection('users').findOne({ _id: event.userId });
      await mail.sendMail({
        from: 'Shop <no-reply@example.com>',
        to: user.email,
        subject: 'Your file was uploaded',
        text: `We received your file ${event.key}.`,
      });
      await inbox.insertOne({ _id: id, userId: event.userId, type: event.type, read: false, createdAt: new Date() });
    } catch (err) {
      const topic = attempts >= 3 ? 'notifications-dlq' : 'notifications';
      await producer.send({ topic, messages: [{ value: message.value, headers: { attempts: String(attempts + 1) } }] });
    }
  },
});

Design points:

  • At-least-once delivery. Kafka can deliver a message twice after a rebalance, so the handler checks whether the event was already processed.
  • Retries with a limit, then a dead-letter topic that someone reviews.
  • The same record in MongoDB powers in-app notifications (a bell icon showing unread items).
  • In production, SMTP goes through Amazon SES or your own relay with SPF, DKIM and DMARC configured, or messages will land in spam.

5. Deploy the API, worker and UI

# 30-apps.yaml
apiVersion: v1
kind: ConfigMap
metadata: { name: shop-config, namespace: apps }
data:
  AWS_REGION: ap-south-1
  S3_ENDPOINT: http://localstack.platform.svc.cluster.local:4566
  UPLOAD_BUCKET: shop-uploads
  KAFKA_BROKERS: kafka-0.kafka.data.svc.cluster.local:9092
  SMTP_HOST: mailpit.platform.svc.cluster.local
  SMTP_PORT: "1025"
  REDIS_HOST: redis.data.svc.cluster.local
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop-api, namespace: apps }
spec:
  replicas: 1
  selector: { matchLabels: { app: shop-api } }
  template:
    metadata: { labels: { app: shop-api } }
    spec:
      containers:
        - name: api
          image: shop-api:dev
          imagePullPolicy: IfNotPresent
          envFrom: [{ configMapRef: { name: shop-config } }, { secretRef: { name: shop-secrets } }]
          ports: [{ containerPort: 3000 }]
          readinessProbe: { httpGet: { path: /health, port: 3000 }, periodSeconds: 5 }
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
---
apiVersion: v1
kind: Service
metadata: { name: shop-api, namespace: apps }
spec:
  selector: { app: shop-api }
  ports: [{ port: 80, targetPort: 3000 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: notify-worker, namespace: apps }
spec:
  replicas: 1
  selector: { matchLabels: { app: notify-worker } }
  template:
    metadata: { labels: { app: notify-worker } }
    spec:
      containers:
        - name: worker
          image: notify-worker:dev
          imagePullPolicy: IfNotPresent
          envFrom: [{ configMapRef: { name: shop-config } }, { secretRef: { name: shop-secrets } }]
          resources:
            requests: { cpu: 50m, memory: 128Mi }
            limits: { memory: 256Mi }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop-ui, namespace: apps }
spec:
  replicas: 1
  selector: { matchLabels: { app: shop-ui } }
  template:
    metadata: { labels: { app: shop-ui } }
    spec:
      containers:
        - name: ui
          image: shop-ui:dev
          imagePullPolicy: IfNotPresent
          ports: [{ containerPort: 80 }]
          volumeMounts:
            - { name: runtime-config, mountPath: /usr/share/nginx/html/config.json, subPath: config.json }
      volumes:
        - name: runtime-config
          configMap:
            name: shop-ui-config
---
apiVersion: v1
kind: ConfigMap
metadata: { name: shop-ui-config, namespace: apps }
data:
  config.json: |
    { "apiUrl": "https://app.local.test/api", "cdnUrl": "https://app.local.test/cdn" }
---
apiVersion: v1
kind: Service
metadata: { name: shop-ui, namespace: apps }
spec:
  selector: { app: shop-ui }
  ports: [{ port: 80 }]

shop-secrets holds the database URLs and passwords, created with kubectl create secret as in Part 1.

6. Ingress with TLS

Use mkcert for a locally trusted certificate, so the browser accepts HTTPS without warnings:

mkcert -install                      # run on Windows too, so the Windows browser trusts it
mkcert app.local.test
kubectl -n apps create secret tls app-tls --cert=app.local.test.pem --key=app.local.test-key.pem
# 40-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: apps
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
spec:
  tls:
    - hosts: [app.local.test]
      secretName: app-tls
  rules:
    - host: app.local.test
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend: { service: { name: shop-api, port: { number: 80 } } }
          - path: /
            pathType: Prefix
            backend: { service: { name: shop-ui, port: { number: 80 } } }

Add 127.0.0.1 app.local.test to C:\Windows\System32\drivers\etc\hosts. With mirrored networking, https://app.local.test in a Windows browser reaches Traefik inside WSL.

7. Test the whole flow

  1. Open https://app.local.test, sign in and upload a file.
  2. Check it arrived: aws s3 ls s3://shop-uploads/users/ --recursive (with the LocalStack endpoint).
  3. Open Mailpit (kubectl -n platform port-forward svc/mailpit 8025) and see the email.
  4. Check the in-app notification record in MongoDB.
  5. Stop Mailpit, upload again, and confirm the message retries and then lands in notifications-dlq.

Moving the same app to AWS

Local AWS
LocalStack S3, S3_ENDPOINT set Real S3, S3_ENDPOINT unset, IAM role for the pod
nginx cache CloudFront with origin access control
Mailpit Amazon SES or an SMTP relay with SPF, DKIM and DMARC
Kafka StatefulSet Amazon MSK or a self-hosted cluster
mkcert certificate ACM certificate on the load balancer

Because only configuration differs, what passes on the laptop is the same code that ships.