Last week I got a message from a parent: "Hey, my 9 year old son uses ModelRift for creating things for his 3D printer, it's great! Product feedback: You should probably ask me to pay now, I feel like I've used it enough."

That one landed. Let me tell you how this thing came to exist.

The rabbit hole

I bought a 3D printer. Turned out to be the best purchase I'd made in years.

After printing a few things from Makerworld and Printables, I fell headfirst into parametric CAD design. Fusion 360, flanges, hinges, chamfers, fillets, loft, sweep, revolve. I learned words I'd never needed before. At some point I realized I was more interested in designing models for other people than printing them myself.

Then I found OpenSCAD. It's essentially a DSL for 3D modeling: you describe geometry as code, and it renders the result. As a programmer, this felt like home. No mouse, no GUI, just code and math.

The problems started when I wanted complex geometry. Sinusoidal rib distributions, procedurally generated mazes, things that are genuinely hard to express in static parametric drawings. I turned to ChatGPT and Gemini. They can generate basic OpenSCAD "skeletons" reasonably well. But the geometry is broken in roughly 90% of cases. The code is syntactically perfect. It runs without errors. It just produces... something that doesn't look anything like what you asked for.

So the workflow became: ask ChatGPT for code, paste into OpenSCAD desktop app, render, stare in horror, take a screenshot, draw arrows on it in some image editor, paste the screenshot back into ChatGPT, repeat. Ten iterations per model if you were lucky.

What I built

ModelRift is a browser-based OpenSCAD editor with an embedded AI chat. The core loop: you describe what you want, the AI writes .scad code, OpenSCAD renders it server-side and returns a 3D preview. If the result is wrong, you click "Annotate", draw directly on the rendered model - arrows, rectangles, text labels - and send the annotated screenshot back to the AI.

That annotation step is where most of the work went. Weeks of iteration on just the annotation mode. It uses Fabric.js under the hood, and getting the overlay to composite correctly with the Three.js viewport took longer than I want to admit. But it solves the actual problem: LLMs understand spatial feedback from visual annotations much better than text descriptions like "the left side is too wide."

The model viewer gives you real-time orbit controls, and after each iteration you can see the updated geometry without leaving the page. The diff viewer shows exactly what changed in the .scad code between versions. You can revert to any previous revision.

Technical bits worth mentioning

The stack is React + Three.js (for .stl model preview) on the frontend, Node.js + PostgreSQL on the backend. For the job queue I used pg-boss instead of Redis - it runs on the same Postgres instance and eliminates one infrastructure dependency. Simple tradeoff that I'm happy with.

The AI model is currently Gemini Flash with thinking mode enabled. I evaluated several options and Gemini Flash produced the fewest geometric errors at reasonable cost for this specific use case. The backend runs OpenSCAD as a CLI process, generates multi-view PNG renders using Sharp, and streams progress back to the frontend via Server-Sent Events.

One feature I'm genuinely proud of is SVG import. If you want to engrave a logo or artwork onto a 3D model, you upload an SVG and ModelRift converts it to OpenSCAD geometry. I wrote the converter from scratch: a regex-based SVG path parser that handles M, L, H, V, C, S, A, Z commands, interpolates Bezier curves, and outputs polygon() primitives with correct point indices. No external library. It runs in three modes (polygon, lines, or hybrid) and auto-detects which one fits the input. The result drops directly into the agent's context, so the AI can use your artwork as geometry in the generated model.

One more thing: the intro video for the launch (the logo spinning and title typing in) was generated frame-by-frame from a plain HTML animation using Puppeteer at 60fps, then assembled into an mp4 with ffmpeg. No After Effects, no video editor. Just a Node.js script and some CSS animations.

What shipped since launch

The initial v0.1 was the bare minimum: browser editor, live 3D preview, STL export. Since then:

• v0.2 added a side-by-side code editor with live preview and a diff viewer for tracking changes
• v0.3 added a public model gallery and user profiles - you can browse and remix models without touching the AI assistant at all
• v0.3.1 added revision history (revert to any previous version of your SCAD code)
• v0.3.3 improved SVG import significantly, adding proper polygon and line conversion modes

The public gallery is at modelrift.com/models. Yeah it still does not instantly enable you build a complex assemblies, which took weeks to grap in Fusion 360 or OnShape. But still worth a look even if you're skeptical about AI-generated geometry.

First payment

I got my first payment for ModelRift in 3 weeks after launch.
I still get shivers on first payment - even after launching a lot of SaaS products. This proves the thing is useful to someone and brings value.

First model printed by the community

One of the early things that told me people were actually using this: someone published a real, practical model on the gallery and printed it.

Exhaust hose adapter for small skylight by viewprintlab - a parametric adapter for venting a 3D printer through a small skylight. Adjustable width from 18 to 22 inches, compatible with a 4-inch flexible hose, designed to attach without drilling using command strips. Two parts that print separately and slide together.

It's exactly the kind of model that makes no sense to look for on Printables - too specific to one person's window size - but takes maybe 20 minutes to generate in ModelRift and then works. That's the use case I was building for.

Where it stands

Every new user gets 150 free credits, enough for roughly 15-30 models (depends on the complexity). After that it costs money because the LLM costs money and I'd prefer not to go bankrupt.

This wasn't a weekend project. It took many days and nights, and there are still plenty of things to improve. But my family uses it for our own printing needs now, which is the most honest endorsement I can give.

If you try it, I'd genuinely appreciate feedback: modelrift.com

n8n is a no-code (low-code) automation platform - similar to Zapier but more technical - and it can be self-hosted. I love n8n (read my older detailed writeup about it): it makes you feel powerful on day one — drag a few nodes, connect an API, and suddenly you’ve automated a whole process. I've built a lot with it myself, including custom nodes, e.g. ScrapeNinja web scraping API integration node. I also maintain the Awesome n8n GitHub repo. For many non-technical indie builders and small teams, n8n feels like magic, and I see many agencies popping up that set up self-hosted n8n instances for their customers.
n8n took over Zapier in Google Trends and in people minds in 2024-2025 - it got this momentum mostly because its self-hosted distribution model is brilliant and serves as an awesome marketing channel; and because AI influencers all over Youtube used n8n as a platform for "Run your own AI chat bot in 5 min" educational videos.

But eventually, some of us hit the ceiling.

After building a few internal tools and SaaS prototypes with n8n, and after talking with lots of smart folks doing the same, I noticed a pattern: the moment you start calling it a “product” or "SaaS" rather than an “automation,” you’re probably past the point where n8n fits.
It is important to note that most of the points mentioned in this post are relevant for most of no-code and low-code platforms.
Here’s when that happens.

1. You Need a Real UX

n8n shines behind the scenes, not in front of customers. When launching products, once I think "would I use n8n here for MVP?" (By the way, MVP abbr stands for Minimum Viable Product) this one is the major blocker. More abbreviations for not-too-technical readers - UX is a shortcut for User Experience, UI stands for User Interface.

What n8n can do well in “UX land”:

• Webhooks: it’s totally fine to keep n8n as your webhook receiver for internal flows or to glue services together. Treat it like a background worker with HTTP ingress. There is a trick to emulate HTML page output via n8n webhook but believe me, this does not end well for real products.
• Forms: there is a built-in Form node for quick data capture, approvals, or small operational tools. To my taste, they are ugly, but they do the job.
• Chat UI - n8n has custom chat UI which allows you to embed AI assistant into HTML page. I don't really like its UX but it works - and is great as a starter kit to experiment with AI agents!

For custom UIs, deploy separately — another no‑code tool or a code app.

Where it starts to hurt:

• Once you’re building user dashboards, onboarding flows, sessions, roles/permissions, or anything that must feel polished, the visual workflow metaphor fights you. You’ll spend more time wiring webhooks and shaping JSON than building a focused UI.

At that stage, you’re not automating anymore — you’re engineering. A small code app (Next.js/Express/FastAPI) with a proper auth layer will be faster to iterate on and easier to test, version, and review.

If you never shipped real code to production, some news for you: modern frontend is a sticky mess. If you decide to go this route - I recommend to check shadcn UI which is definitely one of best UI frameworks available on the market right now. I also recommed to check out Lovable and Bolt which can mitigate your pain of getting started with UX; and then switch over to Claude Code or Cursor for editing.

2. Payments Are Serious

Subscriptions are the major thing which turns your workflows or code into real SaaS Product.

Most no-code workflows can handle simple Stripe calls, but as soon as you add subscription logic, idempotency keys, retries, proration, or multi-currency handling, things get fragile. The “Stripe node + webhook” combo works for MVPs, not for a real SaaS with thousands of transactions and audit requirements.

Stripe API is not what it used to be in 2010 - it's really big now, caters for different use cases, and it takes some time to figure out how it works.

In code, you can keep webhooks thin, enqueue events, process with a job queue, and attach observability. That level of control around billing, error handling, and audit trails is hard to replicate in a visual editor.

3. n8n Community License Does Not Fit SaaS

Recently, someone told me they were building a multi‑customer AI chatbot “on top of n8n” because “n8n developers are cheaper than regular devs.” That’s an MVP‑stage shortcut, not a production strategy. Many folks don’t realize the self‑hosted Community Edition license isn’t designed for building and selling a SaaS on top of n8n.

Will n8n find out if you violate the license? Don’t gamble. Do the right thing from the start—either budget for a commercial license or move the product logic to code.

Licensing context and links:

• n8n is fair‑code, with the Sustainable Use License and a separate Enterprise License. Read: Sustainable Use License, Enterprise License, and the license explainer.
• The SUL typically allows internal business use and consulting. Hosting n8n as the paid product or embedding it as a revenue‑generating feature requires a commercial agreement. n8n also offers an Embed program that, as of this writing, starts around $50,000/year.

Direct quote from the SUL: docs

Our license restricts use to "internal business purposes". In practice this means all use is allowed unless you are selling a product, service, or module in which the value derives entirely or substantially from n8n functionality.

Put simply: the moment you accept payments for a SaaS built on top of n8n, the Community Edition is not permitted.

Real‑world threads on multi‑tenant credentials and scale:

• Using n8n to Power a Scalable SaaS for AI Assistants – Self‑Hosted or Cloud?
• Licensing clarifications from users and staff: Reddit

Takeaway: once you need per‑tenant secrets, scoped permissions, and contractual clarity—and you’re not budgeting for Enterprise/Embed—a small code backend with proper auth, a secrets manager, and per‑tenant queues is safer than stretching a single n8n instance.

4. You Need Versioning That Actually Works

Yes, workflows are JSON and you can export them. Some folks even automate export-to-Git from n8n itself or rely on built-in history. But merges and reviews remain painful compared to code.

Community anecdotes and constraints:

• Users report syncing JSON to Git or using history; full Git integration is an enterprise feature. Reddit thread: Version control in n8n
• Try merging two branches of a big workflow and you’ll see why text diffs, typed models, and PRs still win.

At any point, git diff beats “export workflow as JSON.” If your team depends on code review, CI, and tests, you’ve outgrown the visual editor.
Don't even get me started on modern AI coding tools which can review git history and commit code on your behalf... it's just another level and you should try it as soon as possible.

5. When Architecture Gets Complex

Once you’re orchestrating multiple APIs, queues, webhooks, and databases, n8n starts feeling like one giant script rather than a set of testable modules.

In code, you can separate layers, add tests, monitor performance, and scale horizontally. In n8n, complexity accumulates in a single visual flow, and operational concerns (retries, DLQs, idempotency, tracing) are harder to centralize.

The “47 nodes at 2 AM” anecdote shows up often for a reason—visual debt is real. If you need SLAs and clean rollback paths, code gives you the controls.

• About the Code node: the Code node is excellent for quick JavaScript/Python snippets, but for non‑trivial logic you quickly run into dependency and runtime constraints.
• Anecdote: I had a list of freelancers waiting for crypto payouts. Using n8n, I built a workflow that scanned Google Sheets and created payouts on the BEP‑20 (EVM‑compatible) blockchain, then wrote back transaction hashes or errors. I needed blockchain npm packages (like ethers), so I built a custom Docker image for n8n with those dependencies baked in. It worked—but managing deps, image updates, and security patches felt like running a bespoke backend rather than “no‑code”.
• AI constraints: self‑hosted Code node doesn’t include AI enhancements. Writing code manually in a browser in 2025 is dated; AI assistance for the Code node is currently available to n8n Cloud users, not typical self‑hosted setups.
• Signal to move: if you’re repeatedly adding external SDKs or complex libraries, move that logic to a small service with its own repo, dependencies, tests, and CI. Call it from n8n via HTTP/queues—or replace the flow with code entirely.

6. Performance Concerns

Performance in n8n is about the whole workflow lifecycle: triggers, network calls, payload sizes, execution storage, and scaling — not just the database.

• Item fan‑out and node count:

  • Every node hop serializes/deserializes data and adds latency. Long chains with large items get slow and memory‑hungry.
  • Collapse trivial transforms into a single Code node where it reduces hops, or pre‑aggregate upstream so you pass fewer, smaller items.

• Network chattiness and batching:

  • Loops that call external APIs per row are the classic bottleneck. Prefer bulk endpoints, batch inserts/updates, and rate‑aware concurrency.
  • Use Split In Batches to cap concurrency and add backoff; avoid tight polls when webhooks/events are available.

• Database as a bottleneck - in a lot of cases, Google Sheets and Airtable are used as a database and while these products are awesome getting started; it is a very poor choice even for a medium load. Luckily, in 2025 n8n has finally launched data tables - built-in db which should perform way better compared to Google Sheets.

• Scale the runner, not just the editor:

  • For sustained throughput, run Queue Mode with Redis and multiple workers (N8N_EXECUTIONS_MODE=queue, set worker concurrency) to process jobs in parallel.
  • Keep heavy workflows off the main/editor instance; scale workers horizontally and monitor queue depth/latency.

• Triggers and external systems:

  • Polling (e.g., Google Sheets) adds unpredictable latency and hits quotas. Prefer webhooks and event sources where possible.
  • Sheets are fine for prototypes; for hot paths use a real datastore (PostgreSQL) and let n8n orchestrate, not store, the workload.

• Data integrity under retries:

  • Per‑node retries can create duplicates. Use idempotency keys, unique constraints/UPSERTs downstream, or move critical write logic behind a small API that enforces exactly‑once semantics.

Bottom line: optimize workflows for fewer hops, smaller payloads, event‑driven triggers, and queue‑backed execution. Use Postgres (and/or a thin API) for durability and idempotency, and scale n8n with workers when throughput matters.

7. Deployment

One of n8n’s strongest advantages is skipping DevOps entirely: a single Docker container and you’re productive. For many teams, that’s perfect for internal automations and prototypes.

When DevOps becomes necessary:

• Compliance and backups: scheduled DB/file backups, retention policies, and auditable change management for credentials and workflows.
• High availability and scale: queue mode with Redis, multiple workers, external Postgres, and rolling upgrades; pin versions and test migrations.
• Security and access: TLS, SSO, RBAC, network rules (VPC/egress allowlists), and secret management (e.g., HashiCorp Vault or AWS Secrets Manager).
• Custom runtimes: Code node dependencies via a custom image; private registries; base image patching for CVEs.
• Isolation: separate instances per environment/tenant; constrain cross‑tenant access by design.

If you’re regularly touching these concerns, you’ve crossed into “DevOps needed” territory—either budget time/skills for it or move core product logic to a service you can operate with standard SRE practices.

8. Observability

The built‑in Execution view is a highlight of n8n: you can open a run and see exactly which node failed and why. It’s excellent for ad‑hoc debugging — but sometimes it just doesn’t scale to concurrent millions of runs a day.

What to add for scale:

• Centralized logs: ship container logs to Grafana Loki or the ELK Stack.
• Metrics and dashboards: expose app and worker metrics to Prometheus and graph throughput, queue depth, execution latency (p95/p99), and error rate in Grafana.
• Request visibility: keep reverse‑proxy (nginx/Traefik) access logs and forward them to the same sink; correlate with executionId to trace inputs to outcomes.
• Retention and pruning: tune n8n’s execution data retention so the UI stays responsive and the DB doesn’t balloon; rely on your log/metrics backends for long‑term history.

If your ops questions sound like “which tenant’s runs are failing most this week?” or “why did queue latency spike?”, it’s time to treat n8n like any other production workload and invest in real logging and metrics.

Final Thoughts

n8n is perfect for what it was built for — automating processes, validating ideas, stitching APIs.
But when your project turns into a customer-facing product with real uptime, billing, and architecture needs, it’s time to graduate to code.

AI coding is already here and becoming software engineer to ship real, sophisticated SaaS products is possible for everyone. It will be painful, but the results are well worth it.

You don’t have to throw n8n away. Keep it for what it’s good at — internal automations, monitoring, glue work. But for your core product? Build it right.

Practical Migration Patterns

• My best advice: get on a consulting call with adequate software engineer and describe your case. It's great if you have a working n8n workflow (Minimal Viable Product) and some customers at this stage, it simplifies things a lot! I can help you to convert a n8n workflow into product, as well.
• My 2nd best advice: talk to ChatGPT and descibe what you are doing and your major pains. Ask it how your workflow could be gradually migrated to real code.

AI Tooling Makes Code Faster Than Clicks

If you’ve avoided code because you thought it would be slower than dragging nodes, that’s increasingly outdated.

• Claude Code, Cursor, and Codex are insanely powerful for day‑to‑day engineering. They scaffold endpoints, write tests, refactor safely, and keep context across large changes.
• Chat-driven coding is often more convenient than navigating a complex web UI. You can iterate on logic, add retries, or change data models with a single prompt and get runnable diffs.
• These tools make the “rewrite” path viable: migrate a workflow to a small service with proper types, tests, and observability in hours rather than days.

My Recommendation

• Keep n8n for internal automations and glue.
• For customer-facing product logic, billing, and multi-tenant auth, use real code: clear modules, a queue, typed models, tests, and real version control.
• Lean on AI coding tools (Claude Code, Cursor, Codex) to make the jump quickly and safely.

Related Posts

• My experience using n8n, from a developer perspective — tradeoffs and tips: https://pixeljets.com/blog/n8n/
• Self-hosted is awesome — why I prefer self-hosting and how it applies to n8n deployments: https://pixeljets.com/blog/self-hosted-is-awesome/
• Web scraping in n8n — practical patterns, selectors, proxies, and caveats: https://pixeljets.com/blog/web-scraping-in-n8n/

Why AI Products Need Sandboxing

Sandboxing has become a core feature of modern AI-powered development tools. As AI coding assistants and autonomous agents become more sophisticated, they generate and execute code that needs to run safely in isolated environments.

In my recent Lovable.dev and Bolt.new blog post I described two different approaches to this task. Lovable.dev uses Fly.io containers with Firecracker MicroVMs for stronger isolation, while Bolt.new uses WebAssembly-based WebContainers that run directly in the browser. Very different approaches, with different outcomes: and both solutions highlight the critical need for secure code execution when dealing with AI-generated code.

If you enjoy sandboxing topic as I do, you should check out my Awesome Code Sandboxes github page.

The rise of AI coding tools has made sandboxing essential for:

• Safe AI Code Execution: Running untrusted code generated by AI models
• Rapid Prototyping: Creating and testing applications quickly without security risks
• Educational Platforms: Allowing users to experiment with code safely
• Development Environments: Providing isolated workspaces for teams

AS a big fan of self-hosting, for me and my companies, the choice of sandboxing technology becomes even more critical, as I prefer products which are easier to deploy on cloud nodes of Hetzner / Netcup, but which still provide a certain level of security and all the features we might need.

Section 1: Executive Summary

1.1. Overview

This blog post compares two platforms for running code in isolated environments: Daytona and Microsandbox. Both are designed for safe code execution, especially AI-generated code, but they take very different approaches.

Daytona is a full development platform backed by venture capital. It handles the complete development process - from creating workspaces to team collaboration and deployment. It targets enterprise teams and AI workflows, offering both a managed cloud service and self-hosted options. Daytona aims to standardize development across organizations.

Microsandbox is an open-source tool focused on one thing: running untrusted code securely. It provides strong isolation using virtual machines with fast startup times. Rather than being a complete platform, it's designed as a building block for other systems.

1.2. Key Technical Differences

The main difference is how they handle security and isolation. Daytona uses containers (Docker/OCI) by default, which are fast and convenient but share the host system's kernel. It can use virtual machines for stronger isolation, but containers are the standard approach.

Microsandbox uses micro-virtualization with the libkrun library. Every sandbox gets its own virtual machine with a dedicated kernel (using KVM on Linux or HVF on macOS). This provides much stronger security than containers since there's no shared kernel that could be compromised.

1.3. Which Should You Choose?

Choose Daytona if you want a complete, supported platform for team development environments and you're comfortable with container-based security. It's good for organizations that need lots of features and don't mind the complexity of Kubernetes deployment.

Choose Microsandbox if security is your top priority and you need to run truly untrusted code. It's better for building custom systems where you need maximum isolation, even if the feature set is more limited.

1.4. Key Differences Table

This table shows the main differences between the two platforms:

Section 2: How They Work

2.1. Daytona Architecture

Daytona is built as a complex, multi-component system. Most of it (80-90%) is written in TypeScript, including the web backend (Next.js) and dashboard (React). The CLI and performance-critical parts use Go (10% of the code). This setup allows fast feature development with TypeScript while using Go for system-level tasks.

Daytona is designed to run and test AI coding assistants in controlled environments. It integrates with enterprise tools and runs on Kubernetes (managed with Helm charts) and uses Terraform for infrastructure. This makes it powerful but complex - you need DevOps expertise to run it yourself.

For security, Daytona uses Docker containers by default. This provides basic isolation that works for most development scenarios. You can configure stronger isolation with tools like Sysbox or VMs, but containers are the standard. While Daytona claims "zero risk," containers share the host kernel, so a sophisticated attack could potentially escape the container and affect the host system.

2.2. Microsandbox Architecture

Microsandbox takes a different approach, focusing on security and simplicity over features. It's a lean server that creates and manages virtual machines for code execution. It's written almost entirely in Rust, which provides memory safety and performance - important for security software.

Microsandbox's key feature is hardware-level isolation. It uses libkrun, a library that works with hypervisors (KVM on Linux, HVF on macOS). Each sandbox gets its own virtual machine with a dedicated kernel. This means if an attacker breaks out of one sandbox, they only compromise that single VM, not the host system or other sandboxes. This is much more secure than containers.

The project is built as separate modules: a core library, server, CLI, and specialized libraries for filesystems and data. This modular design makes it easier to integrate into custom systems, unlike Daytona's all-in-one platform approach.

2.3. How libkrun Works

Microsandbox uses libkrun, a library that makes VM-based isolation easy to use. The goals are simplicity, low resource usage, and fast boot times. It includes its own Virtual Machine Monitor (VMM) so it doesn't need external tools like QEMU. It borrows code from optimized projects like Firecracker and only includes essential features, keeping it lightweight and secure.

libkrun has two important features for Microsandbox: networking and filesystem access.

For networking, it uses Transparent Socket Impersonation (TSI). Instead of creating virtual network cards, it intercepts network calls from the VM and handles them on the host. This means all traffic looks like it's coming from the Microsandbox process itself. This avoids the complexity of traditional VM networking - no need for virtual switches or NAT rules.

For file access, it uses virtio-fs to share a host directory directly with the guest VM. This is faster than emulating a full disk and allows easy file sharing between host and guest - which is how Microsandbox saves files.

These different technical choices reflect different goals. Daytona uses many technologies (TypeScript, Go, React, Kubernetes) to build features quickly and integrate with enterprise systems. It's trying to be a complete platform. Microsandbox uses fewer technologies (Rust and libkrun) to focus on security and performance for one specific job.

Choosing between them means choosing different philosophies. Daytona gives you a complete platform with managed workflows. Microsandbox gives you a security component that you build around. They also require different skills: Daytona needs Kubernetes and DevOps knowledge, while Microsandbox needs systems programming and Rust skills. Your choice depends on your team's expertise and goals.

Section 3: Performance Comparison

3.1. Startup Time Claims

Both platforms emphasize fast startup times, which matters for short-lived tasks and AI workflows that need quick iteration.

Daytona claims startup times under 90ms. Detailed benchmarks show 71ms for creation, 67ms for execution, and 59ms for cleanup (197ms total). Other sources mention 200ms startup times. The variation suggests 90ms is a best-case scenario when container images are already downloaded.

Microsandbox claims startup times under 200ms. This is impressive because it's starting a full virtual machine with its own kernel, which traditionally takes many seconds. The speed comes from the lightweight libkrun VMM and modern hardware virtualization.

3.2. Warm vs Cold Starts

These startup times are for "warm" starts after the initial setup. The first time you create an environment ("cold" start), there will be a delay while downloading the base image. Both platforms mention this in their documentation.

• Daytona's "creation time" of 71ms most likely corresponds to the execution of a docker run command or an equivalent container runtime API call on an image that is already present on the host. This is a measure of the container runtime's efficiency.
• Microsandbox's "boot time" of under 200ms is a measure of the entire microVM initialization process: allocating memory, loading a minimal Linux kernel and initrd, and starting the first user-space process inside the guest. Achieving this speed for a VM likely involves advanced techniques, such as loading a pre-booted kernel state directly into memory from a snapshot, a common optimization in the microVM space to bypass the time-consuming hardware initialization phase of a traditional boot sequence (similar to how Firecracker achieves fast startup).

3.3. Performance Beyond Startup

For long-running processes, the initial startup time becomes less critical than sustained runtime performance. Here, the architectural differences lead to different performance profiles.

Daytona, relying on containers, offers near-native execution performance for CPU-bound tasks. Since there is no hardware virtualization layer between the application and the host CPU, instructions run directly on the processor. The primary source of overhead comes from the security layers of the container runtime. System calls (syscalls) made by the application must pass through security filters like seccomp-bpf, and I/O operations for networking and filesystems are mediated by the container daemon. This introduces a small but measurable overhead compared to running directly on the host.

Microsandbox, using microVMs, introduces a layer of hardware virtualization. CPU-bound tasks will incur some overhead from the constant switching between "guest mode" (running the sandboxed code) and "host mode" (running the VMM). Modern CPU virtualization extensions (Intel VT-x, AMD-V) are highly optimized to minimize this overhead, but it is not zero. I/O operations are handled by virtio paravirtualized drivers. These drivers are designed for high efficiency in virtualized environments, but they are not entirely zero-cost compared to a native syscall on the host. The libkrun documentation acknowledges that its virtio-fs implementation, while flexible, does not match the performance of dedicated block-based storage devices.

Both projects focus heavily on fast startup times, which suggests they're designed for short-lived tasks. This works well for AI model testing where you might create and destroy thousands of sandboxes quickly, or for serverless functions.

However, for long-running processes (1+ hours), startup time doesn't matter much. A 110ms difference is less than 0.003% of an hour-long task. For long-running workloads, you should focus on runtime stability, I/O performance, resource usage, and how well data persists over time.

Section 4: Setup and Usage

This section includes practical examples showing how to create and manage sandboxes with both platforms.

4.1. Setting Up Daytona

Setting up Daytona involves multiple steps. You install the CLI, start the server, connect to Git providers (GitHub/GitLab), and configure where workspaces will run (Docker, AWS, etc.). Finally, you set targets for where to create workspaces.

Daytona needs substantial infrastructure: at least 4 vCPUs, 16GB RAM, 200GB disk space, and a Kubernetes cluster. It uses Helm charts for deployment. While this setup is powerful and flexible for complex environments, it requires significant DevOps expertise to manage.

Example: Creating a Sandbox with Daytona SDK

// daytona-example.ts
import { Daytona } from "@daytonaio/sdk";

// Initialize the Daytona client
const daytona = new Daytona({
  apiKey: process.env.DAYTONA_API_KEY,
  serverUrl: "https://api.daytona.io"
});

async function createPythonSandbox() {
  try {
    // Create a new sandbox
    const sandbox = await daytona.sandbox.create({
      name: "python-analysis",
      image: "python:3.11",
      envVars: {
        "PYTHONPATH": "/workspace",
        "DATA_SOURCE": "https://api.example.com/data"
      }
    });

    console.log(`Sandbox created: ${sandbox.id}`);

    // Upload a Python script
    await daytona.sandbox.uploadFile(sandbox.id, {
      filePath: "/workspace/analyze.py",
      content: `
import pandas as pd
import requests
import os

def analyze_data():
    # Fetch data from API
    response = requests.get(os.getenv('DATA_SOURCE'))
    data = response.json()
    
    # Process with pandas
    df = pd.DataFrame(data)
    result = df.describe()
    
    # Save results
    result.to_csv('/workspace/results.csv')
    print("Analysis complete!")

if __name__ == "__main__":
    analyze_data()
`
    });

    // Execute the script
    const execution = await daytona.sandbox.execute(sandbox.id, {
      command: "pip install pandas requests && python analyze.py"
    });

    console.log("Output:", execution.output);

    // Download results
    const results = await daytona.sandbox.downloadFile(
      sandbox.id, 
      "/workspace/results.csv"
    );
    
    console.log("Results downloaded:", results);

  } catch (error) {
    console.error("Error:", error);
  }
}

createPythonSandbox();

Example: devcontainer.json Configuration

{
  "name": "AI Analysis Environment",
  "image": "python:3.11-slim",
  "features": {
    "ghcr.io/devcontainers/features/common-utils:2": {
      "installZsh": true,
      "configureZshAsDefaultShell": true
    },
    "ghcr.io/devcontainers/features/python:1": {
      "version": "3.11",
      "installJupyterlab": true
    }
  },
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
        "ms-toolsai.jupyter"
      ]
    }
  },
  "postCreateCommand": "pip install pandas numpy matplotlib requests",
  "remoteUser": "vscode",
  "workspaceFolder": "/workspace",
  "mounts": [
    "source=${localWorkspaceFolder}/data,target=/workspace/data,type=bind"
  ],
  "forwardPorts": [8888, 5000],
  "portsAttributes": {
    "8888": {
      "label": "Jupyter Lab",
      "onAutoForward": "openBrowser"
    }
  }
}

4.2. Setting Up Microsandbox

Microsandbox is much simpler to install. You run one command (curl -sSL https://get.microsandbox.dev | sh) to download and install it, then start the server with msb server start.

The trade-off is that Microsandbox has specific hardware requirements. It needs Linux with KVM enabled, or macOS with Apple Silicon (M1/M2/M3/M4). Windows support is planned but not available yet. Instead of managing complex infrastructure like Kubernetes, you just need to ensure your host machine has the right virtualization features.

Example: Creating a Sandbox with Microsandbox SDK

// microsandbox-example.ts
import { Microsandbox } from "@microsandbox/sdk";

// Initialize the Microsandbox client
const msb = new Microsandbox({
  serverUrl: "http://localhost:8080" // Default local server
});

async function createPythonAnalysisSandbox() {
  try {
    // Create a new sandbox
    const sandbox = await msb.sandbox.create({
      image: "python:3.11-alpine",
      command: "sh",
      env: {
        "PYTHONPATH": "/workspace",
        "DATA_SOURCE": "https://api.example.com/data"
      },
      workdir: "/workspace"
    });

    console.log(`Sandbox created: ${sandbox.id}`);

    // Write Python analysis script
    await msb.sandbox.writeFile(sandbox.id, "/workspace/analyze.py", `
import json
import urllib.request
import os
import csv
from collections import Counter

def fetch_and_analyze():
    # Fetch data from API
    url = os.getenv('DATA_SOURCE', 'https://jsonplaceholder.typicode.com/posts')
    
    with urllib.request.urlopen(url) as response:
        data = json.loads(response.read().decode())
    
    # Simple analysis
    user_counts = Counter(post['userId'] for post in data)
    
    # Save results
    with open('/workspace/analysis.csv', 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['User ID', 'Post Count'])
        for user_id, count in user_counts.items():
            writer.writerow([user_id, count])
    
    print(f"Analyzed {len(data)} posts from {len(user_counts)} users")
    return user_counts

if __name__ == "__main__":
    result = fetch_and_analyze()
    print("Top 3 users:", dict(list(result.most_common(3))))
`);

    // Execute the analysis
    const result = await msb.sandbox.exec(sandbox.id, {
      command: "python analyze.py",
      timeout: 30000
    });

    console.log("Execution output:", result.stdout);
    
    if (result.stderr) {
      console.error("Errors:", result.stderr);
    }

    // Read the results file
    const analysisResults = await msb.sandbox.readFile(
      sandbox.id, 
      "/workspace/analysis.csv"
    );
    
    console.log("Analysis results:", analysisResults);

    // Clean up
    await msb.sandbox.destroy(sandbox.id);
    console.log("Sandbox destroyed");

  } catch (error) {
    console.error("Error:", error);
  }
}

createPythonAnalysisSandbox();

Example: Sandboxfile Configuration

# Sandboxfile
name: python-analysis-env
image: python:3.11-alpine

# Environment setup
setup:
  - apk add --no-cache curl git
  - pip install --no-cache-dir requests pandas numpy

# Working directory
workdir: /workspace

# Environment variables
env:
  PYTHONPATH: /workspace
  PYTHON_ENV: sandbox
  TZ: UTC

# Resource limits
resources:
  memory: 512m
  cpu: 0.5
  disk: 1g

# Networking (optional)
network:
  internet: true
  ports:
    - 8000:8000  # Expose port for web services

# Persistence (optional)
volumes:
  - ./data:/workspace/data:ro
  - ./output:/workspace/output

# Security settings
security:
  read_only_root: false
  no_new_privileges: true
  user: 1000:1000

Example: Quick Command-Line Usage

# Create ephemeral sandbox for quick tasks
msx python:3.11 "pip install requests && python -c 'import requests; print(requests.get(\"https://api.github.com\").status_code)'"

# Create persistent project sandbox
msr python:3.11
# This creates ./menv directory for persistence

# Run commands in persistent sandbox
msr exec "pip install pandas"
msr exec "python data_analysis.py"

# View sandbox status
msr status

# Stop sandbox (keeps files in ./menv)
msr stop

4.3. Configuration and Management

Configuration and daily management also reflect the platforms' differing philosophies.

Daytona uses devcontainer.json files for project configuration. This is a widely adopted format, especially in VS Code, making it familiar to many developers. Management is offered through a comprehensive CLI for scripting and automation (daytona create, daytona code, etc.) and a full-featured web dashboard for graphical management. This provides a polished user experience catering to different preferences.

Microsandbox uses a custom Sandboxfile for defining project-based environments. This file is similar to a Dockerfile or Vagrantfile. Management is performed exclusively through its command-line tools (msb, msr, msx, msi), which is ideal for automation and users comfortable in the terminal but lacks a graphical user interface.

The concept of "ease of deployment" is therefore subjective and highly dependent on the user's role and the scale of the deployment. Daytona's workflow is analogous to a modern PaaS platform. It abstracts the underlying infrastructure from the end-developer, providing them with a simple, standardized environment, while giving platform administrators powerful tools (Kubernetes, Helm) for management and control. For a platform engineering team tasked with providing a managed service to an entire organization, Daytona's structured, Kubernetes-native approach may be considered "easier" to integrate into their existing CI/CD pipelines and infrastructure-as-code practices.

Conversely, Microsandbox offers the experience of a classic developer command-line tool. It is direct, transparent, and highly scriptable. The user interacts directly with the server process running on their local or remote machine. For an individual developer or a small team needing to get a secure sandbox up and running immediately on a compatible machine, Microsandbox's single installation script is undeniably "easier." The choice reflects the intended scale: Daytona is built for organizational scale, abstracting complexity for the many. Microsandbox is built for developer or small-team scale, offering simplicity for the few.

Section 5: Long-Running Workloads

How well each platform handles long-running, I/O-heavy tasks depends on their filesystem and networking systems.

5.1. Filesystem Persistence

Daytona provides a comprehensive API-driven approach to filesystem management. Its SDKs for Python and TypeScript offer methods for listing files, creating directories, changing permissions, and uploading/downloading files between the local machine and the remote sandbox. Daytona supports "Volumes" for persistent storage that's separate from individual sandboxes, similar to Kubernetes. The platform's sandboxes are "stateful," preserving the filesystem across multiple interactions. This API-first approach is ideal for building automated workflows.

Microsandbox implements persistence more directly. When using its "project sandbox" feature, any changes made to the filesystem inside the guest VM are automatically saved to a ./menv directory on the host machine. This uses virtio-fs technology to map the host directory into the guest. While this makes the sandbox's state easily inspectable from the host, it's subject to the maturity of the virtio-fs implementation. Some GitHub issues report I/O errors when mounting single files and problems with Unix sockets. This suggests the filesystem layer is still maturing and may have rough edges for complex I/O patterns.

5.2. Networking Capabilities

Daytona offers flexible networking tools for developers building network services. The daytona forward CLI command allows port forwarding from a workspace to your local machine for testing. It can also generate shareable public URLs for ports by routing traffic through Daytona's proxy service, which is valuable for collaboration and demos. The platform also supports VPN tunnels to development environments. This feature set is well-suited for web application development.

Microsandbox's networking is handled by libkrun's Transparent Socket Impersonation (TSI). This system provides zero-configuration networking, which is a major advantage for simple use cases. However, it may be more limited than Daytona's explicit tooling. TSI is limited to IPv4 UDP and TCP sockets, which covers most common use cases but may exclude others. Some GitHub issues report unexpected hangs when processes try to make requests to 127.0.0.1, pointing to potential networking problems. Community feedback also indicates desire for more advanced network configurations, like per-sandbox VPN routing, which isn't currently supported.

5.3. Managing Long-Running Processes

Daytona has a first-class API concept called "Sessions" for managing background and long-running processes. The SDK provides functions to create, monitor (getSession), and delete these sessions. This makes the platform well-suited for running services, daemons, or any task that needs to persist in the background, as it provides the necessary programmatic hooks for lifecycle management.

Microsandbox supports long-running processes through its stateful persistence model. A user can execute a command to run in the background within the guest VM, and as long as the sandbox isn't stopped or destroyed, that process will continue to run. Because the filesystem state is preserved across stop/start cycles, a service can be configured to restart automatically when the sandbox is brought back up. However, Microsandbox currently lacks a high-level API like Daytona's "Sessions" for managing these background processes. Instead, it relies on standard Linux process management tools (e.g., nohup, systemd) within the sandbox itself.

The comparison of I/O capabilities reveals a classic trade-off between maturity and innovation. Daytona employs well-established, battle-tested patterns for filesystem and network I/O: API-driven file management, explicit port forwarding, and concepts like Volumes borrowed from mature container orchestrators. These methods are reliable and feature-rich. Microsandbox, on the other hand, leverages innovative but less mature technologies like virtio-fs for direct host mapping and TSI for zero-configuration networking. While these approaches are elegant and powerful in theory, the evidence from the project's issue tracker suggests they still have rough edges that need to be smoothed out.

For a production workload that runs for over an hour and is highly sensitive to I/O stability and features—such as a database server or a public-facing web application—Daytona's mature and explicit I/O model currently presents a lower risk. Microsandbox's approach is highly promising and may become more robust over time, but for now, it appears better suited for long-running, compute-centric tasks where I/O requirements are simpler and the paramount concern is the security of the execution environment itself.

Section 6: Project Status and Licensing

6.1. Community and Support

The two projects exist at vastly different stages of maturity and are supported by different ecosystem models.

Daytona is a well-funded commercial product, having raised over $5 million in venture capital. This financial backing is reflected in the project's polish and scale. It has a large and active community, with over 21,000 stars and 191 contributors on GitHub. Daytona offers a commercial PaaS product, Daytona Cloud, alongside its self-hostable version. The platform provides enterprise-grade capabilities like GPU support for machine learning workloads and role-based access control (RBAC). The project maintains extensive documentation and shows high activity with frequent releases and a clear public changelog.

Microsandbox is a younger, community-driven open-source project. It has a smaller but growing community with approximately 3,300 GitHub stars. The project is highly active, but development appears to be driven by a smaller core team. It's positioned as a self-hosted, open-source alternative to commercial cloud sandboxing solutions. Its long-term viability depends on its ability to foster community engagement and achieve wider adoption. It is currently in beta state, indicating it's not yet considered production-ready by its authors.

6.2. Licensing Implications

The choice of open-source license is a critical and strategic differentiator between the two projects.

Daytona is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). This is a "strong copyleft" license with a specific clause for network-accessible services. It mandates that if a modified version of the Daytona software is run on a server and made accessible to users over a network, the complete source code of that modified version must also be made publicly available. This license is a strategic choice designed to create a commercial funnel. It makes it very difficult for another company to use Daytona's open-source code to build a competing commercial PaaS offering. This effectively encourages users to either contribute their modifications back to the public project or purchase a commercial license from Daytona.

Microsandbox is licensed under the Apache License 2.0. This is a permissive license that places very few restrictions on users. It allows developers to freely use, modify, and distribute the software, and to incorporate it into proprietary, closed-source commercial products without any obligation to release their modifications.

This licensing difference is not a minor detail; it is a core element of each project's business model and philosophy. Daytona's AGPL-3.0 license is designed to protect its commercial interests and drive sales. Microsandbox's Apache-2.0 license is designed to encourage the widest possible adoption and integration, even within commercial products. For an organization evaluating these tools, this presents a critical build-versus-buy decision point. Using Microsandbox is a "build" decision; it provides a foundational component that can be freely built upon to create a custom, proprietary platform. Using the open-source version of Daytona forces a "buy-in" decision—either buying into the open-source obligations of the AGPL or buying a commercial license. This makes Microsandbox a more flexible choice for organizations that intend to create and sell a product that includes a sandboxing component.

6.3. Documentation and Developer Experience

The developer experience and quality of documentation also reflect the different maturity levels and resource allocations of the projects.

Daytona provides extensive, well-structured, and polished documentation. The documentation site is built with modern tools like Astro and MDX and is maintained as a dedicated repository. It includes detailed guides, a full CLI reference, and complete SDK references for both Python and TypeScript, making it easy for developers to get started. The SDKs are professionally packaged and distributed on standard repositories like npm and pip, ensuring a smooth installation process.

Microsandbox's documentation is more basic. The primary source of information is the comprehensive README.md file in the main GitHub repository. While it has a documentation website (docs.microsandbox.dev), it was inaccessible during research. The SDKs are also available on npm and pip, but the overall developer experience is that of a powerful, but still-in-beta, tool focused on core functionality rather than polished presentation.

Section 7: Related Technologies & Alternatives

Beyond Daytona and Microsandbox, the sandboxing ecosystem includes several other notable technologies and platforms worth considering:

7.1. Core Sandboxing Technologies

Firecracker: The AWS-developed VMM that powers many platforms including e2b. Creates lightweight microVMs with 125ms startup times and minimal memory overhead.

gVisor: Google's application kernel that intercepts system calls, providing stronger isolation than containers without requiring hardware virtualization.

libkrun: The library powering Microsandbox. Enables embedded microVM creation with hardware isolation and fast startup times.

7.2. Alternative Platforms

e2b: AI-focused sandboxing platform using Firecracker microVMs. Offers both SaaS and self-hosted options with Apache-2.0 licensing. Targets similar AI use cases as Daytona.

WebContainers: StackBlitz's browser-native Node.js runtime using WebAssembly. Eliminates server infrastructure by running entirely in browsers, ideal for lightweight prototyping.

Cloudflare Workers: Edge computing platform using V8 Isolates with 0ms cold starts. Best for short-running functions distributed globally.

Kata Containers: Hybrid container/VM runtime providing OCI compatibility with VM-level security. Integrates with Kubernetes while offering hardware isolation.

7.3. Development Environment Platforms

Gitpod: Open-source CDE using containers with zero-trust architecture. AGPL-3.0 licensed like Daytona but focuses on Git-based ephemeral environments.

Coder: Enterprise CDE using Terraform for flexible provisioning. Can deploy across containers, Kubernetes, or VMs depending on security needs.

CodeSandbox: Hybrid approach using browser sandboxes for frontend and microVMs for full-stack development. Popular for prototyping and code sharing.

7.4. How They Compare

These alternatives highlight different trade-offs:

• e2b offers similar AI focus to Daytona but with microVM security and permissive licensing
• WebContainers eliminates infrastructure costs but limits functionality to browser environments
• Cloudflare Workers provides global edge distribution but restricts to short-running functions
• Kata Containers delivers VM security with container compatibility but requires Kubernetes expertise
• Gitpod/Coder focus on development environments rather than general code execution

Section 8: Final Recommendations

8.1. Summary

Choosing between Daytona and Microsandbox involves fundamental trade-offs based on your priorities around security, operations, and licensing:

• Complete Platform vs. Security Tool: Daytona manages the entire development process. Microsandbox focuses on one thing: running untrusted code securely.
• Containers vs. Virtual Machines: Daytona uses containers (fast, compatible, shared kernel). Microsandbox uses VMs (more secure, isolated kernels, newer technology).
• Web Interface vs. Command Line: Daytona has a web UI and abstracts complexity. Microsandbox is CLI-focused and more direct.
• Commercial vs. Open Source: Daytona is commercially backed with restrictive licensing. Microsandbox is community-driven with permissive licensing.
• Proven vs. Innovative: Daytona uses established technologies. Microsandbox uses newer approaches that may have rough edges.

8.2. Which to Choose

Based on this analysis:

Choose Daytona if:

• You want a complete development platform for your team
• You're running mostly trusted code from internal developers
• You need lots of features, web UI, team management, and cloud integrations
• You're okay with AGPL licensing or plan to buy a commercial license
• You have DevOps and Kubernetes expertise

Choose Microsandbox if:

• Security is your top priority and you need to run fully untrusted code
• You're building a custom platform and need a secure execution component
• You want permissive licensing (Apache-2.0) for commercial use
• You prefer simple deployment and CLI tools
• You have systems expertise and can work with beta-stage software

8.3. Future Outlook

Both projects are on promising but divergent trajectories.

Daytona is well-funded and positioned to become a major player in AI agent infrastructure and enterprise development environment management. Its future development will likely focus on expanding enterprise features, deepening integrations with cloud providers and AI tools, and further polishing its user experience to solidify its commercial leadership.

Microsandbox represents accessible, lightweight virtualization. Its future success will depend on its ability to stabilize and mature its I/O systems (networking and filesystem persistence) and to cultivate a larger, more active contributor community. If it can overcome these hurdles, it has the potential to become a standard open-source component for secure sandboxing, similar to the role Firecracker plays for AWS Lambda.

Ultimately, the choice between them today is a choice between a mature, feature-rich, and commercially-oriented product (Daytona) and a newer, technologically-focused, and security-focused tool (Microsandbox).

Okay, so if you're not into building a big startup with investors and splitting equity, you can try going solo and bootstrapped. Will it work for you? I don't know. It works for me, as far as I can tell - and I'm a tech person - by no means a genius, but I still can package technology into a product.
This post is an attempt to organize my scattered and fragmented observations and thoughts about bootstrapping small SaaS products - into a long (and still disjointed) text.

The freedom of being solo

Being the only founder means no messy partner agreements or confusing profit splits. Bootstrapped means we're not running around pitching to VCs - we're just focusing on solving a real user problem. If there's no buyer, there's no revenue, end of story. This simplicity brings clarity: you can make decisions fast without endless discussions.

Sure, places like Y Combinator prefer multiple co-founders (if one burns out, the other steps in). But that often translates into compromise after compromise, especially when you're both technical. Instead of pushing the product and landing deals, you might end up arguing about frameworks and code reviews. Sometimes it works (like early Google), but more often it stalls the project, and then you discover that no one actually wants the product.
When you're a single founder, the buck really does stop with you. Mistakes? You learn immediately and fix them. That's motivating.

Personal health == Company health

You quickly realize that your own health—both mental and physical - is your company’s most critical asset. It’s not just “personal well-being” anymore; it’s about keeping the product alive. Skip a run or bail on that cold plunge, and your brain feels stuck in low gear. Sure, you can still show up physically, but your focus and creativity tank - an impossible situation when you’re trying to build long-term products and tackle tough decisions.

There’s also a kind of unspoken rule: if you consistently skip your exercise or your daily mind-clear ritual, it’s almost like not coming to work at all. You can squeak by for a few days, especially if you see others cruising on coffee and sporadic bursts of energy. But a year or two down the line, neglecting your physical and mental well-being means your product suffers. You’re half-asleep, burning daylight in an office chair, and your spark is gone. If you want to keep innovating and thinking several steps ahead, your “personal health engine” has to fire on all cylinders.

Mornings can be rough - maybe it’s cold, maybe you’re sore, and the last thing you want is a run. But you know the difference it makes: once you’ve pushed through that resistance, your mind lights up, you’re more present, and you’re able to consistently do the heavy lifting that a single-founder role demands. Think of it less like a luxury and more like essential maintenance on the machine that’s powering your company.

The sales reality check

Sales become a huge reality check once you’re a solo tech founder. Suddenly, “Who are my customers, and why should they pay me?” is an urgent daily question. It’s a crash course in marketing, positioning, and real human conversations—no longer an abstract problem for some “business partner” to handle. If you’ve never had to sell before, it can feel both terrifying and exhilarating. You realize it’s your job to answer the toughest question: “Does anyone actually want this?”

Yet there’s a common piece of advice—“Don’t build if you don’t know how to sell”—that can be a trap. If you take it too literally, you might end up building something dull, just because you think it will be easy to sell. That kills your motivation in the long run; boredom is the enemy of good products. Sure, you need to understand how to reach your audience, but if you’re not at least partially passionate about the solution, it’s going to show. People pick up on that lack of excitement; it makes you a less convincing salesperson.

I built ScrapeNinja web scraping API because I needed to scrape a specific site - and found the process exciting. That passion pushed me to keep iterating, figuring out the marketing and sales along the way. Sure, it was challenging, but having a genuine spark for your own product makes selling a more natural, less forced experience. When you care about what you’ve built, you’ll do whatever it takes to connect with the right people. That drive—not a formulaic sales strategy—is what truly keeps you and your product alive.

Dealing with isolation

But watch out for loneliness. When you are the entire company, you can't rely on coworkers to bounce ideas off - you are all the coworkers. You have to be comfortable spending time on your own. At the same time, you need community outside your own head: talk to friends, family (if they're not toxic), and other founders. Those conversations spark fresh insights and remind you you're not the only crazy person trying to do something big.

The art of delegation

Delegating is non-negotiable. Doing the same repetitive tasks over and over again is a huge time drain and kills your motivation. Once you understand a process and it's working, hand it off to someone else. If you absolutely hate sales, at least do it yourself enough times to figure out the basics, then hire or outsource.

Choosing the right niche

Picking a niche that’s too ambitious or capital-intensive can sink an indie dev before they even start. If you’re building a super high-tech AI solution requiring an army of PhDs or trying to dethrone a mega-corporation with billions in funding, you’ll get stuck scaling those massive complexities. On the flip side, if the scope is so tiny that it’s basically a weekend hobby project, you won’t see real traction - or enough revenue to keep you motivated. Aim for a middle ground: a problem that’s valid, clear, and solvable with your current skill set and resources.

You want something you can launch quickly (I never launch a product if I cannot implement the main, core feature of the product in 3 weeks max), pivot if needed, and see tangible results. Focus on niches you find interesting enough to stick with during the inevitable tough times. Solve a problem that a specific user base actually cares about, but don’t go for a target market of ten people. Likewise, avoid fads that might disappear next month. Steady, proven demand plus your personal curiosity is the sweet spot that can sustain an indie dev journey.

Minimalistic lifestyle & cost control

Keeping your personal expenses low is one of the smartest moves you can make as a solo founder. The lower your costs, the longer your runway — meaning you don’t have to scramble to raise money or get desperate for quick revenue. Bootstrapping is a maraphon. It’s about being content with a simpler lifestyle so you can focus your resources on the business. That could mean sharing an apartment, cooking meals at home, or getting rid of monthly subscriptions you don’t actually need. Little sacrifices add up, and the peace of mind you get from having more time is priceless. But, you’re not trying to punish yourself or prove how frugal you can be - if I really want this bike right now, I just get it.

Minimalism also frees up mental space. When you’re not juggling a bunch of bills or worrying about living paycheck to paycheck, you can allow yourself to be “bored” and just think. That boredom, ironically, is where some of the best ideas often come from. Instead of constantly rushing between tasks, you can take leisurely walks in the park—giving your brain the breathing room it needs to solve problems creatively.

The Freedom is the real wealth

I have a friend who (from time to time) suggests, “Why do you keep messing around alone with these indie, bootstrapped projects? Let’s hire five developers, get some funding, and sell for tens of millions at least.” And for a while, that proposition made me feel bad. I couldn’t explain why, but I knew deep down that path wasn’t for me, at least not now. But why I was feeling bad? Part of me felt like maybe I was being “unambitious” by not building the classic venture-backed startup. But the more I thought about it, the more I realized that a glossy, investor-friendly narrative just wasn’t my personal compass. I found genuine fulfillment doing things exactly the way I wanted — no board meetings, no forced decisions, no external scripts to follow.

"Unambitous"...
Boostrapping is not about rejecting ambition altogether. Ambition is crucial; without it, we’re just couch potatoes going nowhere. The real trick is recognizing and honoring your own ambitions rather than the ones people try to pin on you. There’s a certain kind of wealth that doesn’t show up on your bank statements or in a piece of real estate in Dubai - it’s the wealth of deciding what you do with your own time, every day, without being "lost" in life in philosophical sense - you have a clear trajectory. Your own trajectory. When you’re not locked into someone else’s objectives, you can pivot, pause, or double down the moment you sense it’s right. Some folks assume venture capital is an easy way to “do nothing” while your employees build your dream. In reality, it’s a high-stakes game with a completely different set of pressures. By staying solo, you’re never a hostage to outside demands; you work like crazy, but on your own terms, fueled by your own passion. That, in my book, is real freedom.

Proceed to the code and installation instructions: https://github.com/restyler/n8n-nodes-scrapeninja

Over the past few months, I’ve tested different approaches for cleaning raw HTML and creating efficient scraping pipelines using n8n. Since I have some experience building complex web scrapers using real coding, I quickly realized that n8n ecosystem can be significantly improved with better web scraping tools As a result, I have built ScrapeNinja n8n community node.
Read my blog post about web scraping in n8n - and in this post, I will talk about new feature of ScrapeNinja node: web crawler.

The Need for Specialized LLM Knowledge Bases

Large language models have changed how we interact with data, but they’re often missing domain-specific insights. By structuring documentation into a single Markdown file, we give LLMs the context they need to produce accurate, detailed answers.

I’ve observed that using a curated knowledge base fed into an LLM as a prompt (I am so glad we now get 128k+ context windows!) dramatically improves response quality and code generation. This realization led me to build a tool that automates the process of building knowledgebases from websites in a scalable, repeatable way.

Why n8n?

I love n8n: it is an open-source, self hosted workflow engine where complex automations can be built using low code. It has an amazing community.

Web Scraping Challenges

There are plenty of web scraping tutorials for n8n, but real world web scraping is much harder once you try to do something useful. Common issues include:

• Messy HTML Output: Raw HTML often includes scripts, styles, and irrelevant tags that confuse both humans and LLMs.

• Manual URL Management: Listing URLs by hand became tedious, especially as documentation grew.

• Resource-Heavy Operations: Spinning up new browser sessions for every page was slow and used too many resources.

I have built ScrapeNinja web scraping API, and I know how painful it is to maintain a web scraper.
As I have 10+ years of experience working as a software developer, most of my web scrapers are built using code (Node.js) but I have recently started using n8n more and more for my own tasks, since it is sometimes easier to maintain scrapers in n8n compared to maintaining a node.js application. Once something breaks, checking n8n execution logs is so much better compared to a situation when I realize that I don't even remember where my particular node.js process is running - as I literally have 10+ cloud servers. n8n also allows to use JS code anywhere and this is a very nice feature to have, compared to other no-code platforms which are "too no-code" for me. I don't like to feel restricted. n8n is also self-hosted and runs perfectly well via docker compose file. Just try it.

The Birth of the ScrapeNinja n8n Crawler

ScrapeNinja crawler was another evolutional step for me: I have already deployed ScrapeNinja scraping n8n node and got some experience building complex n8n nodes. Crawler is way more complex compared to my Scrape n8n operations which can be used to scrape a single page (included within the same n8n node).

Github repo of the Scrapeninja n8n community node

I developed the ScrapeNinja n8n crawler to address these issues by combining workflow automation with flexible scraping capabilities. Key highlights include:

• Recursive Traversal: It follows links based on URL rules until a page limit is reached or no new URLs are found.

• Hybrid Extraction Techniques: It uses the ScrapeNinja API for /scrape or /scrape-js (fast raw requests or full browser rendering), while a local “Extract primary content” operation runs inside the n8n node.

• Detailed Logging: All actions are logged in Postgres (via Supabase), simplifying debugging and performance checks.

I even used it on ScrapeNinja’s own documentation to confirm it could produce a single Markdown file from multiple pages.

How It Works: A Technical Overview

1. Initialization: The crawler starts with a seed URL.

2. Page Fetching with ScrapeNinja API: Depending on your configuration, it calls /scrape for fast raw requests or /scrape-js for browser-rendered pages, handling JavaScript-heavy sites.

3. Local Content Extraction: Once a page is fetched, the primary content is extracted locally, removing scripts, ads, and unnecessary tags to yield clear Markdown text.

4. Recursive Link Extraction: The crawler extracts and queues links, continuing until it hits the page limit or runs out of new links.

5. Data Storage and Logging: Processed pages and logs (in JSON) are stored in a Postgres database, aiding in visibility and troubleshooting.

Things always go wrong in web scraping

Crawling is a really complex process, many things can go wrong. To mitigate this, crawler node logs everything. I mean, everything. Take a look at these log entries:

[
  {
    "level": "debug",
    "message": "Fetching page \"https://scrapeninja.net\" using ScrapeNinja",
    "metadata": {
      "url": "https://scrapeninja.net",
      "runId": 2
    },
    "created_at": "2025-02-06T11:36:12.011Z"
  },
  {
    "level": "debug",
    "message": "Selected URL \"https://scrapeninja.net\" for processing",
    "metadata": {
      "url": "https://scrapeninja.net",
      "depth": 0,
      "runId": 2,
      "queueId": 17
    },
    "created_at": "2025-02-06T11:36:12.046Z"
  },
  {
    "level": "debug",
    "message": "Sending request to https://scrapeninja.p.rapidapi.com/scrape",
    "metadata": {
      "url": "https://scrapeninja.net",
      "runId": 2,
      "engine": "scrape",
      "marketplace": "rapidapi"
    },
    "created_at": "2025-02-06T11:36:12.072Z"
  },
  {
    "level": "info",
    "message": "First page link analysis for \"https://scrapeninja.net\"",
    "metadata": {
      "runId": 2,
      "links_ignored": 10,
      "crawl_external": false,
      "links_included": 14,
      "exclude_patterns": [],
      "include_patterns": [],
      "total_links_found": 24,
      "sample_ignored_links": [
        "https://www.producthunt.com/posts/scrapeninja?utm_source=badge-top-post-badge&utm_medium=badge&utm_souce=badge-scrapeninja",
        "https://rapidapi.com/restyler/api/scrapeninja",
        "https://rapidapi.com/restyler/api/scrapeninja/pricing",
        "https://pipedream.com/apps/pipedream/integrations/scrapeninja",
        "https://apiroad.net/proxy",
        "https://pixeljets.com/blog/bypass-cloudflare/",
        "https://pixeljets.com/blog/browser-as-api-web-scraping/",
        "https://github.com/restyler/scrapeninja-api-php-client",
        "https://www.make.com/en/integrations/scrapeninja",
        "https://t.me/scrapeninja"
      ],
      "sample_included_links": [
        "https://scrapeninja.net/",
        "https://scrapeninja.net/docs/n8n/",
        "https://scrapeninja.net/scraper-sandbox?slug=hackernews",
        "https://scrapeninja.net/docs/",
        "https://scrapeninja.net/docs/proxy-setup/",
        "https://scrapeninja.net/curl-to-scraper",
        "https://scrapeninja.net/scraper-sandbox",
        "https://scrapeninja.net/cheerio-sandbox",
        "https://scrapeninja.net/docs/make.com/",
        "https://scrapeninja.net/openapi.yaml"
      ]
    },
    "created_at": "2025-02-06T11:36:13.554Z"
  },
  {
    "level": "debug",
    "message": "ScrapeNinja response info for \"https://scrapeninja.net\"",
    "metadata": {
      "url": "https://scrapeninja.net",
      "runId": 2,
      "pageTitle": "ScrapeNinja Web Scraping API: Turns websites into data, on scale. 🚀",
      "statusCode": 200
    },
    "created_at": "2025-02-06T11:36:13.580Z"
  },
  {
    "level": "debug",
    "message": "Queued 14 new URLs for crawling",
    "metadata": {
      "runId": 2,
      "parentUrl": "https://scrapeninja.net",
      "linksQueued": 14,
      "currentDepth": 0
    },
    "created_at": "2025-02-06T11:36:13.620Z"
  },
  {
    "level": "info",
    "message": "Successfully processed page \"https://scrapeninja.net\"",
    "metadata": {
      "url": "https://scrapeninja.net",
      "depth": 0,
      "run_id": 2,
      "status": "completed",
      "max_pages": 5,
      "latency_ms": 1448,
      "parent_url": null,
      "links_found": 24,
      "queue_stats": {
        "total": 15,
        "failed": 0,
        "pending": 14,
        "completed": 1
      },
      "links_queued": 14,
      "processed_pages": 1
    },
    "created_at": "2025-02-06T11:36:13.638Z"
  },
  {
    "level": "debug",
    "message": "Selected URL \"https://scrapeninja.net/\" for processing",
    "metadata": {
      "url": "https://scrapeninja.net/",
      "depth": 1,
      "runId": 2,
      "queueId": 18
    },
    "created_at": "2025-02-06T11:36:13.655Z"
  },
  {
    "level": "debug",
    "message": "Fetching page \"https://scrapeninja.net/\" using ScrapeNinja",
    "metadata": {
      "url": "https://scrapeninja.net/",
      "runId": 2
    },
    "created_at": "2025-02-06T11:36:13.673Z"
  },
  {
    "level": "debug",
    "message": "Sending request to https://scrapeninja.p.rapidapi.com/scrape",
    "metadata": {
      "url": "https://scrapeninja.net/",
      "runId": 2,
      "engine": "scrape",
      "marketplace": "rapidapi"
    },
    "created_at": "2025-02-06T11:36:13.677Z"
  },
  {
    "level": "debug",
    "message": "ScrapeNinja response info for \"https://scrapeninja.net/\"",
    "metadata": {
      "url": "https://scrapeninja.net/",
      "runId": 2,
      "pageTitle": "ScrapeNinja Web Scraping API: Turns websites into data, on scale. 🚀",
      "statusCode": 200
    },
    "created_at": "2025-02-06T11:36:14.862Z"
  },
  {
    "level": "debug",
    "message": "Queued 0 new URLs for crawling",
    "metadata": {
      "runId": 2,
      "parentUrl": "https://scrapeninja.net/",
      "linksQueued": 0,
      "currentDepth": 1
    },
    "created_at": "2025-02-06T11:36:14.885Z"
  },
  {
    "level": "info",
    "message": "Successfully processed page \"https://scrapeninja.net/\"",
    "metadata": {
      "url": "https://scrapeninja.net/",
      "depth": 1,
      "run_id": 2,
      "status": "completed",
      "max_pages": 5,
      "latency_ms": 1108,
      "parent_url": "https://scrapeninja.net",
      "links_found": 24,
      "queue_stats": {
        "total": 15,
        "failed": 0,
        "pending": 13,
        "completed": 2
      },
      "links_queued": 0,
      "processed_pages": 2
    },
    "created_at": "2025-02-06T11:36:14.901Z"
  },
  {
    "level": "debug",
    "message": "Selected URL \"https://scrapeninja.net/docs/n8n/\" for processing",
    "metadata": {
      "url": "https://scrapeninja.net/docs/n8n/",
      "depth": 1,
      "runId": 2,
      "queueId": 19
    },
    "created_at": "2025-02-06T11:36:14.916Z"
  },
  {
    "level": "debug",
    "message": "Fetching page \"https://scrapeninja.net/docs/n8n/\" using ScrapeNinja",
    "metadata": {
      "url": "https://scrapeninja.net/docs/n8n/",
      "runId": 2
    },
    "created_at": "2025-02-06T11:36:14.933Z"
  },
  {
    "level": "debug",
    "message": "Sending request to https://scrapeninja.p.rapidapi.com/scrape",
    "metadata": {
      "url": "https://scrapeninja.net/docs/n8n/",
      "runId": 2,
      "engine": "scrape",
      "marketplace": "rapidapi"
    },
    "created_at": "2025-02-06T11:36:14.935Z"
  },
  {
    "level": "debug",
    "message": "ScrapeNinja response info for \"https://scrapeninja.net/docs/n8n/\"",
    "metadata": {
      "url": "https://scrapeninja.net/docs/n8n/",
      "runId": 2,
      "pageTitle": "Using ScrapeNinja with n8n | ScrapeNinja",
      "statusCode": 200
    },
    "created_at": "2025-02-06T11:36:18.783Z"
  },
  {
    "level": "debug",
    "message": "Queued 9 new URLs for crawling",
    "metadata": {
      "runId": 2,
      "parentUrl": "https://scrapeninja.net/docs/n8n/",
      "linksQueued": 9,
      "currentDepth": 1
    },
    "created_at": "2025-02-06T11:36:18.816Z"
  },
  {
    "level": "info",
    "message": "Successfully processed page \"https://scrapeninja.net/docs/n8n/\"",
    "metadata": {
      "url": "https://scrapeninja.net/docs/n8n/",
      "depth": 1,
      "run_id": 2,
      "status": "completed",
      "max_pages": 5,
      "latency_ms": 3806,
      "parent_url": "https://scrapeninja.net",
      "links_found": 23,
      "queue_stats": {
        "total": 24,
        "failed": 0,
        "pending": 21,
        "completed": 3
      },
      "links_queued": 9,
      "processed_pages": 3
    },
    "created_at": "2025-02-06T11:36:18.833Z"
  },
  {
    "level": "debug",
    "message": "Selected URL \"https://scrapeninja.net/cdn-cgi/l/email-protection\" for processing",
    "metadata": {
      "url": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "depth": 1,
      "runId": 2,
      "queueId": 31
    },
    "created_at": "2025-02-06T11:36:18.860Z"
  },
  {
    "level": "debug",
    "message": "Fetching page \"https://scrapeninja.net/cdn-cgi/l/email-protection\" using ScrapeNinja",
    "metadata": {
      "url": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "runId": 2
    },
    "created_at": "2025-02-06T11:36:18.877Z"
  },
  {
    "level": "debug",
    "message": "Sending request to https://scrapeninja.p.rapidapi.com/scrape",
    "metadata": {
      "url": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "runId": 2,
      "engine": "scrape",
      "marketplace": "rapidapi"
    },
    "created_at": "2025-02-06T11:36:18.885Z"
  },
  {
    "level": "debug",
    "message": "ScrapeNinja response info for \"https://scrapeninja.net/cdn-cgi/l/email-protection\"",
    "metadata": {
      "url": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "runId": 2,
      "pageTitle": "Email Protection | Cloudflare",
      "statusCode": 200
    },
    "created_at": "2025-02-06T11:36:20.005Z"
  },
  {
    "level": "debug",
    "message": "Queued 0 new URLs for crawling",
    "metadata": {
      "runId": 2,
      "parentUrl": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "linksQueued": 0,
      "currentDepth": 1
    },
    "created_at": "2025-02-06T11:36:20.030Z"
  },
  {
    "level": "info",
    "message": "Successfully processed page \"https://scrapeninja.net/cdn-cgi/l/email-protection\"",
    "metadata": {
      "url": "https://scrapeninja.net/cdn-cgi/l/email-protection",
      "depth": 1,
      "run_id": 2,
      "status": "completed",
      "max_pages": 5,
      "latency_ms": 1113,
      "parent_url": "https://scrapeninja.net",
      "links_found": 4,
      "queue_stats": {
        "total": 24,
        "failed": 0,
        "pending": 20,
        "completed": 4
      },
      "links_queued": 0,
      "processed_pages": 4
    },
    "created_at": "2025-02-06T11:36:20.046Z"
  },
  {
    "level": "debug",
    "message": "Selected URL \"https://scrapeninja.net/curl-to-php\" for processing",
    "metadata": {
      "url": "https://scrapeninja.net/curl-to-php",
      "depth": 1,
      "runId": 2,
      "queueId": 30
    },
    "created_at": "2025-02-06T11:36:20.062Z"
  },
  {
    "level": "debug",
    "message": "Fetching page \"https://scrapeninja.net/curl-to-php\" using ScrapeNinja",
    "metadata": {
      "url": "https://scrapeninja.net/curl-to-php",
      "runId": 2
    },
    "created_at": "2025-02-06T11:36:20.082Z"
  },
  {
    "level": "debug",
    "message": "Sending request to https://scrapeninja.p.rapidapi.com/scrape",
    "metadata": {
      "url": "https://scrapeninja.net/curl-to-php",
      "runId": 2,
      "engine": "scrape",
      "marketplace": "rapidapi"
    },
    "created_at": "2025-02-06T11:36:20.090Z"
  },
  {
    "level": "debug",
    "message": "ScrapeNinja response info for \"https://scrapeninja.net/curl-to-php\"",
    "metadata": {
      "url": "https://scrapeninja.net/curl-to-php",
      "runId": 2,
      "pageTitle": "Convert cURL to PHP Web Scraper",
      "statusCode": 200
    },
    "created_at": "2025-02-06T11:36:21.557Z"
  },
  {
    "level": "debug",
    "message": "Queued 0 new URLs for crawling",
    "metadata": {
      "runId": 2,
      "parentUrl": "https://scrapeninja.net/curl-to-php",
      "linksQueued": 0,
      "currentDepth": 1
    },
    "created_at": "2025-02-06T11:36:21.583Z"
  },
  {
    "level": "info",
    "message": "Successfully processed page \"https://scrapeninja.net/curl-to-php\"",
    "metadata": {
      "url": "https://scrapeninja.net/curl-to-php",
      "depth": 1,
      "run_id": 2,
      "status": "completed",
      "max_pages": 5,
      "latency_ms": 1451,
      "parent_url": "https://scrapeninja.net",
      "links_found": 6,
      "queue_stats": {
        "total": 24,
        "failed": 0,
        "pending": 19,
        "completed": 5
      },
      "links_queued": 0,
      "processed_pages": 5
    },
    "created_at": "2025-02-06T11:36:21.599Z"
  },
  {
    "level": "info",
    "message": "Reached maximum pages (5), stopping crawler",
    "metadata": {
      "maxPages": 5,
      "processedPages": 5
    },
    "created_at": "2025-02-06T11:36:21.605Z"
  },
  {
    "level": "info",
    "message": "Crawler process completed for run \"2\"",
    "metadata": {
      "maxPages": 5,
      "processedPages": 5
    },
    "created_at": "2025-02-06T11:36:21.647Z"
  }
]

API-Powered vs. Local Extraction

A key design choice was splitting remote and local responsibilities:

• Remote Rendering with ScrapeNinja API: For speed or when a full browser environment is required, the /scrape or /scrape-js endpoints handle the page fetch.

• Local “Extract Primary Content” Operation: The actual text extraction runs locally inside n8n, which offers control over processing and reduces load on remote systems.

I want to note that currently you need a ScrapeNinja API key (ScrapeNinja is a cloud SaaS) to launch n8n web crawler - but you do not need ScrapeNinja API key for other locally-running operations. ScrapeNinja has a free plan.

For instance, "Clean up HTML content" and "Extract ..." operations do not require ScrapeNinja subscription, while"Scrape ..." and "Crawl" operations require ScrapeNinja API key.

This setup balances efficiency and flexibility, letting you tailor workflows to your specific requirements.

Getting Started with n8n web crawler: A Step-by-Step Guide

To set up the ScrapeNinja crawler in n8n:

1. Open n8n Dashboard: Go to Settings → Community Nodes.

2. Install the Node: Find and install n8n-nodes-scrapeninja, then restart n8n if prompted.

3. Check the Version: Use 0.4.0 or later for the ScrapeNinja n8n node. You’ll also need your ScrapeNinja API key.

4. Configure Your Workflow: Insert the crawler node, define the seed URL, and set parameters like the maximum page count.

5. Run and Monitor: Execute the workflow, then review logs via JSON output or the Postgres crawler_logs table in Supabase.

I had it running smoothly on my self-hosted instance in a short time.

Building Powerful Knowledge Bases

One standout application is compiling large documentation sets into a unified Markdown file:

• Your Own Docs: I tested it on ScrapeNinja documentation to confirm its capabilities.

• Aggregated Content: Core text from each page is combined into a single file, ideal for LLM ingestion.

• Enhanced LLM Responses: A dedicated knowledge base helps an LLM produce precise, contextual answers that generic training data cannot match.

Monitoring, Debugging, and Maintaining Transparency

Web crawling can take minutes or longer, so good monitoring is essential:

• JSON Output Logs: Every step is logged in real time, making it easy to integrate with external tools or review later.

• Postgres Crawler Logs: Logs are stored in a Postgres database (via Supabase) for persistent records of all operations.

• Resource Monitoring: Memory usage and performance are checked continuously to handle multiple pages in parallel without excessive load.

This has been crucial for debugging and performance reviews. In one test, memory stayed under 119 MB even with ten concurrent browser sessions.

Real-World Applications and Future Directions

Though initially focused on LLM knowledge bases, the crawler can be adapted for:

• E-commerce: Aggregate product data, reviews, and pricing from large catalogs.

• Content Aggregation: Collect articles or forum posts for comprehensive data sets.

• Market Analysis: Scrape competitor sites for trends and product insights.

• Academic Research: Pull data from journals, conferences, or public repositories.

Future improvements may include better error handling, more advanced parallel processing, and closer integration with vector databases for RAG. Community feedback is welcome to shape the project’s roadmap.

The ScrapeNinja Recursive Web Crawler for n8n is a result of practical insights from working on real-world scraping tasks. By combining the ScrapeNinja API (via /scrape or /scrape-js) with local extraction, it creates an efficient path to build domain-specific knowledge bases and improve LLM performance.

Try it on your self-hosted n8n setup. Install the node, configure it with your API key, and convert any web pages into structured, actionable data. Your feedback is welcome as I refine this project.

I am a big fan of n8n and I am using it for a lot of my projects. I love that it provides a self-hosted version and this self-hosted version is not paywalled like if often happens with so-called "open core" products which just use "open source" as a marketing term.

Web scraping in n8n can be both simple and sophisticated, depending on your approach and tools.

In this blog post, I will explore two ways of scraping: basic HTTP requests and advanced scraping techniques using ScrapeNinja n8n integration. Whether you're building a price monitoring system or gathering competitive intelligence, this guide will help you choose the right approach.

n8n is a powerful low-code automation platform that allows building complex workflows without writing code. While it's similar to Zapier and Make.com, it offers more technical flexibility and can be self-hosted, making it perfect for data-intensive operations like web scraping.
Compared to custom-built web scrapers where you can often find yourself digging through cryptic text logs, n8n provides a very nice observabiliy out of the box: "Executions" tab on each scenario allows to explore how everything goes and where/how errors, if any, happen. This applies not just to web scraping scenarios, but any scenario in n8n, of course, but if you ever scraped some real website you know how often web scrapers break and require maintenance - this indeed can be painful!

ScrapeNinja is a web scraping API built to mitigate common challenges in modern web scraping. It provides high-performance scraping capabilities with features like real browser TLS fingerprint emulation, proxy rotation, and JavaScript rendering. All this complexity is packed into two simple API endpoints: /scrape?url=<target_website> and /scrape-js?url=<target_website - there are plenty of params to control the behaviour of ScrapeNinja but n8n node simplifies our life because most of these API params have UI controls and it's rather easy to find out how it works.

The integration between n8n and ScrapeNinja combines the power of workflow automation with enterprise-grade scraping capabilities.

HTTP Node

Understanding the Basics

The HTTP node is your gateway to web scraping (and HTTP requests in general) in n8n. It's the Swiss Army knife of HTTP requests, capable of GET, POST, PUT, and other methods. While it might seem straightforward, there's more than meets the eye when it comes to its configuration and retry capabilities.

Request Handling

Once you start using the HTTP node for real-world web scraping, you quickly realize it's a pretty nice tool with a lot of available settings.

It works fine if you're scraping your own 10-page website. But if you try scraping another website, you'll likely run into some obstacles:

• Default settings don’t make much sense. The n8n HTTP node’s default user agent is axios/xx. I recommend setting it to something more realistic, like the latest Chrome version (copy it from Chrome Dev Tools -> "Copy as cURL" or visit this website and copy it from there).
• HTTP node is concurrent by default. Unless you enable the "Batching" option, the HTTP node will send all your requests simultaneously. Not very intuitive—and a quick way to get your IP banned!
  
• No TLS fingerprinting bypass. All requests are made via the Axios npm package, meaning they share the default Node.js TLS fingerprint. If the target website uses Cloudflare bot protection, it will detect that your request isn’t from a real browser (even with a proper user agent) and return a 403 — no matter what IP you're using!
• No proxy rotation. The HTTP node doesn't support proxy rotation. Not surprising, since it’s a relatively advanced feature found in dedicated web scraping tools, and the HTTP node was never designed for that purpose.

Response Handling

The HTTP node provides several important response configuration options:

• Response Format: Automatically detects and parses various formats (JSON, XML, etc.)
• Response Headers: Option to include response headers in the output
• Response Status: Can be configured to succeed even when status code is not 2xx
• Never Error: When enabled, the node never errors regardless of the HTTP status code
  

Retry Mechanics

The HTTP node comes with built-in retry functionality that can be a lifesaver when dealing with unstable connections or rate-limited APIs. Like all n8n nodes, it includes a generic retry mechanism for handling failures. However, this basic retry system is often too simplistic for real-world web scraping, where you need granular control over retry conditions based on specific response content or status codes.

Here's what you need to know about HTTP node retries:

• Retry Options: You can set both the number of retries and the wait time between attempts
• Generic Nature: The retry mechanism is designed for general HTTP failures, not specialized scraping scenarios

However, these retries are "dumb" - they use the same IP address and request fingerprint, which often isn't enough for serious scraping operations.

Notable Feature: cURL Command Import

One of the most useful features is the ability to import cURL commands directly. This makes it incredibly easy to replicate browser requests - just copy the cURL command from your browser's developer tools and paste it into n8n. I have encountered some failures of cURL import feature, due to outdated npm library which is used by n8n to parse cURL syntax under the hood, but it was happening on a relatively complex requests copy&pasted from Chrome Dev Tools console, chances that you will enounter these is pretty low, at least on simpler requests.

Proxy Support Challenges

While the HTTP node does support proxies, there are known issues with proxies. As mentioned in the n8n community forum and this Github issue, you might get troubles with proper proxy support because underlying npm library which n8n uses for HTTP node (Axios) does not support proxies which require https connection via CONNECT method properly.

ScrapeNinja Node

How It Works

The ScrapeNinja n8n community node is a set of tools designed for web scraping and content extraction.

Some operations retrieve content from target websites, while others simplify content extraction from website responses.
Read more in the ScrapeNinja docs.

Request Flow

For content retrieval, the request flow works as follows:

[your n8n self-hosted instance] → [HTTP Node Helper] → [ScrapeNinja API] → [Target Website]

The ScrapeNinja API includes two powerful scraping engines designed to work reliably while bypassing various anti-scraping protections. All of the requests to target websites are completed via rotating proxies.

The Power of Advanced Scraping

ScrapeNinja transforms n8n from a basic scraping tool into a serious web harvesting platform. It's not just another HTTP client - it's a specialized scraping service that handles the complex challenges of modern web scraping. As a SaaS solution, it requires an API key and offers both free and paid plans to suit different scraping needs.

Core Capabilities

The official ScrapeNinja node for n8n brings several nice capabilities:

• Chrome-like TLS fingerprinting
• Automatic proxy rotation with multiple countries of proxies
• JavaScript rendering
• Built-in HTML parsing (JS Extractors)
• Cloudflare bypass capabilities
  
• Built-in, locally executed crawler, which traverses through website links and retrieves all pages content recursively

Response Structure

ScrapeNinja always returns a consistent JSON structure, making it easy to process responses in your workflows:

{
  "info": {
    "version": "2",
    "statusCode": 200,
    "statusMessage": "",
    "headers": {
      "server": "nginx",
      "date": "Sat, 25 Jan 2025 16:20:22 GMT",
      "content-type": "text/html; charset=utf-8",
      // ... other headers
    },
    "screenshot": "https://scrapeninja.net/screenshots/abc123.png" // when screenshot option is enabled
  },
  "body": "<html>... scraped content ...</html>",
  "extractor": {  // when JS extractor is provided
    "result": {
      "items": [
        [
          "some title",
          "https://some-url",
          "pr337h4m",
          24,
          "2025-01-25T14:47:33",
          // ... extracted data
        ],
        // ... more items
      ]
    }
  }
}

This structured response provides:

• Complete request metadata in the info object
• Original response headers
• HTTP status information
• Screenshot URL (when enabled)
• Raw response body
• Structured data from JS extractors (when JS extractor code is provided in request)

JavaScript Extractors

One of ScrapeNinja's most powerful features is its JavaScript extractor functionality. These are small JavaScript functions that run in the ScrapeNinja cloud to process and extract structured data from scraped content. Here's what makes them special:

• Cloud Processing: Extractors run in ScrapeNinja's cloud environment, reducing load on your n8n instance
• Cheerio Integration: Built-in access to the Cheerio HTML parser for efficient DOM manipulation
• Clean JSON Output: Perfect for no-code environments where structured data is essential
• Reusable Logic: Write once, use across multiple similar pages. Also both ScrapeNinja /scrape and /scrape-js engines are using the same extractors, so switching to real browser rendering later when you suddenly decide to do it, is easy.

AI-Powered Extractor Generation

ScrapeNinja provides a cool Cheerio Sandbox with AI capabilities that helps you create extractors:

1. Automated Code Generation: Paste your HTML and describe what you want to extract
2. Interactive Testing: Test your extractors in real-time against sample data
3. AI-Assisted Improvements: Get suggestions for improving your extractors
4. Optimization Features: The system automatically handles HTML cleanup and compression

Feature Comparison: HTTP Node vs ScrapeNinja Node

Here's a detailed comparison of features between the two nodes:

Setting Up ScrapeNinja in n8n

Getting started with ScrapeNinja in n8n is straightforward:

1. Install the community node (n8n-nodes-scrapeninja)
2. Configure your API credentials (supports both RapidAPI and APIRoad)
3. Start using advanced scraping features

Read more on n8n community forum

Real-World Scraping Scenarios

Let's look at some common scenarios where n8n can be used for web scraping:

AI agent that can scrape webpages

This is an example of real-world workflow where Scrapeninja is probably a better fit compared to HTTP node.

https://n8n.io/workflows/2006-ai-agent-that-can-scrape-webpages/

If you want to get better in n8n, it is useful to check how workflow author is using n8n tools to cleanup HTML so it can be ingested into LLM context, and n8n workflow execute node to split scenario into smaller isolated parts. The HTML cleanup looks rather simplistic and I think using some external API like Article Extractor and Summarizer /extract endpoint may be more bulletproof.

E-commerce Data Collection

When scraping e-commerce sites, you often need to:

• Handle JavaScript-rendered content
• Navigate through pagination
• Extract structured data from complex layouts
• Bypass anti-bot measures

ScrapeNinja handles all these challenges while maintaining a high success rate.

Social Media Monitoring

Social platforms are notoriously difficult to scrape due to:

• Sophisticated bot detection
• Dynamic content loading
• Rate limiting
• Complex authentication requirements

The ScrapeNinja node's advanced fingerprinting and proxy rotation make these challenges manageable.

n8n caveat: HTTP request concurrency control

Let's say you are building a n8n screnario where you get website URLs from Google Sheet and request each URL via HTTP node or ScrapeNinja node, and put the HTML of the response back into Google Sheet. The naive approach would be to just add "Google Sheets (get all rows)" n8n node and "HTTP node" right after it. Let's say there are 100 URLs in your Google sheet. It is not obvious, but in this case n8n will run 100 HTTP requests at the same time. This can easily overload both the target website and you n8n instance. Even worse, if you want to store HTTP results somewhere, even if one of these HTTP requests fails, all the 100 HTTP requests results will be lost and next n8n node won't be executed. To mitigate this, always use built-in n8n Loop node when dealing with more than 10 external APIs or HTTP requests. Do not forget to put node which stores results in the same loop.

Best Practices for Production Scraping

When deploying scraping workflows to production, consider these tips:

1. Error Handling

   • Implement comprehensive error catching
   • Use n8n's error workflows
   • Monitor scraping success rates
   • Use n8n "Executions" tab on a scenario to see what is happening

2. Rate Limiting

   • use Loop n8n node to limit concurrency
   • Respect website terms of service
   • Implement appropriate delays
   • Use ScrapeNinja's built-in rate limiting features

3. Data Validation

   • Verify extracted data integrity
   • Handle missing or malformed data gracefully
   • Implement data cleaning workflows

Conclusion

While n8n's HTTP node is perfect for basic web requests, serious scraping operations benefit significantly from ScrapeNinja integration. The combination provides a powerful, reliable, and scalable solution for modern web scraping challenges.

Remember: successful web scraping isn't just about getting the data - it's about getting it reliably, ethically, and efficiently. With n8n and ScrapeNinja, you have the tools to do just that.

Despite progress, I still spend a lot of time copying and pasting between tools. While VS Code Copilot’s code edits and AI autocomplete help bridge the gap between VS Code with Copilot and tools like Cursor.sh, they’re far from seamless. I also continue to spend time setting up my development environment, configuring databases, managing DNS entries in the Cloudflare control panel, and setting up Nginx virtual hosts.

That's why I am so excited to see lovable.dev and bolt.new launched in 2024.

Lovable.dev and Bolt.new represent a new wave of AI products for developers. The major difference compared to the "old-school" approach (can we call the ChatGPT app old-school now?) of code generation via ChatGPT or Copilot is that they:

• Allow a very quick start and smooth development progress directly from a web browser.
• Mitigate the pain of website deployment.
• Provide a real sandbox for code execution.
• Include a ready-to-use, opinionated database solution.

With these features, LLM models can finally execute code in a controlled environment, interact with databases, and potentially free us from manual database design and endless copy-pasting of code.

Lovable.dev

Lovable.dev is a SaaS built by authors of open-source project https://github.com/gpt-engineer-org/gpt-engineer (50k+ of stars!)

Here is the Lovable.dev CEO and founder:

AntonOsika - Overview

Founder & CTO at Depict (2x top YC startup) Founder & CEO at Lovable.dev, building the last piece of software. Also into Particle Physics. - AntonOsika

GitHub

Looking at the GPT Engineer repo commits, it is obvious that the open source development of the project was active during 2023 and effectively stoppped approximately by the summer of 2024.

Reading GPTengineer code is recommended if you want to understand better how open-source AI code generation and execution works (but, if you hope to find some interesting code sandboxing techniques which could be later re-used under the hood of Lovable, you will be disappointed - there is no any sandboxing here to be seen). Aider is another open-source tool which is actively maintained and allows to leverage AI for coding - I recommend to explore it as well (and read their awesome blog where they analyze and benchmark how recent LLM models write real-world code).

Ok, lets get back to Lovable!

Pricing

Paid plans start at 20 USD / mo.

Free plan is very restrictive - it only has 5 messages allowed per day. I exhausted my limit without building anything resembling a production project.

Starter plan for 20 USD is also pretty restrictive: 100 edits per month. But this should be enough to build something meaningful, especially if you are careful about your prompting!

Code execution: Fly.io

Lovable needs to execute the code which was generated by the AI, to build the preview of the project. Obviously, this code is untrusted and requires some isolated environment to be run in. I have some previous posts on Javascript sandboxing because I had a very similar task of untrusted code isolation while building my ScrapeNinja Web Scraping API where I allow customers to write some JS code for HTML extractors which are executed on ScrapeNinja servers.

Since GPTEngineer github repo does not provide any information regarding sandboxing of the generated code, we have to dig deeper into Chrome Dev Tools and Lovable network requests.

Okay, this was easy: the response headers of Lovable show us that they use Fly.io containers under the hood. And Fly.io leverages Firecracker MicroVMs technology, read more in their awesome architecture post.

Database

Lovable has freshly built-in Supabase connector. It provides a convenient way to tie your Lovable project to Supabase (which is a self-hostable PostgreSQL-powered Firebase alternative). There is no obvious way to attach self-hosted Supabase to Lovable, only cloud version is easy to connect. This means that Supabase pricing also applies to you now.

Supabase is seriously awesome – but it can become expensive if you suddenly realize you need to scale it.

I enjoyed how I asked "please add user auth" and it was added by Lovable using modern Supabase React SDK. I hate adding auth and it could easily take me a couple of hours. With Lovable it takes just 1 minute!

Code editing: NONE

Supabase does not have code editing built in. I mean, you can ask AI to change your code, but there is no way to really edit the files by yourself at the moment when you chat with AI. The only way to edit is to publish the project into Github and launch in-browser github.dev.

But you cannot build and run the project here.

Deployment: Vercel & Netlify

Lovable now allows to easily deploy to Vercel and Netlify.

Experience

I tried Lovable.dev and Bolt.new with identical (poorly written) prompt:

create crypto portfolio tracker. dark theme. sparkline on full width of the coin container with semi-transparent gradient. draggable coins to sort them. BTC and SOL by default, ability to add SOL, HYPE, TRX, BNB

My understanding is Lovable is a bit more opionated in terms of tech stack: it uses React with Shadcn for frontend. The first response provided a working code with minor glitches. Then, I connected to Supabase and asked it to add Supabase auth and it managed to complete this task.

Bolt.new

Bolt.new is a project by Stackblitz.

Open-source version of Bolt: https://github.com/stackblitz-labs/bolt.diy/

Code editing: StackBlitz

StackBlitz is the collaborative browser-based IDE for web developers. This is clearly what gives Bolt an edge compared to Lovable: these guys know how to provide you a real IDE (with terminal) and working npm install – right in your browser.

Code execution: Webcontainers.io

Since webcontainers.io is an integral part of Bolt.new, let's take a look how it works. Webcontainers is a project by Stackblitz, it is essentially a bunch of WASM (Webassembly) files which do real magic: they allow to emulate real OS with Node.js in it. Right in your browser. Read more here:

Introducing WebContainers: Run Node.js natively in your browser

Today we’re excited to announce WebContainers, a new type of WebAssembly-based operating system that boots instantly and enables Node.js environments to run natively in-browser.

Eric Simons CEO

So... while Lovable.dev spends their resources to boot and run a VM in Fly.io to execute code generated by their AI engine, Bolt leverages Webassembly tech to use your own browser and your own machine to run an operation system in a browser.

I have opened Bolt.new terminal and typed npm i axios in it. My Chrome Dev tools showed the axios download. Let me repeat this: entire OS network stack is emulated in browser. npm "thinks" that it is executed in regular server environment. But the actual file download happens via XHR request in my Chrome Dev tools. I still find this hard to believe.

This is a Github issue in Node.js github repo which sheds some light how all this was implemented by Stackblitz team and which corners that had to cut. They had to implement certain parts of Node.js and built their own, very restrictive, wrappers for others, like networking. The sentiment of some Github commenters is rather negative because they think that Stackblitz team developed a proprietary technology on a foundation of open-source products. The issue also contains "an open source implementation of Webcontainers" which I have not checked yet.

StackBlitz code execution restrictions

I have quickly encountered Stackblitz code sandbox restrictions in my second Bolt.new project. I have decided to build a simple proxy checker - a user can enter a proxy URL in "http://user:pw@host:port" format, and the API backend uses npm package like https://www.npmjs.com/package/http-proxy-agent to check if the supplied proxy is functional and what is the latency. Bolt.new admitted it cannot run this - you simply cannot implement this in Chrome network stack - so it's not possible to run this in Stackblitz IDE code execution environment!

Database

I am writing this post on Dec 20 of 2024. And on Dec 19, Bolt.new released their new connector to Supabase.

Just a week ago, Bolt.new team recommended Firebase as a database for their AI. I think they did the right thing. I like Supabase more, compared to Firebase (because I like self-hosted and I like SQL databases compared to NoSQL) so I am excited by official Supabase integration.

Pricing

I think Bolt.new pricing is better compared to Lovable.dev. They calculate the tokens spent on input and output, and give 1M of tokens for free. 20 USD plan gives you 10M tokens. Due to Webassembly sandbox, I think that Bolt.new internal costs should be much smaller compared to Lovable and I think their customer-facing pricing model is better as well.

Experience

I loved Bolt output: the crypto portfolio HTML was better. No minor glitches, almost perfect.

Conclusions

We live in incredible times. By 2024, fully manual coding stopped making sense. The progress of LLM models and AI tooling makes writing code by hand feel strange and outdated. I can’t imagine creating new products without AI anymore. Tools like Bolt.new and Lovable.dev mark another major step into the AI coding era.

If you ask me which one I’d choose out of these two, I was initially hesitant - until I learned that Bolt.new released the Supabase connector. Now I think I’d prefer Bolt because:

 • It feels more mature in terms of UX.

 • It provides a real editor and terminal.

 • It offers better pricing.

 • It leverages cutting-edge tech for code sandboxing (WebAssembly) - though it comes with certain restrictions, as I mentioned earlier. If you need real server-side code execution for your project, Lovable is a better choice due to Fly.io VMs under the hood.

What's your experience?

UPD 2025: more of my n8n blog posts:

• n8n vs real coding for SaaS products
• Web scraping in n8n

My technical background heavily influences my no-code product choices: I don't like it when low-code products try to hide all the complexity of real web development or when code execution is poorly implemented. In a perfect world, I would like to be able to create projects starting from no-code but have the ability to write some fancy JS here and there when I decide to get my hands dirty.

Both Zapier and Make obviously target less technical audience segments, so I gave high praise to Pipedream for targeting developers like me and making code a first-class citizen in their low-code platform. In this blog post, I will describe my experience using the n8n self-hosted engine and occasionally compare it with Make, Zapier, and Pipedream.

Self-hosted services are great in 2024

I have recently wrote a blog post on awesome self hosted products I use almost every day. Self hosted is awesome if you know a bit of Docker. n8n is another example of a self-hosted instrument which provides immense value for exactly zero dollars of recurring cost.

n8n: self-hosted low-code powerhouse

I was a bit hesitant setting up n8n as I am a paying customer of Make.com which is a great platform. But, a lot of ScrapeNinja customers (which is my bootstrapped SaaS API for web scraping) were recently coming with basic support requests from n8n community, asking how my API could be used with n8n, and I wanted to help them, so I had to dive into it.

[UPD Jan 2025]: I have launched a ScrapeNinja n8n integration node for web scraping, read more.

About n8n

n8n is a workflow automation tool. Zapier and Make are famous n8n competitors you probably know. The best thing about n8n is that the whole source code of this massive project is available on Github. Now wonder these guys have 41k+ Github stars - this is a big Typescript codebase and you can learn a lot by exploring it. For example, take a look how they are sandboxing external JS code - they are using a forked vm2 which I have reviewed a few years ago.

Installation (self-hosted, Docker)

I launched n8n with literally a single bash command on my remote Hetzner server, from my VS Code (with VS Code Remote extension). I have chosen Docker n8n setup for my installation.

The whole setup took like... 30 seconds. VS Code also automatically forwarded remote port of the server to my localhost, so I opened http://localhost:5678 in Chrome and saw n8n signup screen which allowed me to create an admin user and log in. So smooth!

Final self-hosted setup

I played with this 30-sec setup of n8n for an hour and realized it can be useful for me on a longer term. Since I installed n8n on a remote Hetzner machine (receive 20 EUR by registering on Hetzner, via my referral link), I've created a subdomain for n8n for one of my domains over Cloudflare and also set up nginx as a reverse proxy to forward all requests from https://myn8n.scrapeninja.net to n8n running instance. This was another time investment of around 20 minutes.

Since I am using Cloudflare, it manages all HTTPS certificates for me.

Here is my nginx reverse proxy config which I took from n8n website:

server {
    listen 80;
    server_name n8n.scrapeninja.net;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name n8n.scrapeninja.net;
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
    location / {
        proxy_pass http://localhost:5678;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade; # websocket support
        proxy_set_header Connection "Upgrade";  # websocket support
        proxy_set_header Host $host;
        chunked_transfer_encoding off;
        proxy_buffering off;
        proxy_cache off;
    }
}

Code as a first class citizen

My biggest nice surprise so far is that n8n embraces JS everywhere: in basic inputs and as a separate code nodes.

So, for instance, if I got a big HTTP response with JSON data, I can put just part of this JSON, serialized, back into my DB (NocoDB or Google Sheets) by marshalling it via JSON.stringify called within the n8n node form input. This is really convenient and I don't need to learn new syntax and functions for simple data transformations (like, in Make).

And if I need to do some heavy processing, I just create a Code node:

Do you see the console.log statement? It is actually printed in Chrome Dev Tools console when I click "Execute workflow" button or "Test step" button.

And here is a huge feature: external npm packages can be launched inside n8n JS code node (only in self-hosted setup, you can't do this in n8n Cloud, at least not in april of 2024). You can't do this in: Make, Zapier. Pipedream is on par with n8n (or even better) in terms of JS code execution.

But, it cannot read/write to file system or network.

Read more about n8n Code node in the documentation.

Not just Javascript, but also Python

n8n allows to execute Python (since Feb 2023). I should note that due to overhead on launching Python environment, for now, it is possible only in code blocks, not in every input field, so JS has to be sprinkled across your workflow even in case you hate it - this is totally not a problem for me - as I am perfectly fine with JS!

Python engine uses Pyodide under the hood, which transpiles the Python code into Webassembly, which is later executed via node.js (so internally it's still executed on V8 Javascript environment).

Performing HTTP requests and web scraping in n8n

Obviously, n8n provides an action to perform HTTP requests. I liked how there is an "import from cURL" link right there, and it works fine for ScrapeNinja API at RapidAPI marketplace. RapidAPI marketplace allows to copy&paste cURL code - so I just put it into n8n:

So easy! Of course if you want to make many requests to ScrapeNinja across your n8n workflows, you should create a new "credential" for ScrapeNinja API key instead of having it hardcoded into particular action.

[UPD Jan 2025] You don't need to fight with ScrapeNinja API anymore to use it with n8n! I have built an official n8n ScrapeNinja integration node, which is seriously awesome and includes a lot of free open source features like smart content extraction and html-to-markdown converter. Read more about built-in recursive web crawler as well.

Con of self-hosted: Google oAuth client

Self-hosted is great, but major issue of having your "glue" automation platform self-hosted is that you need to deal with oAuth process of tier1 players like Google. Let's say you want to read and write data to your Google Sheets. In Make, which is a cloud SaaS, all the dirty work of creating Google oAuth Client has been done by Make maintainers: you just authorize in your Google account and start using Google products.

For n8n, you need to spend around 10-20 minutes configuring Google Console access and setting up an OAuth client. This includes configuring the consent screen and authorizing the "n8n" app to access certain APIs. Nothing too awful, here is the n8n tutorial.

However, for me, the experience of creating a new OAuth client in Google Console has consistently been a pain. Even after doing it five times, I still don't enjoy the process. It's much more cumbersome than just copying and pasting an API key (e.g., from the OpenAI dashboard).

n8n pricing: self-hosted

n8n has two editions: Community edition and Enterprise edition. Both of them could be self-hosted. Let me keep it concise: The difference in features of Community vs Enterprise is thin, so when you get the most basic Community edition of n8n and install it via docker compose on your 3 EUR Hetzner or Vultr server, you get (very roughly) 95% of n8n Enterprise Cloud functionality. Sic!

Here is the list of features that Community edition lacks - the list is rather short, and some of these lacking features can be mitigated using community nodes, for example, I think that https://www.npmjs.com/package/n8n-nodes-globals can be used as an alternative to global custom variables.

n8n is a "fair use" licensed product. Read their license here. Essentially, if you use n8n as a self-hosted product ("community edition"), it is free for you (unless you are building some sort of SaaS product where n8n is exposed to your end customers where they can enter their authentication details into n8n nodes - very rare use case). So, if you automate something for your company, or automate something for your customers, self-hosted n8n is FREE for you. You just pay for your server. I still see that a lot of people on Reddit who are asking "okay, seriously, so how much this self-hosted license costs?". I understand: it's hard to believe that such polished automation product can be free when closed-source cloud platforms with better marketing charge their customers a fortune every month. Check this thread regarding self hosted offering.

n8n pricing: cloud

It is possible to use n8n Cloud as another SaaS.

At the time of writing, Make offered 9USD/mo plan while smallest n8n cloud plan is 20USD - both of these products are very affordable.

The big advantage of n8n Cloud plans is that they do not charge you for the complexity of your workflows, so if your workflow run takes hundreds of actions, you will spend a lot more on Make and Zapier, compared to n8n.

This cost model is very generous and somewhat similar to what Pipedream, another great low-code platform offers (though Pipedream has since implemented a change to their pricing model, they now track the "compute time" of a workflow run - nice idea, which allows them to protect their cloud from abusers, while giving very fair pricing):

Hosting

I use Hetzner Cloud exclusively hosting all my servers nowadays (at the time of writing this article) and I recommend it to everyone - starting from indehackers launching their first website, and ending with medium-sized SaaS companies who come to me for consulting. Hetzner makes self-hosting n8n so affordable. Before Hetzner, I used to host at DigitalOcean, Linode and Google Cloud. Here is the current pricing for 4GB RAM x86 machine across these clouds:

So, it's 4x difference in cost (I know - this is insane!), and the new Hetzner Cloud UI is very swift and pleasant to use - I never felt like I sacrificed anything when using Hetzner as sole founder, for recent 3 years.

Here is my Hetzner referral link if you decide to try it: https://hetzner.cloud/?ref=kW64RqTXiNiN (you  get €⁠20 on sign up, I get €⁠10)

Conclusion

I enjoy using n8n so far, it has great UX, and it seems to be a very stable, mature product which is a great addition to my self-hosted toolbelt.

n8n may have a steeper learning curve compared to Make and Zapier, but just because of this situation with external oAuth. And if you are not too technical, and have no experience with Docker, you can still use n8n as a cloud SaaS product.

If you liked this post, chances are you will enjoy following my Linkedin, where I review SaaS products and bleeding edge tech every week.

The list of self-hosted products I use

These products are awesome and have state-of-art UX, not just "it gets the job done so I can cope with its weird design" kind of feeling – like it used to be 10 years ago, when self-hosted products were often just a couple of scripts cobbled together by some software engineering student. Just a couple of wonderful examples of work-oriented self-hosted products I use:

• Metabase (online dashboard to build SQL reports, we use it at Qwintry);
• Mattermost (self-hosted Slack alternative);
• Directus (turns your SQL database into CMS)
• n8n (developer-friendly Zapier)
• LibreChat (ChatGPT interface for all popular LLMs)
• Mautic (Marketing newsletters)

Oh, and there are also Supabase... and ELK stack... and Zabbix... (read more about our poor-mans SRE setup)

This is a great self-hosted subreddit I recommend. I should note, that a lot of products being discussed there on Reddit are mostly for personal use (like, torrents, etc), but I am mostly seeking for solutions to do my work.

I believe self-hosting has several advantages for indie hackers and developers like myself. When you self-host, you have complete control over your infrastructure and data. This level of control is crucial for me as a developer because I need the flexibility to customize and integrate various tools seamlessly into my workflow.

The Cons

That being said, I think it's important to acknowledge that self-hosting comes with its own set of challenges. Maintenance, security, and scalability become my team responsibility. Ensuring regular updates, implemeting proper security measures, and managing infrastructure can be time-consuming tasks. However, if you have the technical skills and are willing to put in the effort, I believe self-hosting provides a level of independence and control that can be highly beneficial in the long run - it certainly is in our case.

The big question is: how technical do you need to be to embrace self-hosted products? There are no any silver bullets here, and you need to know a lot about how computers work, but I don't think you need to have 5 years of web development or system administration background to self-host effectively and reliably anymore – life got easier!

Docker skills are essential

Luckily, Docker and Docker Compose have greatly simplified the process of self-hosting. I have to admit, as a developer and CTO, in 2014-2018, I was initially hesitant to use Docker for our own web projects (especially the ones in the active development phase). It significantly complicated our deployment workflow and added a lot of overhead while providing marginal value. I didn't fully buy into the concept of 100% reproducible builds, especially considering the increased DevOps complexity that came with it.

Back then, I loved the ability to quickly hack and debug PHP scripts directly on production. While it's definitely a bad practice, there were countless times when this approach proved invaluable while debugging hard-to-reproduce bugs in a live environment: crucial for a small, rapidly changing product with just 1-3 developers!

Moreover, I struggled to wrap my head around Docker's concepts, such as volumes and binds. The cryptic shorthand syntax made it even more challenging to understand how these components worked together. I found myself spending more time trying to decipher Docker's intricacies than focusing on our application's core functionality.

At the time, our team had a well-established workflow that allowed us to develop, test, and deploy our applications efficiently (AND FAST!). Introducing Docker felt like an unnecessary layer of complexity that would disrupt our processes without providing significant benefits in return.

However, our perspective eventually changed and has shifted towards Docker everywhere (with the exception of some development machines): our projects matured and required more predictability; we strived for better CI/CD, dedicated teams of testers, better DevOps practices, and improved horizontal scalability. What's equally important is that Docker matured as well and became a great, boring technology, and a de-facto standard to host everything, starting from API services and ending with databases.

And Docker is what makes running complex software, self-hosted, so enjoyable. These applications, developed by external teams using varying technical stacks, now always come with well-conditioned Docker Compose files (official or community-supported) that make the setup process a breeze. Instead of dealing with the hassle of manual installation and configuration, I simply use Docker's containerization approach to spin up these applications quickly and easily.

You can still be a good web developer without knowing Docker – but it's hard to enjoy self-hosting nowadays without knowing it!

GPT and Claude to the rescue

Besides Docker, to self-host effectively, you will also need good understanding of:

• how Nginx routing works (well, you can use another webserver as an ingress controller, I just prefer Nginx).
• how Linux and networking in general works (firewall, ufw)
• how DNS and/or Cloudflare DNS works

Great news: you don't need to spend months experimenting with these - there is a great shortcut nowadays, with the raise of LLMs, it can be done in a matter of hours! (and it can help with docker errors, too)

Funding self-hosted product development

Developing high-quality, free, and open-source products requires funding. A common approach is to offer a cloud version of the product with some enterprise features to generate revenue. The Directus project had an interesting discussion on GitHub (https://github.com/directus/directus/discussions/17977) where the founder shared ideas on sustaining their product. They eventually adopted a strategy allowing free use of the entire platform unless the legal entity exceeds $5,000,000 USD in annual "total finances." - this is an interesting approach!

Open-core concerns

Not all self-hosted products are created equal. Some cross the line, offering a barely usable free version as a marketing gimmick instead of a genuinely useful product. In my opinion, an acceptable practice is to restrict clearly enterprise features, such as SSO mechanisms (like in n8n), under paid licenses. However, restricting the free product to the point of being nearly unusable is inadequate.

One example of a self-hosted product that faced community backlash is Budibase. They recently restricted their open-source version to a limited number of users, which led to criticism on Reddit (https://www.reddit.com/r/selfhosted/comments/17v48t8/budibase_will_soon_limit_users_on_oss_self_hosted/). Some users blamed this decision on the fact that Budibase is now VC-funded, suggesting that the pursuit of profitability led to the limitation of their free offering.

This incident highlights the balance that open-source projects must strike when seeking to generate revenue while maintaining the trust and support of their community. Restricting the free version too heavily can lead to a loss of goodwill and a perception that the project has abandoned its open-source roots in favor of commercial interests.

My real life approach to self-host a product

Let's say I want to host n8n. These are my steps:

Basic steps for a demo

1. Log in via SSH to my Ubuntu running on a remote Hetzner cloud server (I don't ever host anything in Docker on my Macbook, I do everything on Hetzner via plain SSH and/or awesome VS Code Remote extension, in case I need heavy config edits)
2. Verify docker is available and is running on the server: docker ps
3. Google n8n docker and open docs: https://docs.n8n.io/hosting/installation/docker/#prerequisites
4. Run these two commands highlighted in the manual to run the service
5. Verify docker containers of n8n is running. If not, explore container errors via docker logs xxx and check n8n github repository for similar issues: chances are someone already struggled with this!
6. Port forward the service to my localhost so I can open the service in my Chrome via http://localhost:3333

These steps generally take around 3-5 minutes max to get the service running. Then, I usually play with the service for an hour or two to realize if I want to use it on a longterm basis.

1. If I do, go to Cloudflare to my domain and create a subdomain on one of my domains, with an A record pointing to my Hetzner cloud server (e.g. n8n.scrapeninja.net. HTTPS certificates are handled by Cloudflare, has good DNS UI, and also protects my server ip addresses for occasional DDoS attacks (and it's free!)
2. I create a nginx host and point it to the docker container. Here is an example of such a config:

server {
    listen 80;
    server_name n8n.scrapeninja.net;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name n8n1.scrapeninja.net;
    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
    location / {
        proxy_pass http://localhost:5678;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade; # websocket support
        proxy_set_header Connection "Upgrade";  # websocket support
        proxy_set_header Host $host;
        chunked_transfer_encoding off;
        proxy_buffering off;
        proxy_cache off;
    }
}

Hosting

I use Hetzner Cloud almost exclusively for hosting all my servers nowadays (at the time of writing this article), and I wholeheartedly recommend it to everyone—from indie hackers launching their first website to medium-sized SaaS companies seeking consulting. Hetzner makes self-hosting so affordable. Before Hetzner, I used to host on DigitalOcean, Linode, and Google Cloud. Here is the current pricing for a 4GB RAM x86 machine across these clouds:

So, it's 4x difference in cost (insane, I know!), and the new Hetzner Cloud UI is very swift and pleasant to use - I never felt like I sacrificed anything when using Hetzner, for recent 3 years.

Here is my Hetzner referral link if you decide to try it: https://hetzner.cloud/?ref=kW64RqTXiNiN (you  get €⁠20 on sign up, I get €⁠10).

If you liked this post, chances are you will enjoy checking other posts in this blog, and following my Linkedin, where I review SaaS products and bleeding edge tech.

Observation #1: ChatGPT for Android got a very good speech-to-text and text-to-speech engine based on Whisper, since fall 2023. It understands what you say well and intones its phrases nicely when it speaks. You can have real conversations with it on any topic. Of course, it's still far from being a human tutor in terms of analyzing your pronunciation, because what essentially reaches ChatGPT cloud is not the audio stream of your words, but just recognized text (with some glitches if you consider the sloppy pronunciation of an average beginner).

The conversion from speech to text happens on the local device. (By the way, I see great potential here for neural networks that analyze the audio stream itself and compare it to reference pronunciation - check out https://getfluently.app from fellow founder Yuri Rebryk, a cool idea, but so far implemented only for English language, as far as I know.)

Observation #2: ChatGPT allows you to extend your vocabulary in different industries and niches. Sure, it loses to a human tutor in not "hearing" your pronunciation, but it infinitely wins in the depth and breadth of its knowledge. Today it can give you 50 words from Harry Potter, and tomorrow it can act out a dialogue where you go to a restaurant and discuss the nuances of pairing white wine with oysters with the french chef. Or an angry landlord shows up because you haven't paid rent for 3 months. The possibilities are endless.

Here's an example of my basic prompt #1:

I study French. My level is A2. Please create a simple dialogue happening in a French restaurant between two chefs who argue on ____. Don't use person labels.

(The last phrase is so that each line doesn't have a person label prefix, which is distracting in the audio stream).

It's worth noting that ChatGPT can switch languages decently even within one line, BUT only if that line is long enough. So it can't pronounce a list of French words with their English translations well - it constantly mixes them up, trying to pronounce the words in a French or English manner - it's fine for reading, but almost impossible to listen to. But if you ask it to repeat each dialog line in English right after the French, then it sounds decent.

Prompt #2:

I study French. My level is A2. Please create a simple dialogue happening in a French restaurant. Add an English translation after each phrase. Don't use person labels.

ChatGPT can re-read a message out loud, even if you're already in text mode reading. There's a "Read aloud" option in the context menu for that.

If I encounter an expression I don't understand, I copy that snippet into the Yandex Translate app, translate it there, and listen to it separately. (Yandex is better than Google Translate in that it can pronounce even long lines, has fewer glitches, and provides more usage examples).

After going through the whole dialogue, I dictate a new prompt in the same conversation with ChatGPT (it's already in context):

Prompt #3

Now let's replicate this dialog, I will answer your questions one by one.

And I try to reproduce the phrases and words I just learned. The dialog inevitably deviates from the original version, making it even more exciting and challenging.

The main advantage of this approach over watching YouTube is that the vocabulary is actively used by me, not just passively consumed, bringing it closer in usefulness to conversation clubs - except a conversation club meets once a week if I can make it, while ChatGPT is available at any moment, like during a walk.

Project lifecycle: MVP -> growth -> maintenance

What does a full-stack engineer of a new project think about? Exciting questions: what programming language to choose, Bootstrap vs Tailwind, React vs vue.js, services or microservices, and so on. "How to build the product"? Those days are certainly wonderful, but there comes a time when rapidly deploying features using the latest framework is no longer as important and cool as maintaining the reliability and stability of the entire system - year after year, preferably without any major f**k-ups.

Now that Qwintry can no longer call itself a startup, the focus shifts from introducing new features to SRE (Site Reliability Engineering) - ensuring the availability, reliability and performance of the product. Some might say it's grim and boring, but in my opinion, the ability to sit down and meticulously investigate, analyse, unravel the threads of logs and reconstruct events in production is what makes a person a valuable employee and a true engineer (and detective). And you can get a kick out of it too.

Poor man's approach to SRE of a small business

We don't have a dedicated SRE team - we're no longer a startup, but we're still a long way from being a rich corporation. So our approach is very different from what bigger companies do. When things go south, we plug the holes as best we can: our development people, our testing people, our devops people and myself - we all get involved in some way. In this situation, the stack of products we use to monitor, analyse and manage our services comes to the fore. Why is that? If you have to run to the server and grep through logs from the console every time you want to analyse a problem - no one wants to do that, it's inconvenient, and it also severely limits the number of people who can do that analysis.

Our stack of boring and predictable SRE tools

Zabbix

For cloud and physical server monitoring and Telegram alerts, we've been using Zabbix for a very long time, which allows us to track the health of servers, networks and services. It's not very ergonomic in terms of adding new hosts and creating alerts, to be honest, a lot of clicks and taps - but it gets the job done. Zabbix was launched in 2001 (sic) and is still actively developed and improved - a perfect example of a great open source product.

I think Prometheus + Grafana now allow to build a very similar dashboard – but Zabbix just works for us and we never had a lot of reasons to switch.

ELK Stack for log storage and analysis

All logs from all servers and services flow to a machine with the ELK stack (Elasticsearch, Logstash, Kibana) - it happily eats hundreds of gigabytes of RAM, but for the last couple of years it hasn't crashed or caused any problems. (My recent Linkedin post regarding our primary database migration to Hetzner uses ELK visualisation)

Okay, I was not a huge fan of ElasticSearch 4 years ago when running it on our machines was cumbersome and expensive. Clickhouse seemed to be a very promising alternative for log storage tasks (and it indeed is!). But, ecosystem (Kibana!) is what makes ELK stack so great and strong - it just works with minimal effort. Now, when hardware and RAM got much more affordable for us, ELK stack is still rocking and I appreciate it a lot.

I think Grafana and recently Loki for log storage are a very competitive open source (open core?) players on the market which I still hope to try on some new project.

Metabase: SQL dashboard

For SQL reports, we use Metabase - a great open-source product with a great design that allows us to query databases and build reports based on them. (Don't forget to create a read-only user in your database and limit the export of users' personal data!)

Metabase is so nice: not just for SRE needs; but also for financial department, for support department - we build a lot of dashboards and reports there and this saved us countless hours of development work (as before Metabase, all reports were implemented in our primary app PHP+MySQL backend).

Metabase killer feature for us is that allows to build SQL reports which are parametrized: you put certain [[symbols]] and these {{pieces}} become a convenient UI filters which can be used later by non-technical operators:

SELECT
  count(*)
FROM
  products
WHERE
  category = {{category}}

They also have visual GUI to replace SQL, but this is not what we use often: people who cannot write SQL usually just ask IT guys to create a new report for them.

I am pretty sure we will improve our report building workflow in 2024: with rapid LLM models evolution, I am pretty sure our finance department will soon be able to ask basic questions and GPT or Claude (200k tokens input is finally there!) will be able to spit out a usable SQL report for them.

Sentry: application error monitoring

For monitoring code errors in production, we use Sentry.

A few lines of code - and all our PHP, React-Native, Node.js apps are now reporting all error and exception stack traces into beautiful dashboard for easy investigations. It's awesome.

Choosing boring self-hosted products for SRE

All of these products are self-hosted and open source. Except for the cloud version of Sentry, which has done an excellent job with its cloud solution in terms of cost control (constant spikes of hundreds of thousands of errors per hour don't affect our invoice too badly), and costs very little money relative to the benefit it provides.

All of these products are boring, in a good sense, not very innovative, work acceptably or well, and it's scary to think how many tens of thousands of dollars we would have shelled out if we had taken the "best on the market", as wealthier companies do.

For example, Datadog and NewRelic dominate the observability systems and log storage market, and they have excellent tools, but they're quite expensive - thousands of dollars per month. The scariest thing is not that it's expensive, but that it can very quickly get out of control when a random log file accidentally grows not by 1GB per day, but by 20GB per day. When you have dozens of machines and hundreds of services - ALWAYS some shit like that happens. We know this pretty well, because we recently paid 26K USD for an API which we even forgot we are using.

A bit of a history

Playwright was created by a guy who was one of authors of Puppeteer.js: Andrey Lushnikov (who was part of Chrome DevTools team back then). Playwright was built on the lessons of Puppeteer: it was cross-browser from scratch (while Puppeteer got Firefox experimental support only recently in 2023), it had better syntax, it had a lot of higher level tools e.g. for test runners.

Node.js or Python?

Node.js: the native habitat

Playwright was born in the Node.js ecosystem, making it a natural habitat for this tool. If you're scaling up, Node.js shines particularly brightly. Why, you ask? It boils down to process management.

Node.js's version of Playwright doesn't spawn a new node process for every browser instance. This is crucial when you're managing multiple browser instances simultaneously. If you've got a system running various scripts at unpredictable intervals, you definitely want to avoid the overhead of spinning up a new node process each time.

I stumbled upon a GitHub issue that put this into perspective. A developer was wrestling with the Python Playwright implementation, which, under the hood, was spinning up a separate node process for each instance. The result? A CPU and memory usage spikes.

Here is another GitHub issue where Playwright maintainers recommend Node.js for heavy lifting: [Question]: Performance benchmarking of python playwright versus node #1289 and another one.

Python: simplicity meets elegance

Python's implementation of Playwright, while elegant and simple for scripting, may not be the best companion for heavy lifting. Each call to sync_playwright() in Python fires up a new node process. Although Python is a delight for quick scripts and data analysis, this behavior might bog down your system when you're trying to keep it light on its feet.

Other Languages: a world of choices

It's worth noting that Playwright isn't just a two-player game. It extends its reach to other languages like Java and C#. If you're already working within these ecosystems, it makes sense to stick to your guns and use what you're comfortable with.

My recommendation: Node.js for scalability

For heavy-duty scraping and testing, especially at scale, Node.js is the recommended route. It keeps your CPU from sweating and your memory from overflowing. Moreover, Node.js being Playwright's native environment, you get first-class support and performance.

Code example: blocking image download

Let's get our hands dirty and take a look how basic code looks like in Node.js and Python versions of Playwright. Here's how you can block images when opening a page using both Node.js and Python with Playwright.

Playwright Node.js version  

const { chromium } = require('playwright'); // or 'firefox' or 'webkit'

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Block images
  await page.route('**/*.{png,jpg,jpeg}', route => route.abort());

  await page.goto('https://example.com');
  // ... you can perform actions on the page here

  await browser.close();
})();

In this Node.js example, we use Playwright's route method to intercept network requests and abort any that are for image resources.

Playwright Python version

For the Python version, you will need to have the Playwright Python package installed and the Playwright browser binaries downloaded. This can be done by running pip install playwright followed by playwright install.

import asyncio
from playwright.async_api import async_playwright

async def block_images_and_open_page():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()

        # Block images
        await page.route('**/*.{png,jpg,jpeg}', lambda route: route.abort())

        await page.goto('https://example.com')
        # ... you can perform actions on the page here

        await browser.close()

asyncio.run(block_images_and_open_page())

In this Python async example, we use the async_playwright context manager to handle the Playwright object lifecycle and route method to intercept and abort image requests.

Both snippets demonstrate how to block image loading, which can be particularly useful when scraping websites where images are not needed, thus speeding up page load times and reducing bandwidth usage.

Playwright for web scraping: fingerprinting and stealth modes

If we talk about using Playwright for real-world web scraping, this is where things get interesting, especially in the context of comparing Node.js with Python.

Official Playwright packages simply do not have a task to remove all traces of automation on the browser, as the primary goal of programmatically controlled browser was always UI testing, and not web scraping.

Most of big websites implement basic or sophisticated anti-scraping measures which try to fingerprint your web browser and block it if it looks like an automated one (so stock versions of Playwright are blocked by most of modern websites which have any kind of anti-scraping protection, even the most basic one). There are huge companies which are involved in detecting scrapers on multiple levels:

• on TLS fingerprint level;
• on HTTP/2 level;
• and on browser capabilities level.

For Node.js, there exists a whole ecosystem of stealth plugins, and you can use these stealth improvements to make your Playwright instance look more like a regular human browser:

Fingerprint Suite from Apify

https://github.com/apify/fingerprint-suite

Playwright Extra

This is basically an interoperability layer with Puppeteer stealth packages.

https://www.npmjs.com/package/playwright-extra

These two sets of tools are not a silver bullet and do not guarantee successful scraping process, for example Apify Suite and Playwright Extra perform much worse than ScrapeNinja /scrape-js in terms of detection, but at least they are trying to help you in this regard.

Of course, Python developers want to have similar solutions for Playwright Python – here is a Github issue about it, which was closed without any good resolution.

If you still need a stealthy Python browser, your best bet is Undetected Chromedriver (it is based on heavily hacked Selenium)

Conclusion

Choosing the right tool for the job is as important as the job itself. In the world of Playwright, Node.js offers the best scalability, stealth mode and performance, especially for complex, resource-intensive tasks like web scraping. Python may seduce with its syntax and simplicity, but when it comes to Playwright, the Node.js version holds the upper hand for scaling. Python and other language bindings have their place, but for the heavy-duty browser automation, Node.js is the way to go. But.. if you are comfortable with Python and you are new to Node.js ecosystem, I would definitely recommend to stick to the tool you know until you hit any real performance issues.

If you liked this post, you will probably enjoy reading my blog post where I used low code to web scrape Apple.com for refurbished iPhones and to get push alerts and how I built specialized browser API for web scraping.

In the summer of 2023, I had a client with a custom project where my task was to extract data from a notoriously dynamic single page application (SPA). The site was a huge bunch of JavaScript with robust anti-scraping measures. It felt like a digital game of cat and mouse, and I was the determined mouse. That's when Playwright, a framework for automated browser testing, came to the rescue. It wasn't just about rendering SPAs or managing dynamic content; Playwright excelled at bypassing JavaScript-based anti-scraping mechanisms.

I used Node.js version of Playwright (it is also available as Python package, do not confuse these two!) and stealth npm packages to blend in seamlessly, making my scraping activities indistinguishable from regular browsing. The interesting part was that I used 4g proxies to avoid getting captchas, but it made page loads very slow – so I did my best to increase the speed of page loads by blocking unneccesary media and images, and I am going to show you some code how this could be done.

Playwright: A Quick Overview

Playwright, for those who might not be familiar, is a framework for automated browser testing. But its utility extends far beyond that, especially in web scraping. I've personally found it incredibly effective in handling SPAs and navigating through complex scenarios which requires button clicking and other web page interactions – it simulates a user's interaction with a web page, making it a powerful tool for scraping dynamic content (my best one is endless scrolling!).

The Role of route() in Playwright

To block images on page download, you need to learn about one super-useful essential Playwright feature: route() function. This function allows us to intercept and modify network requests. In my journey with web scraping, I've used route() in various scenarios – from modifying request headers to injecting scripts. It offers an unprecedented level of control over how the browser interacts with a website, giving us the power to tailor our scraping strategy to specific needs. Compared to Puppeteer, another great Node.js package for programmatical browser control, Playwright route() function is much more concise and convenient. For example, it allows to use JS regexes out of the box! Please refer to route docs.

The Challenge: Proxies and Resource Management

As I've said, I used 4g proxies for my project.  They're essential for maintaining anonymity and bypassing IP-based restrictions. However, proxies can be slow and, if you're not careful, expensive in terms of traffic. This is where resource management becomes crucial.

Blocking unnecessary resources in Playwright

In many scraping tasks, we don't need to load images, fonts, or even CSS. These resources consume bandwidth and slow down our scraping process. By blocking these resources, we can significantly speed up our scraping jobs and reduce costs.

Node.js Playwright code to block images, css, and fonts

Here's a practical example of how to implement resource blocking in Playwright. This Node.js code uses top level async/await (so please use Node.js 16+) , ES6 imports (so don't forget to use type:module in package.json of your project), and the route() function to block images, fonts, and CSS.

import { chromium } from 'playwright';



const browser = await chromium.launch();
const page = await browser.newPage();

// Use route() to intercept and block certain types of resources
await page.route('**/*', (route) => {
    const resourceType = route.request().resourceType();
    if (['image', 'stylesheet', 'font'].includes(resourceType)) {
        route.abort();
    } else {
        route.continue();
    }
});

// Navigate to the website
await page.goto('https://example.com');

// Perform your scraping tasks here

await browser.close();

In this code, we launch a Chromium browser, create a new page, and set up a route handler. The handler checks the resource type of each network request and aborts it if it's an image, stylesheet, or font. This significantly reduces the amount of data downloaded and processed, leading to faster and more cost-effective scraping.

Extending blocking: get rid of unneccesary tracking and JS

Let's top it up a notch and get rid of various tracking scripts which usually slow down the page loading times significantly.

import { chromium } from 'playwright';

// Using top-level async/await
(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Define a list of common ad network patterns
  const adNetworkPatterns = [
    '**/*doubleclick.net/**',
    '**/*googleadservices.com/**',
    '**/*googlesyndication.com/**',
    // Add more patterns as needed
  ];

  // Use route() to intercept and block certain types of resources and ad network requests
  await page.route('**/*', (route) => {
    const requestUrl = route.request().url();
    const resourceType = route.request().resourceType();

    // Check if the request is for an ad network
    const isAdRequest = adNetworkPatterns.some(pattern => requestUrl.match(pattern));

    if (['image', 'stylesheet', 'font'].includes(resourceType) || isAdRequest) {
      route.abort();
    } else {
      route.continue();
    }
  });

  // Navigate to the website
  await page.goto('https://example.com');

  // Perform your scraping tasks here

  await browser.close();
})();

Conclusion

Playwright's flexibility, particularly with functions like route(), makes it an incredible tool for web scraping and automation. By understanding and leveraging these capabilities, we can efficiently navigate through the complexities of modern web applications. Remember, in the world of web scraping, it's not just about getting the data; it's about getting it efficiently and responsibly. Keep exploring, and happy scraping!