WebWeave — Semantic Site Crawler & Data Extractor

Parses any website into structured, classified, RAG-ready data.

Discuss a Similar ProjectOpen full report (PDF)
Domain
Web & Data
Year
2025
Type
Applied build — internal engineering work

This is internal engineering and R&D work, not a client project. It is published to show technical depth — the modelling, evaluation, and tooling behind the AI automation we build for clients. No client is associated with it.

A web application that crawls arbitrary websites and extracts structured information — contact info, company overview, services, legal sections — using keyword-driven logic. The cleaned, deduplicated, formatted output is stored in MongoDB for later retrieval or chatbot integration.

Tools & methods

  • Python
  • BeautifulSoup
  • Selenium
  • MongoDB
  • Regex
Full technical report

Objectives

  • Extract meaningful content from arbitrary websites.
  • Tag and classify data by category (overview, services, team, legal, contact).
  • Persist results to MongoDB with an SHA-256 key to identify crawls uniquely.

Process

  • User submits a URL; crawler loads the root page and discovers internal links.
  • Categorize content with predefined keyword buckets; deduplicate to avoid repetition.
  • Save outputs as a file-saving pipeline and upload final data with a hash key.
  • Return an easily-consumable JSON response to the frontend.

Results

  • Successfully extracted structured content from real-world domains.
  • High keyword-categorization coverage with clean, deduplicated JSON output.

What we took from it

  • Robust crawling must handle inconsistent structure and noise.
  • Memory control on large pages + dedup keep the pipeline efficient.

From the report

  • WebWeave — Semantic Site Crawler & Data Extractor overview
    WebWeave — Semantic Site Crawler & Data Extractor overview

Filter by category

Every project and study. Showing 26 of 27.

Products & client engagements

  • OpenInterview marketing site shown on a laptop — upload a résumé to generate a shareable video profile
    Client engagement

    US-based HR technology startup · HR technology & recruitment

    OpenInterview

    OpenInterview is a video interview SaaS platform that turns an uploaded résumé into a shareable candidate profile with a video introduction and availability. Barakode built résumé parsing, video processing, public profiles, and Stripe subscriptions.

    Delivered
    4 services
    Timeline
    August 2024 – November 2024
    • React.js
    • Node.js
    • REST API
    • AI résumé processing
    • Cloud storage

    Platform launched and live at openinterview.me, serving job seekers internationally.

    View case study
  • Beyut Libya real-estate platform on a laptop — sign-in screen with the Add New Property wizard behind it
    Client engagement

    Real-estate marketplace operator · Real estate

    Beyut Libya

    Beyut Libya is an Arabic and English real estate marketplace for residential and commercial listings. Buyers use property filters to explore verified listings, while agents manage inventory through a secure admin panel.

    Delivered
    4 services
    Timeline
    October 2024 – January 2025
    • Next.js
    • React.js
    • Node.js
    • REST API
    • MySQL

    Platform live and operating nationwide at beyutlibya.com.

    View case study
  • Gman tailoring platform dashboard shown on screen
    Client engagement

    Growing tailoring & fashion-retail business · Tailoring & fashion retail

    Gman Stitching Platform

    Gman is custom tailoring management software that connects customer records, measurements, orders, inventory, billing, and accounting. The platform replaced a slow PHP/Laravel system that was difficult to maintain.

    Delivered
    3 services
    Timeline
    January 2023 – March 2023
    • React.js
    • Redux
    • Ant Design
    • Node.js
    • Express.js

    Approximately 70% improvement in application response time after the rebuild.

    View case study
  • Barakah Accounts inventory items screen
    Client engagement

    Restaurant operations team · Hospitality & food service

    Barakah Accounts

    Barakah Accounts is a restaurant ERP that brings order management, inventory, billing, reporting, and the handoff between front-of-house and back-office teams into one system.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live restaurant ERP available at app.barakahaccounts.com.

    View case study
  • Ophir Jewellery purchase orders screen
    Client engagement

    Jewellery business · Jewellery retail & operations

    Ophir Jewellery ERP

    Ophir Jewellery ERP connects product records, inventory, customer activity, orders, and business reporting in one workspace for daily jewellery operations.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live jewellery ERP available at ophir.eaquirstech.com.

    View case study
  • Turner 10 sales workspace screen
    Client engagement

    Real-estate operations team · Real estate

    Turner 10 Real Estate ERP

    Turner 10 is a real estate ERP that connects property records, enquiries, availability, client information, and the operational work behind each deal.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live real-estate ERP available at turner10.eaquirstech.com.

    View case study
  • HMS hospital management purchase returns workspace with settlement, credit note, replacement, and refund summaries
    Client engagement

    Hospital operations team · Healthcare administration

    HMS

    HMS is a hospital management system that connects patient administration, appointments, billing, pharmacy, procurement, and reporting in one operational workspace. Teams can also follow supplier purchase returns.

    Delivered
    3 services
    Timeline
    Hospital project
    • NestJS
    • PostgreSQL
    • Redis
    View case study
  • Merchant product hunt showing source diagnostics for a pet gadgets search
    Independent product

    Independent SaaS product · E-commerce research

    Merchant

    Merchant is a multilingual product research platform that searches ten sources, compares supplier listings, and ranks products by cost, estimated margin, and demand signals. Sellers can inspect sources in English, Urdu, or Arabic.

    Delivered
    4 services
    Timeline
    Production-build demo
    • Next.js 16
    • TypeScript
    • NestJS 12
    • Zod
    • REST

    In the documented demo, a seller enters one niche and receives a ranked list with supplier cost, estimated margin and minimum order beside each product.

    View case study

Engineering & AI R&D

This is internal and academic engineering work. Each study includes its full technical report.

Need this kind of capability in a real product?

Tell us the workflow you want to automate and we will map the practical version of it — grounded, reviewable, and connected to the tools you already run.

Book a Free Project Discovery CallExplore AI Automation
Contact via WhatsApp+92 300 8698044
Cost Estimator
Message us on WhatsApp