WebWeave — Semantic Site Crawler & Data Extractor

Parses any website into structured, classified, RAG-ready data.

Discuss a Similar ProjectOpen full report (PDF)
Domain
Web & Data
Year
2025
Type
Applied build — internal engineering work

This is internal engineering and R&D work, not a client project. It is published to show technical depth — the modelling, evaluation, and tooling behind the AI automation we build for clients. No client is associated with it.

A web application that crawls arbitrary websites and extracts structured information — contact info, company overview, services, legal sections — using keyword-driven logic. The cleaned, deduplicated, formatted output is stored in MongoDB for later retrieval or chatbot integration.

Tools & methods

  • Python
  • BeautifulSoup
  • Selenium
  • MongoDB
  • Regex
Full technical report

Objectives

  • Extract meaningful content from arbitrary websites.
  • Tag and classify data by category (overview, services, team, legal, contact).
  • Persist results to MongoDB with an SHA-256 key to identify crawls uniquely.

Process

  • User submits a URL; crawler loads the root page and discovers internal links.
  • Categorize content with predefined keyword buckets; deduplicate to avoid repetition.
  • Save outputs as a file-saving pipeline and upload final data with a hash key.
  • Return an easily-consumable JSON response to the frontend.

Results

  • Successfully extracted structured content from real-world domains.
  • High keyword-categorization coverage with clean, deduplicated JSON output.

What we took from it

  • Robust crawling must handle inconsistent structure and noise.
  • Memory control on large pages + dedup keep the pipeline efficient.

From the report

  • WebWeave — Semantic Site Crawler & Data Extractor overview
    WebWeave — Semantic Site Crawler & Data Extractor overview

Filter by category

Every project and study. Showing 25 of 26.

Client engagements

  • OpenInterview marketing site shown on a laptop — upload a résumé to generate a shareable video profile
    Client engagement

    US-based HR technology startup · HR technology & recruitment

    OpenInterview

    OpenInterview is a video interview platform that turns a candidate résumé into a recruiter ready profile with a video introduction availability details and one shareable link. Barakode built the product from résumé parsing and video processing to public profiles and Stripe subscriptions.

    Delivered
    4 services
    Timeline
    August 2024 – November 2024
    • React.js
    • Node.js
    • REST API
    • AI résumé processing
    • Cloud storage

    Platform launched and live at openinterview.me, serving job seekers internationally.

    View case study
  • Beyut Libya real-estate platform on a laptop — sign-in screen with the Add New Property wizard behind it
    Client engagement

    Real-estate marketplace operator · Real estate

    Beyut Libya

    Beyut Libya is a bilingual property marketplace for Libya. Buyers can find verified homes with live filters while agents manage their inventory through a secure admin panel. The platform supports residential and commercial property across the country.

    Delivered
    4 services
    Timeline
    October 2024 – January 2025
    • Next.js
    • React.js
    • Node.js
    • REST API
    • MySQL

    Platform live and operating nationwide at beyutlibya.com.

    View case study
  • Gman tailoring platform dashboard shown on screen
    Client engagement

    Growing tailoring & fashion-retail business · Tailoring & fashion retail

    Gman Stitching Platform

    A full-cycle tailoring and order-management platform: customers, measurements, orders, inventory, billing, and accounting in one system. It replaced a PHP/Laravel application that had become too slow to use and too brittle to change.

    Delivered
    3 services
    Timeline
    January 2023 – March 2023
    • React.js
    • Redux
    • Ant Design
    • Node.js
    • Express.js

    Approximately 70% improvement in application response time after the rebuild.

    View case study
  • Barakah Accounts inventory items screen
    Client engagement

    Restaurant operations team · Hospitality & food service

    Barakah Accounts

    A restaurant ERP built to bring orders, stock, billing, reporting, and the day-to-day handover between front-of-house and back-office teams into one working system.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live restaurant ERP available at app.barakahaccounts.com.

    View case study
  • Ophir Jewellery purchase orders screen
    Client engagement

    Jewellery business · Jewellery retail & operations

    Ophir Jewellery ERP

    An ERP for a jewellery operation that brings product records, stock, customer activity, order handling, and business reporting into one calmer working view.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live jewellery ERP available at ophir.eaquirstech.com.

    View case study
  • Turner 10 sales workspace screen
    Client engagement

    Real-estate operations team · Real estate

    Turner 10 Real Estate ERP

    A real-estate ERP that gives the team one practical place to follow properties, enquiries, availability, client records, and the internal work that supports every deal.

    Delivered
    3 services
    Timeline
    Live product
    • NestJS
    • PostgreSQL
    • Redis

    Live real-estate ERP available at turner10.eaquirstech.com.

    View case study
  • HMS hospital management purchase returns workspace with settlement, credit note, replacement, and refund summaries
    Client engagement

    Hospital operations team · Healthcare administration

    HMS

    HMS brings hospital administration, patient services, billing, pharmacy, and procurement into one operational workspace. Teams can review daily work and follow supplier purchase returns from a clear shared interface.

    Delivered
    3 services
    Timeline
    Hospital project
    • NestJS
    • PostgreSQL
    • Redis
    View case study

Engineering & AI R&D

This is internal and academic engineering work. Each study includes its full technical report.

Need this kind of capability in a real product?

Tell us the workflow you want to automate and we will map the practical version of it — grounded, reviewable, and connected to the tools you already run.

Book a Free Project Discovery CallExplore AI Automation
Contact via WhatsApp+92 300 8698044
Cost Estimator
Message us on WhatsApp