Large-scale property data ingestion and processing infrastructure
High-frequency crawling and multi-stage processing pipeline designed for scale, traceability and operational control.
Context
Property market data is fragmented across multiple listing sources, each presenting information in different formats. To enable meaningful analysis, structured data collection and processing infrastructure was required — capable of ingesting high volumes of listing data in near real-time. The platform infrastructure was designed to operate independently of third-party APIs, and is self-hosted for operational control.
The challenge
- Scraping search result pages across multiple property sources
- Extracting listing URLs at scale
- Polling listing pages regularly for changes
- Handling thousands of scrapes, inserts and processing operations per minute
- Storing raw data snapshots for traceability
- Cleaning and normalising data into structured formats
- Ensuring idempotent upsert logic
- Maintaining stability under sustained concurrency
- Self-hosted operational reliability
Our role
GreyTwo designed and built the end-to-end infrastructure. This included multi-source crawling architecture, search parsing logic, listing URL scheduling, raw JSON storage, secondary processing pipeline for normalisation, clean data upsert logic, high-frequency job orchestration, and self-hosted deployment and infrastructure management. The system was engineered for scale, resilience and controlled processing.
Key areas of delivery
Multi-stage data pipeline
Structured phases: discovery, URL extraction and scheduling, raw ingestion and storage, secondary normalisation, structured upsert into clean models. This enables auditability, reprocessing, schema evolution and controlled scaling.
High-frequency scheduling
Job orchestration designed to distribute workload safely, prevent redundant processing, and maintain performance stability under sustained throughput.
Self-hosted operational control
Self-hosted deployment provides cost predictability, infrastructure visibility and operational independence.
Impact
- Reliable high-frequency ingestion of listing data
- Structured separation between raw and processed datasets
- Scalable upsert logic for continuously evolving records
- Stability under sustained workload
- Full operational control through self-hosted infrastructure