Skip to content

Architecture and Tools

Jump to bottom
Abid Ali Awan edited this page Dec 2, 2025 · 1 revision

🏗️ Architecture and Tools

This page provides a deep dive into the technical architecture of ECom Intel, including its components, data flow, and the tools it uses.

📁 Project Structure

ECom-Intel/
├── app.py               # Main Streamlit application (dashboard)
├── database.py          # SQLite database operations
├── firecrawl_client.py  # Firecrawl API integration
├── review_analyzer.py   # OpenAI analysis logic
├── requirements.txt     # Python dependencies
├── .env.example         # Environment variables template
├── reviews.db           # SQLite database (generated)
└── README.md            # Project documentation

🔄 Data Flow Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   User Input    │────▶│  Streamlit App   │────▶│   Check Cache   │
│  (Product URL)  │     │    (app.py)      │     │  (database.py)  │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                         │
                              ┌───────────────────────────┤
                              │                           │
                              ▼                           ▼
                    ┌─────────────────┐         ┌─────────────────┐
                    │  Cache Hit:     │         │  Cache Miss:    │
                    │  Return Results │         │  Scrape Reviews │
                    └─────────────────┘         └─────────────────┘
                                                         │
                                                         ▼
                                               ┌─────────────────┐
                                               │   Firecrawl     │
                                               │ (firecrawl_     │
                                               │  client.py)     │
                                               └─────────────────┘
                                                         │
                                                         ▼
                                               ┌─────────────────┐
                                               │  OpenAI GPT-4o  │
                                               │ (review_        │
                                               │  analyzer.py)   │
                                               └─────────────────┘
                                                         │
                                                         ▼
                                               ┌─────────────────┐
                                               │  Save to DB &   │
                                               │ Display Results │
                                               └─────────────────┘

🧩 Core Components

1. Streamlit App (app.py)

The main entry point and user interface for the application.

Key Responsibilities:

  • Render the web dashboard UI
  • Handle user input (product URL, settings)
  • Orchestrate the analysis workflow
  • Display results with interactive charts
  • Manage session state and caching options

Key Functions:

Function Purpose
main() Main application entry point
validate_url() Validates URL format
extract_product_name() Extracts product name from URL
create_sentiment_chart() Creates Plotly pie chart for sentiment
create_rating_chart() Creates Plotly bar chart for ratings

2. Database Module (database.py)

Handles all SQLite database operations for persistent storage.

Database Schema:

-- Products table
CREATE TABLE products (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url TEXT UNIQUE NOT NULL,
    title TEXT,
    brand TEXT,
    price TEXT,
    image_url TEXT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

-- Reviews table
CREATE TABLE reviews (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    product_id INTEGER REFERENCES products(id),
    review_text TEXT NOT NULL,
    rating INTEGER,
    reviewer_name TEXT,
    review_date TEXT,
    source_url TEXT,
    sentiment_score REAL,
    sentiment_label TEXT,
    created_at TIMESTAMP
);

-- Analysis results table
CREATE TABLE analysis (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    product_id INTEGER REFERENCES products(id),
    sentiment_distribution TEXT,  -- JSON
    key_insights TEXT,            -- JSON
    pros TEXT,                    -- JSON
    cons TEXT,                    -- JSON
    rating_summary TEXT,          -- JSON
    total_reviews INTEGER,
    average_rating REAL,
    created_at TIMESTAMP
);

Key Methods:

Method Purpose
get_or_create_product() Get existing or create new product
add_reviews() Add reviews with duplicate detection
save_analysis() Store analysis results
get_reviews() Retrieve reviews for a product
get_analysis() Retrieve analysis results
get_recent_products() Get recently analyzed products

3. Firecrawl Client (firecrawl_client.py)

Handles web scraping using the Firecrawl API.

Key Features:

  • Search for review pages related to products
  • Scrape and extract review content
  • Handle multiple review formats and patterns
  • Rate normalization (1-5 scale)
  • Duplicate detection

Key Methods:

Method Purpose
search_reviews() Find review pages via Firecrawl search API
scrape_reviews() Scrape content from a specific URL
extract_reviews_from_content() Parse reviews from scraped markdown
get_product_reviews() Main method to get all reviews for a product

Review Extraction Patterns:

  • Star ratings: 5 stars, 4.5 stars
  • Slash ratings: 4/5, 3/5
  • Unicode stars: ★★★★★
  • Rating labels: Rating: 4

4. Review Analyzer (review_analyzer.py)

Performs AI-powered analysis using OpenAI's GPT-4o-mini model.

Key Features:

  • Individual review sentiment analysis
  • Batch insight generation
  • Pros/cons extraction
  • Recommendation generation
  • Product comparison (multi-product)

Key Methods:

Method Purpose
analyze_reviews() Main analysis method returning comprehensive results
_analyze_sentiment() Analyze sentiment of a single review
_generate_insights() Generate key insights, pros, cons, recommendations
_calculate_sentiment_distribution() Calculate sentiment percentages
_calculate_rating_summary() Calculate rating distribution
get_review_summary() Generate human-readable summary
compare_products() Compare multiple products

Analysis Output Structure:

{
    "total_reviews": 150,
    "average_rating": 4.2,
    "sentiment_distribution": {
        "positive": 65.5,
        "negative": 15.3,
        "neutral": 19.2
    },
    "key_insights": ["insight 1", "insight 2"],
    "pros": ["pro 1", "pro 2"],
    "cons": ["con 1", "con 2"],
    "rating_summary": {
        "5_star": 45.0,
        "4_star": 25.0,
        "3_star": 15.0,
        "2_star": 10.0,
        "1_star": 5.0
    },
    "recommendations": ["recommendation 1", "recommendation 2"]
}

🛠️ Technologies & Tools

Core Technologies

Tool Version Purpose
Python 3.8+ Primary programming language
Streamlit Latest Web dashboard framework
SQLite Built-in Local database storage

APIs & Services

Service Purpose Model/Features
OpenAI AI Analysis GPT-4o-mini for sentiment & insights
Firecrawl Web Scraping Search API, Scrape API

Data & Visualization

Library Purpose
Pandas Data manipulation and tables
Plotly Interactive charts (pie, bar)
python-dotenv Environment variable management
requests HTTP client for API calls

🔐 Security Considerations

Aspect Implementation
API Keys Stored in .env file, never committed to git
Data Storage Local SQLite database only
User Privacy Reviews processed anonymously
No Third-Party Sharing Data stays on local machine

📊 Performance Optimizations

  1. Caching System

    • SQLite stores scraped reviews and analysis results
    • Toggle to reuse cached data and save API credits
    • Duplicate review detection prevents redundant storage

  2. API Efficiency

    • Limits reviews to 50 for insight generation (token optimization)
    • Configurable max pages for scraping
    • JSON response format for structured AI outputs

  3. UI Performance

    • Progress indicators for long operations
    • Lazy loading of recent analyses
    • Efficient Plotly chart rendering

🔧 Extension Points

ECom Intel can be extended in several ways:

Extension Implementation Approach
New E-commerce Sites Add URL patterns to firecrawl_client.py
Different AI Models Modify model parameter in review_analyzer.py
Export Functionality Add export methods to database.py
Additional Visualizations Add new chart functions to app.py
Multi-language Support Add language detection and translation

Back to: Home | Getting-Started

Clone this wiki locally