Kettler Data Analysis

AI-Powered property management licensing investigation platform. Python-first architecture with advanced machine learning capabilities.

Last Updated: December 10, 2025

About this project

This platform helps investigate property management licensing compliance across multiple states. It searches licenses, analyzes connections between firms and individuals, and generates research outputs for regulatory compliance investigations.

What you can do:

🤖 AI-Powered Analysis: ML-enhanced violation detection, clustering, and risk scoring
🔍 Semantic Search: Vector embeddings for intelligent similarity matching
🗺️ Multi-state License Search: Search licenses across 15 states
🔗 Connection Mapping: Graph theory and network analysis
🚨 Anomaly Detection: ML models identify unusual patterns and fraud
📄 Evidence Extraction: AI-powered PDF and Excel document processing
📊 Predictive Analytics: Time series analysis and violation prediction
📈 Comprehensive Reports: ML-enhanced research reports with explainable AI

Quick Start

Choose your path based on what you need to do:

Filing administrative complaints

Start here if you're preparing regulatory complaints:

VA DPOR Complaint Files - Complete complaint research
Research Index - Master research index
Research README - Research directory guide

Understanding findings

Start here to explore research results:

Research Index - Master file index
VA DPOR Complaint Files - Complaint research
Research Reports - Summary reports

Data analysis

Start here for data exploration:

Firm Data - 38 firms
Connections - Connection analyses
Research Reports - Analysis reports

🤖 AI/ML capabilities

Explore advanced machine learning features:

ML Pipeline - Advanced ML pipeline with TensorFlow
ML Analysis Results - Clustering, anomaly detection, classification
Graph Theory Analysis - NetworkX graph analysis
Embedding Analysis - Vector similarity results
Implementation Summary - Complete ML feature documentation

System overview

Architecture Diagram

graph TB
    subgraph "Data Sources"
        A["📁 Source Files<br/><a href='data/source/'>View Data</a>"]
        B["📊 Research Data<br/><a href='research/'>Explore Research</a>"]
        C["🔍 License Databases<br/><a href='research/license_searches/'>Search Licenses</a>"]
    end

    subgraph "ETL Pipeline"
        D["📥 Extract"]
        E["🔄 Transform"]
        F["💾 Load"]
    end

    subgraph "AI/ML Analysis"
        G["🧠 NLP & Embeddings<br/>Sentence Transformers<br/><a href='scripts/analysis/advanced_ml_pipeline.py'>View Pipeline</a>"]
        H["📈 Graph Theory<br/>NetworkX Analysis<br/><a href='data/processed/graph_theory_analysis.json'>View Analysis</a>"]
        I["🤖 ML Pipeline<br/>Clustering & Classification<br/><a href='scripts/analysis/ml_tax_structure_analysis.py'>View ML</a>"]
        I2["🚨 Anomaly Detection<br/>Isolation Forest, LOF<br/><a href='data/processed/ml_tax_structure_analysis.json'>View Results</a>"]
    end

    subgraph "Outputs"
        J["📄 Research Reports<br/><a href='research/reports/'>View Reports</a>"]
        K["📊 Visualizations<br/><a href='data/processed/'>View Data</a>"]
        L["✅ Compliance Data<br/><a href='data/processed/cross_referenced_violations.json'>View Violations</a>"]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H
    F --> I
    F --> I2
    G --> J
    H --> K
    I --> L
    I2 --> L

    style A fill:#fbbf24,stroke:#f59e0b,stroke-width:3px
    style B fill:#fbbf24,stroke:#f59e0b,stroke-width:3px
    style C fill:#fbbf24,stroke:#f59e0b,stroke-width:3px
    style D fill:#60a5fa,stroke:#3b82f6,stroke-width:3px
    style E fill:#60a5fa,stroke:#3b82f6,stroke-width:3px
    style F fill:#60a5fa,stroke:#3b82f6,stroke-width:3px
    style G fill:#8b5cf6,stroke:#7c3aed,stroke-width:4px
    style H fill:#34d399,stroke:#10b981,stroke-width:3px
    style I fill:#3b82f6,stroke:#2563eb,stroke-width:4px
    style I2 fill:#ef4444,stroke:#dc2626,stroke-width:4px
    style J fill:#4ade80,stroke:#22c55e,stroke-width:3px
    style K fill:#4ade80,stroke:#22c55e,stroke-width:3px
    style L fill:#4ade80,stroke:#22c55e,stroke-width:3px

System Components

Aspect	Description
Purpose	Multi-state license search, connection analysis, and regulatory compliance investigation
Architecture	Python-first with unified core modules, ETL pipeline, and optional API/web frontend
Data Flow	Source → Extract → Clean → Analyze → Research Outputs
Processing	Parallel processing with 32 workers (ARM M4 MAX optimized)
Throughput	~5,000 files/second processing speed

Installation

Install dependencies and run the pipeline:

git clone https://github.com/1digitaldesign/kettler-data-analysis.git
cd kettler-data-analysis
pip install -r requirements.txt
python bin/run_pipeline.py

See INSTALLATION.md for detailed setup instructions.

Requirements: Python 3.14 or higher

Usage

Run the full pipeline

python bin/run_pipeline.py

This runs the complete data processing pipeline:

Data extraction
Data cleaning
Connection analysis
Data validation
Report generation

Run individual scripts

python bin/analyze_connections.py  # Connection analysis
python bin/validate_data.py        # Data validation
python bin/clean_data.py          # Data cleaning
python bin/generate_reports.py    # Report generation

Run AI/ML Analysis

# Advanced ML pipeline with TensorFlow
python scripts/analysis/advanced_ml_pipeline.py

# ML tax structure analysis (clustering, anomaly detection)
python scripts/analysis/ml_tax_structure_analysis.py

# Embedding-based similarity analysis
python scripts/analysis/embedding_violation_analysis.py

# Graph theory network analysis
python scripts/analysis/graph_theory_analysis.py

# Complete violation analysis pipeline
python scripts/analysis/run_complete_violation_analysis.py

# Create comprehensive visualization suite
python scripts/analysis/create_all_visualizations.py

Generate Visualizations

# Create all available visualizations
python scripts/analysis/create_all_visualizations.py

# Visualizations are saved to:
# research/texas/analysis/visualizations/

This creates interactive visualizations using:

Plotly: Interactive web charts (HTML)
Bokeh: Browser-based visualizations
Altair: Statistical charts
Seaborn: Statistical plots (PNG)
Comprehensive Dashboard: All visualizations in one HTML file

Documentation

Getting started

INSTALLATION.md - Setup guide
QUICK_START.md - Quick start
STATUS.md - Current status

System documentation

System Architecture - Complete architecture (components, data flow, structure, diagrams)
Repository Structure - Detailed file organization
System Analyst Guide - System analyst guide

Data documentation

Data structure:

Schema - FK/PK relationships
Data Dictionary - Field definitions
Ontology - Conceptual relationships
Ancestry - Data lineage
Metadata - Global metadata

Data governance:

Data Catalog - Comprehensive data catalog (discoverability, metadata, quality)
Data Governance - Governance framework (policies, compliance, security)

Documentation index

Documentation Index - All documentation
Documentation Graph - Interactive documentation network

Documentation network:

graph LR
    README[README.md] --> INDEX[docs/INDEX.md]
    INDEX --> ARCH[docs/SYSTEM_ARCHITECTURE.md]
    INDEX --> DATA[data/DATA_DICTIONARY.md]
    INDEX --> RESEARCH[research/README.md]

    style README fill:#C8E6C9,stroke:#4CAF50,stroke-width:3px
    style INDEX fill:#B3E5FC,stroke:#2196F3,stroke-width:2px

Research status

Status: 100% complete. All critical areas documented, evidence compiled, ready for complaint filing.

Statistics

Metric	Value	Status
Total Files	350 JSON + 30 MD	✅ Complete
Research Categories	19 categories	✅ Categorized
License Searches	285 files across 15 states	✅ Searched
Firms	38 firms	✅ Analyzed
Individual Licenses	40+ licenses	✅ Documented
Connections	100+ connections	✅ Mapped
Processing Speed	~5,000 files/second	🚀 Optimized
Data Quality	99.3%	✅ Excellent

Research Distribution

pie title Research Files by Category
    "Texas Data" : 5353
    "License Searches" : 580
    "Analysis" : 22
    "VA DPOR Complaint" : 22
    "Company Registrations" : 20
    "Other Categories" : 88

Key findings

Finding	Value	Impact
Regulatory Violations	8 violations across 11 states	🔴 Critical
Principal Broker Gap	10.5 years	⚠️ Significant
Geographic Violation	1,300 miles	🔴 Critical
Unlicensed Personnel	16 (7 property managers)	⚠️ High Risk
Property Value Managed	$4.75B	💰 Substantial

Violation Analysis

graph LR
    A["🚨 8 Violations<br/><a href='data/processed/cross_referenced_violations.json'>View Details</a>"] --> B["🗺️ 11 States<br/><a href='research/company_registrations/'>View States</a>"]
    A --> C["⚠️ 16 Unlicensed<br/><a href='research/analysis/'>View Analysis</a>"]
    A --> D["💰 $4.75B Property<br/><a href='research/financial/'>View Financial</a>"]

    B --> E["⚖️ Regulatory Risk<br/><a href='research/reports/'>View Reports</a>"]
    C --> E
    D --> E

    style A fill:#ef4444,stroke:#dc2626,stroke-width:4px
    style B fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style C fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style D fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style E fill:#dc2626,stroke:#991b1b,stroke-width:4px

System structure

Directory Structure

graph TD
    A["📦 Repository Root<br/><a href='https://github.com/1digitaldesign/kettler-data-analysis'>GitHub</a>"] --> B["⚙️ bin/<br/><a href='bin/'>Entry Points</a>"]
    A --> C["📜 scripts/<br/><a href='scripts/'>View Scripts</a>"]
    A --> D["💾 data/<br/><a href='data/'>View Data</a>"]
    A --> E["🔬 research/<br/><a href='research/'>View Research</a>"]
    A --> F["📚 docs/<br/><a href='docs/'>View Docs</a>"]

    B --> B1["🚀 Entry Points"]

    C --> C1["🔧 core/"]
    C --> C2["📊 analysis/<br/><a href='scripts/analysis/'>View Analysis</a>"]
    C --> C3["🔄 etl/<br/><a href='scripts/etl/'>View ETL</a>"]
    C --> C4["🛠️ utils/"]

    D --> D1["📥 source/"]
    D --> D2["✅ processed/<br/><a href='data/processed/'>View Processed</a>"]
    D --> D3["🧹 cleaned/"]
    D --> D4["🔢 vectors/"]

    E --> E1["📈 analysis/"]
    E --> E2["🔍 license_searches/<br/><a href='research/license_searches/'>View Searches</a>"]
    E --> E3["🏢 company_registrations/"]

    F --> F1["🏗️ System Architecture<br/><a href='docs/SYSTEM_ARCHITECTURE.md'>View Docs</a>"]
    F --> F2["📑 Documentation Index<br/><a href='docs/INDEX.md'>View Index</a>"]

    style A fill:#fbbf24,stroke:#f59e0b,stroke-width:4px
    style B fill:#60a5fa,stroke:#3b82f6,stroke-width:3px
    style C fill:#34d399,stroke:#10b981,stroke-width:3px
    style D fill:#4ade80,stroke:#22c55e,stroke-width:3px
    style E fill:#f87171,stroke:#ef4444,stroke-width:3px
    style F fill:#a78bfa,stroke:#8b5cf6,stroke-width:3px

Component Breakdown

Directory	Purpose	Files
bin/	Entry points and executables	Pipeline scripts
scripts/core/	Unified core modules	Shared utilities
scripts/analysis/	Analysis scripts	ML, graph theory, violations
scripts/etl/	ETL pipeline	Data processing
data/	All data files	Source, processed, vectors
research/	Research outputs	6,085+ JSON files
docs/	Documentation	Architecture, guides

Advanced Visualization Libraries

This project uses modern, interactive visualization libraries for publication-quality charts and dashboards:

Library	Purpose	Features	Visualization Types
Plotly (5.18.0+)	Interactive web visualizations	3D plots, animations, dashboards	Scatter, 3D scatter, heatmaps, box plots, violin plots, sunburst, treemap, parallel coordinates, Sankey diagrams, network graphs
Plotly Express	High-level interface	Simplified API for common charts	All Plotly chart types with simplified syntax
Dash (2.14.0+)	Interactive web dashboards	Python web apps, real-time	Full dashboard applications with Bootstrap components
Bokeh (3.3.0+)	Browser-based interactive charts	Real-time updates, streaming data	Scatter plots, network graphs, time series
Altair (5.2.0+)	Declarative statistical viz	Grammar of graphics, JSON export	Scatter, bar, line charts, statistical visualizations
Seaborn (0.13.0+)	Statistical data visualization	Beautiful default styles	Pair plots, correlation matrices, distribution plots
NetworkX (3.2.0+)	Graph visualization	Network analysis, layouts	Network graphs, community detection visualizations
Kaleido	Static image export	Export Plotly to PNG/SVG	All Plotly charts as static images

Available Visualization Types

Interactive Charts (Plotly):

✅ 2D & 3D Scatter Plots
✅ Cluster Visualizations
✅ Correlation Heatmaps
✅ Box Plots & Violin Plots
✅ Sunburst & Treemap Charts
✅ Parallel Coordinates
✅ Sankey Diagrams
✅ Network Graphs
✅ Time Series Charts
✅ Anomaly Detection Visualizations

Statistical Charts (Altair):

✅ Scatter Plots
✅ Bar Charts
✅ Line Charts
✅ Statistical Distributions

Statistical Plots (Seaborn):

✅ Pair Plots
✅ Correlation Matrices
✅ Distribution Plots

Network Visualizations:

✅ Interactive Network Graphs (Plotly)
✅ Browser-based Networks (Bokeh)
✅ Community Detection Visualizations

Dashboards:

✅ Comprehensive HTML Dashboards
✅ Interactive Web Applications (Dash)
✅ Multi-chart Dashboards

All visualizations are:

🎨 Interactive: Hover, zoom, pan, click interactions
📊 Publication-ready: Professional styling and themes
💾 Exportable: HTML, PNG, SVG, PDF formats
🌓 Theme-aware: Works in light and dark modes
📱 Responsive: Adapts to different screen sizes
⚡ Modern: Latest visualization technologies

AI & Machine Learning Capabilities

🤖 Advanced ML Pipeline

This platform leverages state-of-the-art AI/ML technologies for intelligent analysis:

graph TB
    subgraph "AI/ML Stack"
        A["🧠 Sentence Transformers<br/>all-MiniLM-L6-v2<br/><a href='scripts/analysis/advanced_ml_pipeline.py'>View Pipeline</a>"]
        B["⚡ TensorFlow<br/>Parallel Processing<br/><a href='scripts/analysis/advanced_ml_pipeline.py'>View Config</a>"]
        C["📊 Scikit-Learn<br/>ML Algorithms<br/><a href='scripts/analysis/ml_tax_structure_analysis.py'>View Analysis</a>"]
    end

    subgraph "Clustering"
        D["🎯 K-Means<br/>Optimal Cluster Detection<br/><a href='data/processed/ml_tax_structure_analysis.json#clustering'>View Results</a>"]
        E["🔍 DBSCAN<br/>Density-Based Clustering<br/><a href='data/processed/ml_tax_structure_analysis.json#clustering'>View Results</a>"]
        F["🌳 Hierarchical<br/>Tax Structure Analysis<br/><a href='data/processed/ml_tax_structure_analysis.json#clustering'>View Results</a>"]
        G["📈 Spectral<br/>Network-Based Clustering<br/><a href='data/processed/ml_tax_structure_analysis.json#clustering'>View Results</a>"]
    end

    subgraph "Anomaly Detection"
        H["🚨 Isolation Forest<br/>Unusual Patterns<br/><a href='data/processed/ml_tax_structure_analysis.json#anomaly'>View Results</a>"]
        I["⚠️ Local Outlier Factor<br/>Abnormal Entities<br/><a href='data/processed/ml_tax_structure_analysis.json#anomaly'>View Results</a>"]
        J["🔬 One-Class SVM<br/>Shell Company Detection<br/><a href='data/processed/ml_tax_structure_analysis.json#anomaly'>View Results</a>"]
    end

    subgraph "Classification & Explainability"
        K["🌲 Random Forest<br/>Feature Importance<br/><a href='data/processed/ml_tax_structure_analysis.json#classification'>View Results</a>"]
        L["⚡ XGBoost<br/>Gradient Boosting<br/><a href='data/processed/ml_tax_structure_analysis.json#classification'>View Results</a>"]
        M["💡 SHAP Values<br/>Explainable AI<br/><a href='data/processed/ml_tax_structure_analysis.json#classification'>View Results</a>"]
    end

    subgraph "Network & Embeddings"
        N["🌐 NetworkX<br/>Graph Analysis<br/><a href='data/processed/graph_theory_analysis.json'>View Analysis</a>"]
        O["🔢 Vector Embeddings<br/>Semantic Similarity<br/><a href='data/processed/embedding_similarity_analysis.json'>View Results</a>"]
        P["📉 UMAP<br/>Dimensionality Reduction<br/><a href='data/processed/ml_tax_structure_analysis.json#dimensionality'>View Results</a>"]
    end

    A --> D
    A --> E
    A --> F
    A --> G
    B --> H
    B --> I
    B --> J
    C --> K
    C --> L
    C --> M
    A --> O
    N --> O
    O --> P

    style A fill:#8b5cf6,stroke:#7c3aed,stroke-width:4px
    style B fill:#3b82f6,stroke:#2563eb,stroke-width:4px
    style C fill:#10b981,stroke:#059669,stroke-width:4px
    style D fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style E fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style F fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style G fill:#f59e0b,stroke:#d97706,stroke-width:3px
    style H fill:#ef4444,stroke:#dc2626,stroke-width:3px
    style I fill:#ef4444,stroke:#dc2626,stroke-width:3px
    style J fill:#ef4444,stroke:#dc2626,stroke-width:3px
    style K fill:#06b6d4,stroke:#0891b2,stroke-width:3px
    style L fill:#06b6d4,stroke:#0891b2,stroke-width:3px
    style M fill:#06b6d4,stroke:#0891b2,stroke-width:3px
    style N fill:#a78bfa,stroke:#8b5cf6,stroke-width:3px
    style O fill:#a78bfa,stroke:#8b5cf6,stroke-width:3px
    style P fill:#a78bfa,stroke:#8b5cf6,stroke-width:3px

ML Capabilities Overview

Category	Technology	Use Case	Status
Embeddings	Sentence Transformers (all-MiniLM-L6-v2)	Semantic similarity, violation matching	✅ Active
Parallel Processing	TensorFlow	High-performance batch processing	✅ Optimized
Clustering	K-Means, DBSCAN, Hierarchical, Spectral	Pattern discovery, entity grouping	✅ 4 Algorithms
Anomaly Detection	Isolation Forest, LOF, One-Class SVM	Fraud detection, unusual patterns	✅ 3 Methods
Classification	Random Forest, XGBoost	Risk scoring, violation prediction	✅ 2 Models
Explainability	SHAP Values	Model interpretability	✅ Available
Network Analysis	NetworkX	Graph theory, community detection	✅ Complete
Dimensionality Reduction	PCA, UMAP	Feature visualization	✅ 2 Methods
Vector Search	Cosine Similarity	Similar violation discovery	✅ Active
Risk Scoring	Multi-model Ensemble	ML-enhanced risk assessment	✅ Production
Visualizations	Plotly, Bokeh, Altair	Interactive, publication-quality	✅ Modern

AI-Powered Features

🧠 Natural Language Processing

Sentence Embeddings: Transform text into 384-dimensional vectors using state-of-the-art transformer models
Semantic Similarity: Find similar violations using cosine similarity on embeddings
Document Understanding: Extract meaning from legal documents, forms, and violations

🎯 Intelligent Clustering

K-Means with Elbow Method: Automatically determine optimal cluster count
DBSCAN: Density-based clustering for outlier detection
Hierarchical Clustering: Build tax structure hierarchies
Spectral Clustering: Network-based pattern discovery

🚨 Anomaly Detection

Isolation Forest: Detect unusual tax structures and patterns
Local Outlier Factor (LOF): Identify abnormal entities
One-Class SVM: Find shell company patterns

🤖 Predictive Analytics

Random Forest: Feature importance analysis and classification
XGBoost: Gradient boosting for high-accuracy predictions
SHAP Values: Explain model decisions with interpretable AI

🌐 Graph Intelligence

NetworkX Analysis: Community detection, centrality measures
Graph Theory: Shortest path algorithms (Dijkstra, all simple paths)
PageRank: Identify critical nodes in violation networks

📊 Advanced Analytics

Time Series Analysis: Trend detection and future violation predictions
UMAP Visualization: High-dimensional data visualization
PCA: Feature reduction and analysis

Performance Metrics

Metric	Value	Technology
Embedding Model	all-MiniLM-L6-v2	Sentence Transformers
Vector Dimensions	384	Optimized for speed/accuracy
Parallel Workers	32 (ARM M4 MAX)	TensorFlow optimized
Batch Processing	128 items/batch	Memory optimized
Clustering Speed	<1 second for 1000 entities	Scikit-learn optimized
Anomaly Detection	Real-time	Isolation Forest
Model Accuracy	High (ensemble methods)	Multi-model approach

Features

Core Capabilities

mindmap
  root((Kettler Analysis))
    AI/ML Powered
      Sentence Transformers
      TensorFlow Processing
      Clustering Algorithms
      Anomaly Detection
      Explainable AI
    License Search
      15 States
      285 Searches
      Bar Licenses
    Connection Mapping
      Graph Theory
      Network Analysis
      Community Detection
    Anomaly Detection
      ML Models
      Isolation Forest
      Pattern Recognition
    Evidence Extraction
      PDF Parsing
      Excel Analysis
      Document Processing
    Data Analysis
      Vector Embeddings
      Timeline Analysis
      Schema Validation

Feature Matrix

Feature	Status	Performance	AI/ML Enhanced
Multi-state License Search	✅ Complete	15 states covered	🔍 Semantic search
Connection Mapping	✅ Complete	Graph theory analysis	🧠 ML-powered clustering
Anomaly Detection	✅ Complete	ML-enhanced detection	🤖 3 ML algorithms
Evidence Extraction	✅ Complete	PDF/Excel support	📊 NLP processing
Vector Embeddings	✅ Complete	Semantic search ready	🧠 Transformer models
Timeline Analysis	✅ Complete	Temporal patterns	📈 Time series ML
Schema Validation	✅ Complete	99.3% quality score	✅ Automated
Risk Scoring	✅ Complete	Multi-model ensemble	🤖 ML-enhanced

Research Status: 100% Complete - Ready for Complaint Filing

Name		Name	Last commit message	Last commit date
Latest commit History 218 Commits
bin		bin
data		data
docs		docs
kubernetes		kubernetes
outputs/reports		outputs/reports
ref/law		ref/law
research		research
scripts		scripts
services		services
web-nextjs		web-nextjs
.cursorrules		.cursorrules
.gitignore		.gitignore
.python-version		.python-version
CODE_FIXES_SUMMARY.md		CODE_FIXES_SUMMARY.md
INSTALLATION.md		INSTALLATION.md
INSTALL_PYTHON_3.14.md		INSTALL_PYTHON_3.14.md
INVESTIGATION_READY.md		INVESTIGATION_READY.md
QUICK_START.md		QUICK_START.md
README.md		README.md
STATUS.md		STATUS.md
SUMMARY.md		SUMMARY.md
requirements.txt		requirements.txt
state_dpor_registry.csv		state_dpor_registry.csv

1digitaldesign/kettler-data-analysis

Folders and files

Latest commit

History

Repository files navigation