Metadata-Version: 2.4
Name: xgdb-connect
Version: 1.2.2
Summary: Connector framework for xrayGraphDB — consume external sources, write over xrayProtocol
Author: eMTAi LLC
Author-email: Damon Sauer <deepvoice@gmail.com>
License-Expression: LicenseRef-Proprietary
Requires-Python: >=3.11
Requires-Dist: pyyaml>=6.0
Provides-Extra: all
Requires-Dist: aiohttp>=3.9; extra == 'all'
Requires-Dist: azure-cosmos>=4.5; extra == 'all'
Requires-Dist: azure-eventhub-checkpointstoreblob-aio>=1.1; extra == 'all'
Requires-Dist: azure-eventhub>=5.11; extra == 'all'
Requires-Dist: azure-servicebus>=7.11; extra == 'all'
Requires-Dist: boto3>=1.34; extra == 'all'
Requires-Dist: confluent-kafka>=2.3; extra == 'all'
Requires-Dist: grpcio-reflection>=1.60; extra == 'all'
Requires-Dist: grpcio>=1.60; extra == 'all'
Requires-Dist: msgpack>=1.0; extra == 'all'
Requires-Dist: mysql-replication>=1.0; extra == 'all'
Requires-Dist: nats-py>=2.7; extra == 'all'
Requires-Dist: paho-mqtt>=2.0; extra == 'all'
Requires-Dist: protobuf>=4.25; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: redis>=5.0; extra == 'all'
Requires-Dist: websockets>=12.0; extra == 'all'
Provides-Extra: cosmosdb
Requires-Dist: azure-cosmos>=4.5; extra == 'cosmosdb'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: eventhub
Requires-Dist: azure-eventhub-checkpointstoreblob-aio>=1.1; extra == 'eventhub'
Requires-Dist: azure-eventhub>=5.11; extra == 'eventhub'
Provides-Extra: grpc
Requires-Dist: grpcio-reflection>=1.60; extra == 'grpc'
Requires-Dist: grpcio>=1.60; extra == 'grpc'
Requires-Dist: protobuf>=4.25; extra == 'grpc'
Provides-Extra: kafka
Requires-Dist: confluent-kafka>=2.3; extra == 'kafka'
Provides-Extra: mqtt
Requires-Dist: paho-mqtt>=2.0; extra == 'mqtt'
Provides-Extra: msgpack
Requires-Dist: msgpack>=1.0; extra == 'msgpack'
Provides-Extra: mysql
Requires-Dist: mysql-replication>=1.0; extra == 'mysql'
Provides-Extra: nats
Requires-Dist: nats-py>=2.7; extra == 'nats'
Provides-Extra: parquet
Requires-Dist: pyarrow>=14.0; extra == 'parquet'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Provides-Extra: servicebus
Requires-Dist: azure-servicebus>=7.11; extra == 'servicebus'
Provides-Extra: webhook
Requires-Dist: aiohttp>=3.9; extra == 'webhook'
Provides-Extra: websocket
Requires-Dist: websockets>=12.0; extra == 'websocket'
Description-Content-Type: text/markdown

# xgdb-connect -- Connector Framework for xrayGraphDB

Standalone Python processes that consume data from external sources and write
to xrayGraphDB over xrayProtocol (port 7689).

## Architecture

```
xgdb_connect/           Core framework
  protocol.py           xrayProtocol v1 binary client
  base.py               BaseConnector (connect, transform, send, checkpoint)
  config.py             YAML config loader
  transform.py          Message-to-Cypher transformer

connectors/             Source-specific implementations
  nats_connector.py     NATS/JetStream
  redis_connector.py    Redis Streams (XREADGROUP)
  kafka_connector.py    Apache Kafka (consumer groups)
  postgres_connector.py PostgreSQL logical replication (pgoutput CDC)
  mysql_connector.py    MySQL binlog CDC
```

## Installation

```bash
# Core only
pip install .

# With specific connector dependencies
pip install ".[nats]"
pip install ".[redis]"
pip install ".[kafka]"
pip install ".[postgres]"
pip install ".[mysql]"

# Everything
pip install ".[all]"
```

## Usage

Each connector reads a YAML config file and runs as a standalone process:

```bash
python -m connectors.nats_connector --config nats.yaml
python -m connectors.redis_connector --config redis.yaml
python -m connectors.kafka_connector --config kafka.yaml
python -m connectors.postgres_connector --config postgres.yaml
python -m connectors.mysql_connector --config mysql.yaml
```

## Config Format

```yaml
source: nats
connection:
  url: nats://localhost:4222
  subject: "events.>"

xraygraphdb:
  host: localhost
  port: 7689
  user: admin
  password: secret

transform:
  node:
    label: Event
    id_field: event_id
    properties:
      - source_field: timestamp
        target: ts
      - source_field: type
        target: event_type
  edge:
    type: TRIGGERED_BY
    from_field: event_id
    to_field: parent_id

batch_size: 1000
checkpoint_file: /var/lib/xgdb-connect/nats.checkpoint
```

## Features

- **Batch writes** -- configurable batch size, UNWIND-based Cypher for efficiency
- **Checkpoint tracking** -- saves last offset/position to disk for restart recovery
- **Backpressure** -- batches are flushed when full, with configurable size
- **Graceful shutdown** -- handles SIGTERM/SIGINT, flushes remaining batch, saves checkpoint
- **Structured logging** -- timestamped log output with configurable level
- **Metrics** -- messages/sec, latency, error counts logged at configurable intervals
- **Reconnection** -- automatic reconnect to xrayGraphDB with configurable retry (max 2 attempts)
- **Transform engine** -- configurable field mappings with dot-path extraction and type casting

## Copyright

Copyright 2026 eMTAi LLC. All rights reserved. Patent pending.
