Metadata-Version: 2.4
Name: kafka_collector
Version: 0.4.0
Summary: collect kafka messages from multiple topics
License: MIT
License-File: LICENSE
Keywords: kafka
Author: Siak Hooi
Author-email: siakhooi@gmail.com
Requires-Python: >=3.10
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: Flask
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: System :: Logging
Classifier: Typing :: Typed
Requires-Dist: flask (>=3.0.0)
Requires-Dist: kafka-python-ng (>=2.2.0)
Project-URL: Bug Tracker, https://github.com/siakhooi/kafka-collector/issues
Project-URL: Documentation, https://github.com/siakhooi/kafka-collector/wiki
Project-URL: Homepage, https://github.com/siakhooi/kafka-collector
Project-URL: Repository, https://github.com/siakhooi/kafka-collector
Description-Content-Type: text/markdown

# kafka-collector

collect kafka messages from multiple topics

## Installation
```
pip install kafka_collector
```
## Usage
```
$ kafka-collector -h
usage: kafka-collector [-h] [-v] [-t TOPICS] [-b BOOTSTRAP_SERVER] [-g GROUP]
                       [-o OUTPUT] [-c CAPTURE_DIR] [-m {cli,service}]
                       [-p PORT]

collect kafka messages from multiple topics

options:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -t, --topics TOPICS   comma separated list of kafka topics to be listened to
  -b, --bootstrap-server BOOTSTRAP_SERVER
                        kafka bootstrap server (default: localhost:9092)
  -g, --group GROUP     consumer group id (default: random uuid)
  -o, --output OUTPUT   output file path, use '-' for stdout (default: stdout)
  -c, --capture-dir CAPTURE_DIR
                        capture directory for service mode (default:
                        /tmp/kafka-collector)
  -m, --mode {cli,service}
                        run mode: cli or service (default: cli)
  -p, --port PORT       service port for service mode (default: 8080)

```
## Environment Variables
- `KAFKA_TOPICS` - comma separated list of topics
- `KAFKA_BOOTSTRAP_SERVER` - bootstrap server address
- `KAFKA_GROUP` - consumer group id
- `COLLECTOR_CAPTURE_DIR` - capture directory for service mode
- `COLLECTOR_MODE` - run mode (cli or service)
- `COLLECTOR_SERVICE_PORT` - service port (must be integer)

## Documentation

- [CLI Mode](docs/cli-mode.md) - Default mode for streaming messages to stdout or file
- [Service Mode](docs/service-mode.md) - HTTP service mode with REST API for capture management

## Deployment Examples

- [Docker Compose](docker-compose/) - Run with Docker Compose
- [Kubernetes](kubernetes/) - Deploy to Kubernetes

## Links
- https://pypi.org/project/kafka_collector/
- https://github.com/siakhooi/kafka-collector
- https://sonarcloud.io/project/overview?id=siakhooi_kafka-collector
- https://qlty.sh/gh/siakhooi/projects/kafka-collector

## Badges
![GitHub](https://img.shields.io/github/license/siakhooi/kafka-collector?logo=github)
![GitHub last commit](https://img.shields.io/github/last-commit/siakhooi/kafka-collector?logo=github)
![GitHub tag (latest by date)](https://img.shields.io/github/v/tag/siakhooi/kafka-collector?logo=github)
![GitHub issues](https://img.shields.io/github/issues/siakhooi/kafka-collector?logo=github)
![GitHub closed issues](https://img.shields.io/github/issues-closed/siakhooi/kafka-collector?logo=github)
![GitHub pull requests](https://img.shields.io/github/issues-pr-raw/siakhooi/kafka-collector?logo=github)
![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed-raw/siakhooi/kafka-collector?logo=github)
![GitHub top language](https://img.shields.io/github/languages/top/siakhooi/kafka-collector?logo=github)
![GitHub language count](https://img.shields.io/github/languages/count/siakhooi/kafka-collector?logo=github)
![GitHub repo size](https://img.shields.io/github/repo-size/siakhooi/kafka-collector?logo=github)
![GitHub code size in bytes](https://img.shields.io/github/languages/code-size/siakhooi/kafka-collector?logo=github)

![Workflow](https://img.shields.io/badge/Workflow-github-purple)
![workflow](https://github.com/siakhooi/kafka-collector/actions/workflows/build.yaml/badge.svg)
![workflow](https://github.com/siakhooi/kafka-collector/actions/workflows/workflow-deployments.yml/badge.svg)

![Release](https://img.shields.io/badge/Release-github-purple)
![GitHub release (latest by date)](https://img.shields.io/github/v/release/siakhooi/kafka-collector?label=GPR%20release&logo=github)
![GitHub all releases](https://img.shields.io/github/downloads/siakhooi/kafka-collector/total?color=33cb56&logo=github)
![GitHub Release Date](https://img.shields.io/github/release-date/siakhooi/kafka-collector?logo=github)

![Quality-Qlty](https://img.shields.io/badge/Quality-Qlty-purple)
[![Maintainability](https://qlty.sh/gh/siakhooi/projects/kafka-collector/maintainability.svg)](https://qlty.sh/gh/siakhooi/projects/kafka-collector)
[![Code Coverage](https://qlty.sh/gh/siakhooi/projects/kafka-collector/coverage.svg)](https://qlty.sh/gh/siakhooi/projects/kafka-collector)

![Quality-Sonar](https://img.shields.io/badge/Quality-SonarCloud-purple)
[![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Bugs](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=bugs)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Technical Debt](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=sqale_index)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=security_rating)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=reliability_rating)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=ncloc)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=siakhooi_kafka-collector&metric=coverage)](https://sonarcloud.io/summary/new_code?id=siakhooi_kafka-collector)
![Sonar Violations (short format)](https://img.shields.io/sonar/violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (short format)](https://img.shields.io/sonar/blocker_violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (short format)](https://img.shields.io/sonar/critical_violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (short format)](https://img.shields.io/sonar/major_violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (short format)](https://img.shields.io/sonar/minor_violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (short format)](https://img.shields.io/sonar/info_violations/siakhooi_kafka-collector?server=https%3A%2F%2Fsonarcloud.io)
![Sonar Violations (long format)](https://img.shields.io/sonar/violations/siakhooi_kafka-collector?format=long&server=http%3A%2F%2Fsonarcloud.io)

[![Wise](https://img.shields.io/badge/Funding-Wise-33cb56.svg?logo=wise)](https://wise.com/pay/me/siakn3)
![visitors](https://hit-tztugwlsja-uc.a.run.app/?outputtype=badge&counter=ghmd-kafka-collector)

