# Tailpipe: A New Era in SIEM, Powered by DuckDB and Parquet

> The SIEM landscape is evolving. Tailpipe redefines what a SIEM can be: open source, local-first, and built for speed with DuckDB and Parquet.

By Turbot Team
Published: 2025-01-30


SIEM (Security Information and Event Management) is evolving. From legacy platforms to cloud-native systems, and now to open source, local-first solutions, this progression  mirrors the broader story of big data: more flexibility, more accessibility, and more control.

[Tailpipe](/) is the next step in this journey. Lightweight, fully open source, and powered by [DuckDB](https://duckdb.org) and [Parquet](https://parquet.apache.org/), it stretches the boundaries of SIEM. Tailpipe is fast, flexible, and developer-first: a natural fit for DevOps teams constrained by heavy, vendor-locked tools.

## Three eras of SIEM

SIEM has evolved through three distinct phases, each bringing new approaches to security data management.

### Legacy

Centralized systems like Splunk, ArcSight, and QRadar pioneered log collection and analysis. But they require heavy  infrastructure, use specialized query languages, incur high costs, and lock users into proprietary ecosystems.

### Cloud database

Cloud-native platforms like Panther introduce familiar standards (SQL, Python) and use the cloud (Snowflake) for large-scale storage. They are more accessible, but still require heavy infrastructure and impose hard trade-offs on cost vs. scale.

### Open, local-first

Tailpipe represents a new approach: local-first, open source, standards-based. Logs are collected, enriched, standardized, and filtered using open source plugins. Data is stored in optimized Parquet format with a simple hive structure that's easy to manage locally or in low-cost cloud object storage. DuckDB and SQL put lightning-fast query right at your fingertips.

## Why is Tailpipe different?

Tailpipe overcomes the limitations of traditional systems while empowering developers.

### Open source and transparent

Everything about Tailpipe — from its [CLI](https://github.com/turbot/tailpipe) to its [plugins](https://github.com/topics/tailpipe-plugin) and [mods](https://github.com/topics/tailpipe-mod) — is open source. This transparency enables collaboration, spurs innovation, and nurtures a thriving community in which developers contribute and build together.

With ready-to-go plugins for [AWS](https://hub.tailpipe.io/plugins/turbot/aws), [GCP](https://hub.tailpipe.io/plugins/turbot/gcp), [Azure](https://hub.tailpipe.io/plugins/turbot/azure), and more, Tailpipe makes log collection simple and extensible.

### Logs at your fingertips with DuckDB + Parquet

Tailpipe brings the power of modern analytics directly to your laptop.

**Blazing-fast queries**: DuckDB ("SQLite for analytics") delivers in-memory performance: slice gigabytes of data in seconds.

**Optimized storage**: Parquet’s columnar format minimizes I/O and works seamlessly with Tailpipe’s hive-style directory structure, enabling rapid query across specific time ranges or datasets.

This "practical scale" — analyzing hundreds of millions of log entries locally —
is a game changer for day-to-day work and lays the foundation for large-scale
programs.

### Actionable insights with "Detections as Code"

When combined with [Detection mods in Powerpipe](https://powerpipe.io/blog/powerpipe-detection-mods), Tailpipe queries become actionable insights that help teams move from reactive troubleshooting to proactive security and operational excellence.

Its open source detection mods align with frameworks like [MITRE ATT&CK](https://attack.mitre.org/), providing a structured approach to identifying anomalies, unauthorized access, or misconfigurations. Pre-built benchmarks offer a starting point for quick wins while remaining fully customizable to meet unique organizational needs.

With Tailpipe, the transition from exploratory analysis to continuous monitoring is seamless, from SQL queries to reusable benchmarks that drive automation. Teams gain insights they can act on to stay ahead of risks and optimize cloud operations.

## Join the next generation

Tailpipe isn’t just another SIEM, it’s an open source platform that redefines how we approach log collection, detection, and analysis. With its local-first design, SQL-driven workflows, and the unmatched power of DuckDB + Parquet, Tailpipe sets a new standard for flexibility, speed, and cost-efficiency. Ready to experience the future of SIEM? [Download Tailpipe](/downloads) today and [join our community](https://turbot.com/community/join).