# Collect and query custom logs with Tailpipe

> Custom log parsing to collect and query any log type.

By Turbot Team
Published: 2025-04-16

Tailpipe's set of [plugins](https://hub.tailpipe.io/) lets you collect and query logs from a variety of sources, including [AWS](https://hub.tailpipe.io/plugins/turbot/aws), [GCP](https://hub.tailpipe.io/plugins/turbot/gcp), [Azure](https://hub.tailpipe.io/plugins/turbot/azure), and more. But what if you have  logs that don't fit into any of these categories?

The latest release of Tailpipe can now collect and query any log type, even if it doesn't have a plugin. This is done by creating a [custom table](https://tailpipe.io/docs/collect/custom-tables) that defines the schema for your logs. This unlocks the ability to use the power of Tailpipe for pretty much any log type you can think of.

## Configure Tailpipe to collect custom logs

For a custom log, you need to define a [partition](https://tailpipe.io/docs/reference/config-files/partition), [format](https://tailpipe.io/docs/reference/config-files/format), and [table](https://tailpipe.io/docs/reference/config-files/table). Tailpipe has several built-in formats, and formats can also be provided by plugins. This example uses the `jsonl` format, which is one of the core formats (others include `grok`, `regex`, and `delimited`). While Tailpipe already has a [plugin for GitHub audit logs](https://hub.tailpipe.io/plugins/turbot/github), we can also use the custom feature to have more flexibility (and to demonstrate this feature).

```hcl
partition "my_github" "demo" {
  source "file" {
    paths       = ["/Users/mattstratton/logs_tailpipe/custom"]
    file_layout = `%{DATA}.json.gz`
  }
}

table "my_github" {
  format = format.jsonl.default

  column "tp_timestamp" {
    transform = `to_timestamp("@timestamp" / 1000)`
  }
}
```

The above table definition uses the `jsonl.default` format preset, which means we don't have to tell Tailpipe anything about the structure of the logs - use of the `default` preset tells Tailpipe to expect a simple JSON object per line.

We do need to tell Tailpipe where to find the timestamp for our logs, however. In this case, the timestamp of our example logs stores the timestamp as Unix epoch time in milliseconds. We use the `transform` function to convert this to a standard DuckDB timestamp format.

Take note also of the use of backticks (\`) around the `transform` function. This is a new feature in Tailpipe that allows us to use backticks instead of escaping quotes, making the configuration cleaner and easier to read.

That's all we need to start collecting logs!

## Collecting the custom logs
  
We collect our logs with `tailpipe collect`, just like any other logs. 

```shell
$ tailpipe collect my_github.demo --from 2024-04-01
Collecting logs for my_github.demo from 2024-04-01

Artifacts:
  Discovered: 2
  Downloaded: 2 1.3MB
  Extracted:  2

Rows:
  Received: 21,718
  Enriched: 21,718
  Saved:    21,074
  Filtered:    644

Files:
  Compacted: 676 => 345

Completed: 4s
```

If we select from this table, we can see there are a lot of columns:

```shell
$ tailpipe query "select * from my_github limit 5;"
+---------------+------------------------+----------------------+--------+-------------------+----------+--------------+----------------+----------------+--------+----------+--------------+-----------------------------------+--------------------------------------+-----------------------+------------------------+--------+--------+----------+-------------+------------+--------+------------------+---------------+-------------------------------+-------------------+--------+--------+---------+----------------------------------------------+-------------+----------+---------+------------------------------------+-------------+---------------+---------+---------+----------------------------------------------+-----------------------------+-------------------------------+-------------------------------+------------+--------+--------+----------------------+----------+----------------+----------------+------------+---------+------------------+--------+------------+------------+--------+-----------------------------------+------------+--------------+--------------+----------------+-------------+-----------------+-----------------+----------------------------------------+------------------------------------------------------------------------+---------------------------------------------------------+--------+---------+----------+---------------------------+----------+--------------------+--------------------------+----------------------+----------------------------+---------------+----------------------+--------------------------------+------------------+---------------------------+----------------------------+---------------------------------+----------------------------------------+-----------------------------------------------------+------------------------------------------+-----------+---------------+------------+-------------+--------------------------+-----------------------------------------+------------+--------------------------------------+--------+--------+-------------+--------------+--------+------------+-----------+---------------------------------------------------------------------------------------------------------------+----------+------------+-----------------------------+-------------------------------+-------------+-----------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
| @timestamp    | _document_id           | action               | active | actor             | actor_id | actor_is_bot | actor_location | admin_enforced | after  | alert_id | alert_number | allow_deletions_enforcement_level | allow_force_pushes_enforcement_level | application_client_id | authorized_actor_names | before | branch | business | business_id | comment_id | config | create_protected | created_at    | dismiss_stale_reviews_on_push | enforcement_level | event  | events | ghsa_id | hashed_token                                 | head_branch | head_sha | hook_id | ignore_approvals_from_contributors | integration | invitation_id | invitee | inviter | linear_history_requirement_enforcement_level | lock_allows_fetch_and_merge | lock_branch_enforcement_level | merge_queue_enforcement_level | multi_repo | name   | number | oauth_application_id | old_name | old_permission | operation_type | org        | org_id  | overridden_codes | owner  | owner_type | permission | plan   | programmatic_access_type          | project_id | project_kind | project_name | public_project | public_repo | publicly_leaked | pull_request_id | pull_request_reviews_enforcement_level | pull_request_title                                                     | pull_request_url                                        | reason | reasons | referrer | repo                      | repo_id  | repositories_added | repositories_added_names | repositories_removed | repositories_removed_names | repository_id | repository_selection | request_access_security_header | request_category | require_code_owner_review | require_last_push_approval | required_approving_review_count | required_deployments_enforcement_level | required_review_thread_resolution_enforcement_level | required_status_checks_enforcement_level | review_id | reviewer_type | run_number | secret_type | secret_type_display_name | signature_requirement_enforcement_level | started_at | strict_required_status_checks_policy | team   | title  | token_id    | token_scopes | topic  | trigger_id | user      | user_agent                                                                                                    | user_id  | visibility | vulnerability_alert_rule_id | vulnerability_alert_rule_name | workflow_id | workflow_run_id | tp_akas | tp_date             | tp_destination_ip | tp_domains | tp_emails | tp_id                                | tp_index | tp_ingest_timestamp | tp_ips | tp_partition | tp_source_ip | tp_source_location | tp_source_name | tp_source_type | tp_table  | tp_tags | tp_timestamp        | tp_usernames |
+---------------+------------------------+----------------------+--------+-------------------+----------+--------------+----------------+----------------+--------+----------+--------------+-----------------------------------+--------------------------------------+-----------------------+------------------------+--------+--------+----------+-------------+------------+--------+------------------+---------------+-------------------------------+-------------------+--------+--------+---------+----------------------------------------------+-------------+----------+---------+------------------------------------+-------------+---------------+---------+---------+----------------------------------------------+-----------------------------+-------------------------------+-------------------------------+------------+--------+--------+----------------------+----------+----------------+----------------+------------+---------+------------------+--------+------------+------------+--------+-----------------------------------+------------+--------------+--------------+----------------+-------------+-----------------+-----------------+----------------------------------------+------------------------------------------------------------------------+---------------------------------------------------------+--------+---------+----------+---------------------------+----------+--------------------+--------------------------+----------------------+----------------------------+---------------+----------------------+--------------------------------+------------------+---------------------------+----------------------------+---------------------------------+----------------------------------------+-----------------------------------------------------+------------------------------------------+-----------+---------------+------------+-------------+--------------------------+-----------------------------------------+------------+--------------------------------------+--------+--------+-------------+--------------+--------+------------+-----------+---------------------------------------------------------------------------------------------------------------+----------+------------+-----------------------------+-------------------------------+-------------+-----------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
| 1710905872881 | isf3hi6IKXPncM45PN1KhA | issue_comment.update | <null> | netlify[bot]      | 40209326 | true         | <null>         | <null>         | <null> | <null>   | <null>       | <null>                            | <null>                               | <null>                | <null>                 | <null> | <null> | <null>   | <null>      | <null>     | <null> | <null>           | 1710905872881 | <null>                        | <null>            | <null> | <null> | <null>  | ro7FUz99uqrzSOr7spjAo/aBDYWPHWfoDoRSh6ouwCo= | <null>      | <null>   | <null>  | <null>                             | <null>      | <null>        | <null>  | <null>  | <null>                                       | <null>                      | <null>                        | <null>                        | <null>     | <null> | <null> | <null>               | <null>   | <null>         | modify         | devopsdays | 2938293 | <null>           | <null> | <null>     | <null>     | <null> | GitHub App server-to-server token | <null>     | <null>       | <null>       | <null>         | true        | <null>          | <null>          | <null>                                 | <null>                                                                 | <null>                                                  | <null> | <null>  | <null>   | devopsdays/devopsdays-web | 45844314 | <null>             | <null>                   | <null>               | <null>                     | <null>        | <null>               | <null>                         | <null>           | <null>                    | <null>                     | <null>                          | <null>                                 | <null>                                              | <null>                                   | <null>    | <null>        | <null>     | <null>      | <null>                   | <null>                                  | <null>     | <null>                               | <null> | <null> | 85582235393 | <null>       | <null> | <null>     | <null>    | Octokit Ruby Gem 4.22.0                                                                                       | <null>   | <null>     | <null>                      | <null>                        | <null>      | <null>          | <null>  | 2024-03-19 00:00:00 | <null>            | <null>     | <null>    | 0dcd7918-6cec-4a35-9a6b-d449de932116 | default  | 2025-04-14 23:45:12 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-03-20 03:37:52 | <null>       |
| 1710905763388 | DK8s-uWa6n5yGL7e-JE07Q | issue_comment.update | <null> | netlify[bot]      | 40209326 | true         | <null>         | <null>         | <null> | <null>   | <null>       | <null>                            | <null>                               | <null>                | <null>                 | <null> | <null> | <null>   | <null>      | <null>     | <null> | <null>           | 1710905763388 | <null>                        | <null>            | <null> | <null> | <null>  | ro7FUz99uqrzSOr7spjAo/aBDYWPHWfoDoRSh6ouwCo= | <null>      | <null>   | <null>  | <null>                             | <null>      | <null>        | <null>  | <null>  | <null>                                       | <null>                      | <null>                        | <null>                        | <null>     | <null> | <null> | <null>               | <null>   | <null>         | modify         | devopsdays | 2938293 | <null>           | <null> | <null>     | <null>     | <null> | GitHub App server-to-server token | <null>     | <null>       | <null>       | <null>         | true        | <null>          | <null>          | <null>                                 | <null>                                                                 | <null>                                                  | <null> | <null>  | <null>   | devopsdays/devopsdays-web | 45844314 | <null>             | <null>                   | <null>               | <null>                     | <null>        | <null>               | <null>                         | <null>           | <null>                    | <null>                     | <null>                          | <null>                                 | <null>                                              | <null>                                   | <null>    | <null>        | <null>     | <null>      | <null>                   | <null>                                  | <null>     | <null>                               | <null> | <null> | 85582235393 | <null>       | <null> | <null>     | <null>    | Octokit Ruby Gem 4.22.0                                                                                       | <null>   | <null>     | <null>                      | <null>                        | <null>      | <null>          | <null>  | 2024-03-19 00:00:00 | <null>            | <null>     | <null>    | c4cf4071-3648-408f-8fda-b682c77d0d5c | default  | 2025-04-14 23:45:12 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-03-20 03:36:03 | <null>       |
| 1710902220282 | BVHf7Sh57OnUc1lWhUW9UQ | issue_comment.update | <null> | netlify[bot]      | 40209326 | true         | <null>         | <null>         | <null> | <null>   | <null>       | <null>                            | <null>                               | <null>                | <null>                 | <null> | <null> | <null>   | <null>      | <null>     | <null> | <null>           | 1710902220282 | <null>                        | <null>            | <null> | <null> | <null>  | mYeGZ0CV9mayrnILSentfVXziqQUddTqmWnNdHmjn38= | <null>      | <null>   | <null>  | <null>                             | <null>      | <null>        | <null>  | <null>  | <null>                                       | <null>                      | <null>                        | <null>                        | <null>     | <null> | <null> | <null>               | <null>   | <null>         | modify         | devopsdays | 2938293 | <null>           | <null> | <null>     | <null>     | <null> | GitHub App server-to-server token | <null>     | <null>       | <null>       | <null>         | true        | <null>          | <null>          | <null>                                 | <null>                                                                 | <null>                                                  | <null> | <null>  | <null>   | devopsdays/devopsdays-web | 45844314 | <null>             | <null>                   | <null>               | <null>                     | <null>        | <null>               | <null>                         | <null>           | <null>                    | <null>                     | <null>                          | <null>                                 | <null>                                              | <null>                                   | <null>    | <null>        | <null>     | <null>      | <null>                   | <null>                                  | <null>     | <null>                               | <null> | <null> | 85574914084 | <null>       | <null> | <null>     | <null>    | Octokit Ruby Gem 4.22.0                                                                                       | <null>   | <null>     | <null>                      | <null>                        | <null>      | <null>          | <null>  | 2024-03-19 00:00:00 | <null>            | <null>     | <null>    | 283bfacc-5176-430f-a949-e51665bff519 | default  | 2025-04-14 23:45:12 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-03-20 02:37:00 | <null>       |
| 1710902079531 | -jBgkLRhNJcJ6snU4Q1Jrw | issue_comment.update | <null> | unfurl-links[bot] | 33112876 | true         | <null>         | <null>         | <null> | <null>   | <null>       | <null>                            | <null>                               | <null>                | <null>                 | <null> | <null> | <null>   | <null>      | <null>     | <null> | <null>           | 1710902079531 | <null>                        | <null>            | <null> | <null> | <null>  | WwELTHjStmlKP70VtHH5R2k4QZxYzwQ1UecRrgwONFU= | <null>      | <null>   | <null>  | <null>                             | <null>      | <null>        | <null>  | <null>  | <null>                                       | <null>                      | <null>                        | <null>                        | <null>     | <null> | <null> | <null>               | <null>   | <null>         | modify         | devopsdays | 2938293 | <null>           | <null> | <null>     | <null>     | <null> | GitHub App server-to-server token | <null>     | <null>       | <null>       | <null>         | true        | <null>          | <null>          | <null>                                 | <null>                                                                 | <null>                                                  | <null> | <null>  | <null>   | devopsdays/devopsdays-web | 45844314 | <null>             | <null>                   | <null>               | <null>                     | <null>        | <null>               | <null>                         | <null>           | <null>                    | <null>                     | <null>                          | <null>                                 | <null>                                              | <null>                                   | <null>    | <null>        | <null>     | <null>      | <null>                   | <null>                                  | <null>     | <null>                               | <null> | <null> | 85577577999 | <null>       | <null> | <null>     | <null>    | octokit.js/15.18.3 Node.js/12.22.11 (Linux 5.15; x64)                                                         | <null>   | <null>     | <null>                      | <null>                        | <null>      | <null>          | <null>  | 2024-03-19 00:00:00 | <null>            | <null>     | <null>    | 62c5d893-1fae-49d1-84bd-474eae69738f | default  | 2025-04-14 23:45:12 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-03-20 02:34:39 | <null>       |
| 1710902052772 | jaQrW36AmX81QuJrhrmQMA | pull_request.create  | <null> | skryukova         | 32201499 | false        | <null>         | <null>         | <null> | <null>   | <null>       | <null>                            | <null>                               | <null>                | <null>                 | <null> | <null> | <null>   | <null>      | <null>     | <null> | <null>           | 1710902052772 | <null>                        | <null>            | <null> | <null> | <null>  | <null>                                       | <null>      | <null>   | <null>  | <null>                             | <null>      | <null>        | <null>  | <null>  | <null>                                       | <null>                      | <null>                        | <null>                        | <null>     | <null> | <null> | <null>               | <null>   | <null>         | create         | devopsdays | 2938293 | <null>           | <null> | <null>     | <null>     | <null> | <null>                            | <null>     | <null>       | <null>       | <null>         | true        | <null>          | 1781234047      | <null>                                 | [HOU-2024] Added registration link and dates, removed organizer, fixe… | https://github.com/devopsdays/devopsdays-web/pull/13841 | <null> | <null>  | <null>   | devopsdays/devopsdays-web | 45844314 | <null>             | <null>                   | <null>               | <null>                     | <null>        | <null>               | <null>                         | <null>           | <null>                    | <null>                     | <null>                          | <null>                                 | <null>                                              | <null>                                   | <null>    | <null>        | <null>     | <null>      | <null>                   | <null>                                  | <null>     | <null>                               | <null> | <null> | <null>      | <null>       | <null> | <null>     | skryukova | Mozilla/5.0 (Windows NT 10.0 Win64 x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 | 32201499 | <null>     | <null>                      | <null>                        | <null>      | <null>          | <null>  | 2024-03-19 00:00:00 | <null>            | <null>     | <null>    | 718a85ca-4532-4b38-b325-0c1119e16079 | default  | 2025-04-14 23:45:12 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-03-20 02:34:12 | <null>       |
+---------------+------------------------+----------------------+--------+-------------------+----------+--------------+----------------+----------------+--------+----------+--------------+-----------------------------------+--------------------------------------+-----------------------+------------------------+--------+--------+----------+-------------+------------+--------+------------------+---------------+-------------------------------+-------------------+--------+--------+---------+----------------------------------------------+-------------+----------+---------+------------------------------------+-------------+---------------+---------+---------+----------------------------------------------+-----------------------------+-------------------------------+-------------------------------+------------+--------+--------+----------------------+----------+----------------+----------------+------------+---------+------------------+--------+------------+------------+--------+-----------------------------------+------------+--------------+--------------+----------------+-------------+-----------------+-----------------+----------------------------------------+------------------------------------------------------------------------+---------------------------------------------------------+--------+---------+----------+---------------------------+----------+--------------------+--------------------------+----------------------+----------------------------+---------------+----------------------+--------------------------------+------------------+---------------------------+----------------------------+---------------------------------+----------------------------------------+-----------------------------------------------------+------------------------------------------+-----------+---------------+------------+-------------+--------------------------+-----------------------------------------+------------+--------------------------------------+--------+--------+-------------+--------------+--------+------------+-----------+---------------------------------------------------------------------------------------------------------------+----------+------------+-----------------------------+-------------------------------+-------------+-----------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
```

If we don't need all those columns from the logs, we can limit this to something more reasonable, by updating our table config:

```hcl
table "my_github" {
  format = format.jsonl.default

  column "tp_timestamp" {
    transform = `to_timestamp("@timestamp" / 1000)`
  }

  map_fields = [
    "action",
    "actor",
    "event",
    "org",
    "owner",
    "repo",
    "team",
    "user",
    "visibility"
  ]
}
```

The `map_fields` list tells Tailpipe to only collect the fields in the list, and ignore the rest (unless they are explicitly defined in the table). This is a great way to reduce the amount of data collected, and also make it easier to query.

With our new table definition, we can run the same query again:

```shell
$ tailpipe query "select * from my_github limit 5;"
+----------------------+--------------+------------------------+--------+------------+--------+------------------------------+--------+----------+------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
| action               | actor        | document_id            | event  | org        | owner  | repo                         | team   | user     | visibility | tp_akas | tp_date             | tp_destination_ip | tp_domains | tp_emails | tp_id                                | tp_index | tp_ingest_timestamp | tp_ips | tp_partition | tp_source_ip | tp_source_location | tp_source_name | tp_source_type | tp_table  | tp_tags | tp_timestamp        | tp_usernames |
+----------------------+--------------+------------------------+--------+------------+--------+------------------------------+--------+----------+------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
| pull_request.merge   | somatorio    | 7qruFL5RXaHqzkJvjmf2aQ | <null> | devopsdays | <null> | devopsdays/devopsdays-assets | <null> | strund3r | <null>     | <null>  | 2024-04-10 00:00:00 | <null>            | <null>     | <null>    | d747dea0-75f0-4b37-be19-04a0c838cd41 | default  | 2025-04-15 10:27:52 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-04-10 21:44:56 | <null>       |
| issue_comment.update | netlify[bot] | uKH0SL6-OYwXEa4p9RO9UQ | <null> | devopsdays | <null> | devopsdays/devopsdays-web    | <null> | <null>   | <null>     | <null>  | 2024-04-10 00:00:00 | <null>            | <null>     | <null>    | 625b0acd-91d8-43a4-ad1d-82da840a1b7a | default  | 2025-04-15 10:27:52 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-04-10 21:34:50 | <null>       |
| issue_comment.update | netlify[bot] | 0Bde27xtYnL0m0p4h2gsTA | <null> | devopsdays | <null> | devopsdays/devopsdays-web    | <null> | <null>   | <null>     | <null>  | 2024-04-10 00:00:00 | <null>            | <null>     | <null>    | c2211040-b665-4871-8c73-674ba3fccd11 | default  | 2025-04-15 10:27:52 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-04-10 21:33:00 | <null>       |
| issue_comment.update | netlify[bot] | m1yY5hNf-aozIreUq39K-Q | <null> | devopsdays | <null> | devopsdays/devopsdays-web    | <null> | <null>   | <null>     | <null>  | 2024-04-10 00:00:00 | <null>            | <null>     | <null>    | e9de6e2e-b1ee-4064-ac29-0749f95b5345 | default  | 2025-04-15 10:27:52 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-04-10 21:30:29 | <null>       |
| issue_comment.update | netlify[bot] | R_jOWSKzP-5lJb96hVoAsQ | <null> | devopsdays | <null> | devopsdays/devopsdays-web    | <null> | <null>   | <null>     | <null>  | 2024-04-10 00:00:00 | <null>            | <null>     | <null>    | 123fd5af-c48f-44ce-9422-6000e316e68b | default  | 2025-04-15 10:27:52 | <null> | demo         | <null>       | <null>             | <null>         | <null>         | my_github | <null>  | 2024-04-10 21:28:12 | <null>       |
+----------------------+--------------+------------------------+--------+------------+--------+------------------------------+--------+----------+------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+
```

Now we only see the columns we have defined, although note that the columns starting with `tp_` are still there. These are the [common columns](https://tailpipe.io/docs/reference/config-files/table#common-columns) populated by Tailpipe to enable you to correlate values across different logs in a standard way, and are not affected by the `map_fields` list.

## Define your own format

We aren't limited to the formats that come with Tailpipe. We can define our own! In this case, we create a [`format` block](https://tailpipe.io/docs/reference/config-files/format) using the `regex` type. This is a little more work, but it gives us a lot of flexibility.

```hcl
partition "my_syslog" "demo" {
  source "file"  {
    paths = ["/Users/kai/tailpipe_data/demo/syslogs"]
    file_layout = `.log`
  }
}

table "my_syslog" {
  format = format.regex.example
  
  column "tp_timestamp" {
    source = "timestamp"
  }
}

format "regex" "example" {
  layout = `^(?P<timestamp>\S+) (?P<host>\S+) (?P<service>\w+)\[(?P<pid>\d+)\]: \[(?P<level>[A-Z]+)\] (?P<message>.+)$`
}
```

In the `table` block, we define the `format` as `format.regex.example`, which tells Tailpipe to use the regex format we defined in the `format` block. The `layout` is a regex pattern that matches the log lines, and captures the fields we want to extract.

## See it in action

<div className="flex justify-center">
<iframe 
    class="youtube-video" 
    src="https://www.youtube-nocookie.com/embed/iqnc7fJQ924"
    frameBorder="0" 
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Collect and query custom logs with Tailpipe"
>
</iframe>
</div>

## No plugin, no problem!

With the Tailpipe's new capability to collect custom logs, you can now collect logs from any source, even if there is no plugin available. This can open up vast possiblities to collect logs from any source, and use them in your queries. [Download Tailpipe today](https://tailpipe.io/downloads) and [let us know](https://turbot.com/community/join) what you're collecting!