Custom Tables

Tailpipe plugins define tables for common log sources and formats. You don't need to define these tables; simply create one or more partition for the table and begin collecting logs.

But what if your logs are not in a standard format or are not currently supported by a plugin? No problem! Custom tables enable you to collect data from arbitrary log files and other sources.

The process is straightforward:

• First define a format that describes how to extract the fields from the source.
• Next, define a table that transforms and maps the source format into a destination table structure.
• Create one or more partition for your table, specifying a source from which to collect logs
• Collect and query as you would for any tailpipe table.

Define the Format

The format block enables you to define source formats for custom tables. Formats describe the layout of the source data so that it can be collected into a table.

Format blocks have two labels:

• The format type. This can be a core format type (grok, regex, delimited, or jsonl) or any format type in any installed plugin.
• A name for the format

For example, Steampipe plugin logs are syslog-style text files. EVERY row has a timestamp, timezone, severity, and message. Most log lines will also contain plugin-specific data - the plugin name, plugin severity, and plugin timestamp in epoch seconds:

We can define a format that defines this structure. In this example, we use the grok format type.

Tip

Use backticks (`) to delimit the layout. Tailpipe treats anything in backticks as a non-interpolated string, so you don't have to escape quotes, backslashes, etc.

Tip

Use the Grok Debugger to help create and test your grok expressions.

Define the Table

Custom tables are defined in a table block. Table blocks have a single label, which defines the name of the table.

You may define the format of the source. In this example, we will use the format that we created previously. By default, all the fields defined in format will be included as columns in the table. If you want, you can use map_fields to include only specific columns.

You may also use one or more column definitions to map fields to map and transform data from the source.

In our example, the source format does not define a field named tp_timestamp. Since tp_timestamp is a required column, we will add a tp_timestamp column and map the timestamp from the source. Also, the source includes a plugin_timestamp, but it is parsed as a number because it is epoch milliseconds. We will transform it to a timestamp data type.

Note

You cannot set the tp_index mapping in the table definition. The tp_index can only be configured through the partition config, where it defaults to "default" if not specified.

Create Partitions

Now that we have the table and format defined, we can create a partition and add a source from which to collect.

The partition has two labels:

• The table name. The table name must match a table name for an installed plugin or a custom table. In this case, we will use the table we created earlier.
• A partition name. The partition name must be unique for all partitions in a given table (though different tables may use the same partition names).

You can use any source from any plugin. In our example, we will use the file source from the core plugin that ships with Tailpipe. Note that if your directory structure and/or file names are arranged by date/time, you should specify the structure in the file_layout so that Tailpipe can use it in the collection state to optimize the collection process.

Collect & Query

You can collect custom logs with tailpipe collect, just like any other logs.

The next time you run tailpipe query, your table will be available to query!