data-prepper

packaged by Chainguard

Chainguard Container for data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.

Chainguard Containers are regularly-updated, secure-by-default container images.

Download this Container Image

For those with access, this container image is available on cgr.dev:

docker pull cgr.dev/ORGANIZATION/data-prepper:latest

Be sure to replace the ORGANIZATION placeholder with the name used for your organization's private repository within the Chainguard Registry.

Compatibility Notes

Chainguard's data-prepper image uses an OpenJDK 17 JRE and the same configuration paths as the upstream opensearchproject/data-prepper image. It runs as data_prepper (UID and GID 1000) from /usr/share/data-prepper and includes the upstream default configuration and keystore. The JRE omits upstream JDK compilation and diagnostic tools.

File permissions differ from upstream: most files under /usr/share/data-prepper are owned by root and cannot be modified by data_prepper. The writable directories are:

• /usr/share/data-prepper/log/ and /usr/share/data-prepper/logs/ for logs.
• /usr/share/data-prepper/data/ for plugin data.
• /var/log/data-prepper/ for custom logging configurations.

The bundled keystore.p12 is also owned by data_prepper. For file sinks or plugins that write elsewhere, use one of the writable directories above or mount a volume writable by UID 1000.

Getting Started

Data Prepper reads its pipeline definitions from /usr/share/data-prepper/pipelines/pipelines.yaml. Create a pipeline that accepts HTTP log events and writes them to the container logs:

cat > pipelines.yaml <<'EOF'
log-pipeline:
  source:
    http:
      ssl: false
  sink:
    - stdout: {}
EOF

Mount the pipeline and start the image:

docker run -d --name data-prepper \
    -p 2021:2021 -p 4900:4900 \
    -v "$(pwd)/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml:ro" \
    cgr.dev/ORGANIZATION/data-prepper:latest

Send a log event and inspect the output for hello from Data Prepper:

curl --fail-with-body --retry 10 --retry-connrefused --retry-delay 2 \
    -H 'Content-Type: application/json' \
    -d '[{"message":"hello from Data Prepper"}]' \
    http://localhost:2021/log/ingest
docker logs data-prepper

The default configuration serves the core API over HTTPS on port 4900 using the bundled self-signed keystore.p12, with HTTP basic authentication admin/admin.

Replace the default credentials and keystore before production use by mounting your own data-prepper-config.yaml and keystore.

To override the server configuration, mount your own file to /usr/share/data-prepper/config/data-prepper-config.yaml.

Configuration

The image sets the following environment variables, matching upstream:

• DATA_PREPPER_PATH=/usr/share/data-prepper
• ENV_CONFIG_FILEPATH=/usr/share/data-prepper/config/data-prepper-config.yaml
• ENV_PIPELINE_FILEPATH=/usr/share/data-prepper/pipelines/pipelines.yaml

Pass additional JVM options with the JAVA_OPTS environment variable. Any directory you mount for writable data, such as logs, must be writable by UID 1000.

Data Prepper's core API listens on port 4900 by default; the ports used by sources (for example 2021 for the HTTP source) depend on your pipeline configuration.

The upstream layout is preserved through links: /usr/share/data-prepper points to the installation at /usr/share/java/data-prepper, and /usr/bin/data-prepper points to its launcher. Paths under /usr/share/data-prepper, including bind mounts for configuration and pipeline files, work the same as in the upstream image.

Deploying with Helm

You can deploy this image with the OpenSearch Data Prepper Helm chart. Create a values.yaml file that points the chart at the Chainguard image:

cat > values.yaml <<'EOF'
image:
  repository: cgr.dev/ORGANIZATION/data-prepper
  tag: "latest"
EOF

Be sure to replace ORGANIZATION with the name used for your organization's private repository within the Chainguard registry.

Then install the chart:

helm repo add opensearch https://opensearch-project.github.io/helm-charts
helm repo update

helm install data-prepper opensearch/data-prepper \
  --namespace data-prepper \
  --create-namespace \
  --values values.yaml \
  --wait

The chart mounts data-prepper-config.yaml, log4j2-rolling.properties, and your pipeline configuration under /usr/share/data-prepper. Set your pipelines with the chart's pipelineConfig values.

Documentation and Resources

• Data Prepper Documentation
• Data Prepper GitHub Repository
• OpenSearch Helm Charts

What are Chainguard Containers?

Chainguard's free tier of Starter container images are built with Wolfi, our minimal Linux undistro.

All other Chainguard Containers are built with Chainguard OS, Chainguard's minimal Linux operating system designed to produce container images that meet the requirements of a more secure software supply chain.

The main features of Chainguard Containers include:

• Minimal design, without unnecessary software bloat
• Daily builds to ensure container images are up-to-date with available security patches
• High quality build-time SBOMs attesting to the provenance of all artifacts within the image
• Verifiable signatures provided by Sigstore
• Reproducible builds with Cosign and apko (read more about reproducibility)

For cases where you need container images with shells and package managers to build or debug, most Chainguard Containers come paired with a development, or -dev, variant.

In all other cases, including Chainguard Containers tagged as :latest or with a specific version number, the container images include only an open-source application and its runtime dependencies. These minimal container images typically do not contain a shell or package manager.

Although the -dev container image variants have similar security features as their more minimal versions, they include additional software that is typically not necessary in production environments. We recommend using multi-stage builds to copy artifacts from the -dev variant into a more minimal production image.

Need additional packages?

To improve security, Chainguard Containers include only essential dependencies. Need more packages? Chainguard customers can use Custom Assembly to add packages, either through the Console, chainctl, or API.

To use Custom Assembly in the Chainguard Console: navigate to the image you'd like to customize in your Organization's list of images, and click on the Customize image button at the top of the page.

Learn More

Refer to our Chainguard Containers documentation on Chainguard Academy. Chainguard also offers VMs and Libraries — contact us for access.

Trademarks

This software listing is packaged by Chainguard. The trademarks set forth in this offering are owned by their respective companies, and use of them does not imply any affiliation, sponsorship, or endorsement by such companies.