apache-beam-python-sdk

Apache Beam is a unified programming model for Batch and Streaming data processing.This image provides the Python SDK of Apache Beam. The Python SDK for Apache Beam provides a simple, powerful API for building batch and streaming data processing pipelines.

Download this Image

The image is available on cgr.dev:

docker pull cgr.dev/chainguard/apache-beam-python-sdk:py3.11-v2.58

Get started with the Beam Python SDK quickstart to set up your Python development environment, get the Beam SDK for Python, and run an example pipeline.

To get you quickly started, we are giving you an example that uses DirectRunner. The Apache Beam examples directory has many examples. All examples can be run locally by passing the required arguments described in the example script.

For example, run wordcount.py with the following command:

container_id=$(docker run -d \
    --entrypoint bash \
    "cgr.dev/chainguard/apache-beam-python-sdk:py3.11-v2.58" \
    -c "
      echo 'Running Apache Beam wordcount with DirectRunner...';
      python -m apache_beam.examples.wordcount \
        --input=/etc/profile \
        --output=/output/part \
        --runner=DirectRunner;
      echo 'Listing /output inside container:';
      ls -l /output;
    ")
``
Once the container process is over, copy the output file from the container to the local system

```bash
docker cp "${container_id}:/output/" "${OUTPUT_DIR}"

Here ${OUTPUT_DIR} is any directory you want the output to be copied at, the output would be of format ${OUTPUT_DIR}/part-00000-of-00001, you can ls the content of the output and that will show you the word count map of each word and how many times it occured in that input file.

We have another example of using PortableRunner documented in our TESTING.md, you can also refer to official docs for further experiments