S3 files support in Parquet Java

Jerónimo López
Jerónimo López
7 min read

When I started working with Apache Parquet, out-of-the-box support in Java was largely limited to local files and Hadoop’s HDFS. However, because Parquet is designed for efficient processing of large-scale datasets, it is most commonly used in cloud-native and distributed environments where files reside in object stores like Amazon S3.

Historically, reading and writing Parquet files directly in S3 required integrating the hadoop-aws module. While functional, this approach forces your project to bring in heavy Hadoop dependencies—introducing a large footprint, complex dependency trees, and potential security vulnerabilities—all while requiring you to configure Hadoop configurations and use s3a:// paths instead of standard s3:// URIs.

Alternative workarounds, such as downloading S3 objects to local temporary disk before reading or writing locally before uploading, introduce unnecessary I/O overhead and latency.

Although Parquet originated within the Hadoop ecosystem alongside MapReduce and Spark, the file format itself has completely decoupled from Hadoop infrastructure over time. Modern Java applications can read and write Parquet files standalone and stream data directly to and from cloud storage.

To bridge this gap cleanly, Carpet (along with other libraries built on top of parquet-java) can now interact directly with Amazon S3. The new carpet-s3 module provides a lightweight bridge between Parquet’s input/output stream abstractions and the AWS SDK for Java, eliminating the need for Hadoop dependencies altogether.

Adding S3 support

The carpet-s3 module is published as an independent artifact that does not depend on the core carpet-record library. Because it directly implements parquet-java’s InputFile and OutputFile abstractions, you can use it in any Java application built on top of parquet-java, whether you are using Carpet or not.

To get started, add the carpet-s3 module alongside your existing Parquet or Carpet setup.

With Maven:

<dependency>
    <groupId>com.jerolba</groupId>
    <artifactId>carpet-record</artifactId>
    <version>0.8.2</version>
</dependency>
<dependency>
    <groupId>com.jerolba</groupId>
    <artifactId>carpet-s3</artifactId>
    <version>0.8.2</version>
</dependency>

Or with Gradle:

implementation 'com.jerolba:carpet-record:0.8.2'
implementation 'com.jerolba:carpet-s3:0.8.2'

The module relies on the AWS SDK for Java v2. By default, it resolves credentials and regions automatically using the standard AWS credential provider chain, so applications running with IAM roles, environment variables, or local AWS profiles work out of the box without extra configuration.

Reading a Parquet file from S3

Reading Parquet files directly from S3 requires only swapping the standard InputFile implementation for S3InputFile:

record Event(long id, String name, long timestamp) { }

InputFile inputFile = S3InputFile.of("s3://my-bucket/events/data.parquet");
List<Event> events = new CarpetReader<>(inputFile, Event.class).toList();

S3InputFile parses the s3:// URI to identify the bucket and object key, handling all low-level HTTP range requests under the hood while Carpet maps the Parquet schema to your Java record.

Optimizing transfers with Projection

Because Parquet is a columnar format, you only pay for the data you read. When you map a file to a record containing only a subset of the available fields, Carpet requests only the specific column chunks required for those fields:

record EventName(String name) { }

List<String> names = new CarpetReader<>(inputFile, EventName.class)
    .stream()
    .map(EventName::name)
    .toList();

In this case, S3InputFile fetches only the byte ranges corresponding to the name column over HTTP. Unread columns are never downloaded, significantly reducing bandwidth usage and I/O latency when querying large datasets in S3.

Writing a Parquet file to S3

Writing Parquet files directly to S3 follows the exact same pattern. Simply pass an S3OutputFile instance to the writer:

OutputFile outputFile = S3OutputFile.of("s3://my-bucket/events/output.parquet");

try (CarpetWriter<Event> writer = new CarpetWriter<>(outputFile, Event.class)) {
    writer.write(events);
}

Under the hood, S3OutputFile handles streaming the generated Parquet data directly to Amazon S3 using multipart uploads, ensuring efficient resource usage without requiring local staging files or manual interaction with the AWS S3 client.

The resulting artifact is a standard Parquet file fully compatible with any other tool or engine in the ecosystem—such as Apache Spark, DuckDB, or AWS Athena.

What happens behind the scenes?

Amazon S3 is an object store, not a traditional file system. Because objects are retrieved over HTTP, treating a remote S3 object like a local random-access file requires deliberate optimization.

Parquet’s architecture relies on reading the file’s footer—located at the very end of the file—to parse the schema, row group locations, and column statistics before any data can be read.

S3InputFile optimizes reading through a two-step approach:

  • Footer Caching: It issues an initial HTTP ranged request to eagerly fetch the last 8 MiB of the file. In the vast majority of cases, this single request retrieves the entire footer and metadata, eliminating multiple network round-trips before data processing even begins.
  • Parallel Ranged Reads: Once the footer is parsed, data pages are retrieved on demand using targeted HTTP range requests. Only the exact byte ranges belonging to the requested columns are fetched, and these requests can be issued in parallel.
  • Speculative Prefetching: While the CPU decodes and processes the current row group’s data, S3InputFile asynchronously fetches the byte ranges for the next row group in the background. Overlapping I/O and CPU execution minimizes processing time.

Writing to S3 presents a different challenge. Parquet files are written sequentially, but S3 requires objects to be committed cleanly. S3OutputFile solves this using S3 Multipart Uploads: data is buffered into memory chunks and uploaded concurrently while the writer continues generating the next blocks.

All of these internal optimizations—caching, parallel ranged reads, and multipart streaming—are encapsulated behind parquet-java’s standard InputFile and OutputFile interfaces. From the application’s perspective, reading and writing from/to S3 feels identical to working with local disk files.

Customizing the S3 client

For most standard S3 deployments, the default AWS SDK auto-configuration works out of the box:

InputFile inputFile = S3InputFile.of("s3://my-bucket/events/data.parquet");

Internally, the AWS SDK for Java v2 resolves credentials, regions, and endpoint configurations automatically via the default provider chain—checking environment variables, system properties, local profiles, container metadata, or IAM execution roles.

Advanced configuration with S3Client

When your application requires fine-grained control—such as setting explicit regions, connecting to custom S3-compatible endpoints (e.g., MinIO or Ceph), tuning timeouts, or configuring retries—you can supply your own pre-configured S3Client:

S3Client customClient = S3Client.builder()
    .region(Region.EU_WEST_1)
    .credentialsProvider(ProfileCredentialsProvider.create("prod-profile"))
    .build();

InputFile inputFile = S3InputFile.builder("s3://my-bucket/events/data.parquet")
    .withS3Client(customClient)
    .build();

Tuning I/O performance

The builder pattern also provides direct tuning options for I/O operations, allowing you to optimize throughput according to your application’s memory footprint and concurrency requirements:

  • Multipart upload chunk sizes: Adjust the buffer size for parallel S3 multipart uploads when writing files.
  • Concurrency limits: Set the maximum number of parallel HTTP GET and PUT requests per reader and writer.
  • Custom Executor: Provide a dedicated ExecutorService for async I/O tasks and background prefetching, ensuring carpet operations stay isolated within your managed thread pools.

S3-compatible services

Because carpet-s3 relies entirely on the official AWS SDK for Java v2, it works seamlessly out of the box with any S3-compatible object storage solution—including LocalStack for local testing, Cloudflare R2, MinIO, or Ceph.

To target an alternative object store, simply configure the S3Client with a custom endpoint override and your target credentials:

S3Client client = S3Client.builder()
    .endpointOverride(URI.create("https://<account-id>.r2.cloudflarestorage.com"))
    .credentialsProvider(StaticCredentialsProvider.create(
        AwsBasicCredentials.create("access-key", "secret-key")
    ))
    .region(Region.US_EAST_1)
    .build();

OutputFile outputFile = S3OutputFile.builder("s3://my-bucket/events/output.parquet")
    .withS3Client(client)
    .build();

Seamless local development fallback

A key design feature for developer experience is transparent local path fallback. S3InputFile and S3OutputFile automatically inspect the target URI scheme: if the path does not start with s3://, the library automatically delegates operations to the local file system implementation.

// Uses local filesystem under the hood
InputFile inputFile = S3InputFile.of("/tmp/events.parquet");
OutputFile outputFile = S3OutputFile.of("/tmp/output.parquet");

This eliminates the need for if/else environment toggles or complex filesystem abstraction wrappers in your business logic. You can write your data pipelines using S3InputFile/S3OutputFile everywhere, execute fast unit/integration tests against local files or temporary directories, and deploy the exact same code path to production targeting Amazon S3.

Conclusion

Storing Parquet files in Amazon S3 is standard practice in modern data pipelines. However, interacting with them from Java applications historically forced a trade-off: either accept the heavy weight and dependency footprint of Hadoop’s filesystem abstractions, or manage inefficient local disk round-trips.

carpet-s3 provides a lightweight, zero-Hadoop alternative. By integrating directly at Parquet’s low-level I/O layer, it delivers built-in footer caching, parallel ranged GETs, background multipart uploads, and seamless local fallbacks through a clean, intuitive API. Crucially, because it implements parquet-java’s native interfaces, it can be adopted by any Java application using parquet-java, regardless of whether you use Carpet’s record mapper.

Most importantly, the programming model remains clean and unified: switching from a local filesystem to cloud storage is as simple as changing a path string to an s3:// URI, without modifying your data models, schemas, or processing logic.

Next Steps

To explore advanced configurations, including fine-tuning upload/download buffers, custom thread pool execution, system properties, and pre-download modes, check out the Carpet S3 Documentation.

The full source code and issue tracker are hosted on GitHub, and artifacts are available directly via Maven Central.