When processing large-scale data, Pandas often faces issues like running out of memory or running slowly. Polars is a high-performance DataFrame library built on Rust that can boost processing speed by 10 to 100 times while keeping a similar API. Polars 2.0 has released its first candidate version, with the stable release expected in the coming weeks. This article will guide you from scratch to master the core usage of Polars, and provide practical steps for migrating from Pandas.

Why Choose Polars

• Speed: Polars is 5–10 times faster than Pandas on typical tasks, and can reach 10–100 times faster in some benchmarks.
• Memory Efficiency: Uses Apache Arrow columnar storage, supports lazy evaluation and streaming, resulting in lower memory footprint.
• Open Source: MIT license, active community, with over 6 million downloads and 19,000+ GitHub stars as of the announcement.
• Parallelization: Native multi-threading and SIMD optimization, utilizing multiple cores without extra configuration.

Installation and Version

bash pip install polars

Importing convention:

python import polars as pl print(pl.version) # e.g., 1.44.2

Getting Started

Creating a DataFrame

python df = pl.DataFrame({ 'name': ['Alice', 'Bob', 'Charlie'], 'age': [25, 30, 35], 'salary': [8200, 9500, 12000], })

Reading CSV

python df = pl.read_csv('employees.csv')

Basic Operations

python

Select columns

df.select('name', 'salary')

Filter

df.filter(pl.col('age') > 25)

Add columns

df.with_columns((pl.col('salary') / 1000).alias('salary_k'))

Sort

df.sort('salary', descending=True)

Group and aggregate

df.group_by('dept').agg( pl.col('salary').mean().alias('avg_salary'), pl.len().alias('count'), )

Lazy Evaluation: The Key to Performance

For large files, it is recommended to use scan_csv instead of read_csv, as it only builds a query plan and does not load the data immediately.

python

Lazy reading

lazy_df = pl.scan_csv('large_data.csv')

Build query

result = ( lazy_df .filter(pl.col('amount') > 1000) .select('user_id', 'amount') .group_by('user_id') .agg(pl.col('amount').sum().alias('total')) .sort('total', descending=True) .head(100) .collect() # Actual execution )

Polars' query optimizer automatically performs optimizations like predicate pushdown and projection pushdown, reading only the necessary data.

Migrating from Pandas

Conversion:

python

Pandas -> Polars

pl_df = pl.from_pandas(pd_df)

Polars -> Pandas

pd_df = pl_df.to_pandas()

Key Features

• Immutability: All operations return a new DataFrame, eliminating the confusion of inplace=True.
• Strict Typing: null and NaN are distinguished, and integers are not implicitly converted to floats.
• Copy-on-Write: Cloning is almost zero-cost.
• Nested Lists: Native support, whereas Pandas requires object dtype.

Important Notes

1. Row Order Not Guaranteed: In Polars 2.0, the streaming engine does not guarantee row order for operations like join and group_by by default. If order preservation is needed, set maintain_order=True.
2. Strict Type Conversion: Version 2.0 removes many implicit conversions. For example, type mismatches in is_in will throw errors, requiring explicit conversion.
3. Engine Affinity: You can fall back to the old engine using pl.Config.set_engine_affinity('in-memory').

Next Steps

• Try replacing read_csv with scan_csv in your existing Pandas scripts to experience lazy evaluation.
• Add maintain_order=True to operations that require order preservation.
• Consult the official migration guide: https://pola.rs/posts/announcing-polars-2/
• Use pl.Config.set_engine_affinity to switch engines based on your needs.

With the steps above, you can achieve significant performance improvements without changing too much code. Polars' design philosophy is "fast and clean", making it ideal for processing large-scale datasets.