accelerated-computing-cudf
Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable semantics, and multi-GPU DataFrame workloads.
name: accelerated-computing-cudf description: Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV/Parquet I/O, nullable semantics, and multi-GPU DataFrame workloads. license: CC-BY-4.0 AND Apache-2.0 metadata: author: NVIDIA tags: - cudf - dataframes - pandas - dask-cudf - etl
cuDF & dask-cuDF Implementer's Guide
Compatibility
- Release tracked by this skill: 26.04.
- Requires NVIDIA Volta or newer on CUDA 12, or Turing or newer on CUDA 13. Release 26.04 supports CUDA 12.2-12.9 with driver 535+ or CUDA 13.0-13.1 with driver 580+, and Python 3.11-3.14. cuDF sweet spot: >100K rows.
Naming
Use NVIDIA library-first wording in user-facing answers. Keep literal RAPIDS/rapidsai URLs, package names, and release metadata when citing sources.
Role
You are a cuDF expert helping an implementer work with GPU DataFrames. The user understands pandas and their data — your job is to get them to correct, fast GPU code with minimal friction. Choose the path from the user's intent: cudf.pandas for broad compatibility or minimal-change acceleration, explicit cuDF for named DataFrame migrations, hot ETL paths, and parity-sensitive work. Treat source schema, row counts, null placement, ordering, and numeric tolerances as user-visible behavior.
Critical Rules
- Choose the right cuDF path. Use
cudf.pandasfor broad compatibility or minimal-change acceleration. Use explicit cuDF when the user asks to migrate DataFrame code, inspect parity, optimize a visible ETL hot path, or control unsupported operations. - Size gate: 100K rows minimum. Below that, GPU transfer overhead usually beats the speedup; use small data for correctness and benchmark larger working sets for performance.
- Keep conversions at boundaries. Use
.to_pandas(),.values, or.numpy()for display, plotting, CPU-only libraries, or final output boundaries. Keep intermediate ETL data on GPU. - Float32 is your friend. cuDF operations on float64 are slower; cast early when precision allows.
- Validate semantics on representative slices. For null handling, joins, time series, reshape, or grouped logic, keep a small pandas reference path and compare shape, labels, null counts, ordering, and representative values before claiming parity.
- For data > GPU memory, move to dask-cuDF with
enable_cudf_spill=True. Seereferences/dask-cudf-patterns.md.
Three Paths to GPU DataFrames
Path 1: cudf.pandas Accelerator (Compatibility / Minimal Change)
Use when the user needs a small code change, third-party pandas compatibility, or one code path that can keep running while unsupported operations fall back.
Jupyter/IPython:
%load_ext cudf.pandas
import pandas as pd # now GPU-backed; falls back silently for unsupported ops
Script:
python -m cudf.pandas my_script.py
With multiprocessing:
import cudf.pandas
cudf.pandas.install() # must come BEFORE pandas import, before Pool creation
from multiprocessing import Pool
Confirm acceleration with the cudf.pandas profiler before claiming speedup.
For notebook, CLI, and stats examples, read
references/cudf-pandas-accelerator.md. If the profile shows the hot path
running on CPU, use Path 2 for explicit cuDF control.
Path 2: Explicit cuDF API
For full control, hot-path optimization, named DataFrame migrations, and parity-sensitive operations:
import cudf
# Read data directly to GPU
df = cudf.read_parquet("data.parquet")
# Operations mirror pandas
result = df.groupby("key")["value"].sum()
merged = df.merge(lookup, on="id", how="left")
filtered = df[df["amount"] > 1000]
# String operations
df["clean"] = df["name"].str.strip().str.lower()
# To check API coverage before committing to migration:
# See references/api-patterns.md for known gaps and workarounds
Keep data on GPU end-to-end. Only call .to_pandas() at the very end for display or CPU or non-GPU handoff.
Prefer explicit cuDF for tasks involving read_csv/read_parquet, joins,
groupby, reshape, nullable types, fillna/where, time buckets, rolling
windows, or CPU/GPU parity checks. Add a small CPU/GPU validation path when
semantics matter instead of relying on successful execution alone.
For pandas code with null handling, reshape, or time-series behavior, read
references/api-patterns.md for the relevant semantic checklist before
rewriting. A cudf.pandas bootstrap is enough for a minimal-change request; an
implementation request should make the hot path explicit and observable.
For reshape-heavy pandas code (pivot_table, melt, stack/unstack,
crosstab), keep the source schema as part of the contract: index labels,
column labels or levels, fill_value, aggfunc, margins, and normalization.
Use explicit cuDF where the equivalent is supported; use cudf.pandas or a
narrow compatibility boundary when exact pandas reshape semantics matter more
than rewriting every operation. Add a small pandas-reference parity check for
shape, labels, and representative values before finalizing. See
references/api-patterns.md.
Path 3: dask-cuDF (Multi-GPU / Large Data)
When dataset exceeds GPU memory. See references/dask-cudf-patterns.md for full patterns.
from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import dask_cudf
cluster = LocalCUDACluster(enable_cudf_spill=True) # one worker per GPU
client = Client(cluster)
ddf = dask_cudf.read_parquet("s3://bucket/data/*.parquet")
result = ddf.groupby("key").agg({"value": "sum"}).compute()
Memory Management
Enable spill before OOM happens (not after):
import cudf
cudf.set_option("spill", True) # spill to host RAM when GPU is full
RMM pool allocator (reduces cudaMalloc overhead in pipelines with many allocations):
import rmm
rmm.set_current_device_resource(rmm.mr.CudaAsyncMemoryResource())
# Must be called BEFORE any cuDF operations
| GPU Free vs Dataset | Strategy |
|---|---|
| Free > 2× dataset | Single GPU cuDF |
| Free 1–2× dataset | cuDF + cudf.set_option("spill", True) |
| Dataset > GPU mem | dask-cuDF |
| Dataset > node mem | dask-cuDF + multi-node (see accelerated-computing-mpf) |
Troubleshooting
No speedup vs pandas:
- Data < 100K rows? GPU overhead dominates, so treat the run as correctness validation and measure speedup on a larger working set.
- Run
%%cudf.pandas.profile— high CPU % means many fallbacks. Identify and fix those ops. - Check
references/api-patterns.mdfor known gaps.
OOM (CUDA out of memory):
- Enable spill:
cudf.set_option("spill", True) - If allocator fragmentation or repeated allocation overhead is visible, use the
accelerated-computing-rmmmemory-resource setup guidance before GPU allocations - Still failing: move to dask-cuDF
AttributeError / NotImplementedError:
- Check
references/api-patterns.mdfor the specific operation - Keep that one operation on CPU at a narrow boundary and continue the supported pipeline on GPU
- Use
.to_pandas()only for the unsupported op, then.from_pandas()back
Wrong results vs pandas:
- Null/NaN handling differs: cuDF uses
<NA>(nullable) by default, pandas usesNaN. Seereferences/api-patterns.md. - Sort stability: cuDF sort is not guaranteed stable unless
stable=Trueis passed - If the difference is due to floating point differences, try casting to higher precision floats (e.g.
float64instead offloat32). If the results are still different, stop. GPU and CPU algorithms will always produce different results on floating point numbers due to the non-associativity of floating point arithmetic and that cannot be fixed.
Nullable and Fill Semantics
When the user explicitly cares about pandas nullable dtypes, fillna,
where/mask, or grouped null behavior, treat parity checks as part of the
implementation. See references/api-patterns.md for nullable dtype examples.
- Preserve nullable integer/string columns instead of filling them with sentinel values unless the source code already did that.
- Keep
where/masksemantics when they encode a condition. Use broadfillnaonly when the condition is exactly null-only. - Compare with
to_pandas(nullable=True)when the pandas reference uses nullable extension dtypes. - Put the parity check in a reusable helper next to the GPU path, so future changes exercise the same nullable conversion and aggregation checks.
- Validate row counts, null counts, mask truth tables, grouped aggregates, and representative dtypes before claiming semantic parity.
Reference Files
references/cudf-pandas-accelerator.md— Profiling, fallback detection, cudf.pandas deep divereferences/api-patterns.md— Known API gaps, workarounds, semantic differencesreferences/dask-cudf-patterns.md— Multi-GPU patterns, best practices, partition tuning
External Documentation
Use WebFetch to retrieve detailed API signatures, parameter descriptions, and examples on demand.
- cuDF Documentation: https://docs.rapids.ai/api/cudf/stable/
- dask-cuDF API Reference: https://docs.rapids.ai/api/dask-cudf/stable/api/
- GitHub: https://github.com/rapidsai/cudf
- CHANGELOG: https://github.com/rapidsai/cudf/blob/main/CHANGELOG.md
Related
More Other skills
find-skills
vercel-labs
Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
tdd
mattpocock
Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
appinsights-instrumentation
microsoft
Guidance for instrumenting webapps with Azure Application Insights. Provides telemetry patterns, SDK setup, and configuration references. WHEN: how to instrument app, App Insights SDK, telemetry patterns, what is App Insights, Application Insights guidance, instrumentation examples, APM best practices.
azure-kusto
microsoft
Query and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis. WHEN: KQL queries, Kusto database queries, Azure Data Explorer, ADX clusters, log analytics, time series data, IoT telemetry, anomaly detection.
lark-approval
open.feishu.cn
飞书审批:查询和处理审批待办/已办/实例,搜索可发起审批定义、查看定义详情并发起原生审批实例。当用户要处理审批任务、查看审批实例、搜索或发起审批时使用。审批待办不是飞书任务;非审批类待办走 lark-task。不负责创建审批定义;三方审批定义不走原生提单。
lark-doc
open.feishu.cn
飞书云文档(Docx / Wiki 文档):读取和编辑飞书文档内容。当用户给出文档 URL 或 token,或需要查看、创建、编辑文档、插入或下载文档图片附件时使用。文档中嵌入的电子表格、多维表格、画板,先用本 skill 提取 token 再切到对应 skill。当用户给出 doubao.com 的 /docx/ 或 /wiki/ URL/token 时,也应直接使用本 skill;路由依据是 URL 路径模式和 token,而不是域名。不负责文档评论管理,也不负责表格或 Base 的数据操作。当用户明确要操作飞书思维笔记时,也使用本 skill。