dbcturbo is a high-performance decompression engine for
DATASUS .dbc files, built from scratch in C99 and designed
for epidemiological Big Data. Unlike read.dbc, no
full dataset is loaded into RAM — records are processed in
streaming batches and written directly to disk.
DBC files from Brazil’s DATASUS system (SINAN, SIM, SINASC, SIH, SIA)
can contain millions of records. dbcturbo handles them
efficiently and outputs clean UTF-8 encoded files.
Before converting, check how many records and columns the file contains:
n_max)read_dbc() decodes, types, and transcodes directly in C
without temporary files. Use n_max to stop decompression
early when previewing large files, or cols to read only
selected columns:
library(dbcturbo)
# Instant preview of the first 100 rows (stops decompression early)
preview <- read_dbc("DENGBR23.dbc", n_max = 100)
# Read the full dataset (or only specific columns)
df <- read_dbc("DENGBR23.dbc", cols = c("DT_NOTIFIC", "SG_UF_NOT", "NU_IDADE_N"))
head(df)
dim(df)For files too large to materialize at once, process them in
memory-bounded chunks with read_dbc_chunks(), or verify
file integrity without allocating a data frame using
dbc_validate():
Important: Microsoft Excel supports a maximum of 1,048,576 rows. Many DATASUS files — such as national Dengue or SINASC datasets — contain over 1.6 million records and cannot be fully opened in Excel.
The recommended approach is to load the data in R and filter it before exporting to Excel:
library(dbcturbo)
# Load the full dataset into R
df <- read_dbc("DENGBR23.dbc")
# Filter to a single state before exporting
# (use the two-digit IBGE state code in SG_UF_NOT)
df_rs <- df[df$SG_UF_NOT == "43", ] # Rio Grande do Sul
df_sp <- df[df$SG_UF_NOT == "35", ] # São Paulo
df_rj <- df[df$SG_UF_NOT == "33", ] # Rio de Janeiro
# Export filtered subset — this will fit comfortably in Excel
write.csv(df_rs, "dengue_2023_RS.csv", row.names = FALSE)You can also filter by year, municipality, or any other variable:
For files too large to fit in RAM, stream directly to a CSV file:
dbc_to_csv(
input_file = "DENGBR23.dbc",
output_file = "dengue_2023.csv",
batch_size = 10000L, # records per batch
encoding = "latin1" # default for modern DATASUS files (use "CP850" for legacy DOS files)
)The output CSV is automatically encoded as UTF-8 with BOM, ensuring correct display of Portuguese characters (ã, ç, é, etc.) in Excel and other tools.
Parquet is ideal for large epidemiological datasets.
dbc_to_parquet() streams typed chunks directly from
read_dbc_chunks() into
arrow::ParquetFileWriter using an explicit DBF-derived
schema — with no intermediate CSV file: - Compressed
(3–5× smaller than CSV) - Columnar (fast for analytical queries) -
Supports column projection via cols - Compatible with R
(arrow), Python (pandas, polars),
Power BI, DuckDB
library(dbcturbo)
dbc_to_parquet(
input_file = "DENGBR23.dbc",
output_file = "dengue_2023.parquet", # must end in .parquet
cols = c("DT_NOTIFIC", "SG_UF_NOT", "NU_IDADE_N")
)
# Read it back in R
library(arrow)
df <- read_parquet("dengue_2023.parquet")Note: Do not pass a
.csvpath todbc_to_parquet()— that function always writes binary Parquet format. Usedbc_to_csv()for CSV output.
The conversion engine has no global mutable C state. On macOS and
Linux, files can be processed concurrently with
parallel::mclapply():
library(parallel)
files <- list.files("datasus/", pattern = "\\.dbc$", full.names = TRUE)
mclapply(files, function(f) {
out <- sub("\\.dbc$", ".csv", f)
dbc_to_csv(f, out)
}, mc.cores = 4L)On Windows, use dbc_batch_to_csv() sequentially or a
Windows-compatible parallel backend instead of
mclapply().
By default (coerce_types = TRUE),
read_dbc() automatically converts DBF date fields
(D, stored as YYYYMMDD) into native R
Date objects:
DATASUS encodes age in a single 4-digit integer where the first digit indicates the unit:
| First digit | Unit | Example | Meaning |
|---|---|---|---|
| 1 | Hours | 1012 | 12 hours |
| 2 | Days | 2015 | 15 days |
| 3 | Months | 3006 | 6 months |
| 4 | Years | 4025 | 25 years |
# Decode NU_IDADE_N into age in years
decode_age <- function(x) {
x <- as.integer(x)
unit <- x %/% 1000 # first digit
value <- x %% 1000 # remaining digits
age_years <- ifelse(unit == 4, value, # already in years
ifelse(unit == 3, value / 12, # months to years
ifelse(unit == 2, value / 365, # days to years
ifelse(unit == 1, value / 8760, # hours to years
NA_real_))))
age_years
}
df$age_years <- decode_age(df$NU_IDADE_N)
# Age group (5-year bands)
df$age_group <- cut(df$age_years,
breaks = c(0, 5, 15, 25, 35, 45, 55, 65, Inf),
labels = c("<5", "5-14", "15-24", "25-34", "35-44", "45-54", "55-64", "65+"),
right = FALSE
)Blank DBF fields are automatically returned as NA by
read_dbc():
# Cases by sex
table(df$CS_SEXO)
# Cases by race/ethnicity (CS_RACA)
# 1=White, 2=Black, 3=Yellow, 4=Brown, 5=Indigenous
table(df$CS_RACA)
# Cases by state (SG_UF_NOT)
sort(table(df$SG_UF_NOT), decreasing = TRUE)
# Cross-tabulation: sex by outcome (EVOLUCAO)
# 1=Cure, 2=Death, 3=Death by other causes, 9=Unknown
table(Sex = df$CS_SEXO, Outcome = df$EVOLUCAO)
# Incidence rate table: cases per state per year
aggregate(cbind(cases = TP_NOT) ~ SG_UF_NOT + NU_ANO,
data = df,
FUN = length)Some DATASUS systems distribute data in monthly files. Combine them easily:
library(dbcturbo)
# List all monthly DBC files in a folder
files <- list.files("dbc/2023/", pattern = "\\.dbc$", full.names = TRUE)
# Read and combine all into a single data frame
df_year <- do.call(rbind, lapply(files, read_dbc))
cat("Total records:", nrow(df_year), "\n")
# Alternative with data.table (faster for large files)
library(data.table)
dt_year <- rbindlist(lapply(files, read_dbc))For very large datasets, use DuckDB to run SQL directly on Parquet files without loading everything into RAM:
library(duckdb)
library(dbcturbo)
# Convert once
dbc_to_parquet("DENGBR23.dbc", "dengue_2023.parquet")
# Query without loading the full file
con <- dbConnect(duckdb())
result <- dbGetQuery(con, "
SELECT
SG_UF_NOT,
COUNT(*) AS total_cases,
SUM(CASE WHEN EVOLUCAO = '2' THEN 1 ELSE 0 END) AS deaths
FROM 'dengue_2023.parquet'
GROUP BY SG_UF_NOT
ORDER BY total_cases DESC
")
print(result)
dbDisconnect(con)When your filtered data fits within Excel’s limits, export to
.xlsx with formatting using the openxlsx
package:
library(dbcturbo)
library(openxlsx)
df <- read_dbc("DENGBR23.dbc")
# Filter to a manageable subset
df_rs_2023 <- df[df$SG_UF_NOT == "43" & df$NU_ANO == "2023", ]
# Create a formatted workbook
wb <- createWorkbook()
addWorksheet(wb, "Dengue_RS_2023")
# Style for the header row
header_style <- createStyle(
fontColour = "#FFFFFF",
fgFill = "#2E4057",
halign = "CENTER",
textDecoration = "Bold"
)
writeData(wb, "Dengue_RS_2023", df_rs_2023)
addStyle(wb, "Dengue_RS_2023", header_style,
rows = 1, cols = 1:ncol(df_rs_2023), gridExpand = TRUE)
saveWorkbook(wb, "dengue_RS_2023.xlsx", overwrite = TRUE)| Approach | Peak RAM | Time (1M rows) | Parallelisable |
|---|---|---|---|
read.dbc |
~4 GB | ~45 s | No |
dbcturbo::read_dbc |
~600 MB | ~15 s | Yes |
dbcturbo::dbc_to_csv |
< 50 MB | ~12 s | Yes |
A .dbc file from DATASUS is a .dbf (dBase)
file compressed with the PKWare Implode algorithm (also
known as blast in the technical literature).
Binary structure:
[Bytes 0-7] -> DBF header (version, date, record count, header size)
[Bytes 8-9] -> uint16_t little-endian: total DBF header size
[Bytes 10-N] -> Field descriptors (32 bytes each) + 0x0D terminator
[N+1 .. N+4] -> CRC32 (4 bytes — ignored during decompression)
[N+5 .. EOF] -> Compressed payload using PKWare Implode
The blast() function (Mark Adler, 2003) decodes this
payload using a 4096-byte sliding window (MAXWIN) with
canonical Huffman coding.
All engine state is stack-allocated (struct state in
blast.c). There are no global mutable variables.
dbc_to_csv() and dbc2dbf() can be called in
parallel via parallel::mclapply() without race
conditions.