Technical appendix
Engineering behind rodado
The public-facing site is a sociological reading of Brazilian public data. This page is the engineering appendix for readers interested in how the platform behind it is built — ingestion, ontology, AI query layer, and a direct comparison to how the same architecture maps onto Palantir Foundry.
Engineering competencies demonstrated
Each capability below is backed by a concrete, deployed piece of this project — not a claim, a running system.
Ingestion pipeline + semantic layer + API + browser UI — fully deployed and running end-to-end as a single-engineer project.
533 tables normalized to a typed ontology with a canonical join-key graph and partition-aware query patterns.
8 business object types (Municipality, Company, Person, Contract…) with typed relationships and explicit cardinalities.
SQL query API + browser DuckDB shell + Rust NL→SQL TUI — all production-deployed behind Caddy + auth.
LLM-powered NL→SQL with semantic table selection over an 11.4 MB embedding index. Backends: Gemini, OpenRouter, Ollama.
HMAC-SHA256 auth, Caddy forward auth, read-only DuckDB enforcement. CPF/CNPJ handled as sensitive identifiers.
Persistent DuckDB connection collocated with S3. Resumable pipeline. Docker + haloy deploy. Automated schema generation.
CPF (individual) and CNPJ (company) tax IDs — read-only access, no PII export endpoints, credential isolation per service.
Delivered case studies
Five shipped applications over Brazilian public data — each with a stakeholder problem, a data model, an application built, and a measurable outcome.
Investigators needed to trace beneficial ownership chains and detect undisclosed conflicts of interest across thousands of interconnected companies and individuals — previously weeks of manual registry lookups. Live demo ↗
Data model: Company (CNPJ) ↔ Person (CPF) via br_me_cnpj.socios — multi-hop graph traversal across ownership layers. Application: interactive force-directed network, nodes as entities, edges as relationships.
A public administration team needed a real-time operational picture of formal economic activity across Rio's 96 neighborhoods. Live map ↗
Data model: 1M+ CNPJ establishments joined to IBGE geocodes → spatial index → tile-rendered point layer per CNAE sector. Rendered client-side with sub-100ms pan/zoom.
Regulators and investors needed to map Swiss corporate presence across Brazil — subsidiary structures, holding layers, undisclosed connections. Live network ↗
Data model: CNPJ country-of-origin codes → Swiss-linked Company objects → director Person nodes → subsidiary chains via cnpj_basico.
A regional economic development agency needed to know which sectors closed first and which survived longest during COVID-19 lockdowns.
Data model: CNPJ open/close event timestamps × CNAE codes → temporal event stream partitioned by sector and municipality. Application: sequential closure heatmap.
A governance team needed to map a senator's full corporate network to identify conflicts between business interests and public budget votes, regulatory decisions, and contract awards.
Data model: TSE candidatos (CPF) → CNPJ socios → CNPJ empresas → CGU contratos — four-entity join across electoral, corporate, and procurement registries.
User workflows
Three analyst scenarios showing the full data → insight → decision arc.
Compliance — company integrity check
A compliance team verifies whether companies awarded public contracts have directors appearing in other sanctioned or flagged entities.
SELECT e.razao_social, e.cnpj, s.nome_socio,
COUNT(DISTINCT e2.cnpj) AS other_entities,
SUM(c.valor_contrato) AS total_contracts_brl
FROM br_me_cnpj.estabelecimentos e
JOIN br_me_cnpj.socios s ON e.cnpj_basico = s.cnpj_basico
JOIN br_me_cnpj.socios s2 ON s.cnpj_cpf_socio = s2.cnpj_cpf_socio
AND s2.cnpj_basico <> s.cnpj_basico
JOIN br_me_cnpj.estabelecimentos e2 ON s2.cnpj_basico = e2.cnpj_basico
JOIN br_cgu_compras_governamentais.contratos c ON e.cnpj = c.cnpj_contratado
WHERE e.sigla_uf = 'SP' AND c.ano = 2023
GROUP BY 1,2,3 HAVING other_entities > 2
ORDER BY total_contracts_brl DESC
Decision: flag companies for manual review; route to procurement governance team.
Policy — infrastructure gap prioritization
A state health secretariat identifies municipalities with critically low hospital bed coverage to prioritize federal budget allocation.
SELECT m.nome, m.sigla_uf, pop.populacao,
ROUND(cnes.leitos_sus * 1000.0 / NULLIF(pop.populacao,0), 2) AS leitos_por_mil,
ideb.nota_media AS ideb_fundamental
FROM br_bd_diretorios_brasil.municipio m
JOIN br_ibge_populacao.municipio pop ON m.id_municipio = pop.id_municipio AND pop.ano = 2022
JOIN ( SELECT id_municipio, SUM(leitos) AS leitos_sus
FROM br_ms_cnes.estabelecimento WHERE ano = 2023
GROUP BY id_municipio ) cnes ON m.id_municipio = cnes.id_municipio
JOIN br_inep_ideb.municipio ideb ON m.id_municipio = ideb.id_municipio AND ideb.ano = 2021
WHERE pop.populacao > 10000
ORDER BY leitos_por_mil ASC, ideb_fundamental ASC
LIMIT 50
Decision: top 50 ranked shortlist delivered to budget committee; top 10 flagged for emergency transfer.
Investigation — electoral spending anomalies
An investigative journalist tracks whether campaign spending correlates with post-election public contract awards for companies linked to the same candidate.
SELECT cand.nome_candidato, cand.sigla_partido,
SUM(desp.valor_despesa) AS campaign_spend,
SUM(cont.valor_contrato) AS post_election_contracts,
COUNT(DISTINCT cont.cnpj_contratado) AS linked_companies
FROM br_tse_eleicoes.candidatos cand
JOIN br_tse_eleicoes.despesas_candidato desp ON cand.id_candidato = desp.id_candidato
JOIN br_me_cnpj.socios s ON cand.cpf_candidato = s.cnpj_cpf_socio
JOIN br_cgu_compras_governamentais.contratos cont
ON s.cnpj_basico = SUBSTR(cont.cnpj_contratado,1,8) AND cont.ano > cand.ano
WHERE cand.ano = 2022 AND cand.sigla_uf = 'SP'
GROUP BY 1,2 HAVING post_election_contracts > 1000000
ORDER BY post_election_contracts DESC
Decision: shortlist of 12 candidates forwarded to editorial team with source data for verification.
Domain ontology
Brazilian public data modeled as typed entities with explicit join keys.
| Join key | Tables | Object |
|---|---|---|
id_municipio | 195 | Municipality |
sigla_uf | 245 | State |
cnpj / cnpj_basico | 18 | Company |
id_setor_censitario | 27 | CensusSector |
id_municipio_tse | 23 | ElectoralZone |
cbo_2002 | 6 | OccupationClass |
cnae_2_subclasse | 6 | EconomicActivity |
ano | 261 | Temporal partition |
Architecture
Data → semantic model → access control → application → users. Each layer independently deployable.
Analysts · policy teams · researchers · journalists
db.ミ.xyz browser SQL shell · ask.ミ.xyz AI NL→SQL TUI · POST /query SQL API
DuckDB views · schema.json (533 tables) · join_keys.md · table_embeddings.json (11.4 MB) · overview/ (34 domain narratives)
auth.py (HMAC-SHA256) · Caddy (TLS + forward auth) · read-only enforcement
Hetzner Object Storage · Parquet + zstd · 533 tables · ~675 GB · httpfs on-demand reads
BigQuery export → GCS → Hetzner (rclone) · resumable, dry-run, GCP VM option
AI-powered query layer
Semantic table selection over 533 datasets prevents hallucination on a large schema.
$ ask "Qual o município com maior mortalidade infantil no Nordeste em 2021?"
→ SELECT m.nome, m.sigla_uf, s.taxa_mortalidade_infantil
FROM br_ms_sim.municipio s
JOIN br_bd_diretorios_brasil.municipio m ON s.id_municipio = m.id_municipio
WHERE m.sigla_uf IN ('BA','PE','CE','MA','PI','RN','PB','AL','SE')
AND s.ano = 2021
ORDER BY s.taxa_mortalidade_infantil DESC
LIMIT 1
Palantir Foundry mapping
This is not a Foundry deployment — it's an open-source system that reproduces the same architectural layers and mental model, for readers who know Foundry and want a quick translation.
| rodado component | Foundry equivalent |
|---|---|
| BigQuery export scripts | External data connectors |
| Parquet files on Hetzner S3 | Foundry datasets |
| DuckDB engine + views | Foundry query engine |
| basedosdados-schema.json | Ontology schema registry |
| join_keys.md entity graph | Object type links / property mappings |
| table_embeddings.json | Semantic search index |
| POST /query endpoint | Foundry Functions |
| Browser SQL shell | Workshop application |
| ask/ NL→SQL layer | AIP-powered application |
| scripts/roda.sh | Foundry pipeline |
| Caddy + auth.py | Access control + audit layer |
| overview/ domain narratives | Business context documentation |
Data quality & governance
Large tables (100M+ rows) require ano, sigla_uf, or mes filters. 261 tables partitioned by ano, 245 by sigla_uf.
cpf (individual), cnpj (company) — read-only, no PII export endpoints, credentials isolated per service.
Data freshness varies (CNPJ monthly, census 2010/2022, health 12–18 month lag). CPF joins can produce high cardinality.
State expected order of magnitude, flag outliers, verify counts two independent ways before reporting.
Stack
Full engineering reference (environment variables, deploy, services, pipeline commands): see TECHNICAL.md in the repository.