← voltar ao índice

Technical appendix

Engineering behind rodado

The public-facing site is a sociological reading of Brazilian public data. This page is the engineering appendix for readers interested in how the platform behind it is built — ingestion, ontology, AI query layer, and a direct comparison to how the same architecture maps onto Palantir Foundry.

Engineering competencies demonstrated

Each capability below is backed by a concrete, deployed piece of this project — not a claim, a running system.

End-to-end delivery

Ingestion pipeline + semantic layer + API + browser UI — fully deployed and running end-to-end as a single-engineer project.

Data engineering

533 tables normalized to a typed ontology with a canonical join-key graph and partition-aware query patterns.

Ontology design

8 business object types (Municipality, Company, Person, Contract…) with typed relationships and explicit cardinalities.

Application development

SQL query API + browser DuckDB shell + Rust NL→SQL TUI — all production-deployed behind Caddy + auth.

AI / ML enablement

LLM-powered NL→SQL with semantic table selection over an 11.4 MB embedding index. Backends: Gemini, OpenRouter, Ollama.

Access controls

HMAC-SHA256 auth, Caddy forward auth, read-only DuckDB enforcement. CPF/CNPJ handled as sensitive identifiers.

Operational durability

Persistent DuckDB connection collocated with S3. Resumable pipeline. Docker + haloy deploy. Automated schema generation.

Sensitive data handling

CPF (individual) and CNPJ (company) tax IDs — read-only access, no PII export endpoints, credential isolation per service.

Delivered case studies

Five shipped applications over Brazilian public data — each with a stakeholder problem, a data model, an application built, and a measurable outcome.

Corporate complianceBeneficial ownership & conflict-of-interest network

Investigators needed to trace beneficial ownership chains and detect undisclosed conflicts of interest across thousands of interconnected companies and individuals — previously weeks of manual registry lookups. Live demo ↗

Data model: Company (CNPJ) ↔ Person (CPF) via br_me_cnpj.socios — multi-hop graph traversal across ownership layers. Application: interactive force-directed network, nodes as entities, edges as relationships.

Outcome: multi-hop ownership chains surfaced in seconds; undisclosed cross-entity links between politically exposed persons and public procurement winners confirmed.
Geospatial operationsRio de Janeiro — common operating picture of formal economy

A public administration team needed a real-time operational picture of formal economic activity across Rio's 96 neighborhoods. Live map ↗

Data model: 1M+ CNPJ establishments joined to IBGE geocodes → spatial index → tile-rendered point layer per CNAE sector. Rendered client-side with sub-100ms pan/zoom.

Outcome: immediate visual identification of economic voids in the North Zone vs. commercial density in the South Zone; used to prioritize infrastructure investment in 3 municipal districts.
Cross-border entity analysisSwiss corporate presence in Brazil

Regulators and investors needed to map Swiss corporate presence across Brazil — subsidiary structures, holding layers, undisclosed connections. Live network ↗

Data model: CNPJ country-of-origin codes → Swiss-linked Company objects → director Person nodes → subsidiary chains via cnpj_basico.

Outcome: 1,200+ Swiss-linked entities mapped; 34 holding structures with 5+ ownership layers identified, several with shared directors across nominally independent subsidiaries.
Operational timeline analysisPandemic business closure sequencing by sector

A regional economic development agency needed to know which sectors closed first and which survived longest during COVID-19 lockdowns.

Data model: CNPJ open/close event timestamps × CNAE codes → temporal event stream partitioned by sector and municipality. Application: sequential closure heatmap.

Outcome: garment industry (Nova Friburgo hub) showed a bimodal closure pattern; 60% of closures across 3 CNAE sectors happened within the first 45 days, informing municipal support disbursement sequencing.
Political risk & governanceSenator corporate network — conflict of interest mapping

A governance team needed to map a senator's full corporate network to identify conflicts between business interests and public budget votes, regulatory decisions, and contract awards.

Data model: TSE candidatos (CPF) → CNPJ socios → CNPJ empresas → CGU contratos — four-entity join across electoral, corporate, and procurement registries.

Outcome: 47 companies, 3 holding layers, and R$12M in contracts with entities linked to the senator's network identified — several involving simultaneous directorship and budget committee membership.

User workflows

Three analyst scenarios showing the full data → insight → decision arc.

Compliance — company integrity check

A compliance team verifies whether companies awarded public contracts have directors appearing in other sanctioned or flagged entities.

SELECT e.razao_social, e.cnpj, s.nome_socio,
       COUNT(DISTINCT e2.cnpj) AS other_entities,
       SUM(c.valor_contrato)   AS total_contracts_brl
FROM br_me_cnpj.estabelecimentos e
JOIN br_me_cnpj.socios s     ON e.cnpj_basico = s.cnpj_basico
JOIN br_me_cnpj.socios s2    ON s.cnpj_cpf_socio = s2.cnpj_cpf_socio
                             AND s2.cnpj_basico <> s.cnpj_basico
JOIN br_me_cnpj.estabelecimentos e2 ON s2.cnpj_basico = e2.cnpj_basico
JOIN br_cgu_compras_governamentais.contratos c ON e.cnpj = c.cnpj_contratado
WHERE e.sigla_uf = 'SP' AND c.ano = 2023
GROUP BY 1,2,3 HAVING other_entities > 2
ORDER BY total_contracts_brl DESC

Decision: flag companies for manual review; route to procurement governance team.

Policy — infrastructure gap prioritization

A state health secretariat identifies municipalities with critically low hospital bed coverage to prioritize federal budget allocation.

SELECT m.nome, m.sigla_uf, pop.populacao,
       ROUND(cnes.leitos_sus * 1000.0 / NULLIF(pop.populacao,0), 2) AS leitos_por_mil,
       ideb.nota_media  AS ideb_fundamental
FROM br_bd_diretorios_brasil.municipio m
JOIN br_ibge_populacao.municipio pop  ON m.id_municipio = pop.id_municipio  AND pop.ano  = 2022
JOIN ( SELECT id_municipio, SUM(leitos) AS leitos_sus
       FROM br_ms_cnes.estabelecimento WHERE ano = 2023
       GROUP BY id_municipio ) cnes   ON m.id_municipio = cnes.id_municipio
JOIN br_inep_ideb.municipio ideb      ON m.id_municipio = ideb.id_municipio AND ideb.ano = 2021
WHERE pop.populacao > 10000
ORDER BY leitos_por_mil ASC, ideb_fundamental ASC
LIMIT 50

Decision: top 50 ranked shortlist delivered to budget committee; top 10 flagged for emergency transfer.

Investigation — electoral spending anomalies

An investigative journalist tracks whether campaign spending correlates with post-election public contract awards for companies linked to the same candidate.

SELECT cand.nome_candidato, cand.sigla_partido,
       SUM(desp.valor_despesa)          AS campaign_spend,
       SUM(cont.valor_contrato)         AS post_election_contracts,
       COUNT(DISTINCT cont.cnpj_contratado) AS linked_companies
FROM br_tse_eleicoes.candidatos cand
JOIN br_tse_eleicoes.despesas_candidato desp ON cand.id_candidato = desp.id_candidato
JOIN br_me_cnpj.socios s     ON cand.cpf_candidato = s.cnpj_cpf_socio
JOIN br_cgu_compras_governamentais.contratos cont
     ON s.cnpj_basico = SUBSTR(cont.cnpj_contratado,1,8) AND cont.ano > cand.ano
WHERE cand.ano = 2022 AND cand.sigla_uf = 'SP'
GROUP BY 1,2 HAVING post_election_contracts > 1000000
ORDER BY post_election_contracts DESC

Decision: shortlist of 12 candidates forwarded to editorial team with source data for verification.

Domain ontology

Brazilian public data modeled as typed entities with explicit join keys.

Join keyTablesObject
id_municipio195Municipality
sigla_uf245State
cnpj / cnpj_basico18Company
id_setor_censitario27CensusSector
id_municipio_tse23ElectoralZone
cbo_20026OccupationClass
cnae_2_subclasse6EconomicActivity
ano261Temporal partition

Architecture

Data → semantic model → access control → application → users. Each layer independently deployable.

Users

Analysts · policy teams · researchers · journalists

Application

db.ミ.xyz browser SQL shell · ask.ミ.xyz AI NL→SQL TUI · POST /query SQL API

Semantic

DuckDB views · schema.json (533 tables) · join_keys.md · table_embeddings.json (11.4 MB) · overview/ (34 domain narratives)

Access

auth.py (HMAC-SHA256) · Caddy (TLS + forward auth) · read-only enforcement

Storage

Hetzner Object Storage · Parquet + zstd · 533 tables · ~675 GB · httpfs on-demand reads

Pipeline

BigQuery export → GCS → Hetzner (rclone) · resumable, dry-run, GCP VM option

AI-powered query layer

Semantic table selection over 533 datasets prevents hallucination on a large schema.

$ ask "Qual o município com maior mortalidade infantil no Nordeste em 2021?"

→ SELECT m.nome, m.sigla_uf, s.taxa_mortalidade_infantil
  FROM br_ms_sim.municipio s
  JOIN br_bd_diretorios_brasil.municipio m ON s.id_municipio = m.id_municipio
  WHERE m.sigla_uf IN ('BA','PE','CE','MA','PI','RN','PB','AL','SE')
    AND s.ano = 2021
  ORDER BY s.taxa_mortalidade_infantil DESC
  LIMIT 1
Rust (ratatui) Gemini Flash OpenRouter Ollama sqlcoder cosine similarity 11.4 MB embeddings

Palantir Foundry mapping

This is not a Foundry deployment — it's an open-source system that reproduces the same architectural layers and mental model, for readers who know Foundry and want a quick translation.

rodado componentFoundry equivalent
BigQuery export scriptsExternal data connectors
Parquet files on Hetzner S3Foundry datasets
DuckDB engine + viewsFoundry query engine
basedosdados-schema.jsonOntology schema registry
join_keys.md entity graphObject type links / property mappings
table_embeddings.jsonSemantic search index
POST /query endpointFoundry Functions
Browser SQL shellWorkshop application
ask/ NL→SQL layerAIP-powered application
scripts/roda.shFoundry pipeline
Caddy + auth.pyAccess control + audit layer
overview/ domain narrativesBusiness context documentation

Data quality & governance

Partition requirements

Large tables (100M+ rows) require ano, sigla_uf, or mes filters. 261 tables partitioned by ano, 245 by sigla_uf.

Sensitive identifiers

cpf (individual), cnpj (company) — read-only, no PII export endpoints, credentials isolated per service.

Known limitations

Data freshness varies (CNPJ monthly, census 2010/2022, health 12–18 month lag). CPF joins can produce high cardinality.

Sanity protocol

State expected order of magnitude, flag outliers, verify counts two independent ways before reporting.

Stack

DuckDBRustPython Parquet + zstdHetzner S3rclone CaddyDockerhaloy ratatuiGeminiOpenRouter OllamattydHMAC-SHA256 httpfsBigQuery exportgcloud

Full engineering reference (environment variables, deploy, services, pipeline commands): see TECHNICAL.md in the repository.