How do companies store and query business data at scale without a dedicated tool?

How companies store and query business data at scale without a dedicated tool Companies don't necessarily need a specialized business intelligence (BI) platform or dedicated data-management tool to manage large amounts of business data. Many build their own data infrastructure using databases, cloud storage, APIs, and open-source technologies. The key idea is to separate data storage, data processing, and data querying rather than relying on one application to do everything. The typical…

1. PostgreSQL — PostgreSQL · MySQL · S3 · Data lake · Data warehouse.
2. MySQL — PostgreSQL · MySQL · S3 · Data lake · Data warehouse.
3. Trino — SQL · Python · Spark · Trino · DuckDB.
4. DuckDB — SQL · Python · Spark · Trino · DuckDB.
5. BigQuery — Google Cloud's lakehouse architecture also demonstrates how different processing engines can work with shared storage and centralized metadata.
6. Snowflake
7. Metabase
8. dbt — dbt for SQL-based transformations.
9. Apache Spark — Apache Spark for distributed processing.

Companies typically build a data stack from general-purpose infrastructure rather than buying one dedicated business-data tool: Collect data from applications, databases, SaaS systems, logs, and files using scheduled scripts, APIs, change-data capture, or lightweight ETL/ELT pipelines. Store raw data in inexpensive cloud object storage such as S3-compatible storage, usually as Parquet files. Open formats keep data usable across multiple query engines and reduce vendor lock-in. Catalog and…

1. dbt — Transform and model it into clean tables—often facts, dimensions, and aggregates—using SQL, Python, dbt, Spark, or distributed processing.
2. DuckDB — Query it with a SQL engine such as DuckDB, Trino, Presto, Spark SQL, or a cloud warehouse.
3. Trino — Query it with a SQL engine such as DuckDB, Trino, Presto, Spark SQL, or a cloud warehouse.
4. PostgreSQL — PostgreSQL for operational data.
5. Airbyte — Scheduled Python or Airbyte jobs for ingestion.
6. Metabase — Metabase, Superset, or a custom dashboard for reporting.

1. Snowflake — When companies grow beyond simple spreadsheets or flat files but haven't yet adopted a dedicated enterprise data warehouse (like Snowflake or BigQuery) or a specialized database (like Oracle or Teradata), they often…
2. BigQuery — When companies grow beyond simple spreadsheets or flat files but haven't yet adopted a dedicated enterprise data warehouse (like Snowflake or BigQuery) or a specialized database (like Oracle or Teradata), they often…
3. Oracle Autonomous Data Warehouse — When companies grow beyond simple spreadsheets or flat files but haven't yet adopted a dedicated enterprise data warehouse (like Snowflake or BigQuery) or a specialized database (like Oracle or Teradata), they often…
4. Teradata — When companies grow beyond simple spreadsheets or flat files but haven't yet adopted a dedicated enterprise data warehouse (like Snowflake or BigQuery) or a specialized database (like Oracle or Teradata), they often…
5. Microsoft Azure — Object Storage Lakes: Companies dump raw CSVs, JSON lines, and Parquet files into cloud object stores like Amazon S3, Google Cloud Storage , or Azure Blob Storage.
6. PostgreSQL — Operational Database Replicas: They route read-only operational traffic or analytical queries to read-replicas of their primary production databases (like PostgreSQL or MySQL) or use point-in-time database backups…
7. MySQL — Operational Database Replicas: They route read-only operational traffic or analytical queries to read-replicas of their primary production databases (like PostgreSQL or MySQL) or use point-in-time database backups…
8. AWS — Serverless Query Engines: Tools like AWS Athena or Trino/Presto allow engineers to run standard SQL directly over raw files sitting in object storage without provisioning a database cluster.
9. Trino — Serverless Query Engines: Tools like AWS Athena or Trino/Presto allow engineers to run standard SQL directly over raw files sitting in object storage without provisioning a database cluster.
10. DuckDB — Command-Line & Scripting Utilities: Data is processed and filtered using local or containerized scripts written in Python (pandas, duckdb ) or shell scripting (awk, sed, grep ), pulling data in chunks or loading it into…
11. Apache Spark — Distributed Compute Frameworks: For massive workloads, engineers write code in Apache Spark or Dask to distribute query execution across a cluster of commodity virtual machines rather than relying on a single database…