Last week, we got a new, key component of Floe running: FloeScan. It enables point queries, aka needle-in-haystack queries, to run in milliseconds or less against massive Lakehouse tables.
Databricks demo’d a very early version of their “Rayden engine” in their conference, where they showed a concurrent, low-latency query demo against 100GB and 1TB data sets – data sets so small they can fit on an iPhone or a thumb drive. Floescan enables this sort of thing against Petabyte-scale data sets – for security, telemetry and other enterprise data warehousing workloads.
I recorded a small video with our man Thomas aka the Database Doctor, talking about the problem FloeScan is solving, which you can watch here. We’ll be releasing it along with Floe, in preview later this year, and will probably open source a FloeScan plugin for DuckDB and DataFusion too.
What else have we been up to? A key task has been getting Floe ready to start using internally, since we are our own Customer Number One. We chose test data and observability data – telemetry and log analytics – as the first use case, and have instrumented all of our microservices to generate OpenTelemetry (OTel) formatted data. The OTel data flows through Databricks Zerobus ingest into Delta Lake, where FloeScan and Floe consume it. To query the data, we added support for VARIANT data types, added more predicate pushdowns to our query planner, and sped up regular expressions. Finally, we wrapped the whole thing in Cloudflare zero-trust networking and kicked off our first SOC-2 compliance audit.
Do please send a note to neil@floedb.ai if you’d like to query your log analytics, telemetry and observability data directly from the Lakehouse, with all the JOINs you want, without having to duplicate the data into Splunk, Clickhouse, Elastic or elsewhere. We’d love to talk to you about previewing this functionality.
For the summer, we’re focusing on remaining query planner and statistics work, adding some automatic data indexing, and taking query latency out of the end-to-end stack. We’ll have some great demos by September and are expecting to have a Preview of Floe out before the end of the year. Stay tuned!
- The Floe team