Do you think the Iceberg spec was intended to solve all those problems?
I think of specs and protocols like HTTP or gRPC. They define contracts for systems to communicate to one another, but there's no built-in definition for how long a request/response should take. That varies wildly across use cases.
So yeah, I agree that a semantic spec isn't sufficient for Iceberg as a technology. But that's the case with everything, right? I can't look at some service and say "oh, they provide RESTful API endpoints, that's all I need to know that it will work for my use case!" But it's sure nice that most web services use standard protocols, rather than having bespoke binary protocols for every integration under the sun.
+1. In general, I agree that a protocol spec isn’t sufficient to build a production system, there are many other aspects. But Ananth, what’s your expectation here - that the OSS Iceberg REST Catalog API spec would define performance SLAs for all implementations? Do you have any successful industry examples of OSS specs defining latency SLAs?
OTEL is an excellent example of a successful implementation with widespread adoption. Instead of defining the SLA in a spec, OTEL implemented the client SDK. Between the two catalogs, as of now, there is a lot of polling going on to sync, which is conceptually the same as OTEL implementations. I expect that the specs are great, but it won't succeed without a standard implementation.
Without a standard implementation for the spec, it trickles down to the catalog war and commercial aspects, leading to more friction in integration. Essentially, Iceberg Rest Catalog created the very own problem it is trying to solve.
My thoughts are that semantic specs are well-suited for contract definition and to define the structure of an object. In this specific case, the Iceberg REST catalog defines behaviours or actions such as List Table and Get Table. It's more like a Buy Now button with no response time contract on when the transaction will complete.
Do you think the Iceberg spec was intended to solve all those problems?
I think of specs and protocols like HTTP or gRPC. They define contracts for systems to communicate to one another, but there's no built-in definition for how long a request/response should take. That varies wildly across use cases.
So yeah, I agree that a semantic spec isn't sufficient for Iceberg as a technology. But that's the case with everything, right? I can't look at some service and say "oh, they provide RESTful API endpoints, that's all I need to know that it will work for my use case!" But it's sure nice that most web services use standard protocols, rather than having bespoke binary protocols for every integration under the sun.
+1. In general, I agree that a protocol spec isn’t sufficient to build a production system, there are many other aspects. But Ananth, what’s your expectation here - that the OSS Iceberg REST Catalog API spec would define performance SLAs for all implementations? Do you have any successful industry examples of OSS specs defining latency SLAs?
OTEL is an excellent example of a successful implementation with widespread adoption. Instead of defining the SLA in a spec, OTEL implemented the client SDK. Between the two catalogs, as of now, there is a lot of polling going on to sync, which is conceptually the same as OTEL implementations. I expect that the specs are great, but it won't succeed without a standard implementation.
Without a standard implementation for the spec, it trickles down to the catalog war and commercial aspects, leading to more friction in integration. Essentially, Iceberg Rest Catalog created the very own problem it is trying to solve.
My thoughts are that semantic specs are well-suited for contract definition and to define the structure of an object. In this specific case, the Iceberg REST catalog defines behaviours or actions such as List Table and Get Table. It's more like a Buy Now button with no response time contract on when the transaction will complete.
Thank you! Very helpful!
This was interesting
Excellent points.
Interesting observation thanks
True that. I hope the community addresses these issues.