Love the comparison to microwaves. I've reduced my own microwave usage to a single button: +30 seconds. Sometimes it results in more button presses, sometimes less. But it's a single function that's always guaranteed to work, no matter my situation.
You're right that the data community tends to be very opinionated in how they build their solutions. It sounds like you take the perspective that most data tools first solve one thing then try to expand, offering too many functionalities that feel half-baked.
But what if the one thing the tool solves initially is the ability to build and launch custom solutions (a.k.a code)? I still hold the belief that data orchestration is the ultimate all-in-one tool. Its sole functionality is repeatable code execution. The more teams that figure out it can be used to run any solution under one roof (ETL pipelines, machine learning models, AI processes, etc.), the more it's going to gain in popularity and usage.
I think the line for where a product should stop trying to do the jobs of other tools varies... Orchestration is interesting in that the world's most basic orchestration tool, cron, continues to be a cornerstone for tons of things. We sorta tolerate it because it's so simple and "good enough" that it just lets us execute whatever custom-made atrocities we want. Improving that experience with DAGs and automatic retries and triggers feels natural.
But I also wonder if those tools might also one day feature creep somehow... maybe start including their own execution engine or scripting language... or try to take over the entire OS a-la SystemD... Who knows...
When I was a product manager we started trying to find things to get rid of. Our inspiration was Steve Jobs (and I am not an Apple person) when he said “We’re trying to make great products for people, and we have at least the courage of our convictions to say we don’t think this is part of what makes a great product, we’re going to leave it out.”
I've worked on a LOT of products over 15 years and only worked on removing a feature like.... maybe once or twice... It's an unnatural thing for most teams because someone inevitably complains that we killed their favorite feature
One question that I've had is whether or not these "one-stop-shop" data products result because data products tend to be sold from the top down (i.e. selling to the head of data to implement on their data) and thus necessitate having a lot of features in the product in order to encourage them to incorporate it. I'm not too familiar with many data products that are sold from the bottom up (i.e. catering to improve productivity for data engineers/scientists/analysts). Not sure if that's the case and, if so, why that's the case (decision makers are at the top? data is too complicated? some other reason?) Would like to hear other people's thoughts
For any enterprise-y- product there's always a weird disjoint between "person who writes the check" and "person who uses the product"... but it's not like they'll willingly keep renewing expensive contracts if everyone complains how the product isn't delivering on important promises...
What I do very often see is that large enterprises have the clout to influence feature creation, and so enterprise products often pick up lots of really niche features that most people don't have any need for. That tends to contribute to extra bloat. But even w/o that pressure, I think its the nature of software products to just keep snowballing
So these enterprise-y things play a factor but I’m guessing the problem of bloated software just echoes the adage that if you try to accommodate everyone (to make more money) you end up accommodating no one in particular. Does this mean that there is some optimal point where the product provides the maximum value to the most amount of people? (Does this mean there’s an inherent limit to the scale of companies?)
I think there's not an optimum, but a point of diminishing returns for product design... for example, the designs of hammers, chefs knives, forks, are all more or less centuries old. There's still plenty of innovation in metallurgy, manufacturing cost, and fashion, but most new innovations are mostly gimmicks compared to the core functionality. So I suppose the question is... where is the product in the maturation cycle?
Yeah although maybe its time when we empower all these tools with "AI" (kidding of course but wouldn't be surprised if it happened/is happening). Wonder if there are any useful heuristics to detect/get a feel for the maturation level of certain product (aside from the obvious ones like usage and adoption)?
Ooh ok I have lots of thoughts that I need about, uh, three days to put together. But the short version is, do we blame VCs? (Probably not but kinda?) And can we solve this with suites? (Probably not but kinda?)
Maybe blame VCs... Maybe blame those fat enterprise customers who request weird one-off features to integrate w/ their weirdo setup. Sometimes blame just blame teams that only use their stuff internally and lose sight of what the rest of the world/market actually does... there's SO MUCH BLAME to share
I think modularity and consistency can help a lot. But there will always be a learning curve between disparate systems. Creating less friction always creates more maintenance debt which needs to be paid each time a system is changed.
I agree with the premise that all-in-one tools are more likely to overpromise and underdeliver. Databases themselves are a sort of one-stop shop and they are very good at many things. A jack-of-all algorithms. But once you need to optimize a certain code path you may need to switch some parts to use custom code. This custom code might be able to run inside of the database or it might not. I think all-in-one tools fit the same kind of category.
The main differentiator between these things--why all-in-one apps are more doomed to fail--are because most MBAs aren't going out there building new databases. They are building all-in-one apps and so the problem is often tackled with the wrong direction or scale.
There's definitely a lot of things going on here, and honestly I'm pretty sure I sorta wrote around in circles without getting anywhere. But there's definitely a difference in fundamental design/development principles between "thing sold as product" which seems to constantly feature creep, and unix-like programs where if it accomplishes the job well enough people just stop working on it
I think the main thing is that all-in-one apps can only succeed if they are a monolith to begin with. If the all-in-one app is just wrangling in data from a bunch of different places then there is huge friction to get started and maintain integration across disparate data-generating/capturing tools. The task is to become Sisyphus.
I distribute my own tools as one python package even though they would probably be more popular if I split them up into different projects: https://github.com/chapmanjacobd/library
But I don't really want to manage a bunch of different projects. There would be a huge amount of overhead to do that. Maintenance is centralized in this case because all the data-generating and data-analyzing activities happen from within the same system. The edges of the system are the most likely to flail away like the Red Inflatable Tube Man, ready to sail across the universe into full entropy.
But some people have a different mentality: https://tildes.net/~tildes/1azo/possible_solution_for_the_perennial_what_app_issue but maybe at some point we are getting into Ted Nelson Xanadu territory and the world moves out from under our feet and we are nearing death's door so there is nothing left to do but concede to the great HTML god
Startups are usually in search mode so they are constantly adding and removing requirements as they talk with customers about what needs they have. CLI tools are usually written by people who are themselves users of the tool. The feedback loop is 10x to 100x quicker.
But I have to say many unix-tools are in constant development as well (and databases only stop being developed when they become obscure and die out of use). There are new features added to many userspace tools every year. Just pick your tool and there have probably been new releases within the past few months (but of course if you use something like Mac OS Apple won't bother to update any of it--at least until the next major OS release*):
But yes, in terms of software stability it is much less likely for these 30 or 40 year old tools to have completely different interfaces. Although the same is probably true for 30 or 40 year old products that other businesses sell. The problem is that most software doesn't live that long. I can only think of enterprise tools like Oracle Database... although a lot has changed in the past 40+ years that it has existed. I think the main thing preventing old software from being sold for so long is that customers want something new and exciting. Only boring technologies are allowed to age and mature but still be respected enough to pay for.
* To prove your point: some of this is because it increases the risk of regressions or bugs in new features but the biggest reason is that nobody complains about it. People aren't refreshing the changelog for these tools every day because they only use each tool to do one thing that they know it can do. It's boring technology. They would only care if there is added friction from something breaking or them needing to spend time thinking about how an interface or functionality change will affect their workflows (like when ncdu updated recently and now I need to always do "NO_COLOR=1 ncdu" in order to have a pleasant experience)
Yeah, while it could be argued that even open source projects compete in a notional marketplace of ideas, it's nowhere similar to the capitalistic 'we must sell widgets ever day to survive' that fuels commercial software to devote resources into making their products more attractive to buyers.
I suppose one core lament of this whole post is that I don't like that product development often flows down this path of least resistance because it... doesn't seem to work but takes 5-10 years to play out. I suppose it pays the bills while people realize it.... but resources probably could've have been used on something of more potential value if only we learned when to stop riding the one trick pony...
Love the comparison to microwaves. I've reduced my own microwave usage to a single button: +30 seconds. Sometimes it results in more button presses, sometimes less. But it's a single function that's always guaranteed to work, no matter my situation.
You're right that the data community tends to be very opinionated in how they build their solutions. It sounds like you take the perspective that most data tools first solve one thing then try to expand, offering too many functionalities that feel half-baked.
But what if the one thing the tool solves initially is the ability to build and launch custom solutions (a.k.a code)? I still hold the belief that data orchestration is the ultimate all-in-one tool. Its sole functionality is repeatable code execution. The more teams that figure out it can be used to run any solution under one roof (ETL pipelines, machine learning models, AI processes, etc.), the more it's going to gain in popularity and usage.
I think the line for where a product should stop trying to do the jobs of other tools varies... Orchestration is interesting in that the world's most basic orchestration tool, cron, continues to be a cornerstone for tons of things. We sorta tolerate it because it's so simple and "good enough" that it just lets us execute whatever custom-made atrocities we want. Improving that experience with DAGs and automatic retries and triggers feels natural.
But I also wonder if those tools might also one day feature creep somehow... maybe start including their own execution engine or scripting language... or try to take over the entire OS a-la SystemD... Who knows...
When I was a product manager we started trying to find things to get rid of. Our inspiration was Steve Jobs (and I am not an Apple person) when he said “We’re trying to make great products for people, and we have at least the courage of our convictions to say we don’t think this is part of what makes a great product, we’re going to leave it out.”
I've worked on a LOT of products over 15 years and only worked on removing a feature like.... maybe once or twice... It's an unnatural thing for most teams because someone inevitably complains that we killed their favorite feature
And I caught some flack. Had to tread that path carefully.
One question that I've had is whether or not these "one-stop-shop" data products result because data products tend to be sold from the top down (i.e. selling to the head of data to implement on their data) and thus necessitate having a lot of features in the product in order to encourage them to incorporate it. I'm not too familiar with many data products that are sold from the bottom up (i.e. catering to improve productivity for data engineers/scientists/analysts). Not sure if that's the case and, if so, why that's the case (decision makers are at the top? data is too complicated? some other reason?) Would like to hear other people's thoughts
For any enterprise-y- product there's always a weird disjoint between "person who writes the check" and "person who uses the product"... but it's not like they'll willingly keep renewing expensive contracts if everyone complains how the product isn't delivering on important promises...
What I do very often see is that large enterprises have the clout to influence feature creation, and so enterprise products often pick up lots of really niche features that most people don't have any need for. That tends to contribute to extra bloat. But even w/o that pressure, I think its the nature of software products to just keep snowballing
So these enterprise-y things play a factor but I’m guessing the problem of bloated software just echoes the adage that if you try to accommodate everyone (to make more money) you end up accommodating no one in particular. Does this mean that there is some optimal point where the product provides the maximum value to the most amount of people? (Does this mean there’s an inherent limit to the scale of companies?)
I think there's not an optimum, but a point of diminishing returns for product design... for example, the designs of hammers, chefs knives, forks, are all more or less centuries old. There's still plenty of innovation in metallurgy, manufacturing cost, and fashion, but most new innovations are mostly gimmicks compared to the core functionality. So I suppose the question is... where is the product in the maturation cycle?
Yeah although maybe its time when we empower all these tools with "AI" (kidding of course but wouldn't be surprised if it happened/is happening). Wonder if there are any useful heuristics to detect/get a feel for the maturation level of certain product (aside from the obvious ones like usage and adoption)?
Ooh ok I have lots of thoughts that I need about, uh, three days to put together. But the short version is, do we blame VCs? (Probably not but kinda?) And can we solve this with suites? (Probably not but kinda?)
Maybe blame VCs... Maybe blame those fat enterprise customers who request weird one-off features to integrate w/ their weirdo setup. Sometimes blame just blame teams that only use their stuff internally and lose sight of what the rest of the world/market actually does... there's SO MUCH BLAME to share
Yes, but the important thing for everyone to know is that it’s not my fault.
There's a lot of things going on here.
I think modularity and consistency can help a lot. But there will always be a learning curve between disparate systems. Creating less friction always creates more maintenance debt which needs to be paid each time a system is changed.
I agree with the premise that all-in-one tools are more likely to overpromise and underdeliver. Databases themselves are a sort of one-stop shop and they are very good at many things. A jack-of-all algorithms. But once you need to optimize a certain code path you may need to switch some parts to use custom code. This custom code might be able to run inside of the database or it might not. I think all-in-one tools fit the same kind of category.
The main differentiator between these things--why all-in-one apps are more doomed to fail--are because most MBAs aren't going out there building new databases. They are building all-in-one apps and so the problem is often tackled with the wrong direction or scale.
There's definitely a lot of things going on here, and honestly I'm pretty sure I sorta wrote around in circles without getting anywhere. But there's definitely a difference in fundamental design/development principles between "thing sold as product" which seems to constantly feature creep, and unix-like programs where if it accomplishes the job well enough people just stop working on it
I think the main thing is that all-in-one apps can only succeed if they are a monolith to begin with. If the all-in-one app is just wrangling in data from a bunch of different places then there is huge friction to get started and maintain integration across disparate data-generating/capturing tools. The task is to become Sisyphus.
I distribute my own tools as one python package even though they would probably be more popular if I split them up into different projects: https://github.com/chapmanjacobd/library
But I don't really want to manage a bunch of different projects. There would be a huge amount of overhead to do that. Maintenance is centralized in this case because all the data-generating and data-analyzing activities happen from within the same system. The edges of the system are the most likely to flail away like the Red Inflatable Tube Man, ready to sail across the universe into full entropy.
But some people have a different mentality: https://tildes.net/~tildes/1azo/possible_solution_for_the_perennial_what_app_issue but maybe at some point we are getting into Ted Nelson Xanadu territory and the world moves out from under our feet and we are nearing death's door so there is nothing left to do but concede to the great HTML god
Startups are usually in search mode so they are constantly adding and removing requirements as they talk with customers about what needs they have. CLI tools are usually written by people who are themselves users of the tool. The feedback loop is 10x to 100x quicker.
But I have to say many unix-tools are in constant development as well (and databases only stop being developed when they become obscure and die out of use). There are new features added to many userspace tools every year. Just pick your tool and there have probably been new releases within the past few months (but of course if you use something like Mac OS Apple won't bother to update any of it--at least until the next major OS release*):
https://fossies.org/linux/coreutils/ChangeLog
https://fossies.org/linux/sed/ChangeLog
https://busybox.net/
https://dev.yorhel.nl/ncdu/changes
But yes, in terms of software stability it is much less likely for these 30 or 40 year old tools to have completely different interfaces. Although the same is probably true for 30 or 40 year old products that other businesses sell. The problem is that most software doesn't live that long. I can only think of enterprise tools like Oracle Database... although a lot has changed in the past 40+ years that it has existed. I think the main thing preventing old software from being sold for so long is that customers want something new and exciting. Only boring technologies are allowed to age and mature but still be respected enough to pay for.
* To prove your point: some of this is because it increases the risk of regressions or bugs in new features but the biggest reason is that nobody complains about it. People aren't refreshing the changelog for these tools every day because they only use each tool to do one thing that they know it can do. It's boring technology. They would only care if there is added friction from something breaking or them needing to spend time thinking about how an interface or functionality change will affect their workflows (like when ncdu updated recently and now I need to always do "NO_COLOR=1 ncdu" in order to have a pleasant experience)
Yeah, while it could be argued that even open source projects compete in a notional marketplace of ideas, it's nowhere similar to the capitalistic 'we must sell widgets ever day to survive' that fuels commercial software to devote resources into making their products more attractive to buyers.
I suppose one core lament of this whole post is that I don't like that product development often flows down this path of least resistance because it... doesn't seem to work but takes 5-10 years to play out. I suppose it pays the bills while people realize it.... but resources probably could've have been used on something of more potential value if only we learned when to stop riding the one trick pony...