Platform for stateful frame-by-frame processing
Platform for stateful frame-by-frame processing
2 by robotarchitect | 0 comments on Hacker News.
Hello everyone. I'm looking for a platform to do stateful frame-by-frame(sequential) processing of data. Modern stream processing engines tend to focus on parallel processing of big data. This is great when the desired results are calculated from aggregates over the entire set or subset(filtered) of the whole dataset. In contrast to this I need to process data sequentially, where a change from one row to the next has meaning and changes what has to be done next. All data is time-series data. As an example, imagine a video game generating the results according to the current situation and the inputs of the player. Spark kind of allows this with e.g. mapGroupsWithState(), but I believe it's not what it's designed for, as this case doesn't need distributing data across a cluster to do processing on(because as data needs to be processed sequentially,it should benefit from holding the state in one place and passing all data through a single point of processing close to that state). I've looked at flink but I didn't find anything related to stream processing frame-by-frame. Is anything out there for solving this type of problems? I don't want to re-invent the wheel. Thank you.
2 by robotarchitect | 0 comments on Hacker News.
Hello everyone. I'm looking for a platform to do stateful frame-by-frame(sequential) processing of data. Modern stream processing engines tend to focus on parallel processing of big data. This is great when the desired results are calculated from aggregates over the entire set or subset(filtered) of the whole dataset. In contrast to this I need to process data sequentially, where a change from one row to the next has meaning and changes what has to be done next. All data is time-series data. As an example, imagine a video game generating the results according to the current situation and the inputs of the player. Spark kind of allows this with e.g. mapGroupsWithState(), but I believe it's not what it's designed for, as this case doesn't need distributing data across a cluster to do processing on(because as data needs to be processed sequentially,it should benefit from holding the state in one place and passing all data through a single point of processing close to that state). I've looked at flink but I didn't find anything related to stream processing frame-by-frame. Is anything out there for solving this type of problems? I don't want to re-invent the wheel. Thank you.
Comments
Post a Comment