Python Workflow Design Pattern Python Workflow Design Pattern python python

Python Workflow Design Pattern


I believe that you are both, right and wrong, in doubt of re-inventing the wheel. Maybe different levels of thinking gives you a hint here.

How to eat an elephant?

Level A: software design

At that level, you would want to stick to the best practice that no long operations are done in the UI (and UI thread). You need an UI layer that focuses only on gathering input (including cancellation) and drawing (including in-progress-visualization like progress-bar or hour-glass). This layer should be separated from anything else as dusk and dawn. Any call outside of this layer must be fast if you want intuitiveness and responsiveness.

In tasks as complex as yours, the calls outside of the UI layer are typically:

  1. Schecule some work - the command should be queued to smart layer for it to pick up whenever it gets to it.
  2. Read results - the results should be queued in the smart layer so they could just be "popped out" and rendered.
  3. Cancel/stop/exit - just raise a flag. Smart layer should check this flag now and then.

Don't worry too much that some user operations are getting reaction too slowly - if you have a solid design core then you can adjust priorities of the user input later on. Or add a short-term hour-glass or similar. Or even cancel all long operations that get obsolete after a specific user input.

Level B: the heavy-lifting smart layer

There is no "best" framework for "any" kind of hard work.

So I'd suggest you to design the feeding (by UI) of this layer as simple as possible with no frameworks involved.

Internally, you can implement it using some frameworks but you will have the ability in future to redesign the hard-working elements as needed. For example, in future you could:

  • give some math to GPU
  • share tasks to server-farms
  • involve cloud computing

For complex tasks, picking a framework at the top level of design might prove as an obstacle in the future. Specifically, it may limit your freedom of applying other technologies.

It's hard to tell for sure but it seems to me that you don't have a silver bullet framework for your task. So you should find strong tools (e.g. threads and queues) to implement good design practices(e.g. decoupling).

EDIT as a response to your edit

Your latest edit stresses perfectly the hard challenges that a software designer meets.For your case, the acceptance that there is no silver bullet. I'd suggest you to accept it sooner - better than later...

The hint resides in that you offered the most generic task to be defined by Int's and Float's. This could make you happy for today but it will fail tomorrow. Exactly as locking-in to a super-abstract framework.

The path is right - to have a heavy-lifting "task" base in your design. But it should not define Int or Float. Focus on the above mentioned "start", "read" and "stop" instead. If you don't see the size of the elephant that you are eating then you might fail eating it and end up starving :)

From Level A - design perspective - you could define task to contain something like this:

class AnySuperPowerfulTask:    def run():        scheduleForAThreadToRunMe()    def cancel():        doTheSmartCancellationSoNobodyWouldCrash()

This gives you the basis - neutral, yet clean and decoupled by Level A (desing) perspective.

However, you would need some kind of setting up the task and getting the real result, right? Sure, that would fall into Level B of thinking. It would be specific to a task (or to a group of tasks implemented as an intermediate base). Final task could be something along these lines:

class CalculatePossibilitiesToSaveAllThePandas(SuperPowerfulTask):    def __init__(someInt, someFloat, anotherURL, configPath):        anythingSpecificToThisKindOfTasks()    def getResults():        return partiallyCalculated4DimensionalEvolutionGraphOfPandasInOptimisticEnvoronment()

(The samples are intentionally incorrect by python in order to focus on the design, not syntax).

Level C - abstraction-nirvana

It looks like this level should be mentioned in this post.

Yes, there is such a pitfall that many good designers can confirm. The state where you could endlessly (and without any results) search for a "generic solution", i.e. the silver bullet). I suggest you to take a peek into this and then get out fast before it's too late ;)Falling into this pitfall is no shame - it's a normal development stage of the greatest designers. At least I'm trying to believe so :)

EDIT 2

You said: "Im working on a piece of software design, and im stuck between not having any idea what im doing, and feeling like im reinventing the wheel."

Any software designer can get stuck. Maybe next level of thinking might help you out. Here it comes:

Level D - I'm stuck

Suggestion. Leave the building. Walk into the Cafe next corner, order the best coffee and sit down. Ask yourself the question "What do I need?". Note, that it is different from the question "What do I want?". Ping it until you have eliminated wrong answers and start observing the correct ones:

Wrong answers:

  1. I need a framework that would do X, Y and Z.
  2. I need a screwdriver that could run 200mph and harvest forest in a farm nearby.
  3. I need an amazing internal structure that my user will actually never see.

Right answers (forgive me if I understood your problem wrong):

  1. I need user to be able to give input to the software.
  2. I need user to see that the calculations are in progress.
  3. I need user to visually see the result of calculations.