I don’t have plans for a specific model right now. The new vision system is installed and I’m working on some networking magic so I can talk to it easily. Then I will start collecting data with it and see how the images look.
I imagine a DNN would be used. There’s some very interesting vision work going on with transformers right now, but the networks are often large and I imagine that all needs a few years at least before it could be run on edge computing.
I do absolutely love CLIP for the way it ingests training data.
I have done some work to run real time image segmentation on my personal robot Rover.
I don’t have the latest work I’ve done in that thread, but it describes what I have been working on.
However with Rover the Nvidia example network is slow, so I am planning on checking out this network:
I am very interested in self supervised learning, but I’ve not gone in to what it would take for our application. Ideally I would want a network that can automatically learn to differentiate between different plants without explicit labels. My plan is to start collecting a bit of data from the new cameras and then start talking to researchers.
I just came across this paper today. I’ve only read the abstract but it sounds interesting:
I’ve thought there must be some way to automatically differentiate different plants based on the idea that for a given plant, it will always be near parts of itself, while neighboring plants will often but not always be nearby. However I am a relative novice when it comes to deep learning, so this is an area where open collaboration is a must. Luckily it’s an application I think many will agree is worth working on.
Also we only have a Jetson Nano for compute right now which is fine for collecting data, but I imagine we would upgrade to an Xavier or better for running inference onboard.