Looks like they haven't released their code yet, but my guess is that it's an in house wrapper around CLIP or something similar?

There is a GitHub repo / Python lib called com2fun which exploits this. Allows you to get results from functions that you only pretend exist. (Am on mobile and can’t link to it right now.)

According to the ViperGPT paper their "ImagePatch.find()" uses GLIP.

According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application.

† https://arxiv.org/abs/2206.05836

I guess the idea is to trick the model into generating pseudo code. Which really doesn’t do much more than to act as a “scratchpad“ to focus the attention of the model to reason through the problem.

Besides, the Codex models are free right now. So… one more reason to rephrase questions as coding questions ;-)

Oh, so maybe I misunderstood what I was seeing. It wrote pseudo-code that makes sense conceptually, not code that I can paste in Jupyter and run (given the right imports)?

That sure wasn't obvious from the video.

It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.

It's not actully pseudocode. If you read the paper, these are functions/libraries they introduce that haven't been published to github yet.

it's just a separate vision model. you just have to use a state of the art instance segmentation model, the task shown are really not that hard.

it's not "just a library"

So the code that was written by the AI in the video doesn't actually work as written?

It does. If you red the paper, these are functions/libraries they introduce that haven't been published to github yet.

Almost as interesting as the GPT part of it.