Bringing Vision to the Big Stage with Remyx AI ft. Databricks

Access this guide’s notebook here!

Many groups need help to make sense of all the images and videos lying in cloud blob storage. They’re trying to wrangle “unstructured data” – info that’s not neatly organized and labeled or fitting a tabular format.

It takes a lot of work to turn all those images and video clips into helpful information for the business. Bringing structure to this medium typically means employing deep learning to infer semantic information.

Recent multimodal models like GPT-4/MiniGPT-4/LLaVA demonstrate powerful ad hoc visual question-answering capabilities. But as the scale of your image/video corpus increases, you’ll find these models too slow and unconstrained to analyze your data lake reliably.

By specializing, you can achieve higher throughput at a lower cost with fewer false positives or spurious predictions. Using the Remyx Model Engine, we generate a custom model to analyze images in an S3 bucket. No computer vision expertise is required!

The ability to analyze images and video using MapReduce with Spark is nothing new, but the accessibility of relevant models limits the applications. Most publically available models are optimized for performance on the same research benchmark datasets.

By creating a custom model with our no-code autoML for vision, simple text descriptions are all you need to get an optimized model to process your image and video collections. This notebook shows how easy it is to add structure to your image and video collection with custom models generated on-the-fly. It’s almost as if a model zoo had the precise model for your application.

In the notebook, you’ll find instructions to configure and use the Remyx CLI to download a previously created model. Once loaded, we show how to load the model in an MLflow Custom Pyfunc to apply the model to a dataframe of loaded images.

Once logged, running inference is as easy as:

# Loading the model
model_uri = f'runs:/{run_id}/pyfunc_model'
loaded_model = mlflow.pyfunc.load_model(model_uri)

# Loading a dataset 
inference_df = (
    spark.read.format('binaryFile')
              .option('pathGlobFilter', '*.jpg')
              .load('file:/tmp/interiorspacesid_test_dataset/')
              )
inference_pdf = inference_df.toPandas()
 
# Running inference
inference_pdf['prediction'] = loaded_model.predict(inference_pdf['content'])

Integrating the Remyx AI and Databricks platforms will enable many teams to derive valuable insights from their unstructured data without the need for substantial machine learning infrastructure.