When you try different models, settings, or prompts, it can be difficult to keep track of what produced each result. AI Experiments keeps the settings, measurements, and files recorded by your Python programs together so you can compare them in MLflow.
The guides below follow that process from a first look at the interface to recording and comparing results. The Introduction shows practical uses and connections with other products; Definitions of terms explains the vocabulary used in the later examples.
For a new instance, the sequence below starts with storage and Python preparation, then covers instance creation, client authentication, and two calculation runs. With an existing team instance, the reader can start with Python preparation and client authentication; its owner supplies the tracking URL and membership. Reviewing results already recorded by a colleague needs only browser access to the instance.
We will use Sherlock, available from CloudFerro Cloud, for two comparisons. In the first, two models classify the same messages using the same prompt. In the second, we compare two prompts with the same model to see how the wording of the instructions affects the answers. MLflow records the answers, classification accuracy, and response times for each combination so the results can be compared.
