AI Coder - Remark Labelling

AI Coder - Remark Labelling

Introduction

AI Coder is a powerful tool designed to automate the remark labeling process, making it easier to analyse open-ended responses in your EyeQuestion surveys. By leveraging advanced language models, AI Coder analyses and categorises remarks based on pre-defined labels, streamlining data processing and providing valuable insights. This document provides a step-by-step guide on how to use AI Coder, from setting up labels to reviewing the output, ensuring you can make the most out of this tool for your data analysis needs.

How to Use It

Step 1: Open Remark Labelling
After having collected Data for open remarks, click on Data>Remark Labelling. 

Step 2: Define the Label
To define your label, you can:
  1. Use your product's vocabulary: If your product has specific story and attribute terms, copy and paste each label separated by commas.
  2. Utilise EyeQuestion's pre-made label library: Based on your product category, select the category from the dropdown, and the labels will be automatically added to the text input field. You can modify the list and add new labels as needed.

Step 3: Apply AI Coder
  1. After defining the labels, click Apply AI Coder Label.
Remark Preview
All questions containing remarks sent to AI Coder will appear here for review. You can check the remarks to ensure the correct information is sent to the AI. If any remarks should not be included in the labelling task, you can exclude them.
  1. PII Information: EyeQuestion automatically anonymises remarks flagged as personally identifiable information (PII).


Data Processing Agreement
Before proceeding, check the box to confirm your agreement that the data will be sent to OpenAI for processing. 

Start AI Labelling Job
Click Start AI Labelling to begin. Depending on the number of remarks in the project, the task may take anywhere from a few seconds to a minute.

Output Explanation

  1. Language Model for Analysis: which model is used for analysis.
  2. Question-Specific Output: Select which question’s data to visualize. 
  3. Label Count: See the total number of labels applied across all questions.
4. Sentiment Analysis: Sentiment analysis is the process of determining the emotional tone behind a piece of text, such as whether it expresses a good or positive, negative or bad, or neutral sentiment. You can view sentiment analysis via a spider plot for each product. You can hover over the graph to see label frequency.
The sentiment is color-coded:
  1. Green: Good
  2. Red: Bad
  3. Blue: Neutral
5. You can download these spider plots manually, as EyeQuestion does not store them after the task completes.
6. Summary Table: This table provides a quick overview of panellists' responses. You can download this summary, but it is not stored within EyeQuestion after the task is completed. Copy and paste the content if you need to save it elsewhere.

7. Labels: See the labels applied to each remark with sentiment color indicators.
8. AI Coder will apply the most appropriate label to each remark. Only one label is applied at a time.


Step 4. Review & Apply Labels
Once you review the labels, you can decide whether to apply them to EyeQuestion for further analysis (10) (e.g., EyeOpenR) or start a new AI Coder job.
Note: Deleting a project will remove the all the AI Coder data/job and only labels will be stored in EyeQuestion.



Once the labels are applied, they will appear in the remark labeling table, with "AI" prefixed to those added by the AI Coder.


Applying Multiple Labels
Sometimes, remarks may require multiple labels. In the current version of AI Coder, only one label can be applied at a time. However, after the initial labeling job, you can start a new job to apply additional labels to the same remarks.


How it Works

  1. Summary Generation: a Large Language Model(LLM) are used on a subset of remarks to provide a summary of the general themes present.
  2. Individual Labeling: a Large Language Model(LLM), combined with a predefined set of labels (such as "sweet" or "sour"), labels each remark individually. Users can choose from our provided label sets or create their own custom sets.
  3. Sentiment Analysis: a Large Language Model(LLM) always assigns one of three sentiment-based labels to each remark: bad, neutral, or good.
At this moment we use GPT(Generative Pre-trained Transformer) as our default LLM, the GPT-N series is developed by OpenAI and is based on a deep learning architecture called the Transformer.
Here's a simplified explanation of how GPT works:
  1. Data Collection: GPT is trained on a large dataset of text from various sources on the internet. This dataset contains a wide range of topics and writing styles, providing the model with a diverse understanding of language.
  2. Tokenization: When you input text into GPT, it first breaks down your input into smaller chunks called tokens. These tokens are the basic units of text that the model processes. On average 1000 words will be 750 tokens. 
  3. Contextual Understanding: GPT then uses its deep learning architecture to analyze the tokens in the context of each other. Unlike older language models, GPT models can understand and generate text based on the broader context of the conversation, not just the immediate previous message.
  4. Prediction: Based on its understanding of the context provided by the input tokens, GPT generates a response by predicting the most probable next tokens in the conversation. It does this by assigning probabilities to each token in its vocabulary and selecting the most likely ones based on the context.
  5. Generation: Finally, GPT generates a response by selecting the tokens with the highest probabilities. These tokens are then assembled into coherent text and returned as the model's response to the user's input.
  6. Iterative Learning: The model continually improves through a process called fine-tuning. This involves training the model on additional data and fine-tuning its parameters to improve its performance on specific tasks or domains.
Overall, GPT works by understanding the context of a conversation and generating text based on that understanding, leveraging its extensive training on a large dataset of diverse text from the internet.

Security 

OpenAI places a high priority on data security and privacy. According to the information provided in the extracts:
  1. Data sent to the OpenAI API is not used to train or improve OpenAI models(https://platform.openai.com/docs/models). 
  2. Data submitted through the OpenAI API is not used to improve OpenAI’s service offering (https://openai.com/security/).
  3. OpenAI does not share user content with third parties for marketing purposes (https://openai.com/security/).
  4. To help identify abuse, API data may be retained for up to 30 days, after which it will be deleted (unless otherwise required by law). (https://platform.openai.com/docs/models).
Therefore, your data’s security and privacy should not be compromised when using the OpenAI API. However, it’s always a good practice to review the Privacy Policy (https://openai.com/en-GB/policies/eu-privacy-policy/) and Terms of Use (https://openai.com/policies/terms-of-use/) to ensure you understand how your data is handled.

Note:
  1. Remark or text question's in EyeQuestion that were flagged to be Pii information will not be sent to OpenAI.
  2. Product information that is setup in EyeQuestion will not be sent to OpenAI. 
  3. All remarks selected are sent to OpenAI for processing. To protect personal information, you can exclude them from the analysis specific remarks where panellist have included personal information. 
  4. If an AI Coder job has been run but the labels have not been applied, the previous AI Coder job will be shown. To start fresh, click Delete Project, enter the label, and click Apply AI Coder.

    • Related Articles

    • How to Use Remark Labeling

      Introduction Open-ended comments provide valuable insights into the consumer's perception of a product, and help in identifying areas for improvement. However, analyzing such comments can be time-consuming and challenging. With remark labeling, the ...

    • Introduction to EyeQuestion 6

      We could not be more excited to announce the release of our latest software update, version 6, packed with new features designed to enhance user experience, including: Improved Panel Management: Streamline workflows with our enhanced panel management ...

    • Remarks Representation

      Purpose The remarks representation method can be used to display remark and comment data. Options Split Results On: Choose to split your results on either Judge, Attribute, Product or Session. Alternatively, choose None if you do not wish to split ...

    • Text Highlighting

      Introduction The text highlighting is a novel method for explicit attitude measurement. This method leverages the natural, intuitive action of highlighting to gauge participants' attitudes. Participants are shown a piece of text and are asked to use ...

    • How to Display Feedback of Triangle and Tetrad Test

      Introduction During a Triangle test a panellist is presented with three samples of which two are equal and one is different. A panellist must point out the odd sample. A user can add the screen condition in the triangle test to display message when a ...