Invented by Scott Hellman, Lee Becker, Samuel Downs, Alok Baikadi, William Murray, Kyle Habermehl, Peter Foltz, Mark Rosenstein, Pearson Education Inc

The market for systems and methods for automated machine learning model training for a custom authored prompt is experiencing significant growth and is poised to revolutionize the field of machine learning. As the demand for machine learning solutions continues to rise across various industries, the need for efficient and effective model training methods becomes paramount. Automated machine learning (AutoML) systems offer a solution by streamlining the process of model training, making it accessible to a wider range of users. Traditionally, training machine learning models required a deep understanding of complex algorithms, data preprocessing, feature engineering, and hyperparameter tuning. This process was time-consuming, resource-intensive, and often required a team of experts. However, with the advent of AutoML, the barriers to entry have been significantly lowered. AutoML systems leverage artificial intelligence and machine learning techniques to automate the model training process. These systems can automatically perform tasks such as data preprocessing, feature selection, hyperparameter optimization, and model evaluation. This automation allows users with limited machine learning expertise to train high-quality models quickly and efficiently. One specific area where AutoML is gaining traction is in the training of models for custom authored prompts. Custom authored prompts refer to specific instructions or guidelines given to the model during the training process. These prompts can be tailored to address specific business needs, industry requirements, or even ethical considerations. The market for systems and methods for automated machine learning model training for custom authored prompts is driven by several factors. Firstly, the increasing demand for machine learning solutions across industries such as healthcare, finance, retail, and manufacturing is fueling the need for efficient model training methods. AutoML systems provide a scalable and cost-effective solution to meet this demand. Secondly, the democratization of machine learning is another driving force behind the market growth. AutoML systems empower users with limited machine learning expertise to leverage the power of AI without the need for extensive training or hiring specialized teams. This accessibility opens up opportunities for small and medium-sized businesses to adopt machine learning solutions and gain a competitive edge. Furthermore, the ability to train models with custom authored prompts offers a level of flexibility and control that was previously unavailable. Businesses can now tailor their models to address specific requirements, ensuring that the trained models align with their unique needs and objectives. This customization enhances the accuracy and relevance of the models, leading to improved decision-making and better business outcomes. Several companies are already capitalizing on this market opportunity by developing and offering AutoML systems specifically designed for training models with custom authored prompts. These systems incorporate advanced natural language processing techniques to understand and interpret the prompts accurately. They also integrate with existing machine learning frameworks and provide user-friendly interfaces to simplify the model training process. In conclusion, the market for systems and methods for automated machine learning model training for custom authored prompts is witnessing rapid growth. The demand for efficient and accessible machine learning solutions, coupled with the need for customization and control, is driving the adoption of AutoML systems. As this market continues to evolve, we can expect to see further advancements in the field of machine learning, making AI more accessible and impactful across industries.

The Pearson Education Inc invention works as follows

The present invention discloses systems and methods for automating custom training of a score model. The method includes: receiving responses from a plurality students in response of a given prompt; identifying a machine-learning model that is relevant to the prompt; which model can be trained to output a relevant score for at least portions in a response; creating a training indication that shows a graphic depiction the degree of training the identified evaluation; determining the status of the model, receiving at least a single evaluation input when it is determined the model has not been sufficiently trained; updating the training of evaluation model using the received evaluation input received;

Background for Systems and Methods for Automated Machine Learning Model Training for a Custom Authored Prompt

A computer network, or data network, is a telecommunications system that allows computers to share data. Computer networks allow networked devices to exchange data via network links. Cable media or wireless media are used to establish connections between nodes. “The Internet is the most well-known computer network.

Network nodes are devices on a network that originate, route and terminate data. Nodes include personal computers, phones and servers as well as network hardware. “Two such devices are networked when they can exchange information, regardless of whether they have direct connections to each other.

Computer networks are different in terms of the transmission medium used to transmit their signals, communication protocols used to organize traffic on the network, network size, topology, and organizational intent. Most often, communication protocols are built on top of each other (i.e. Work using) other, more specific or general communication protocols except the physical layer which directly deals with transmission media.

One aspect of this disclosure is a system that allows an evaluation model be customized to match an evaluation style. The system comprises a memory that includes: a content database with a plurality prompts, and evaluation data for each prompt; and a database of models including a plurality evaluation models to automate evaluation of user responses. In some embodiments the evaluation data for each of the plurality prompts include a link to the associated model. The system comprises at least a processor which can: receive responses from a plurality users in response of a given prompt; identify a model of evaluation relevant to that prompt, which includes a machine-learning model trained to provide a score for at least portions in a response; create a training indication, which provides a graphic depiction to the degree of training the identified evaluation models; determine a status of training the identified model.

In some embodiments, an evaluation model may include a plurality. In some embodiments each of the multiple evaluation models is linked to a portion of the prompt that evaluates the model. In certain embodiments, at least one processor determines a first response order. In some embodiments the first response order identifies the order in which responses are provided to the user. In some embodiments the response order is determined by the estimated contribution each response makes towards the completion of training the evaluation model.

In some embodiments the atleast one processor can determine the second training status of a model after updating the training of an evaluation model using the received evaluation input. In some embodiments the processor can: auto evaluate the response if the second training status is determined to be sufficient; determine whether the auto evaluation of the answer is acceptable; and determine a secondary response order if the auto evaluation of the reply is determined to be unacceptable.

In some embodiments, identifying a model of evaluation relevant to a prompt provided includes: identifying portions for prompt evaluation; and retrieving sub-models associated with each portion. In some embodiments the at least one computer can determine the training level of the model. In some embodiments determining the training level of an identified model can include retrieving submodel data for each retrieved submodel; determining the submodel confidence level for every retrieved submodel; and determining the aggregate confidence level by combining the submodel confidence levels.

The present disclosure includes a method for customizing an assessment model to an assessment style. The method comprises: receiving responses from multiple users in response of a prompt; identifying a model that is relevant to the prompt; which model includes a machine-learning model trained to output scores relevant to portions of a reply; generating a “training indicator” which provides a graphic depiction to the degree of training the identified evaluation models; determining the training status of identified model; receiving at least one input of evaluation; updating the training model using the received input of evaluation; and controlling the indicator to reflect how well the model

In some embodiments, an evaluation model may include a plurality. In some embodiments each of the plurality evaluation models is linked to an evaluation portion provided by the prompt. In some embodiments the method comprises determining a response order, wherein this response order identifies the order in which responses are provided to the user for evaluation.

In some embodiments the response order is determined by the estimated contribution each response makes towards the completion of the training of the evaluation models. In some embodiments of the method, the second training status is determined for the identified model after the update of the training model on the basis of at least one evaluation input. In some embodiments of the method, auto-evaluating is performed when the second status of the model is deemed to be sufficiently trained.

In some embodiments, identifying a model of evaluation relevant to a prompt provided includes identifying portions for prompt evaluation and retrieving sub-models associated with each identified portion. In some embodiments the method also includes determining the training level of an identified model. In some embodiments determining the training level of an identified model comprises retrieving submodel data for each retrieved submodel; determining the submodel confidence level for every retrieved submodel; and determining the aggregate confidence level by combining the submodel confidence levels.

One aspect of this disclosure is a system to control the training quality of a model of machine learning. The system includes: a memory containing a content database containing a number of prompts, and evaluation data for each prompt; and a database of models containing a number of evaluation models to automate the evaluation of user responses. At least one processor is included in the system. The at least processor can “receive a plurality responses received from a number of users in response of providing at least one prompt”; “identify an evaluation models relevant to the provided request, which evaluation models includes a machine-learning model trained to output relevant scores to at least portions a response, generate a Training indicator, which provides a graphic depiction to the degree that the identified evaluation is trained, determine a status of training of the identified Model; control the Training indicator to identify a status of training of the identified model, the at least one processor

In some embodiments, a graphical indicator can indicate at least one: a distribution score generated by the model of evaluation; a level of confidence of the model of evaluation; or an accuracy level for the model of evaluation. In some embodiments the graphical indicators identifies outliers scores based on historic user data. In some embodiments, identifying outlier scores based on historic user data includes retrieving historical data, comparing the historical results to the results of evaluating a plurality responses and indicating an “outlier score” when the discrepancy is greater than a threshold.

In some embodiments the historical data include a distribution of historical evaluation results. In some embodiments the results of evaluating a plurality of responses include an evaluation result. In some embodiments the historical data also includes user-historical data. In some embodiments the user historical data is related to at least one user who has previously received evaluation results.

In some embodiments, at least one processor is able to determine the acceptability of an evaluation. In certain embodiments, acceptability is determined by identifying outlier scores. In some embodiments the processor can train the evaluation model further when the evaluation is unacceptable. In certain embodiments, at least a processor can receive a choice of at least ONE response for reevaluation. It will also receive an evaluation input. The evaluation input is then used to update the training of the evaluation.

The present disclosure includes a method for controlling the training quality of a model. The method comprises: receiving a plurality responses from a number of users in response of an at least one prompt. Identifying an evaluation, which includes a machine-learning model that is trained to output scores relevant to portions of a reply; generating an indicator for training, which provides a graphic depiction of how well the evaluated model has been trained; determining the status of training the identified model. Controlling the indicator to determine the status of training the identified model. Automatically evaluating the responses with the evaluation when the model

In some embodiments, a graphical indicator can indicate at least one: a distribution score generated by the model of evaluation; a level of confidence of the model of evaluation; or an accuracy level for the model of evaluation. In some embodiments the graphical indicators identifies outliers scores based on historic user data. In some embodiments, identifying outlier scores based on historic user data includes retrieving historical data, comparing the historical results to the results of evaluating a plurality responses and indicating an “outlier score” when the discrepancy is greater than a threshold.

In some embodiments the historical data include a distribution of historical evaluation results. In some embodiments the results of evaluating a plurality of responses include an evaluation result. In some embodiments the historical data also includes user-historical data. In some embodiments the user historical data is related to at least one user who has previously received evaluation results.

In some embodiments the method also includes determining the acceptability of an evaluation. In some embodiments the acceptability is determined by the outlier scores. In some embodiments the method includes training an evaluation model when the assessment is unacceptable. In some embodiments the method includes receiving at least one selection for reevaluation, receiving an evaluation input and updating the training of the model based on that evaluation input.

One aspect” of the disclosure is a system to train a model on a custom-written prompt. The system comprises a memory that includes: a content database database with a plurality prompts, and a database of models including at least one trained model to evaluate prompts. The system includes a processor that is able to: receive a request; parse it to identify a plurality prompt evaluation portions; identify pre-existing information relevant to an evaluation portion of the requested prompt; and train a model to evaluate responses to the request at least partially based on pre-existing information.

In some embodiments parsing the request includes identifying the complexity of each of a plurality of prompt evaluation parts associated with the received request. In some embodiments the pre-existing information includes: a model that has been trained to evaluate responses for another prompt, and response data generated by responses to other prompts.

Click here to view the patent on Google Patents.