TASK FORCE LIMA
TASK FORCE LIMA
LLM Maturity-Model Workshop
Hosted By: Task Force Lima
Calling all LLM Developers! If you’re a product manager, researcher, designer, engineer, or architect building an LLM product, we want to collaborate with you to co-develop an LLM Maturity Model and associated acceptability criteria to help guide both use of LLM solutions in DoD and non-DoD use cases. Development of the LLM Maturity Model will be the centerpiece of the LLM track at the upcoming Advantage DoD 2024 Symposium and will support community discussion aimed at aligning LLM performance and accuracy with use case needs.
The proposed maturity model will define levels of acceptable capability for LLM solutions across a number of relevant dimensions, such as accuracy, completeness, traceability, focus, creativity, and others to be chosen, and will serve as a decision rubric for those wishing to leverage LLMs by enabling assessment of current LLM capability in the context of use case requirements. We believe defining an LLM maturity model along with associated acceptability criteria for LLM products is key to the responsible and secure adoption of LLMs among both military and non-military users.
As part of the LLM track at Advantage DoD 2024, Task Force Lima will showcase the top LLM use cases collected throughout the DoD, industry, and academia, and conduct a series of workshops to define the LLM Maturity Model and set the acceptability criteria for applying LLMs to the relevant use cases. This project expands beyond the symposium.
By developing the LLM maturity model and acceptability criteria for LLM products, together, we will be able to answer the following questions:
- What is the LLM Maturity level required for given use cases?
- What are the LLM acceptability criteria required for given use cases?
- Are current LLM capabilities acceptable for use case needs?
- Where is additional work needed to improve LLM performance to meet use case needs?
To propose your work to be included in the discussion and development of the LLM Maturity Model and associated acceptability criteria, submit your organization’s proposed design for the LLM Maturity Model on the CDAO’s platform, Tradewinds, by December 7, 2023.
Task Force Lima monitors, develops, evaluates, and recommends the responsible and secure implementation of generative AI capabilities across the DoD.