AI4PublicPolicy Policy Extraction and Recommendation toolkit part 2

Part 2 – Architecture and Procedures

In the first part of our blog series on the Policy Extraction and Recommendation toolkit developed for the AI4PublicPolicy platform, we discussed the design goals. In this second part, we delve into the architecture of the module.

After analyzing state-of-the-art solutions, our team opted to design our system around several independent sub-components, each addressing specific requirements.

The following figure illustrates how the components map onto the AI4PublicPolicy Reference Architecture. The toolkit interacts with the Policy and Data Management component to retrieve policies and datasets, the Policy Interpretation component to extract insights, and dashboards for AI model interaction. Additionally, it interfaces with the AI Security component to ensure data and training trust and with the AutoML component to support public administrators and provide AI models and dashboards to the VPME.

The diagram below depicts the internal architecture of the Policy Extraction Toolkit and how its subcomponents interact with each other.

The Web “User Interface” enables data scientists and policymakers to perform necessary actions throughout the policy lifecycle.

“Code Repository” provides versioning and collaboration for AI projects and templates.

“Open Analytical Environment” is a subcomponent that provides an AI expert the environment for developing and executing AI experiments.

“Pipeline Manager” is responsible for structuring AI experiments in a pipeline and tracking and comparing all the runs of the experiments with related parameters, metrics, and artifacts (i.e. AI models, graphs, KPIs);

“Model Manager” is a subcomponent responsible for managing AI models during the policy extraction lifecycle.

Description of components interactions and policy design process:

Users start their interaction with the toolkit through Web UI, with separate views for AI experts and policymakers. Using a specific view, an AI expert can create an AI project by launching the open analytical environment to develop AI experiments. The AI project includes several files, related to different stages of the experiment (data preparation, training, data visualization), which are stored in the open analytical environment and committed to the code repository when they are updated.

Pipeline manager and model manager are accessible through UI and they provide AI experts the ability to manage pipelines and models. The code developed in the analytical environment is structured in a pipeline that could be managed by the pipeline manager. Models are created as a result of running pipelines and managed by the model manager for versioning and staging in different environments.

The diagrams below show the whole process of designing the policy starting from selecting a policy and creating a project by an AI expert and finishing with the policy maker approving the policy with the related model.

The first diagram shows the workflow that is performed by an AI expert. An AI expert first selects a policy to work on, selects a Key Performance Indicator (KPI), and creates the project, loads a dataset and starts working on the AI model development.

In the second diagram, an AI expert runs several experiments by trying different parameters for the AI model and/or several different AI models. Then the model is stored together with the pipeline that applies this model to a dataset.

The diagram below shows the workflow for a policy maker. Here, they can browse through available pipelines for a selected policy and KPI, run the pipeline and then select a pipeline that has the best performance. Then, they approve this model as the best model for the selected policy and KPI.

In this diagram, the workflow performed by the policy maker is showcased. At this stage, the policy maker selects a policy and provides the required input for every available KPI. Subsequently, the policy maker runs the model. Then, they navigate to the evaluation dashboard, compare models, select the best one, and approve it.