Empowering crisis-affected communities: How Signpost AI is scaling local information services

In 2015, 65 million people were forcibly displaced. Among them were migrants in Greece, the Balkans and Italy, many of whom were seeking asylum or refugee status. To support them with reliable information based on their needs, the International Rescue Committee (IRC) set up the Signpost Project, which establishes digital help centres tailored to the questions the community needs answers to. Locally-based staff curate relevant and publish content that allows clients to access essential services and make informed decisions on the issues that matter most to them. Clients can also ask direct questions to a local community moderator and receive a response in their language in less than 24 hours.

Now in 2024, with the number of displaced people having reached over 117 million, the Signpost team is piloting a generative AI-powered chatbot to considerably scale up services and reach the ever-growing number of displaced people and communities in crisis.

Collaborating directly with impacted communities

It is estimated that over 1.2 billion people could be displaced globally by 2050 due to climate change and conflict. Humans that have been forcibly displaced or impacted by crisis need information to support them in what are often unfamiliar or volatile environments. Where do I go? Where can I get food, shelter, or medical services? How do I apply for asylum?

Responses to these questions need to be hyperlocal and up to date to be effective. The Signpost information service addresses this by collaborating directly with impacted communities, to ensure focus on what is relevant to them. Locally based teams curate information from websites, social media, and other sources, and make it available to clients through digital channels such as WhatsApp or Facebook, or through direct two-way communication with human moderators.

However, the challenge is scaling these services up to meet the rapidly growing need.

This is where Signpost AI comes in.

AI development

The Signpost team hypothesized that using a generative AI-powered chatbot could deliver a number of benefits: It would increase the speed of information curation, allow a level of standard questions to be answered directly by the chatbot at all times, and offer expanded language capabilities. At the same time, the team was aware of the risks using such a tool, such as providing incorrect, inappropriate, or biased responses. Mitigating these risks was going to be essential for the success of the solution, where engagements with clients are high-stakes, and there is little to no margin for error. To provide an overall framework for the ethical use of AI, the team, with the input of 50+ stakeholders, established guiding principles they would follow. Collaborators included local NGOs, research institutions, and technology partners.

A critical success factor was to make sure that the chatbot, named SignpostChat, would communicate with the right language, tone, and context. Taking a community-led approach, with local teams in the pilot countries Greece, Italy, Kenya and El Salvador, the Signpost AI team tested SignpostChat with different large language models (LLMs) to establish which of them would provide the best quality of responses for a specific country. Locally based project teams made sure that the “voice” of SignpostChat would sound familiar to users from the local community, and thus would be able to gain their trust.

The local teams test the chatbot and flag any responses that needed adjusting for correctness, language, cultural sensitivity, or any other concerns. A defined Quality Framework is used to assess the quality of the chatbot’s output and to decide whether responses pass the quality checks or whether the technical support teams need to intervene.

Operationalising AI

The four SignpostChat pilots will run until February 2025 to allow sufficient time to test and mature the solution before scaling. The solution will continue to operate with the “human in the loop” - local moderators who ensure SignpostChat understands and respects the diverse cultures and communities it serves, reacting with empathy and sensitivity to a client’s local context and the question they have asked. SignpostChat is not designed to replace local teams but to enhance their ability to deliver critical services more efficiently, ensuring that high-touch human interactions remain central to the work.

Importantly, the team recognizes that a generative AI powered chatbot will have limitations in terms of the complexity of requests it can handle. Once SignpostChat is released for use by clients, in cases where the chatbot is not likely to provide an appropriate response they will be referred to a human agent to discuss their needs and get support.

SignpostChat is embedded within the infrastructure of Signpost – leveraging the customer success platform, Zendesk, as an omnichannel communication and content management system. The country pilot teams thus use SignpostChat within the user interface with which they are already familiar.

Learnings

Whilst SignpostChat is still in its pilot phase, there are already many learnings documented by the team, which are transparently shared through the Signpost AI Blog. They cover client needs (keep it simple, contextual, empathetic), technology (the performance of different LLMs in specific environments), and how to measure quality of responses.

Some of the key learnings are:

• the importance of local language and context to ensure high quality responses;
• the level of effort to refine and tweak the prompts needed to get SignpostChat to provide appropriate and correct answers; and
• the amount of upskilling needed for the project teams, where moderators in each country, who play key role in validating the AI output and maintaining service quality, need a good level of understanding of the tool. Insights from pilots will guide the expansion to new geographies and partnerships, ensuring the tool meets diverse community needs.

Local language support takes effort, as well – right now the team wants to start supporting Farsi and Somali. However, at the moment, the AI tools are struggling to provide good quality output in those languages.

Plans for the future

The SignpostAI team has an ambitious 3-year target of reaching 50% of the world’s displaced people with critical real-time information. To do this, Signpost aims to onboard up to a thousand local NGOs to boost their capabilities and supplement their workforce with advanced AI agent systems. In turn, this means lower cost per client than human-only support, and improved service quality for their clients due to much reduced waiting times.

As part of their plan to scale up, the team is exploring a “human-on-the-loop” approach with direct to client responses for questions that the bot can answer with the AI, but still supported by human moderators that will intervene in case of issues and respond to questions that are better fielded by humans, e.g., acute crises, or complex issues beyond the scope of the knowledge base.

The first step of this expansion will be to make SignpostChat available to those 300+ local NGOs who are currently using Signpost; this will involve an optimized approach to adapting the tool to local languages and context, and upskilling local moderators so they are able to intervene if there are any issues with SignpostChat’s responses.

Right now, however, the team is very much focused on refining its approach towards responsible AI, i.e., providing a safe, ethical, and efficient AI support to teams, driving greater impact, and reaching more people in need.

Find out more:

Contact: Andre Heller, Director, Signpost Project, andre.heller@rescue.org or visit the SignpostAI Website.