Analyzing Overture Maps Foundation's Places data

The Overture Maps Foundation (OMF) represents a significant collaboration among leading technology companies like Amazon, Meta, Microsoft, and TomTom, which is dedicated to building an open-source platform for geospatial data. Established to create a standardized and widely accessible map dataset, OMF aims to meet the growing demand for accurate and dependable mapping information across various sectors. By combining resources and expertise, the foundation envisions a shared framework capable of supporting diverse applications, ranging from logistics to augmented reality.

This analysis is conducted with great respect for the efforts of the Overture Maps Foundation. We recognize the tremendous potential of this collaborative initiative and are eager to see how it evolves. Our feedback is intended to contribute to the ongoing refinement of the platform, ensuring it can reach its full potential and serve diverse industries effectively.

Although the collaborative and open-source nature of the initiative presents promising opportunities, including enhanced data quality and broader accessibility, the true effectiveness and reliability of OMF’s data are yet to be fully evaluated. As the digital landscape evolves, it becomes increasingly important to critically assess the quality and implications of the data produced by the Overture Maps Foundation, particularly concerning its potential impact on industries that depend on precise geospatial information.

Previous studies on OMF's data quality

The quality of the Overture Maps Foundation’s dataset has been explored in various studies, including a notable analysis by Wille Marcel on Observable. This study offers a detailed examination of OMF’s data, utilizing visualizations and comparisons to assess its performance in practical, real-world scenarios.

However, it’s important to recognize the limitations of this study, as it primarily focused on a specific geographic area—a neighborhood in Salvador de Bahia, Brazil—encompassing 308 places. While the insights provided are valuable, they may not fully capture the overall quality or reliability of OMF’s data across different regions or applications. Consequently, there remains a need for more extensive research to thoroughly evaluate the consistency and accuracy of the data provided by the Overture Maps Foundation on a broader scale.

Scope of Analysis

This analysis focused on several key attributes within the Overture dataset, including Point of Interest (POI) location accuracy, name accuracy, number of POIs, brand association, and confidence scores. These attributes were selected for their critical role in determining the overall reliability and utility of the dataset in real-world applications.

1. POI Location Accuracy: This attribute was analyzed by comparing the geographical coordinates of POIs in Overture with those in Google Maps. Location accuracy is vital for applications like navigation, logistics, and local search, where even small discrepancies can significantly impact user experience and operational efficiency.
2. Name Accuracy: We also checked the accuracy of POI names using the Levenshtein algorithm to measure the similarity between names in Overture and corresponding names in Google Maps. Name accuracy is a key factor for ensuring proper identification of locations, which is essential for users who rely on search engines, business directories, and local services.
3. Number of POIs: The number of POIs in the Overture dataset was compared to other sources, such as Echo Analytics, to gauge the breadth of coverage across different regions. This is important for assessing whether Overture provides a comprehensive dataset, which is particularly useful for industries like urban planning, retail site selection, and logistics.
4. Brand Association: The brand attribute, which links POIs to specific companies, was assessed in terms of the number of reliable branded POIs and the coverage of these brands across different countries. This analysis is essential for market research, retail analytics, and competitive analysis, as it helps gauge the presence and distribution of major brands in various regions.
5. Confidence Scores: Confidence scores, which reflect the likelihood that a POI is accurate and up-to-date, were evaluated to understand their correlation with actual data quality. These scores are critical for decision-making, as higher confidence indicates greater reliability in the information, while lower scores suggest the need for caution.

By focusing on these attributes, the analysis aimed to assess the reliability of the Overture dataset for use in applications that depend on precise geospatial information. Understanding how these key factors perform provides insights into both the strengths and the limitations of the dataset, highlighting areas where more reliable data sources may be needed.

Number of POIs OMF provides

At the extraction moment, OMF was providing 52,849,527 Points of Interest (POIs) across 254 countries. Comparing the number of POIs offered by OMF to those available through Echo Analytics makes it clear that while OMF offers a substantial number of POIs, it is only partially comprehensive. Below is a comparison of the 15 countries with the highest number of OMF POIs.

Methodology

This analysis was conducted across four major countries: France, Great Britain, the United States, and Mexico. For each country, we randomly selected a sample of 445 POIs, resulting in a total of 1,780 POIs. These POIs were manually annotated by comparing them with Google Maps, which served as our reference.

The OMF POIs were categorized as follows:

• Not found: The OMF POI could not be located at the reported site using other references.
• Far: The OMF POI was found but located more than 1 km away from the reported position.
• Closed: The OMF POI was permanently or temporarily closed.
• Open/Good: The OMF POI was found open and located near the reported site.

Findings

The distribution of the POIs in these categories is as follows per country:

A great correlation between overture confidence score (an attribute available by OMF) and the portion of the quality POIs has been observed:

One of Overture’s key advantages is its ability to identify high-quality POIs. Our analysis indicates that this capability significantly enhances the accuracy of POI predictions within the dataset.

The chart below illustrates the geographical distance between POIs in the Overture dataset and their corresponding locations on Google Maps, broken down by confidence score. The data is represented through the first quartile (Q1), median, and third quartile (Q3) for each confidence group. Notably, it reveals that, for instance, 50% of POIs with a confidence score between 75% and 85% are located more than 100 meters away from their actual position.

This figure also illustrates the distance between POI names using the Levenshtein algorithm. The results indicate that the top two groups (95%-100% and 85%-95%) have highly accurate names, whereas the names in the other groups show significant discrepancies.

Based on these criteria, the first two groups with a confidence score of more than 0.85 are proved to be reliable. Here is the percentage of reliable POIs in the top 10 countries which varies from 0% to 50%:

These are the countries with the most reliable POIs:

Brands

Overture assigns a brand attribute to its POIs, linking them to specific companies or brands. In the dataset, 4.9% of all POIs are branded, while this percentage rises to 12.9% among the more reliable POIs, further indicating their higher quality.

The dataset includes a total of 263,706 brands. However, 159,319 of these brands (60.4%) do not have any reliable POIs, and only 20,822 brands (less than 8%) have five or more reliable POIs. This suggests that while Overture covers a large number of brands, its definition of what constitutes a brand is quite broad.

Brand coverage

Let’s define three key terms:

• Market Research Value (MRV): The expected number of POIs for a brand in a specific country, as reported by the brand itself or other reliable third-party sources. This value applies to a brand-country pair.
• Coverage: The ratio of available POIs for a brand in a given country to its MRV. Coverage closer to 100% indicates a more precise representation.
• A Grade A brand has the brand-country coverage between 90% and 110%, representing the best coverage accuracy.

We calculated the brand coverage for 2,131 brand-country coverage using the number of reliable POIs in Overture compared to the MRVs provided by Echo Analytics. Here are the findings:

• These values encompass 1,209 brands across 23 countries.
• There are 53 brand-country pairs with an MRV of zero, yet Overture provides POIs for them. This discrepancy suggests a potential issue, as no POIs for these brands should exist in those countries based on reliable reports.
• Only 17.6% (376) of these brand-country pairs achieve Grade A status.
• The median coverage is 64.7%, and in 86% of cases, the coverage is below 100%, indicating that the dataset often under-represents brands. When including POIs with lower confidence scores, the median coverage increases to 78%, with 78% of coverages still below 100%, showing an improvement but also an increased risk of inaccuracies in low-confidence POIs.

Conclusion

In conclusion, while the Overture Maps Foundation offers a valuable and extensive dataset, its utility for precise analysis and decision-making is currently limited by inconsistencies in data quality and coverage. The dataset’s broad inclusion of brands and POIs provides a wide-ranging resource, but the variability in accuracy, particularly in brand coverage and POI location precision, suggests that it may not yet be fully reliable for critical applications.

To leverage Overture’s data effectively, it is essential to supplement it with more reliable and validated resources, especially when precise geospatial information is crucial. As the foundation continues to evolve and improve its dataset, combining Overture’s offerings with additional, trustworthy sources will provide a more accurate and dependable basis for informed decisions.

While our analysis identifies areas where the Overture Maps Foundation's dataset could benefit from further refinement, it is important to recognize the immense progress already made. We are excited about the future of OMF and look forward to seeing how it continues to innovate and improve, ultimately providing an invaluable resource for a wide range of applications.