summaryrefslogtreecommitdiff
path: root/doc/rfc/rfc8805.txt
diff options
context:
space:
mode:
authorThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
committerThomas Voss <mail@thomasvoss.com> 2024-11-27 20:54:24 +0100
commit4bfd864f10b68b71482b35c818559068ef8d5797 (patch)
treee3989f47a7994642eb325063d46e8f08ffa681dc /doc/rfc/rfc8805.txt
parentea76e11061bda059ae9f9ad130a9895cc85607db (diff)
doc: Add RFC documents
Diffstat (limited to 'doc/rfc/rfc8805.txt')
-rw-r--r--doc/rfc/rfc8805.txt1050
1 files changed, 1050 insertions, 0 deletions
diff --git a/doc/rfc/rfc8805.txt b/doc/rfc/rfc8805.txt
new file mode 100644
index 0000000..ac5d2bc
--- /dev/null
+++ b/doc/rfc/rfc8805.txt
@@ -0,0 +1,1050 @@
+
+
+
+
+Independent Submission E. Kline
+Request for Comments: 8805 Loon LLC
+Category: Informational K. Duleba
+ISSN: 2070-1721 Google
+ Z. Szamonek
+ S. Moser
+ Google Switzerland GmbH
+ W. Kumari
+ Google
+ August 2020
+
+
+ A Format for Self-Published IP Geolocation Feeds
+
+Abstract
+
+ This document records a format whereby a network operator can publish
+ a mapping of IP address prefixes to simplified geolocation
+ information, colloquially termed a "geolocation feed". Interested
+ parties can poll and parse these feeds to update or merge with other
+ geolocation data sources and procedures. This format intentionally
+ only allows specifying coarse-level location.
+
+ Some technical organizations operating networks that move from one
+ conference location to the next have already experimentally published
+ small geolocation feeds.
+
+ This document describes a currently deployed format. At least one
+ consumer (Google) has incorporated these feeds into a geolocation
+ data pipeline, and a significant number of ISPs are using it to
+ inform them where their prefixes should be geolocated.
+
+Status of This Memo
+
+ This document is not an Internet Standards Track specification; it is
+ published for informational purposes.
+
+ This is a contribution to the RFC Series, independently of any other
+ RFC stream. The RFC Editor has chosen to publish this document at
+ its discretion and makes no statement about its value for
+ implementation or deployment. Documents approved for publication by
+ the RFC Editor are not candidates for any level of Internet Standard;
+ see Section 2 of RFC 7841.
+
+ Information about the current status of this document, any errata,
+ and how to provide feedback on it may be obtained at
+ https://www.rfc-editor.org/info/rfc8805.
+
+Copyright Notice
+
+ Copyright (c) 2020 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (https://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document.
+
+Table of Contents
+
+ 1. Introduction
+ 1.1. Motivation
+ 1.2. Requirements Notation
+ 1.3. Assumptions about Publication
+ 2. Self-Published IP Geolocation Feeds
+ 2.1. Specification
+ 2.1.1. Geolocation Feed Individual Entry Fields
+ 2.1.1.1. IP Prefix
+ 2.1.1.2. Alpha2code (Previously: 'country')
+ 2.1.1.3. Region
+ 2.1.1.4. City
+ 2.1.1.5. Postal Code
+ 2.1.2. Prefixes with No Geolocation Information
+ 2.1.3. Additional Parsing Requirements
+ 2.2. Examples
+ 3. Consuming Self-Published IP Geolocation Feeds
+ 3.1. Feed Integrity
+ 3.2. Verification of Authority
+ 3.3. Verification of Accuracy
+ 3.4. Refreshing Feed Information
+ 4. Privacy Considerations
+ 5. Relation to Other Work
+ 6. Security Considerations
+ 7. Planned Future Work
+ 8. Finding Self-Published IP Geolocation Feeds
+ 8.1. Ad Hoc 'Well-Known' URIs
+ 8.2. Other Mechanisms
+ 9. IANA Considerations
+ 10. References
+ 10.1. Normative References
+ 10.2. Informative References
+ Appendix A. Sample Python Validation Code
+ Acknowledgements
+ Authors' Addresses
+
+1. Introduction
+
+1.1. Motivation
+
+ Providers of services over the Internet have grown to depend on best-
+ effort geolocation information to improve the user experience.
+ Locality information can aid in directing traffic to the nearest
+ serving location, inferring likely native language, and providing
+ additional context for services involving search queries.
+
+ When an ISP, for example, changes the location where an IP prefix is
+ deployed, services that make use of geolocation information may begin
+ to suffer degraded performance. This can lead to customer
+ complaints, possibly to the ISP directly. Dissemination of correct
+ geolocation data is complicated by the lack of any centralized means
+ to coordinate and communicate geolocation information to all
+ interested consumers of the data.
+
+ This document records a format whereby a network operator (an ISP, an
+ enterprise, or any organization that deems the geolocation of its IP
+ prefixes to be of concern) can publish a mapping of IP address
+ prefixes to simplified geolocation information, colloquially termed a
+ "geolocation feed". Interested parties can poll and parse these
+ feeds to update or merge with other geolocation data sources and
+ procedures.
+
+ This document describes a currently deployed format. At least one
+ consumer (Google) has incorporated these feeds into a geolocation
+ data pipeline, and a significant number of ISPs are using it to
+ inform them where their prefixes should be geolocated.
+
+1.2. Requirements Notation
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
+ "OPTIONAL" in this document are to be interpreted as described in
+ BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
+ capitals, as shown here.
+
+ As this is an informational document about a data format and set of
+ operational practices presently in use, requirements notation
+ captures the design goals of the authors and implementors.
+
+1.3. Assumptions about Publication
+
+ This document describes both a format and a mechanism for publishing
+ data, with the assumption that the network operator to whom
+ operational responsibility has been delegated for any published data
+ wishes it to be public. Any privacy risk is bounded by the format,
+ and feed publishers MAY omit prefixes or any location field
+ associated with a given prefix to further protect privacy (see
+ Section 2.1 for details about which fields exactly may be omitted).
+ Feed publishers assume the responsibility of determining which data
+ should be made public.
+
+ This document does not incorporate a mechanism to communicate
+ acceptable use policies for self-published data. Publication itself
+ is inferred as a desire by the publisher for the data to be usefully
+ consumed, similar to the publication of information like host names,
+ cryptographic keys, and Sender Policy Framework (SPF) records
+ [RFC7208] in the DNS.
+
+2. Self-Published IP Geolocation Feeds
+
+ The format described here was developed to address the need of
+ network operators to rapidly and usefully share geolocation
+ information changes. Originally, there arose a specific case where
+ regional operators found it desirable to publish location changes
+ rather than wait for geolocation algorithms to "learn" about them.
+ Later, technical conferences that frequently use the same network
+ prefixes advertised from different conference locations experimented
+ by publishing geolocation feeds updated in advance of network
+ location changes in order to better serve conference attendees.
+
+ At its simplest, the mechanism consists of a network operator
+ publishing a file (the "geolocation feed") that contains several text
+ entries, one per line. Each entry is keyed by a unique (within the
+ feed) IP prefix (or single IP address) followed by a sequence of
+ network locality attributes to be ascribed to the given prefix.
+
+2.1. Specification
+
+ For operational simplicity, every feed should contain data about all
+ IP addresses the provider wants to publish. Alternatives, like
+ publishing only entries for IP addresses whose geolocation data has
+ changed or differ from current observed geolocation behavior "at
+ large", are likely to be too operationally complex.
+
+ Feeds MUST use UTF-8 [RFC3629] character encoding. Lines are
+ delimited by a line break (CRLF) (as specified in [RFC4180]), and
+ blank lines are ignored. Text from a '#' character to the end of the
+ current line is treated as a comment only and is similarly ignored
+ (note that this does not strictly follow [RFC4180], which has no
+ support for comments).
+
+ Feed lines that are not comments MUST be formatted as comma-separated
+ values (CSV), as described in [RFC4180]. Each feed entry is a text
+ line of the form:
+
+ ip_prefix,alpha2code,region,city,postal_code
+
+ The IP prefix field is REQUIRED, all others are OPTIONAL (can be
+ empty), though the requisite minimum number of commas SHOULD be
+ present.
+
+2.1.1. Geolocation Feed Individual Entry Fields
+
+2.1.1.1. IP Prefix
+
+ REQUIRED: Each IP prefix field MUST be either a single IP address or
+ an IP prefix in Classless Inter-Domain Routing (CIDR) notation in
+ conformance with Section 3.1 of [RFC4632] for IPv4 or Section 2.3 of
+ [RFC4291] for IPv6.
+
+ Examples include "192.0.2.1" and "192.0.2.0/24" for IPv4 and
+ "2001:db8::1" and "2001:db8::/32" for IPv6.
+
+2.1.1.2. Alpha2code (Previously: 'country')
+
+ OPTIONAL: The alpha2code field, if non-empty, MUST be a 2-letter ISO
+ country code conforming to ISO 3166-1 alpha 2 [ISO.3166.1alpha2].
+ Parsers SHOULD treat this field case-insensitively.
+
+ Earlier versions of this document called this field "country", and it
+ may still be referred to as such in existing tools/interfaces.
+
+ Parsers MAY additionally support other 2-letter codes outside the ISO
+ 3166-1 alpha 2 codes, such as the 2-letter codes from the
+ "Exceptionally reserved codes" [ISO-GLOSSARY] set.
+
+ Examples include "US" for the United States, "JP" for Japan, and "PL"
+ for Poland.
+
+2.1.1.3. Region
+
+ OPTIONAL: The region field, if non-empty, MUST be an ISO region code
+ conforming to ISO 3166-2 [ISO.3166.2]. Parsers SHOULD treat this
+ field case-insensitively.
+
+ Examples include "ID-RI" for the Riau province of Indonesia and "NG-
+ RI" for the Rivers province in Nigeria.
+
+2.1.1.4. City
+
+ OPTIONAL: The city field, if non-empty, SHOULD be free UTF-8 text,
+ excluding the comma (',') character.
+
+ Examples include "Dublin", "New York", and "Sao Paulo" (specifically
+ "S" followed by 0xc3, 0xa3, and "o Paulo").
+
+2.1.1.5. Postal Code
+
+ OPTIONAL, DEPRECATED: The postal code field, if non-empty, SHOULD be
+ free UTF-8 text, excluding the comma (',') character. The use of
+ this field is deprecated; consumers of feeds should be able to parse
+ feeds containing these fields, but new feeds SHOULD NOT include this
+ field due to the granularity of this information. See Section 4 for
+ additional discussion.
+
+ Examples include "106-6126" (in Minato ward, Tokyo, Japan).
+
+2.1.2. Prefixes with No Geolocation Information
+
+ Feed publishers may indicate that some IP prefixes should not have
+ any associated geolocation information. It may be that some prefixes
+ under their administrative control are reserved, not yet allocated or
+ deployed, or in the process of being redeployed elsewhere and
+ existing geolocation information can, from the perspective of the
+ publisher, safely be discarded.
+
+ This special case can be indicated by explicitly leaving blank all
+ fields that specify any degree of geolocation information. For
+ example:
+
+ 192.0.2.0/24,,,,
+ 2001:db8:1::/48,,,,
+ 2001:db8:2::/48,,,,
+
+ Historically, the user-assigned alpha2code identifier of "ZZ" has
+ been used for this same purpose. This is not necessarily preferred,
+ and no specific interpretation of any of the other user-assigned
+ alpha2code codes is currently defined.
+
+2.1.3. Additional Parsing Requirements
+
+ Feed entries that do not have an IP address or prefix field or have
+ an IP address or prefix field that fails to parse correctly MUST be
+ discarded.
+
+ While publishers SHOULD follow [RFC5952] for IPv6 prefix fields,
+ consumers MUST nevertheless accept all valid string representations.
+
+ Duplicate IP address or prefix entries MUST be considered an error,
+ and consumer implementations SHOULD log the repeated entries for
+ further administrative review. Publishers SHOULD take measures to
+ ensure there is one and only one entry per IP address and prefix.
+
+ Multiple entries that constitute nested prefixes are permitted.
+ Consumers SHOULD consider the entry with the longest matching prefix
+ (i.e., the "most specific") to be the best matching entry for a given
+ IP address.
+
+ Feed entries with non-empty optional fields that fail to parse,
+ either in part or in full, SHOULD be discarded. It is RECOMMENDED
+ that they also be logged for further administrative review.
+
+ For compatibility with future additional fields, a parser MUST ignore
+ any fields beyond those it expects. The data from fields that are
+ expected and that parse successfully MUST still be considered valid.
+ Per Section 7, no extensions to this format are in use nor are any
+ anticipated.
+
+2.2. Examples
+
+ Example entries using different IP address formats and describing
+ locations at alpha2code ("country code"), region, and city
+ granularity level, respectively:
+
+ 192.0.2.0/25,US,US-AL,,
+ 192.0.2.5,US,US-AL,Alabaster,
+ 192.0.2.128/25,PL,PL-MZ,,
+ 2001:db8::/32,PL,,,
+ 2001:db8:cafe::/48,PL,PL-MZ,,
+
+ The IETF network publishes geolocation information for the meeting
+ prefixes, and generally just comment out the last meeting information
+ and append the new meeting information. The [GEO_IETF], at the time
+ of this writing, contains:
+
+ # IETF106 (Singapore) - November 2019 - Singapore, SG
+ 130.129.0.0/16,SG,SG-01,Singapore,
+ 2001:df8::/32,SG,SG-01,Singapore,
+ 31.133.128.0/18,SG,SG-01,Singapore,
+ 31.130.224.0/20,SG,SG-01,Singapore,
+ 2001:67c:1230::/46,SG,SG-01,Singapore,
+ 2001:67c:370::/48,SG,SG-01,Singapore,
+
+ Experimentally, RIPE has published geolocation information for their
+ conference network prefixes, which change location in accordance with
+ each new event. [GEO_RIPE_NCC], at the time of writing, contains:
+
+ 193.0.24.0/21,NL,NL-ZH,Rotterdam,
+ 2001:67c:64::/48,NL,NL-ZH,Rotterdam,
+
+ Similarly, ICANN has published geolocation information for their
+ portable conference network prefixes. [GEO_ICANN], at the time of
+ writing, contains:
+
+ 199.91.192.0/21,MA,MA-07,Marrakech
+ 2620:f:8000::/48,MA,MA-07,Marrakech
+
+ A longer example is the [GEO_Google] Google Corp Geofeed, which lists
+ the geolocation information for Google corporate offices.
+
+ At the time of writing, Google processes approximately 400 feeds
+ comprising more than 750,000 IPv4 and IPv6 prefixes.
+
+3. Consuming Self-Published IP Geolocation Feeds
+
+ Consumers MAY treat published feed data as a hint only and MAY choose
+ to prefer other sources of geolocation information for any given IP
+ prefix. Regardless of a consumer's stance with respect to a given
+ published feed, there are some points of note for sensibly and
+ effectively consuming published feeds.
+
+3.1. Feed Integrity
+
+ The integrity of published information SHOULD be protected by
+ securing the means of publication, for example, by using HTTP over
+ TLS [RFC2818]. Whenever possible, consumers SHOULD prefer retrieving
+ geolocation feeds in a manner that guarantees integrity of the feed.
+
+3.2. Verification of Authority
+
+ Consumers of self-published IP geolocation feeds SHOULD perform some
+ form of verification that the publisher is in fact authoritative for
+ the addresses in the feed. The actual means of verification is
+ likely dependent upon the way in which the feed is discovered. Ad
+ hoc shared URIs, for example, will likely require an ad hoc
+ verification process. Future automated means of feed discovery
+ SHOULD have an accompanying automated means of verification.
+
+ A consumer should only trust geolocation information for IP addresses
+ or prefixes for which the publisher has been verified as
+ administratively authoritative. All other geolocation feed entries
+ should be ignored and logged for further administrative review.
+
+3.3. Verification of Accuracy
+
+ Errors and inaccuracies may occur at many levels, and publication and
+ consumption of geolocation data are no exceptions. To the extent
+ practical, consumers SHOULD take steps to verify the accuracy of
+ published locality. Verification methodology, resolution of
+ discrepancies, and preference for alternative sources of data are
+ left to the discretion of the feed consumer.
+
+ Consumers SHOULD decide on discrepancy thresholds and SHOULD flag,
+ for administrative review, feed entries that exceed set thresholds.
+
+3.4. Refreshing Feed Information
+
+ As a publisher can change geolocation data at any time and without
+ notification, consumers SHOULD implement mechanisms to periodically
+ refresh local copies of feed data. In the absence of any other
+ refresh timing information, it is recommended that consumers SHOULD
+ refresh feeds no less often than weekly and no more often than is
+ likely to cause issues to the publisher.
+
+ For feeds available via HTTPS (or HTTP), the publisher MAY
+ communicate refresh timing information by means of the standard HTTP
+ expiration model ([RFC7234]). Specifically, publishers can include
+ either an Expires header (Section 5.3 of [RFC7234]) or a Cache-
+ Control header (Section 5.2 of [RFC7234]) specifying the max-age.
+ Where practical, consumers SHOULD refresh feed information before the
+ expiry time is reached.
+
+4. Privacy Considerations
+
+ Publishers of geolocation feeds are advised to have fully considered
+ any and all privacy implications of the disclosure of such
+ information for the users of the described networks prior to
+ publication. A thorough comprehension of the security considerations
+ (Section 13 of [RFC6772]) of a chosen geolocation policy is highly
+ recommended, including an understanding of some of the limitations of
+ information obscurity (Section 13.5 of [RFC6772]) (see also
+ [RFC6772]).
+
+ As noted in Section 2.1, each location field in an entry is optional,
+ in order to support expressing only the level of specificity that the
+ publisher has deemed acceptable. There is no requirement that the
+ level of specificity be consistent across all entries within a feed.
+ In particular, the Postal Code field (Section 2.1.1.5) can provide
+ very specific geolocation, sometimes within a building. Such
+ specific Postal Code values MUST NOT be published in geofeeds without
+ the express consent of the parties being located.
+
+ Operators who publish geolocation information are strongly encouraged
+ to inform affected users/customers of this fact and of the potential
+ privacy-related consequences and trade-offs.
+
+5. Relation to Other Work
+
+ While not originally done in conjunction with the GEOPRIV Working
+ Group [GEOPRIV], Richard Barnes observed that this work is
+ nevertheless consistent with that which the group has defined, both
+ for address format and for privacy. The data elements in geolocation
+ feeds are equivalent to the following XML structure ([RFC5139]
+ [W3C.REC-xml-20081126]):
+
+ <civicAddress>
+ <country>country</country>
+ <A1>region</A1>
+ <A2>city</A2>
+ <PC>postal_code</PC>
+ </civicAddress>
+
+ Providing geolocation information to this granularity is equivalent
+ to the following privacy policy (the definition of the 'building'
+ Section 6.5.1 of [RFC6772] level of disclosure):
+
+ <ruleset>
+ <rule>
+ <conditions/>
+ <actions/>
+ <transformations>
+ <provide-location profile="civic-transformation">
+ <provide-civic>building</provide-civic>
+ </provide-location>
+ </transformations>
+ </rule>
+ </ruleset>
+
+6. Security Considerations
+
+ As there is no true security in the obscurity of the location of any
+ given IP address, self-publication of this data fundamentally opens
+ no new attack vectors. For publishers, self-published data may
+ increase the ease with which such location data might be exploited
+ (it can, for example, make easy the discovery of prefixes populated
+ with customers as distinct from prefixes not generally in use).
+
+ For consumers, feed retrieval processes may receive input from
+ potentially hostile sources (e.g., in the event of hijacked traffic).
+ As such, proper input validation and defense measures MUST be taken
+ (see the discussion in Section 3.1).
+
+ Similarly, consumers who do not perform sufficient verification of
+ published data bear the same risks as from other forms of geolocation
+ configuration errors (see the discussion in Sections 3.2 and 3.3).
+
+ Validation of a feed's contents includes verifying that the publisher
+ is authoritative for the IP prefixes included in the feed. Failure
+ to verify IP prefix authority would, for example, allow ISP Bob to
+ make geolocation statements about IP space held by ISP Alice. At
+ this time, only out-of-band verification methods are implemented
+ (i.e., an ISP's feed may be verified against publicly available IP
+ allocation data).
+
+7. Planned Future Work
+
+ In order to more flexibly support future extensions, use of a more
+ expressive feed format has been suggested. Use of JavaScript Object
+ Notation (JSON) [RFC8259], specifically, has been discussed.
+ However, at the time of writing, no such specification nor
+ implementation exists. Nevertheless, work on extensions is deferred
+ until a more suitable format has been selected.
+
+ The authors are planning on writing a document describing such a new
+ format. This document describes a currently deployed and used
+ format. Given the extremely limited extensibility of the present
+ format no extensions to it are anticipated. Extensibility
+ requirements are instead expected to be integral to the development
+ of a new format.
+
+8. Finding Self-Published IP Geolocation Feeds
+
+ The issue of finding, and later verifying, geolocation feeds is not
+ formally specified in this document. At this time, only ad hoc feed
+ discovery and verification has a modicum of established practice (see
+ below); discussion of other mechanisms has been removed for clarity.
+
+8.1. Ad Hoc 'Well-Known' URIs
+
+ To date, geolocation feeds have been shared informally in the form of
+ HTTPS URIs exchanged in email threads. Three example URIs
+ ([GEO_IETF], [GEO_RIPE_NCC], and [GEO_ICANN]) describe networks that
+ change locations periodically, the operators and operational
+ practices of which are well known within their respective technical
+ communities.
+
+ The contents of the feeds are verified by a similarly ad hoc process,
+ including:
+
+ * personal knowledge of the parties involved in the exchange and
+
+ * comparison of feed-advertised prefixes with the BGP-advertised
+ prefixes of Autonomous System Numbers known to be operated by the
+ publishers.
+
+ Ad hoc mechanisms, while useful for early experimentation by
+ producers and consumers, are unlikely to be adequate for long-term,
+ widespread use by multiple parties. Future versions of any such
+ self-published geolocation feed mechanism SHOULD address scalability
+ concerns by defining a means for automated discovery and verification
+ of operational authority of advertised prefixes.
+
+8.2. Other Mechanisms
+
+ Previous versions of this document referenced use of the WHOIS
+ service [RFC3912] operated by Regional Internet Registries (RIRs), as
+ well as possible DNS-based schemes to discover and validate geofeeds.
+ To the authors' knowledge, support for such mechanisms has never been
+ implemented, and this speculative text has been removed to avoid
+ ambiguity.
+
+9. IANA Considerations
+
+ This document has no IANA actions.
+
+10. References
+
+10.1. Normative References
+
+ [ISO.3166.1alpha2]
+ ISO, "ISO 3166-1 decoding table",
+ <http://www.iso.org/iso/home/standards/country_codes/iso-
+ 3166-1_decoding_table.htm>.
+
+ [ISO.3166.2]
+ ISO, "ISO 3166-2:2007",
+ <http://www.iso.org/iso/home/standards/
+ country_codes.htm#2012_iso3166-2>.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119,
+ DOI 10.17487/RFC2119, March 1997,
+ <https://www.rfc-editor.org/info/rfc2119>.
+
+ [RFC3629] Yergeau, F., "UTF-8, a transformation format of ISO
+ 10646", STD 63, RFC 3629, DOI 10.17487/RFC3629, November
+ 2003, <https://www.rfc-editor.org/info/rfc3629>.
+
+ [RFC4180] Shafranovich, Y., "Common Format and MIME Type for Comma-
+ Separated Values (CSV) Files", RFC 4180,
+ DOI 10.17487/RFC4180, October 2005,
+ <https://www.rfc-editor.org/info/rfc4180>.
+
+ [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing
+ Architecture", RFC 4291, DOI 10.17487/RFC4291, February
+ 2006, <https://www.rfc-editor.org/info/rfc4291>.
+
+ [RFC4632] Fuller, V. and T. Li, "Classless Inter-domain Routing
+ (CIDR): The Internet Address Assignment and Aggregation
+ Plan", BCP 122, RFC 4632, DOI 10.17487/RFC4632, August
+ 2006, <https://www.rfc-editor.org/info/rfc4632>.
+
+ [RFC5952] Kawamura, S. and M. Kawashima, "A Recommendation for IPv6
+ Address Text Representation", RFC 5952,
+ DOI 10.17487/RFC5952, August 2010,
+ <https://www.rfc-editor.org/info/rfc5952>.
+
+ [RFC7234] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
+ Ed., "Hypertext Transfer Protocol (HTTP/1.1): Caching",
+ RFC 7234, DOI 10.17487/RFC7234, June 2014,
+ <https://www.rfc-editor.org/info/rfc7234>.
+
+ [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
+ 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
+ May 2017, <https://www.rfc-editor.org/info/rfc8174>.
+
+ [W3C.REC-xml-20081126]
+ Bray, T., Paoli, J., Sperberg-McQueen, M., Maler, E., and
+ F. Yergeau, "Extensible Markup Language (XML) 1.0 (Fifth
+ Edition)", World Wide Web Consortium Recommendation REC-
+ xml-20081126, November 2008,
+ <http://www.w3.org/TR/2008/REC-xml-20081126>.
+
+10.2. Informative References
+
+ [GEOPRIV] IETF, "Geographic Location/Privacy (geopriv)",
+ <http://datatracker.ietf.org/wg/geopriv/>.
+
+ [GEO_Google]
+ Google, LLC, "Google Corp Geofeed",
+ <https://www.gstatic.com/geofeed/corp_external>.
+
+ [GEO_ICANN]
+ ICANN, "ICANN Meeting Geolocation Data",
+ <https://meeting-services.icann.org/geo/google.csv>.
+
+ [GEO_IETF] Kumari, W., "IETF Meeting Network Geolocation Data",
+ <https://noc.ietf.org/geo/google.csv>.
+
+ [GEO_RIPE_NCC]
+ Schepers, M., "RIPE NCC Meeting Geolocation Data",
+ <https://meetings.ripe.net/geo/google.csv>.
+
+ [IPADDR_PY]
+ Shields, M. and P. Moody, "Google's Python IP address
+ manipulation library",
+ <http://code.google.com/p/ipaddr-py/>.
+
+ [ISO-GLOSSARY]
+ ISO, "Glossary for ISO 3166",
+ <https://www.iso.org/glossary-for-iso-3166.html>.
+
+ [RFC2818] Rescorla, E., "HTTP Over TLS", RFC 2818,
+ DOI 10.17487/RFC2818, May 2000,
+ <https://www.rfc-editor.org/info/rfc2818>.
+
+ [RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912,
+ DOI 10.17487/RFC3912, September 2004,
+ <https://www.rfc-editor.org/info/rfc3912>.
+
+ [RFC5139] Thomson, M. and J. Winterbottom, "Revised Civic Location
+ Format for Presence Information Data Format Location
+ Object (PIDF-LO)", RFC 5139, DOI 10.17487/RFC5139,
+ February 2008, <https://www.rfc-editor.org/info/rfc5139>.
+
+ [RFC6772] Schulzrinne, H., Ed., Tschofenig, H., Ed., Cuellar, J.,
+ Polk, J., Morris, J., and M. Thomson, "Geolocation Policy:
+ A Document Format for Expressing Privacy Preferences for
+ Location Information", RFC 6772, DOI 10.17487/RFC6772,
+ January 2013, <https://www.rfc-editor.org/info/rfc6772>.
+
+ [RFC7208] Kitterman, S., "Sender Policy Framework (SPF) for
+ Authorizing Use of Domains in Email, Version 1", RFC 7208,
+ DOI 10.17487/RFC7208, April 2014,
+ <https://www.rfc-editor.org/info/rfc7208>.
+
+ [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data
+ Interchange Format", STD 90, RFC 8259,
+ DOI 10.17487/RFC8259, December 2017,
+ <https://www.rfc-editor.org/info/rfc8259>.
+
+Appendix A. Sample Python Validation Code
+
+ Included here is a simple format validator in Python for self-
+ published ipgeo feeds. This tool reads CSV data in the self-
+ published ipgeo feed format from the standard input and performs
+ basic validation. It is intended for use by feed publishers before
+ launching a feed. Note that this validator does not verify the
+ uniqueness of every IP prefix entry within the feed as a whole but
+ only verifies the syntax of each single line from within the feed. A
+ complete validator MUST also ensure IP prefix uniqueness.
+
+ The main source file "ipgeo_feed_validator.py" follows. It requires
+ use of the open source ipaddr Python library for IP address and CIDR
+ parsing and validation [IPADDR_PY].
+
+ <CODE BEGINS>
+ #!/usr/bin/python
+ #
+ # Copyright (c) 2012 IETF Trust and the persons identified as
+ # authors of the code. All rights reserved. Redistribution and use
+ # in source and binary forms, with or without modification, is
+ # permitted pursuant to, and subject to the license terms contained
+ # in, the Simplified BSD License set forth in Section 4.c of the
+ # IETF Trust's Legal Provisions Relating to IETF
+ # Documents (http://trustee.ietf.org/license-info).
+
+ """Simple format validator for self-published ipgeo feeds.
+
+ This tool reads CSV data in the self-published ipgeo feed format
+ from the standard input and performs basic validation. It is
+ intended for use by feed publishers before launching a feed.
+ """
+
+ import csv
+ import ipaddr
+ import re
+ import sys
+
+
+ class IPGeoFeedValidator(object):
+ def __init__(self):
+ self.prefixes = {}
+ self.line_number = 0
+ self.output_log = {}
+ self.SetOutputStream(sys.stderr)
+
+ def Validate(self, feed):
+ """Check validity of an IPGeo feed.
+
+ Args:
+ feed: iterable with feed lines
+ """
+
+ for line in feed:
+ self._ValidateLine(line)
+
+ def SetOutputStream(self, logfile):
+ """Controls where the output messages go do (STDERR by default).
+
+ Use None to disable logging.
+
+ Args:
+ logfile: a file object (e.g., sys.stdout) or None.
+ """
+ self.output_stream = logfile
+
+ def CountErrors(self, severity):
+ """How many ERRORs or WARNINGs were generated."""
+ return len(self.output_log.get(severity, []))
+
+ ############################################################
+ def _ValidateLine(self, line):
+ line = line.rstrip('\r\n')
+ self.line_number += 1
+ self.line = line.split('#')[0]
+ self.is_correct_line = True
+
+ if self._ShouldIgnoreLine(line):
+ return
+
+ fields = [field for field in csv.reader([line])][0]
+
+ self._ValidateFields(fields)
+ self._FlushOutputStream()
+
+ def _ShouldIgnoreLine(self, line):
+ line = line.strip()
+ if line.startswith('#'):
+ return True
+ return len(line) == 0
+
+ ############################################################
+ def _ValidateFields(self, fields):
+ assert(len(fields) > 0)
+
+ is_correct = self._IsIPAddressOrPrefixCorrect(fields[0])
+
+ if len(fields) > 1:
+ if not self._IsAlpha2CodeCorrect(fields[1]):
+ is_correct = False
+
+ if len(fields) > 2 and not self._IsRegionCodeCorrect(fields[2]):
+ is_correct = False
+
+ if len(fields) != 5:
+ self._ReportWarning('5 fields were expected (got %d).'
+ % len(fields))
+
+ ############################################################
+ def _IsIPAddressOrPrefixCorrect(self, field):
+ if '/' in field:
+ return self._IsCIDRCorrect(field)
+ return self._IsIPAddressCorrect(field)
+
+ def _IsCIDRCorrect(self, cidr):
+ try:
+ ipprefix = ipaddr.IPNetwork(cidr)
+ if ipprefix.network._ip != ipprefix._ip:
+ self._ReportError('Incorrect IP Network.')
+ return False
+ if ipprefix.is_private:
+ self._ReportError('IP Address must not be private.')
+ return False
+ except:
+ self._ReportError('Incorrect IP Network.')
+ return False
+ return True
+
+ def _IsIPAddressCorrect(self, ipaddress):
+ try:
+ ip = ipaddr.IPAddress(ipaddress)
+ except:
+ self._ReportError('Incorrect IP Address.')
+ return False
+ if ip.is_private:
+ self._ReportError('IP Address must not be private.')
+ return False
+ return True
+
+ ############################################################
+ def _IsAlpha2CodeCorrect(self, alpha2code):
+ if len(alpha2code) == 0:
+ return True
+ if len(alpha2code) != 2 or not alpha2code.isalpha():
+ self._ReportError(
+ 'Alpha 2 code must be in the ISO 3166-1 alpha 2 format.')
+ return False
+ return True
+
+ def _IsRegionCodeCorrect(self, region_code):
+ if len(region_code) == 0:
+ return True
+ if '-' not in region_code:
+ self._ReportError('Region code must be in ISO 3166-2 format.')
+ return False
+
+ parts = region_code.split('-')
+ if not self._IsAlpha2CodeCorrect(parts[0]):
+ return False
+ return True
+
+ ############################################################
+ def _ReportError(self, message):
+ self._ReportWithSeverity('ERROR', message)
+
+ def _ReportWarning(self, message):
+ self._ReportWithSeverity('WARNING', message)
+
+ def _ReportWithSeverity(self, severity, message):
+ self.is_correct_line = False
+ output_line = '%s: %s\n' % (severity, message)
+
+ if severity not in self.output_log:
+ self.output_log[severity] = []
+ self.output_log[severity].append(output_line)
+
+ if self.output_stream is not None:
+ self.output_stream.write(output_line)
+
+ def _FlushOutputStream(self):
+ if self.is_correct_line: return
+ if self.output_stream is None: return
+
+ self.output_stream.write('line %d: %s\n\n'
+ % (self.line_number, self.line))
+
+
+ ############################################################
+ def main():
+ feed_validator = IPGeoFeedValidator()
+ feed_validator.Validate(sys.stdin)
+
+ if feed_validator.CountErrors('ERROR'):
+ sys.exit(1)
+
+ if __name__ == '__main__':
+ main()
+ <CODE ENDS>
+
+ A unit test file, "ipgeo_feed_validator_test.py" is provided as well.
+ It provides basic test coverage of the code above, though does not
+ test correct handling of non-ASCII UTF-8 strings.
+
+ <CODE BEGINS>
+ #!/usr/bin/python
+ #
+ # Copyright (c) 2012 IETF Trust and the persons identified as
+ # authors of the code. All rights reserved. Redistribution and use
+ # in source and binary forms, with or without modification, is
+ # permitted pursuant to, and subject to the license terms contained
+ # in, the Simplified BSD License set forth in Section 4.c of the
+ # IETF Trust's Legal Provisions Relating to IETF
+ # Documents (http://trustee.ietf.org/license-info).
+
+ import sys
+ from ipgeo_feed_validator import IPGeoFeedValidator
+
+ class IPGeoFeedValidatorTest(object):
+ def __init__(self):
+ self.validator = IPGeoFeedValidator()
+ self.validator.SetOutputStream(None)
+ self.successes = 0
+ self.failures = 0
+
+ def Run(self):
+ self.TestFeedLine('# asdf', 0, 0)
+ self.TestFeedLine(' ', 0, 0)
+ self.TestFeedLine('', 0, 0)
+
+ self.TestFeedLine('asdf', 1, 1)
+ self.TestFeedLine('asdf,US,,,', 1, 0)
+ self.TestFeedLine('aaaa::,US,,,', 0, 0)
+ self.TestFeedLine('zzzz::,US', 1, 1)
+ self.TestFeedLine(',US,,,', 1, 0)
+ self.TestFeedLine('55.66.77', 1, 1)
+ self.TestFeedLine('55.66.77.888', 1, 1)
+ self.TestFeedLine('55.66.77.asdf', 1, 1)
+
+ self.TestFeedLine('2001:db8:cafe::/48,PL,PL-MZ,,02-784', 0, 0)
+ self.TestFeedLine('2001:db8:cafe::/48', 0, 1)
+
+ self.TestFeedLine('55.66.77.88,PL', 0, 1)
+ self.TestFeedLine('55.66.77.88,PL,,,', 0, 0)
+ self.TestFeedLine('55.66.77.88,,,,', 0, 0)
+ self.TestFeedLine('55.66.77.88,ZZ,,,', 0, 0)
+ self.TestFeedLine('55.66.77.88,US,,,', 0, 0)
+ self.TestFeedLine('55.66.77.88,USA,,,', 1, 0)
+ self.TestFeedLine('55.66.77.88,99,,,', 1, 0)
+
+ self.TestFeedLine('55.66.77.88,US,US-CA,,', 0, 0)
+ self.TestFeedLine('55.66.77.88,US,USA-CA,,', 1, 0)
+ self.TestFeedLine('55.66.77.88,USA,USA-CA,,', 2, 0)
+
+ self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,', 0, 0)
+ self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,94043',
+ 0, 0)
+ self.TestFeedLine('55.66.77.88,US,US-CA,Mountain View,94043,'
+ '1600 Ampthitheatre Parkway', 0, 1)
+
+ self.TestFeedLine('55.66.77.0/24,US,,,', 0, 0)
+ self.TestFeedLine('55.66.77.88/24,US,,,', 1, 0)
+ self.TestFeedLine('55.66.77.88/32,US,,,', 0, 0)
+ self.TestFeedLine('55.66.77/24,US,,,', 1, 0)
+ self.TestFeedLine('55.66.77.0/35,US,,,', 1, 0)
+
+ self.TestFeedLine('172.15.30.1,US,,,', 0, 0)
+ self.TestFeedLine('172.28.30.1,US,,,', 1, 0)
+ self.TestFeedLine('192.167.100.1,US,,,', 0, 0)
+ self.TestFeedLine('192.168.100.1,US,,,', 1, 0)
+ self.TestFeedLine('10.0.5.9,US,,,', 1, 0)
+ self.TestFeedLine('10.0.5.0/24,US,,,', 1, 0)
+ self.TestFeedLine('fc00::/48,PL,,,', 1, 0)
+ self.TestFeedLine('fe00::/48,PL,,,', 0, 0)
+
+ print ('%d tests passed, %d failed'
+ % (self.successes, self.failures))
+
+ def IsOutputLogCorrectAtSeverity(self, severity,
+ expected_msg_count):
+ msg_count = self.validator.CountErrors(severity)
+
+ if msg_count != expected_msg_count:
+ print ('TEST FAILED: %s\nexpected %d %s[s], observed %d\n%s\n'
+ % (self.validator.line, expected_msg_count, severity,
+ msg_count,
+ str(self.validator.output_log[severity])))
+ return False
+ return True
+
+ def IsOutputLogCorrect(self, new_errors, new_warnings):
+ retval = True
+
+ if not self.IsOutputLogCorrectAtSeverity('ERROR', new_errors):
+ retval = False
+ if not self.IsOutputLogCorrectAtSeverity('WARNING',
+ new_warnings):
+ retval = False
+
+ return retval
+
+ def TestFeedLine(self, line, warning_count, error_count):
+ self.validator.output_log['WARNING'] = []
+ self.validator.output_log['ERROR'] = []
+ self.validator._ValidateLine(line)
+
+ if not self.IsOutputLogCorrect(warning_count, error_count):
+ self.failures += 1
+ return False
+
+ self.successes += 1
+ return True
+
+
+ if __name__ == '__main__':
+ IPGeoFeedValidatorTest().Run()
+ <CODE ENDS>
+
+Acknowledgements
+
+ The authors would like to express their gratitude to reviewers and
+ early implementors, including but not limited to Mikael Abrahamsson,
+ Andrew Alston, Ray Bellis, John Bond, Alissa Cooper, Andras Erdei,
+ Stephen Farrell, Marco Hogewoning, Mike Joseph, Maciej Kuzniar,
+ George Michaelson, Menno Schepers, Justyna Sidorska, Pim van Pelt,
+ and Bjoern A. Zeeb.
+
+ In particular, Richard L. Barnes and Andy Newton contributed
+ substantial review, text, and advice.
+
+Authors' Addresses
+
+ Erik Kline
+ Loon LLC
+ 1600 Amphitheatre Parkway
+ Mountain View, CA 94043
+ United States of America
+
+ Email: ek@loon.com
+
+
+ Krzysztof Duleba
+ Google
+ 1600 Amphitheatre Parkway
+ Mountain View, CA 94043
+ United States of America
+
+ Email: kduleba@google.com
+
+
+ Zoltan Szamonek
+ Google Switzerland GmbH
+ Brandschenkestrasse 110
+ CH-8002 Zürich
+ Switzerland
+
+ Email: zszami@google.com
+
+
+ Stefan Moser
+ Google Switzerland GmbH
+ Brandschenkestrasse 110
+ CH-8002 Zürich
+ Switzerland
+
+ Email: smoser@google.com
+
+
+ Warren Kumari
+ Google
+ 1600 Amphitheatre Parkway
+ Mountain View, CA 94043
+ United States of America
+
+ Email: warren@kumari.net